Unpacking Post-Market Management and Incident Response for Medical Devices

Christian: Hi, welcome back to the Med Device Cyber Podcast. I'm your host Christian Espinosa along here with our co-host Trevor Slattery. Today we're going to be talking about post-market management and incident response. Like what happens when a medical device is hacked or somebody reports a vulnerability on a coordinated vulnerability disclosure database. What's that process look like? And we're going to unpack that today. So before we dive into it, just uh checking in with you, Trevor. How how's it going this morning? Trevor: It's going great. We're getting some warm weather. The even broke 70 yesterday, so spring has finally arrived. Christian: Yeah, it was up to 100 here in the valley. So I went for a walk and uh it was pretty hot. Trevor: Yeah, you know this year was the earliest year on record that it broke nine- or the earliest day of the year on record that it broke 90 in Phoenix. I think it was February 2nd. Christian: Yeah, doing good. Also, we got our the Blue Goat coffee mugs. I don't know if you got one yet, but... Trevor: Oh, I didn't know we even had those. I got to get one of those. Christian: We have those, yes. We just got back from a conference in California, so we gave a bunch of them away and uh, quite a few other things. But yeah, we'll have to get you some for, for sure. Trevor: Perfect. Yeah, we got a couple more conferences coming up, so have to bring some Blue Goat swag. Christian: That's true. We got to get shipped out to Boston pretty soon. Yeah. Cool. So, with incident response, uh let let maybe you can just walk us through a little bit of the standard, uh, the TIR standard, uh, 97 I believe it is, and then what the high level process is, then we'll unpack that a little bit. Trevor: Well, I think we should sort of walk back into general incident response and then we can dive into the medical side of things. So, incident response is essentially the next steps after a breach, a vulnerability, a hack has been uncovered. What is the corrective action? How do you figure out what led to this hack happening and how do you prevent it from happening again? So, in a traditional cybersecurity perspective, this is usually going to be in a network where uh sys- uh systems administrator or maybe a data engineer or an architect, someone sees that something weird is going on. There's unaccounted for traffic, there's a rogue device, there's something that doesn't line up with their expected baseline. And that's a sign that they've been hacked. And they take a look, you can review, you know, network traffic, you can... Christian: Well, it could be, it could be an anomaly, right? We've had clients that thought they were hacked but it was actually an anomaly and they had to run it to ground. So it's not always somebody hacking into you, it's it's like you said, anomalous behavior we have to figure out what what what's the root cause of it. Trevor: Yeah, sometimes it's just computers being weird, which happens quite a lot. So, it's not always, you know, doomsday situation, you've been hacked. Often times it's just something weird going on. But you want to know whether or not someone was hacked or if it was just weird behavior. So you track it down, you look at, you know, networks on your peri- or components on your perimeter, so your VPN access, your firewalls, things like that. See if there's a problem and you try to get down to the root cause of what's going on. Now, in medical devices, rarely is a device going to be a connected network. It can be in a connected network, but it isn't the network itself. It's a single component, so you're usually going to have a pretty targeted search if something goes wrong. Uh, in the medical space luckily, the more common way that these vulnerabilities come up is through benevolent security researchers as opposed to malicious hackers. Uh, but that's not always the case. It can often just be even a user that notices weird behavior. So they are going to have a process to inform the manufacturer or in some cases inform, you know, whoever's handling their security if that's outsourced that there's a problem in this product. Maybe it's even just through an annual penetration test or, um, more often it'll be a coordinated vulnerability disclosure in the product where they say, I identified that this is an issue. Um, it happens when I do this and this is what I think the impact could be. Christian: How is one of these security researchers I mean, how are they going to do this? They just like find an old device on eBay or something and try to hack into it? But I mean, they're not going to be in a live hospital unless they're a penetration tester like and they happen to run across a medical device. Trevor: More often, it'll be just a user who identifies weird behavior. So, a lot of vulnerability reports even from penetration testers and security researchers are just not problems. They just identify something weird and they say, I think this is a vulnerability and then they disclose it even if it's just an intended behavior they didn't know about or the device is just acting up for some reason. Doesn't necessarily mean it's a vulnerability, but sometimes it does. And so that's why these vulnerability disclosure panels are so important and vulnerability reporting is such a critical process. Um, but yeah, if in the context of a security researcher, this will be pretty common with software as a medical device as opposed to a physical device. Software as a medical device, if it's put out on the open internet, or if it's an app you can get on your phone, then anyone can get it. And so a security researcher might just be, you know, doing it out of the good of their heart. They want to try to push forward that security posture across the industry. They don't want anyone else to get hacked, and so they're trying to help people out. That's what vulnerability disclosure programs are for. That's what bug bounty programs are for. It's security researchers just trying to make the world a little bit more secure. Uh, so I'd say that's the most common use case that we're going to see is under software as a medical device as opposed to a physical device. Well, while it is still possible. Christian: With with the physical devices, one of the things I've noticed is they're often constrained by memory and other things, so the logging functionality isn't the same as like an IT device. So from an incident response perspective, without like solid log files, it's kind of hard to track down the incident, isn't it? Trevor: Yeah, it can be pretty tricky. And, um, you know, some devices will have a full operating system, they're a full, they basically a computer with bonus parts. And then it's not too hard. You'll have those logs usually for, you know, up to 90 or 180 days, whatever is going to be the standard set by the manufacturer. But yeah, sometimes you have just a device that it's running on a real-time operating system and it's generating minimal if any logs at all. And then it can be difficult to figure out what happened without just trying to reproduce it. And so that's where security triage can come into place. A security professional can just keep trying to mess with the device, keep trying to replicate the behavior that was reported. Um, they can work with the security researcher or the doctor, the patient, whoever discovered this problem and say, okay, what were you doing when this happened? You know, what buttons were you pressing? How can we try to make this happen again? As soon as they figure out what caused it and they're able to reliably reproduce it, then it's either a consistent anomaly, or if it has a cybersecurity or a safety impact, then it's a known vulnerability. So, it's a much more difficult process when you're trying to do it by essentially just guessing until you get it, uh, as opposed to having logs, having these resources available where you can pin down exactly what happened and reproduce it very easily through those means. Christian: Right. Let's let's uh, back up for a second, like the sources of vulnerabilities because with postmarket management, you've got the software bill of materials, which could identify a vulnerability. You've got the CISA KEV, you have an annual penetration test, at least an annual one. You have these security researchers reporting things to a potential coordinated vulnerability disclosure database. Um, what else is there, uh, from a perspective of discovering a vulnerability? Trevor: I think one often-overlooked area for vulnerability discovery is static testing and fuzz testing. I guess two areas. Static testing is going over the code to look for, essentially insecure coding practices. Are you using hard-coded credentials? Are you escaping bad data? Things like that. Um, fuzz testing is often going to look for pretty difficult to catch problems and it's super comprehensive. It's very good at covering the vast majority of problems where many other different types of security testing are a little bit more of casting a wide net on a shallow area. But fuzz testing drills deep into a specific input field or a functionality. It tries to do everything that you could possibly do to that input field or that functionality. It sends, you know, up to millions of requests or millions of different, you know, function calls, just under slightly different parameters, slightly different context. It'll try sending 10,000 characters, then, you know, -10 characters or something like that. It'll really try to just break the device. That I think is a great way to uncover difficult to find vulnerabilities and often times, pretty severe vulnerabilities. This can spot out some pretty nasty, uh, especially memory handling problems that can give you pretty unrestricted access to a system. Christian: Yeah, and as as a medical device manufacturer, like, I know we kind of briefly introduced, uh, TIR 97 and the FDA has some guidance on post-market mana- management and incident response. Uh, what else is there? Are those the two main ones? Trevor: Those are two of the main ones that we want to look for. Um, they talk about, you know, how you're handling device's security once it's out in the field. This is going to depend a little bit on your device class, it's going to depend on what you're going for as far as the certification. Um, for an example, with a PMA device, one of the requirements is you need to submit an annual PMA report to the FDA if you are submitting a device under this classification. As you're identifying vulnerabilities in a system, as you are remediating them, as you're scoring them, you need to document all of this information as part of your measures and metrics, which, you know, you should be tracking for all devices, but PMA reports are the only ones where you have to reliably keep giving them to the FDA. As you're collecting that information, when you're identifying vulnerabilities, you should document how you're finding them, where you're finding them, how you're fixing them, who's fixing them, what the priority is, all as much information as you can capture, and then give it to the manufacturer or to the FDA. That burden is a little bit less for a Class II device under a 510(k) program. You usually aren't required to go through as much of that. But what the post-market requirements really want to see is that you have a way to collect all these vulnerabilities, you're tracking this information, and you're doing something with it. As you see, you know, oh, we're getting three vulnerabilities a month that are of high or greater severity. That's a pretty big number. There might be some corrective actions that need to be done to try to bring that down. So just how you're triaging vulnerabilities and essentially learning from it. Christian: And where are these vulnerabilities tracked? Is are these put in the QMS and the CAPA as you alluded to as well? Trevor: It is really going to be up to the manufacturer how they want to track them. Um, the easiest solution and what I usually try to recommend is ticketing software like Jira or Pivotal Tracker. So, you know, many, pretty much any software developer is going to be familiar with Jira, but it may not be as commonplace for a lot of other people. And so essentially what it does is it lets you raise a problem in software or in a device in a development process and say, hey, we need to do X thing, even if it's not a problem, even if it's just we have to add this new functionality, then you put in a ticket, which is that specific request to do this functionality, and then someone is going to implement it, whether it's a security fix or a new feature, and then they'll close that ticket. So these tools usually will let you track how long was the ticket open for, what was the severity, how long are tickets open on average based on severity, which is a great way to fulfill the need of FDA metrics. So that's why I usually recommend that manufacturers just use ticketing software that more often than not they already have in place. Christian: Right. Right. Can, can you talk a little bit about, uh, from my experience of incident response, I've done quite a bit in my career and I used to be in the DoD, we did it there quite a lot. One of the biggest challenges is eliminating false positives. Do you have any, uh, words of wisdom on, because I know with our SAS tools, they're often like super prone to false positives. Do you have any advice on false positives and how to eliminate them? And maybe you can explain what a false positive is first. Trevor: So, a false positive is when a tester, a tool, a scanner, something says there's a problem when there isn't. Uh, this can come up for a lot of reasons. Um, with scanning tools, often times it'll say, 'Oh, like with static testing.' Static testing is reading through your code to look for vulnerabilities. It'll say, 'Oh, you aren't escaping um, bad data in this function. That is a vulnerability.' But if you're escaping what gets fed into that function, then it's likely not going to be a vulnerability altogether. Uh, but the static tool doesn't know that. It doesn't understand how these different parts of your code connect together. It only sees the vulnerability in a vacuum. So, that doesn't mean you have to fix it. I mean it's still going to keep showing up in these static tools, but it's something that you can just say, no, we know we have a mitigation for this. So even a lot of static testing tools like Snyk is a very popular one, and that's what we use. Uh, you can just flag a problem and say, we know this is fine, this is a false positive and Snyk will just ignore it in future scans. Uh, so it's pretty common to see false positives pop up. Now, this can come in in a little bit of a complex situation in the post-market since as you're identifying vulnerabilities, you have to do something about them. In that pre-market phase, you have so many different problems, you're just going through everything, trying to get it down to the minimum that you can and once it's secure, great, you submit it. But in the post-market phase, you have this device, it's fielded, if a vulnerability comes up, you're going to have to push out fixes to a lot of devices. So, getting it right is really important and a lot more important than in the pre-market phase. That's why we really need to run false positives to ground. If it's not going to be an applicable issue, then it's not something that we should waste our energy on. It's a lot of energy to fix a big problem in medical devices. Now, often times, this is going to be through scanning tools that are picking up false positives more so than like manual testing. Like penetration testing is going to be far more accurate since a penetration tester can usually see the immediate impact. But even then, sometimes pen testers get it wrong and they don't understand a specific functionality as well as maybe the manufacturer. Uh, but trying to get through false positives is going to lighten the burden on the manufacturer a little bit and just making sure that everything actually is a false positive is one of the key areas of that. If you're saying something's false positive and just marking it out of scope when it still has an impact, that's going to be a much bigger problem than trying to fix a problem that isn't there. Christian: And the, the converse of a false positive is a true positive. Trevor: Yes. Christian: Which is actually is a vulnerability. And then once a vulnerability is discovered, there has to be a process to figure out what the real risk with that vulnerability is because not every single new vulnerability has to be fixed if it's to an if it has an acceptable risk. Trevor: Right, and that's sort of that triage process. So, when we're looking at a vulnerability, we want to understand what the actual impact is, and often times this is going to be different from what a scanning tool says or something like that. Uh especially in medical device context. You know, as we always talk about, we're less concerned with traditional cyber security metrics and more focused on patient harm, patient data loss, things like that. Those are really important when we're in this medical space. So, if we're using a tool for static testing, it doesn't know how this product is used. It just says, 'Well, this is the vulnerability, here's how we map it to the confidentiality, integrity, availability.' And that's all it's going to do. And then it's going to give it a score based on that. Now, if we look at it and we say, 'Okay, the availability problem is there could be downtime for 15 minutes.' But this is a device to diagnose cancer. 15 minutes is not going to make a difference. It's going to be pretty minimal impact compared to if you have a, you know, a critical system handling finance or something like that, 15 minutes is a massive amount of time in that industry. Uh and so it might say, oh, we think this is a high finding and then we take a look at it and we go, well, not really, cancer can wait 15 minutes, it can't wait, you know, six months, but we're going to say this is a low. And then that gets pushed down to the bottom. And in the same vein, it might say, oh, we discovered this information disclosure vulnerability, we think it's a medium. The scanning tool says, oh, we can, you know, disclose information from this field. And we take a look at that field and it's someone's Social Security number. Yeah, it's a small amount of information, but it's really, really important information or diagnostic results tied to a patient. Having that vulnerability present is, you know, not only super dangerous for the patient but it's a HIPAA violation and so we might say, this is going to get bumped up to a high, even though the scanner says it's only a medium. So having that triage process in place and saying, well, we know what we need to prioritize. These criticals are top of the stack, these lows, we'll get around to it when we can. Christian: And that's why it's important to have a risk methodology to apply because from my experience the scanners, the tools, like you mentioned, are really just a starting point. They do their best to say this is a critical risk and a lot of people take that at face value, but when you actually apply it to your own risk methodology, it may not be a critical risk. It maybe a low risk and the reverse is true too. A low risk could actually be a critical risk as well. Trevor: Definitely. So one interesting thing that comes up in the post-market that I want to talk about a little bit is how your exploitability can change. So, exploitability is essentially what factors must be true for a vulnerability to be realized. When we're looking at the pre-market phase, so exploitability for FDA standards is based off of CVSS 3.0 in a lot of context. Um, and what we're looking at is what level of authentication does the attacker need, what proximity do they need? Can they do it over the internet? Do they need physical access? Uh, how complex is the attack? Are there conditions outside of the attacker's control? And then finally, is user interaction required? So is it an attack against a person or just purely the network, or purely the device? These can change in the post-market. There is more stuff that we can start considering. We're getting information in from a coordinated vulnerability disclosure, we can say, well, how confident are we in this report? Um, is there exploit code available for this? Is there mature exploit code available for this? So, the factors that we're considering can actually shift a little bit in post-market phase. And I don't think that's always known explicitly by manufacturers. And so they're using the exact same process for their pre-market assessments as they are in their post-market assessments. So that's another thing that manufacturers need to be conscious of is as their device goes through its life cycle and as the device reaches maturity in the market, there's more that you have to consider for security. It's not a one-and-done process and you can't just keep using the same process because the environment changed, the context changed. Now there are actual users of the product where before it was just security testers. Christian: Yeah, I think the exploitability is a good point and something that we monitor is the CISA KEV, the known exploitability, uh, vulnerable database. Does that what it stands for? Trevor: Known exploited vulnerabilities. Christian: Known exploited vulnerabilities, yeah. Because, yeah, because you can have a vulnerability but if there's no associated exploit with it, then it doesn't matter as much. And the whole idea behind the CISA KEV is to let you know if somebody came up with a new exploit for an existing vulnerability, which could quickly change the risk rating of that vulnerability. Trevor: Yeah, that's a really good point and that goes into that exploit maturity, which is one of the post-market factors the FDA wants to see manufacturers considering. Um yeah, they're, I think it's less than 1% of CVEs, CVEs are essentially a vulnerability tied to a specific component, device or product. Uh, I think less than 1% are known exploited vulnerabilities. So, more often than not when a CVE comes up, it's a security researcher who's testing a product and identifies a problem, discloses it to the manufacturer and the manufacturer fixes it before anyone else can ever touch it. And so, then it's, you know, between the manufacturer and between the security researcher, what happened and maybe in the case that no one else even knows what the vulnerability could be and nobody else is going to figure it out. That does happen all the time. There are, you know, dozens of CVEs that pop up every single day and almost all of them are like that. Now, sometimes, there'll be a vulnerability that just enters the KEV immediately. Um or something gets promoted to the KEV if someone researches an old vulnerability and decides to disclose how it worked and publish that exploit to the, you know, out onto the internet, then it can get promoted into the KEV. But more often than not, it's going to be a vulnerability in a widely used tool, since those are going to be the more priority targets for hackers. They want to look at something like Google, they want to look at a VPN appliance that tons and tons of different people use. Uh, those are going to be much more fruitful targets for a hacker as opposed to some obscure software library only used in, you know, 200 products. So, those are where we're seeing KEVs come up more often. It's in common tools, something that a hacker's going to research a lot because they know they're going to get a lot of reward out of it. Or a security researcher is going to research a lot so that they can arm other security researchers with the tools to test and verify this vulnerability on a massive scale. Um, but yeah, having something in the KEV, that definitely indicates, you know, it's a mature problem, it's something that really does need to be addressed by the manufacturer. Christian: And before we wrap here, wrap up here, I'd like to talk about the CVD a little bit and what the purpose of the CVD is and what the process looks like on the back end a little bit, once somebody submits something to the coordinated vulnerability disclosure database. Trevor: It depends on how well-managed the CVD is. Of course. I'll talk about our process. So, we have our CVD online which we use for our post-market management clients. Uh, we'll say that, you know, a device or a vulnerability is discovered in XYZ device. The security researcher is going to look at the labeling for the device or it may be on the manufacturer's website, you know, what do I do to disclose a vulnerability? That's going to point them to our CVD page. They open it up, it's going to ask them a ton of questions. says, you know, what product are you looking at? Do you think this affects other products? How did you discover it? What do you think the impact could be? You know, can you explain in exact detail how you reproduce this vulnerability? Can you give us pictures, videos? And we try to collect as much information as we possibly can. That gets sent to us where we take a look at all this information. You know, for any of these devices that we're handling under post-market, we have a copy of the device or we have access to the application, and then we go in and we start trying to triage the vulnerability. Now, it could be just an overzealous security researcher trying to find a problem where there is none and we say, well, there's nothing there. And then we go ahead and say, yeah, this wasn't applicable and close it out. Uh, on the other hand, it could be a big problem. And so we look at it, we reproduce what they said to do and we go, yeah, this is a legitimate finding. We compare it against our risk assessment, which again is going to change a little bit in that post-market phase opposed to the pre-market phase and we say, okay, we're confident, we're going to say this is a high. We'll go to the manufacturer, we say this is the problem, this is how it was discovered, this is a high finding, we need to do something about this. Here's the corrective action. And then the manufacturer fixes it. If they're under responsibility for one reason or another to disclose it to the FDA, they're going to have to disclose it to the FDA. Typically those disclosure windows have to be within 30 days and they have to fix it within 60. Um or if they need to disclose it to users so they can implement a band-aid fix. All that's going to be a little bit more specific on the vulnerability what they have to do next, but that's essentially what that basic process looks like. Christian: Cool. Cool. Any uh, last words of wisdom here? We're kind of running up on time. I like to always ask you for last words of wisdom. Trevor: Yeah, I've noticed. Christian: Put you on the spot, I guess. Trevor: Um, I think just being aware that cybersecurity changes. Uh, you can't have the same process. You can't, you know, start with one thing and you can't have that same thing going on three years later after your device has been cleared. You need to be constantly evolving, constantly on your toes, considering new factors and always keeping an eye on your product once it's out in the market. Christian: Yeah, because a product recall because of a hack could be very costly to a manufacturer. So want to prevent that and if there is a vulnerability you want to get it addressed immediately. All right, awesome. Well, thanks everyone for tuning in. I hope you learned a little bit more about incident response and post-market management with medical devices and we hope to see you next time.