Understanding the Distinction Between Measures and Metrics
In the world of medical device cybersecurity, the terms “measures” and “metrics” are often used interchangeably; however, they represent distinct concepts that are crucial to understand, especially when navigating the regulatory landscape established by the U.S. Food and Drug Administration (FDA).
As Christian Espinosa and Trevor Slattery, the hosts of the Med Device Cyber Podcast, explain, a measure is a quantifiable attribute, such as the time it takes to apply a software patch or the number of security incidents that have occurred. On the other hand, a metric is a calculation derived from one or more measures, typically expressed as a percentage or ratio. For example, the percentage of identified vulnerabilities that have been patched is a metric, while the time it takes to patch those vulnerabilities is a measure.
This distinction is crucial because the FDA has very specific expectations regarding the measures and metrics that medical device manufacturers must track and report. Failing to understand the difference can lead to confusion and potentially jeopardize the regulatory approval process.
What the FDA Expects: Measures and Metrics for Medical Device Cybersecurity
According to the guidance provided by the FDA, there are three key measures and metrics that medical device manufacturers must focus on:
- Percentage of identified vulnerabilities that are updated or patched – This metric represents the proportion of known vulnerabilities that have been addressed through software updates or patches.
- Duration from vulnerability identification to patch availability – This measure tracks the time it takes to develop and release a patch or update to address a vulnerability that has been identified.
- Duration from patch availability to patch deployment – This measure focuses on the time it takes to actually roll out the patch or update to all fielded devices.
These measures and metrics are crucial because they provide the FDA with a clear understanding of how medical device manufacturers proactively identify, address, and mitigate cybersecurity vulnerabilities throughout the product’s lifecycle.
It’s essential to note that the FDA’s expectations regarding these measures and metrics extend beyond the initial device submission process. In fact, the agency requires manufacturers to continue collecting and reporting on this data as part of their post-market surveillance efforts, which are typically submitted through annual reports or other regulatory filings.
Addressing the Challenges of Measure and Metric Collection
While the FDA’s requirements may seem straightforward, the reality of collecting and reporting on these measures and metrics can be more complex, especially for medical device manufacturers that may not have robust cybersecurity monitoring and incident response processes in place.
One of the key challenges is the sheer volume of vulnerabilities that can be identified in medical devices, particularly as the complexity of these devices continues to increase. As Espinosa and Slattery point out, it’s not uncommon for a single device to have hundreds or even thousands of identified vulnerabilities. Triaging and prioritizing these vulnerabilities based on their risk profile is crucial, as the FDA is primarily concerned with the remediation of critical and high-risk vulnerabilities.
Another challenge is the lack of real-time monitoring and alerting capabilities on many medical devices. Unlike traditional IT systems that are often integrated with security operations centers (SOCs) and other monitoring tools, many medical devices operate in relative isolation, with limited visibility into security events and incidents. To address this, Espinosa and Slattery recommend that medical device manufacturers design their products with built-in alerting mechanisms that can notify users of anomalies or security events in a clear and actionable way.
Additionally, the FDA’s focus on measures and metrics related to patch management and deployment highlights the importance of having a well-defined and efficient patching process. This can be particularly challenging for medical devices that may be deployed in a variety of environments, from hospitals to private homes, each with its own unique security considerations and constraints.
Incorporating Risk Profiles and Actionable Insights
While the FDA’s requirements for measures and metrics provide a solid foundation for medical device cybersecurity, Espinosa and Slattery emphasize that these data points should not be viewed as the end goal, but rather as a starting point for more comprehensive security efforts.
One key aspect that the hosts highlight is the importance of incorporating risk profiles into the collection and analysis of cybersecurity measures and metrics. The level of risk associated with a particular vulnerability or security event can vary significantly depending on the device’s intended use, the environment in which it is deployed, and the potential impact on patient safety and care.
For example, a medical device used in a hospital setting may face a higher risk profile than one used in a home environment, due to factors such as the increased likelihood of network-based attacks and the potential for more sophisticated threat actors. By understanding these nuanced risk profiles, medical device manufacturers can prioritize their remediation efforts and ensure that the most critical vulnerabilities are addressed in a timely manner.
Additionally, Espinosa and Slattery stress the importance of making the collected measures and metrics actionable, rather than simply focusing on compliance. This means using the data to drive meaningful improvements in the device’s security posture, such as optimizing patch management processes, enhancing incident response capabilities, and implementing more robust security controls.
Navigating the Regulatory Landscape: When to Include Measures and Metrics
One common misconception that Espinosa and Slattery address is the timing of when medical device manufacturers need to include measures and metrics in their regulatory submissions. The hosts explain that the requirement to provide this data is not always applicable, especially for new devices or those without a predicate device to reference.
For devices that are being submitted for the first time, the FDA does not necessarily expect the manufacturer to have a complete set of measures and metrics ready for the initial submission. Instead, the agency requires that the manufacturer provide a plan for how they intend to collect and report on these data points in the post-market phase.
However, for devices that have a predicate or previously approved version, the FDA may expect the manufacturer to have a more robust set of measures and metrics available, as they should have been collecting this data for the existing product. In these cases, the manufacturer should be prepared to include the relevant measures and metrics as part of their regulatory submission.
Regardless of the device’s history, Espinosa and Slattery emphasize the importance of having a well-defined plan in place for collecting and reporting on cybersecurity measures and metrics, even if the data is not immediately available. This proactive approach can help medical device manufacturers navigate the regulatory landscape more effectively and demonstrate their commitment to ongoing security improvements.
Conclusion: Embracing Cybersecurity Measures and Metrics for Improved Patient Safety
The FDA’s focus on cybersecurity measures and metrics for medical devices is a clear indication of the growing importance of this issue in the healthcare industry. By understanding the distinction between measures and metrics, and aligning their practices with the agency’s expectations, medical device manufacturers can not only navigate the regulatory landscape more effectively, but also enhance the overall security and safety of their products.
As Espinosa and Slattery have highlighted, the collection and analysis of these data points should be viewed as a starting point for a more comprehensive cybersecurity strategy, one that incorporates risk profiles, actionable insights, and a commitment to continuous improvement. By adopting this approach, medical device manufacturers can play a vital role in ensuring patient safety and maintaining the trust of healthcare providers and the public.