Listen now
Key takeaways
- AI development for medical devices demands the same rigorous, risk-aware engineering principles to ensure safety and regulatory compliance.
- The quality of training data is paramount for AI models; poor or biased data can lead to inaccurate diagnoses and compromised safety.
- Emerging certifiable standards like ISO 42001 for AI management systems are crucial for establishing verifiable and traceable frameworks essential for regulatory approval.
- The regulatory scrutiny of an AI system depends significantly on its intended use, ranging from exploratory data analysis to critical diagnostic decisions.
- Deployment architecture (on-device, on-premise, or cloud) impacts an AI model's cost, performance, data privacy, and cybersecurity.
- Determining legal liability for AI systems in medical contexts when a mistake occurs is a complex and largely unresolved issue.
- Converting complex AI models into simpler mathematical forms can make them viable for resource-constrained medical devices and facilitate verification.
- AI systems need to be developed to communicate their uncertainty, moving away from 'hallucinating' answers toward more collaborative interactions with human users.
What does responsible AI implementation look like in medical devices?
This episode explores the intersection of AI, cybersecurity, and medical device regulation with guest Matt Lemay, CEO of Lemay.ai. Hosts Christian Espinosa and Trevor Slattery of Blue Goat Cyber dig into how AI models are trained, certified, and deployed in clinical contexts - and what can go wrong.
Key points:
(7:29) Data, Security, and Deployment Risks
-
Training data inconsistencies and data drift in AI models.
-
Cybersecurity concerns tied to cloud deployment and version control.
(11:48) Can AI Prescribe Medication?
- Legal and liability implications of AI autonomy in healthcare.
(22:35) Risks and Regulation
- Expectations for AI-enabled device regulations in the EU and US.
(33:35) AI Answers
- Thoughts on how AI has a hard time admitting it doesn't know the answer to something.
Notable quotes
“The time for medtech companies to engage with AI was somewhere between not yet and five years ago.”
“The biggest thing that can come up, especially in that diagnostic space, is if the AI is not properly trained on the correct data.”
“There needs to be a bit of adjustments there, but typically we tend to see that as long as people are pursuing the best type of model, then you have a lot of advantages in making sure that your solution is delivered successfully.”
“The regulations will always try to find what is the most amount of protection while balancing innovation, permissiveness with public safety. So together, there's a fine line that you can navigate.”
Frequently asked questions
Bring this work to your device
Need help with penetration testing?
Blue Goat Cyber delivers medical device penetration testing for medical device manufacturers - from threat modeling to FDA-ready reports.
Medical Device Penetration TestingMore on Penetration Testing
Keep listening
-
Episode 66
Vibe Coding Security Risks & Malicious Injection with Jake Rodriguez of Triangle Tech
With Jake Rodriguez
-
Episode 60
How to Move Stakeholders from Awareness to Sustained Adoption Without Friction
With MedTech leader
-
Episode 46
How Market Intelligence Shapes MedTech Growth with Kevin Saem
With Kevin Saem
-
Episode 42
What Is A Medical Cyber Device?