The Hidden Cybersecurity Risks When Doctors Use AI Diagnostics

In this episode of the Med Device Cyber Podcast, hosts Trevor Slattery and Christian Espinosa are joined by special guest Jun Xiang Tan, the owner of TuringLabs, who is currently working with a health-tech startup in Singapore. Jun Xiang brings a unique perspective, with a background in military cybersecurity and network forensics before transitioning into the AI and health-tech space. The conversation centers on the burgeoning use of Artificial Intelligence in healthcare and the significant, often overlooked, cybersecurity challenges it presents. The discussion kicks off by highlighting the alarming trend of 'shadow IT' in clinical settings. Christian Espinosa points out that studies show almost 25% of clinicians use unauthorized AI tools like ChatGPT for diagnostic support. Jun Xiang elaborates on this, noting the convenience for doctors to quickly input patient symptoms, text, or even upload X-ray images to get instant feedback. This practice, however, introduces massive data privacy and compliance risks, as sensitive Protected Health Information (PHI) is fed into public models that may use it for future training, essentially creating a major data breach.

The podcast then delves into the core vulnerabilities of AI systems themselves. A primary concern raised is data poisoning, a type of adversarial attack where the AI's training data is manipulated. The hosts discuss a case where poisoning just 0.001% of the training data resulted in a 5% increase in incorrect outputs. In a healthcare context, such inaccuracies could lead to misdiagnoses and severe patient harm, underscoring the 'garbage in, garbage out' principle. The conversation also scrutinizes the quality of AI-generated code. Citing recent statistics, Christian notes that nearly 50% of code written by AI introduces new security vulnerabilities, such as cross-site scripting. This is largely because AI models are trained on vast repositories of public, open-source code from platforms like Stack Overflow, much of which is outdated, insecure, or written by inexperienced developers. The AI, therefore, learns and replicates these poor security practices, creating bloated and vulnerable codebases that require significant manual effort to clean up and secure.

The episode contrasts the rapid, often unregulated development of commercial AI with the stringent, safety-critical standards of the medical device industry, such as IEC 62304. This standard dictates a rigorous, safe development lifecycle that current AI tools cannot replicate. The hosts and guest conclude that while AI offers powerful capabilities as a support tool—a 'pair programmer' or a clinical decision support system—it cannot be trusted to operate autonomously. The risk of hallucinations, biases, and security flaws necessitates constant human oversight. The ultimate message is to guide the AI, not let it guide you, by providing it with clear requirements and verifying its output, ensuring that patient safety remains the paramount concern.

Key Takeaways

• A significant number of clinicians (almost 25%) are using unauthorized AI tools like ChatGPT for diagnostic help, creating major privacy and compliance risks by uploading sensitive patient data.

• AI models are vulnerable to 'data poisoning,' where a minuscule amount of corrupted training data can lead to a disproportionately high rate of incorrect and potentially harmful outputs.

• Nearly 50% of AI-generated code introduces security vulnerabilities like cross-site scripting because the models are often trained on insecure, outdated, or low-quality public code.

• The convenience of AI is a primary driver of its adoption in healthcare, but it can lead to unintentional data breaches and HIPAA violations when PHI is entered into public systems.

• Medical software development is governed by strict safety standards like IEC 62304, which emphasizes a controlled and secure lifecycle that contrasts with the rapid, less-regulated nature of commercial AI development.

• The 'garbage in, garbage out' principle is critical for AI in healthcare; models trained on unreliable or biased data will produce flawed results with potentially severe consequences for patient safety.

• AI should be treated as a guided tool or a 'pair programmer,' not an autonomous decision-maker. Human oversight is essential to define requirements, verify outputs, and mitigate inherent risks.

Listen on mdcpodcast.com · Watch on YouTube