Generative AI Medical Device Security: LLM Risks and Controls

Reviewed October 13, 2026

Device makers are adding chat-style AI to companion apps, clinician portals, and support tools: summarizing readings, answering patient questions, drafting reports. The model is helpful because it follows instructions written in plain language. That is also the problem. Anyone who can get text in front of the model, through a chat box, an uploaded document, or a record it reads, may be able to change what it does. If the model can see patient data or trigger actions, a few sentences of hostile text can turn into a privacy breach or a wrong instruction to a patient. Most of the security controls you already use still apply. The difference is that the model itself cannot be trusted to enforce rules, so the rules have to live around it.

Why This Matters

The FDA's February 3, 2026 final premarket cybersecurity guidance, "Cybersecurity in Medical Devices: Quality Management System Considerations and Content of Premarket Submissions," expects a threat model, a security risk assessment, and testing that cover the whole system, including connected apps and cloud services. An LLM feature sits inside that system, so it belongs in the same documents. The FDA's January 2025 draft guidance on AI-enabled device software functions goes further and names AI-specific threats, such as data poisoning and model inversion, as things to address in the cybersecurity section. That guidance is still a draft, but it shows where reviewers are looking. Hospitals are asking too: security questionnaires now include questions about what data AI features send to outside services and who can see it. A device team that can show where the model sits, what it can reach, and how it was tested answers those questions once.

Official sources: FDA, Cybersecurity in Medical Devices: Quality Management System Considerations and Content of Premarket Submissions (February 2026); FDA draft guidance, Artificial Intelligence-Enabled Device Software Functions (January 2025); FDA guidance, Predetermined Change Control Plans for AI-Enabled Device Software Functions.

What Makes an LLM Feature Different?

A normal app feature does what its code says. An LLM feature does what its input suggests, within whatever limits you put around it. That changes three things:

1. Instructions and data are mixed. The model cannot reliably tell your instructions apart from text a user or document supplies.
2. Output is unpredictable. The same question can produce different answers, and some will be wrong in a confident tone.
3. New parts to protect. The model file, the system prompt, the retrieval index, and any outside AI service all become part of the attack surface.

The OWASP Top 10 for LLM Applications lists prompt injection as its first risk, followed by issues such as sensitive information disclosure, supply chain risks, and excessive agency. It is a practical checklist for scoping.

Which Threats Matter Most for Devices?

What Controls Should Be in Place?

Keep rules outside the model. Access checks, rate limits, and allowed actions belong in your own code. The system prompt can guide tone and format, but it is not a security control.

Least data, least tools. Pass the model only the records the signed-in user is allowed to see. Give it read-only tools where possible. Any action that changes therapy, settings, or orders needs explicit human confirmation.

Separate untrusted content. Clearly mark retrieved documents and user uploads as data, strip hidden text where you can, and never let them change which tools are available.

Check outputs. Filter for patient data that should not appear, block or flag dose and setting values, and show clinical content with a clear note that a clinician must confirm it.

See also: AI Model Integrity for Medical Devices: Signing and Updates, AI Training Data Security for Medical Devices: FDA Evidence, and Hospital Security Questions for AI Medical Devices: Answers.

Protect the AI supply chain. Sign and version model files and prompts, list models and AI libraries in your SBOM, and control who can change the retrieval index.

Decide where the model runs. If you call an outside AI service, document what data leaves, under what agreement, and how it is retained. Hospitals will ask.

Log and monitor. Record prompts, tool calls, and blocked outputs (without storing more patient data than needed) so you can spot abuse and investigate incidents.

How Do You Test an LLM Feature?

Standard pen testing still covers the app, API, and cloud around the model. On top of that, test the model's behavior directly:

• Direct and indirect prompt injection, including instructions hidden in files and records
• Attempts to pull other users' data, the system prompt, or secrets
• Tool and function-call abuse, including actions the user should not be allowed to trigger
• Unsafe clinical outputs under adversarial and ordinary questions
• Tampering with model files, prompts, and retrieval data in the update path

Because answers vary, run each test many times and record how often it succeeds, not just whether it succeeded once. Rate each finding with CVSS and a patient-harm rating, and trace it to the threat it confirms. See our cybersecurity hazard analysis guide for turning those findings into risk file rows.

How Blue Goat Approaches This

We start with the threat model. The LLM goes in as its own component, with each input and tool drawn at a trust boundary, and each threat tied to a hazard in your ISO 14971 file. We then test the feature the way an attacker would: prompt injection through every input path, data extraction, tool abuse, and model and prompt tampering, alongside the usual app, API, and cloud testing. Findings get CVSS and patient-harm ratings and trace back to the threat model, so the report is evidence a reviewer can follow. Retests are unlimited until findings close. Full details are on our AI/ML Medical Device Security page, and our AI pen testing limits post explains where automated AI testing tools fall short.

FAQ

Does the FDA have specific rules for generative AI in devices?

Not yet as a final guidance. The February 3, 2026 premarket cybersecurity guidance applies to any device software, including LLM features, so the threat model, risk assessment, and testing must cover them. The FDA's January 2025 draft guidance on AI-enabled device software functions names AI-specific threats, but it is still a draft. Treat it as a strong signal of what reviewers will ask, not a binding requirement.

Is a system prompt enough to stop prompt injection?

No. A system prompt tells the model what to do, but a well-crafted input can override it, and no prompt wording has been shown to stop injection reliably. Enforce access checks, allowed actions, and data limits in your own code outside the model. Use the system prompt for tone and format, and assume an attacker can make the model ignore it.

Does using an outside AI service change our submission?

It adds a component to describe. Your architecture views and threat model should show the outside service, what data is sent, and how it is protected in transit and at rest. Your SBOM should list the models and AI libraries you depend on. Hospitals will also ask about data retention and agreements with the provider, so document those before review.

Do LLM features count toward the cyber device definition?

Section 524B covers devices that include software, can connect to the internet, and could be vulnerable to cybersecurity threats. An LLM feature that calls a cloud service usually means the device already meets that definition. Either way, the feature belongs in your threat model and testing, because the FDA expects the whole system to be covered.

How often should we retest an LLM feature?

Retest whenever the model, the system prompt, the retrieval data, or the tools change, since any of these can change behavior. Many teams also add a short automated injection test set to every release. If your device has a PCCP that covers model updates, name the security tests and pass criteria for each planned change in the plan.

CTA

Adding a chat-style AI feature to your device or app? Book a free 30-minute discovery call. We'll review where the model sits, what it can reach, and what testing your submission needs, then send a fixed-fee proposal. Nothing is billed until you approve it.