Threat Modeling AI-Enabled Medical Devices: Worked Example

Reviewed October 14, 2026

Most threat models for AI devices draw the model as a single box inside "the app." That hides the parts attackers go after: the data the model learned from, the file that holds its weights, the inputs it reads at run time, and the process that ships new versions. When those parts are missing, reviewers notice, and the questions arrive as deficiencies.

This post shows how to add the AI parts properly. We walk through a fictional device step by step, from components and boundaries to threats, hazards, controls, and tests. The device, its numbers, and its findings are invented for teaching. They are not a client project and not an FDA decision.

Why This Matters

Section 524B of the FD&C Act requires cyber device makers to show a plan and processes for reasonable assurance of cybersecurity. The FDA's February 3, 2026 premarket cybersecurity guidance explains how: threat modeling across the whole system, a security risk assessment, architecture views, and testing that traces back to the model. It applies to all device software, including AI functions.

The FDA's January 2025 draft guidance on AI-enabled device software functions adds AI-specific threats to consider, such as data poisoning, model inversion, and evasion. It is still a draft and not for implementation, so treat it as a signal of reviewer questions rather than a binding rule. If you plan to update the model after clearance, the FDA's final PCCP guidance for AI-enabled devices expects each planned change to come with its own validation and impact assessment, which includes cybersecurity.

Meet the Example Device

DoseWise (fictional) is a companion app for an insulin pump. A model running in the cloud reads glucose readings, meal entries, and pump history, then suggests a bolus dose. The patient confirms the suggestion in the app, and the app sends the command to the pump over Bluetooth Low Energy. The model is retrained every quarter on new, de-identified data from consenting users.

The parts that matter for security:

1. Pump with firmware that accepts signed commands.
2. Phone app that shows suggestions and relays confirmed commands.
3. Inference service in the cloud that hosts the model.
4. Model file (weights plus a version manifest).
5. Training pipeline: data collection, labeling, training, and validation.
6. Training data store with de-identified records.
7. Model release process that promotes a validated model to production.

Step 1: Draw Trust Boundaries Around the AI Parts

Draw a boundary wherever data or control crosses from one owner or level of trust to another. For DoseWise, the AI-specific boundaries are:

• Phone app to inference service (patient data goes in, a dose suggestion comes out).
• Data collection to training data store (outside data enters the training set).
• Training pipeline to model registry (a new model file is produced).
• Model registry to inference service (a model goes live).

Each boundary gets authentication, integrity checks, and logging. If you cannot say who is allowed to cross a boundary, that is your first finding.

Step 2: Run STRIDE on Each AI Component

Then check the list against the draft guidance's seven AI threats: data poisoning, model inversion and stealing, model evasion, data leakage, overfitting, model bias, and performance drift. Overfitting, bias, and drift are mostly performance topics, but they matter here because an attacker can exploit or hide behind them. A drift monitor that also flags sudden shifts can catch slow poisoning.

Step 3: Turn Threats Into Hazards and Harms

A security finding only gets the right severity when you know what it does to a patient. For DoseWise:

This is the same CIA-to-harm method in our cybersecurity hazard analysis guide, applied to AI parts.

See also: AI Model Integrity for Medical Devices: Signing and Updates, AI Training Data Security for Medical Devices: FDA Evidence, and FDA PCCP Beyond AI: Cybersecurity & Firmware Uses.

Step 4: Choose Controls Outside the Model

The model cannot be trusted to protect itself, so controls sit around it:

• Dose limits in the pump firmware. The pump rejects any bolus above the patient's configured maximum, whatever the model suggests.
• Signed model files. The inference service loads only models signed by the release key, and checks the version against an allowed list.
• Data provenance. Each training record carries its source and ingestion date, so a suspicious batch can be found and removed.
• Separation of duties. The training pipeline cannot deploy. A named person approves each release, and the approval is logged.
• Rate limits and output rounding on the inference API to make membership inference harder.
• Drift and anomaly monitoring on suggestions in production, with a rollback plan.

Step 5: Map Each Control to a Test

These tests sit alongside normal app, API, cloud, and wireless pen testing. Our AI pen testing limits post explains why automated AI scanners alone do not cover them.

Step 6: Cover the Update Path

DoseWise retrains every quarter. If the maker files a PCCP, each planned model change should name the security tests that must pass before release, such as the signing and provenance tests above. A change outside those bounds needs a new submission. Our PCCP cybersecurity example shows how to write those bounds.

How Blue Goat Approaches This

We build the threat model with your engineers, not for them. AI parts go in as separate components with boundaries, and every threat ties to a hazard in your ISO 14971 file. We then test the controls the model depends on: model signing, data provenance, release approvals, API abuse, and the device-side limits that keep a bad suggestion from reaching a patient. Findings get CVSS and patient-harm ratings and trace back to the model. Retests are unlimited until findings close. See our medical device threat modeling and AI/ML Medical Device Security pages, or start a draft with the free Threat Model Starter.

Related reading: What a Medical Device Security Advisor Does After FDA Clearance.

FAQ

Do I need a separate threat model for the AI part?

No. Keep one threat model for the whole system and add the AI parts as components inside it. A separate AI model makes it harder to show how threats move between the app, cloud, and device. Reviewers want to follow one chain from architecture to threat to control to test. Splitting it invites gaps at the seams, which is where many real attacks start.

Is STRIDE enough for AI threats?

STRIDE is a good start because it forces you to think about each component. It does not name AI threats such as poisoning or model inversion directly, so check your list against the seven threats in the FDA's January 2025 draft AI guidance. Many teams map each AI threat to a STRIDE category, for example poisoning as tampering, to keep one consistent method.

How do we score severity for AI threats?

Score exploitability with a method such as CVSS, and score severity from patient harm in your ISO 14971 file. A poisoning attack may be hard to carry out but could harm many patients at once, so its severity stays high. Keep the two scores separate and explain how you combine them, since reviewers check that the method is consistent.

Does the threat model have to cover the training pipeline?

Yes, if the pipeline can change what the device does. Training data, labeling, and the release process all affect model behavior. The February 2026 premarket guidance expects the threat model to cover the whole system, and the draft AI guidance names data poisoning directly. A pipeline that is out of scope is a gap reviewers will likely ask about.

When should we update the threat model?

Update it whenever the model, data sources, architecture, or connections change, and at least once per release. Also update it when a new vulnerability or attack method appears that affects your parts. If your device has a PCCP, review the threat model before each planned model change and record the result in your release file.

CTA

Building an AI-enabled device and unsure your threat model covers the model, data, and updates? Book a free 30-minute discovery call. We'll review your current model and send a fixed-fee proposal. Nothing is billed until you approve it.