On this page
Published:
Key Takeaways
- The FDA's Feb 3, 2026 guidance explicitly enumerates 7 AI cyber threats as 524B obligations.
- Threats span the full ML lifecycle: training data, model weights, inference, and post-deployment drift.
- Each threat needs an entry in your threat model with a documented mitigation and test evidence.
- Generic application security controls are not sufficient, AI threats need AI-specific controls.
- Postmarket monitoring must cover drift and bias, not only CVEs.
- Document everything in your SPDF; the FDA will ask for it during premarket review.
The FDA's February 3, 2026 premarket cybersecurity guidance names seven AI-specific cyber threats manufacturers must address as Section 524B obligations: data poisoning, model inversion and stealing, evasion, data leakage, overfitting, model bias, and performance drift. Each threat needs a dedicated entry in the threat model with documented mitigations and test evidence, since generic application security controls alone do not satisfy these AI-specific risks.
Published June 29, 2026
Why This Matters
The FDA's final premarket cybersecurity guidance dated February 3, 2026 makes one point unmistakable: if your device meets the definition of a "cyber device" under section 524B(c) of the FD&C Act and incorporates AI, the AI is not a separate compliance track. It is part of your cybersecurity submission. The guidance lists seven categories of AI risk that "can be impacted by cybersecurity threats", and the FDA expects each one to be analyzed in your threat model, controlled in your Secure Product Development Framework, and verified before clearance.
This is a meaningful shift. Until recently, manufacturers could treat model quality as a data-science concern and cybersecurity as an IT concern. The 2026 guidance, read with AAMI CR34971 on AI risk management and the FDA's predetermined change control plan (PCCP) framework, fuses them. A reviewer can, and will, issue a cybersecurity deficiency for an AI-enabled device whose threat model does not address data poisoning or adversarial evasion. Standards including IEC 81001-5-1, AAMI TIR57 / ANSI/AAMI SW96:2023, AAMI TIR97, and ISO/IEC 27001 provide the surrounding controls, but it is the threat-modeling and SPDF evidence that the FDA reads first.
The seven threats below are taken directly from the guidance text. Each section explains what the threat is, why the FDA treats it as a cybersecurity issue (not only a model-quality issue), and what control evidence a reviewer expects to see.
At-a-Glance: 7 Threats → Test Types → Standards
This matrix is the fastest way to line up FDA's seven AI cyber threats with the tests you should run and the standards a reviewer will recognize. Every technique ID is from MITRE ATLAS; attack classes map to NIST AI 100-2 E2023, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations; practitioner test lists are from OWASP ML Top 10 and, for LLM-backed devices, OWASP LLM Top 10.
| # | FDA threat | Test types to run | NIST AI 100-2 class | MITRE ATLAS | OWASP |
|---|---|---|---|---|---|
| 1 | Data Poisoning | Training-data provenance verification; signed-manifest negative tests; backdoor / trigger scans on imported weights; canary-sample detection; red-team poisoning injection against drift/outlier detector | Poisoning (targeted, backdoor, availability) | AML.T0020 Poison Training Data; AML.T0018 Backdoor ML Model | ML02 Data Poisoning; LLM03 Training Data Poisoning |
| 2 | Model Inversion & Stealing | Membership-inference tests; attribute-inference tests; model-extraction / surrogate-fidelity tests under a defined query budget; rate-limit and auth pen tests against inference endpoint; output-minimization verification | Privacy (membership inference, model extraction, attribute inference) | AML.T0024 Exfiltration via ML Inference API; AML.T0044 Full ML Model Access; AML.T0057 LLM Data Leakage | ML05 Model Theft; ML03 Model Inversion; LLM10 Model Theft; LLM06 Sensitive Information Disclosure |
| 3 | Model Evasion | Adversarial robustness testing with FGSM, PGD, AutoAttack, C&W, adversarial patches; modality-specific attacks (audio, waveform, tabular); OOD-detector validation; confidence-threshold fallback tests; certified-robustness bounds where warranted | Evasion (white-box, black-box, transferability) | AML.T0015 Evade ML Model; AML.T0043 Craft Adversarial Data | ML01 Input Manipulation Attack; LLM01 Prompt Injection (for LLM-backed devices) |
| 4 | Data Leakage | Authenticated + unauthenticated pipeline pen test (object storage, feature store, model registry, logs, caches); IAM and encryption config scans; inference-endpoint verbose-output tests; debug-log PHI-exfiltration tests | Privacy (data reconstruction, side-channel) | AML.T0035 ML Artifact Collection; AML.T0025 Exfiltration via Cyber Means | ML06 AI Supply Chain Attacks (storage/logs); LLM02 Insecure Output Handling; LLM06 Sensitive Information Disclosure |
| 5 | Overfitting | Held-out and external-cohort validation; train/val gap thresholds gated in CI; decision-boundary sharpness tests; adversarial-overfitting simulations after training-data refresh; reproducible-training pipeline verification | Poisoning (availability) + robustness generalization | AML.T0018 Backdoor ML Model (as exploit path); AML.T0031 Erode ML Model Integrity | ML04 Membership Inference (indirect indicator); ML08 Model Skewing |
| 6 | Model Bias | Stratified subgroup performance tests (sensitivity, specificity, calibration by demographic and clinical subgroup); disparity-threshold gates; backdoor / trojan scanning on imported checkpoints; ML-SBOM provenance verification; postmarket subgroup-drift monitoring validation | Poisoning (backdoor); abuse (bias amplification) | AML.T0019 Publish Poisoned Datasets; AML.T0010 ML Supply Chain Compromise | ML07 Transfer Learning Attack; ML08 Model Skewing; LLM05 Supply Chain |
| 7 | Performance Drift | Drift-detector validation against benign vs. adversarial shifts; reference-distribution regression tests; signed-update negative tests (unsigned rejected); rollback tests; postmarket telemetry-integrity tests | Availability / integrity (adversarial drift) | AML.T0031 Erode ML Model Integrity; AML.T0034 Cost Harvesting | ML09 Output Integrity Attack; LLM04 Model Denial of Service |
GenAI / LLM-backed devices should additionally run: prompt-injection suites (direct and indirect), jailbreak suites, insecure-output-handling tests, and tool-abuse tests. Tooling: IBM Adversarial Robustness Toolbox (ART), Foolbox, CleverHans, Microsoft Counterfit, NVIDIA Garak for LLMs, Microsoft PyRIT for GenAI red-teaming, and Giskard for ML testing. Each threat's per-section table below drills into the specific SPDF artifact and submission evidence expected.
1. Data Poisoning
What it is. Cyber attackers deliberately inject inauthentic or maliciously modified records into training, fine-tuning, or feedback data so the model learns the wrong thing, for example, labeling malignant lesions as benign or shifting decision boundaries in a sepsis predictor.
Why it is a 524B issue. Data poisoning is not just bad data hygiene; it is an integrity attack on the model's training pipeline. The FDA treats the training pipeline as part of the device's attack surface when training artifacts or update channels affect the deployed model.
Mitigations.
- Cryptographically signed dataset manifests and per-record hashes; reject any training run whose corpus does not match the manifest.
- Role separation between data engineers, label reviewers, and model owners; least-privilege access to the training bucket and feature store.
- Statistical drift and outlier detection on inbound training and feedback data, gated before the next training run.
- A documented retrain/revert procedure with a known-good model baseline if poisoning is suspected post-deployment.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | Training-data provenance policy + signed manifest schema | Manifest-signature validation test (negative case rejected) | Data-integrity test report referenced in eSTAR Security Architecture section |
| Secure implementation | Access-control matrix for training infra and feature store | IAM/least-privilege review + log review | Access-review record and IAM export attached to threat model |
| Secure verification | Statistical drift / outlier detector on inbound data | Red-team poisoning injection against detector | Adversarial-test report; STRIDE/ATLAS row "Tampering, AML.T0020 Poison Training Data" |
| Secure release & response | Retrain-from-known-good runbook | Tabletop on poisoning incident | Postmarket response plan + runbook version in submission |
For a deeper treatment, see our dedicated post on AI data poisoning in medical devices.
2. Model Inversion and Model Stealing
What it is. An attacker queries the deployed model with crafted inputs to either (a) reconstruct sensitive attributes of the training data, model inversion, or (b) replicate the model's behavior in a surrogate, model stealing. Both attacks turn the inference endpoint into a side channel.
Why it is a 524B issue. Inversion can leak PHI used during training, which is a confidentiality breach under 524B. Stealing exposes proprietary weights and creates a parallel model an attacker can then attack offline at no cost, accelerating every other threat on this list.
Mitigations.
- Authentication, authorization, and per-user/per-device rate limiting on every inference endpoint.
- Output minimization: return the class or score the clinician needs, not full logits, embeddings, or attention maps.
- Differential-privacy or noise-injection techniques where training data is sensitive.
- Query-pattern anomaly detection for high-volume probing or adversarial query distributions.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | Inference-API threat model entry (Information Disclosure) | Architectural review against ATLAS "AML.T0024 Exfiltration via ML Inference API" | Threat-model row + data-flow diagram |
| Secure implementation | AuthN/AuthZ, rate-limit, and output-minimization config | Endpoint pen test with stolen credentials and probing harness | Pen-test report (AI scope) |
| Secure verification | Model-stealing simulation (query budget, fidelity score) | Surrogate-model fidelity test below documented threshold | Verification report with fidelity metrics |
| Secure release & response | Inference-log retention + anomaly alert rules | Incident tabletop on detected probing campaign | Postmarket monitoring plan section |
See Medical Device AI Model Inversion for the full breakdown.
3. Model Evasion
What it is. An attacker crafts or perturbs an input, often imperceptibly to a clinician, so the model misclassifies it. A pixel-level perturbation on a dermatology image, an adversarial patch on an ECG waveform, or carefully chosen noise in a radiology scan can flip a prediction.
Why it is a 524B issue. Evasion attacks degrade the reliability of clinical output, which the FDA treats as patient-safety-relevant. The 2026 guidance is explicit that cyber controls must protect the integrity of model predictions, not only the system around the model.
Mitigations.
- Adversarial robustness testing against published attack libraries (FGSM, PGD, patch attacks, modality-specific attacks).
- Input validation, range checks, and out-of-distribution (OOD) detection on raw inputs and intermediate features.
- Confidence thresholds with human-in-the-loop fallback when confidence is low or OOD detection fires.
- Adversarial training or certified-robustness techniques where the harm rating warrants it.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | Adversarial-robustness requirement tied to hazard analysis (ISO 14971) | Threat-model row "Tampering, AML.T0015 Evade ML Model" | Hazard-to-control traceability table |
| Secure implementation | OOD detector + confidence-threshold logic in inference path | Unit + integration tests for OOD fallback | Code-review record + test results |
| Secure verification | Adversarial test plan (FGSM, PGD, patch, modality-specific) | Robustness benchmark vs. clean accuracy delta | Adversarial-test report appended to risk-based testing summary |
| Secure release & response | Field-collected misprediction triage workflow | Tabletop on adversarial input cluster in production | Postmarket surveillance plan |
See AI Model Evasion in Medical Devices for examples and test patterns.
4. Data Leakage
What it is. Cyber threats exploit vulnerabilities to access sensitive training data, fine-tuning data, or inference-time data, for example, through misconfigured object storage, over-broad model endpoints, debug logs, or cached intermediate tensors.
Why it is a 524B issue. Data leakage is a direct confidentiality failure under 524B's expectations for protecting electronic information. It is also a regulatory issue under HIPAA and applicable international privacy regimes, but the FDA cybersecurity reviewer will judge it on 524B grounds.
Mitigations.
- Encryption in transit and at rest for every store that holds training, validation, or inference data.
- Strict separation of training infrastructure from production patient data; documented data-minimization at each stage.
- Tamper-evident access logging that captures who read which dataset and when.
- Pen testing that explicitly probes the data pipeline and the inference endpoint, not only the device firmware.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | Data-flow diagram with classification labels (PHI, derived, model artifact) | Architecture review against 524B "protect electronic information" | Architecture section of submission |
| Secure implementation | Encryption + IAM config for every data store and bucket | Config scan and IAM review | Cryptographic-controls summary |
| Secure verification | Pipeline pen test (storage, logs, caches, inference) | Authenticated and unauthenticated tests with leakage findings | AI penetration testing report |
| Secure release & response | Incident-response runbook covering PHI exposure | Tabletop with disclosure and notification path | IR plan + breach-notification SOP |
5. Overfitting
What it is. A model is trained too closely to its training set and generalizes poorly. A cyber attacker can deliberately push a model toward overfitting by injecting narrow patterns into training data, then exploit the brittle decision surface with adversarial inputs at inference time.
Why it is a 524B issue. Overfitting is normally a model-quality concern, but the FDA includes it here because attackers can cause it deliberately and then exploit it. That makes overfitting an indicator of compromise as well as a quality defect.
Mitigations.
- Held-out validation across demographically and clinically diverse cohorts; cross-validation results in the submission.
- Train/validation gap and decision-boundary sharpness treated as security-relevant alerts, not only data-science metrics.
- Regularization, dropout, and ensemble strategies documented in the model card.
- Re-evaluation of overfitting indicators after every training-data refresh or PCCP-authorized update.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | Generalization requirement linked to clinical performance acceptance criteria | Review against intended-use population | Performance requirements traceability |
| Secure implementation | Regularization + ensemble configuration captured in model card | Reproducible training pipeline test | Model card + training-pipeline artifact |
| Secure verification | Subgroup + external-dataset validation; train/val gap threshold | Validation report with diverse cohorts | Clinical/non-clinical performance section |
| Secure release & response | Overfitting-alert rule tied to retraining triggers | Tabletop on suspected adversarial overfitting | Postmarket monitoring plan |
See AI Overfitting Cybersecurity Threats in Medical Devices for detection patterns.
6. Model Bias
See also: Home Use vs Hospital Device Cybersecurity Requirements, Infusion Pump Cybersecurity: FDA Expectations, and Ransomware and Medical Devices: What the FDA Expects.
What it is. Bias enters when training data under-represents populations, when labels carry historical inequities, or when an attacker deliberately skews fine-tuning data, embeds backdoors, or distributes pre-trained models that already carry bias to be amplified downstream.
Why it is a 524B issue. The 2026 guidance treats deliberate bias manipulation, including backdoor triggers and adversarial bias amplification, as a cybersecurity attack on the model. Reviewers expect bias monitoring to be integrated with security monitoring, not run as a separate fairness exercise.
Mitigations.
- Provenance and integrity of any third-party datasets or pre-trained checkpoints; SBOM-style accounting for ML artifacts.
- Subgroup performance reporting (sensitivity, specificity, calibration by demographic and clinical subgroup) in the premarket submission.
- Backdoor scanning on imported model weights and a procedure for re-training if a backdoor is found.
- Postmarket bias monitoring tied to the device's postmarket cybersecurity program.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | ML-SBOM listing datasets, checkpoints, licenses, versions | Review of artifact provenance against SBOM/supply chain requirements | ML-SBOM in submission |
| Secure implementation | Backdoor / trojan scan in CI on imported weights | Negative test with planted trigger | CI scan report |
| Secure verification | Subgroup performance + fairness metrics | Subgroup test report with predefined thresholds | Performance section of submission |
| Secure release & response | Bias-monitoring rules in postmarket telemetry | Tabletop on detected subgroup drift | Postmarket plan |
7. Performance Drift
What it is. Cyber attackers shift the input data distribution over time, through compromised upstream sensors, manipulated EHR fields, or slow injection of skewed feedback, so the model's accuracy decays. Drift can also arise naturally; the threat is that an attacker can cause it on purpose and hide it as ordinary degradation.
Why it is a 524B issue. Performance drift connects directly to postmarket cybersecurity obligations. The FDA expects manufacturers to detect drift, decide whether it is benign or adversarial, and respond within their PCCP or change-control process.
Mitigations.
- Continuous performance monitoring against a held-out reference distribution; alarm thresholds documented in the SPDF.
- Distinction between data drift, concept drift, and adversarial drift, with investigation playbooks for each.
- Integrity controls on the update channel so legitimate retraining cannot be hijacked.
- Reporting paths into the FDA when drift indicates a cybersecurity issue affecting safety or effectiveness.
SPDF control / test mapping.
| SPDF activity | Control artifact | Verification test | Submission evidence |
|---|---|---|---|
| Secure design | Drift-monitoring requirement + reference distribution definition | Architectural review of telemetry path | Postmarket monitoring architecture |
| Secure implementation | Signed update channel + rollback control | Negative test: unsigned update rejected | CI/CD security gates evidence |
| Secure verification | Drift-detector validation against benign and adversarial shifts | Adversarial-drift simulation | Verification report |
| Secure release & response | Drift triage playbook + FDA reporting decision tree | Tabletop covering benign vs. adversarial drift | Postmarket plan + reporting SOP |
See AI Performance Drift in Medical Devices for the monitoring architecture.
Mapping the 7 Threats to Your SPDF and Threat Model
For a 524B submission, each of the seven threats above should appear in three places in your documentation:
- Threat model. Use STRIDE or a STRIDE-plus-MITRE-ATLAS overlay so AI-specific threats sit alongside conventional ones. ATLAS gives you the adversarial ML tactics and techniques; STRIDE keeps the framing consistent with the rest of the device. See our step-by-step threat modeling guide.
- SPDF risk and control matrix. Each threat ties to one or more controls, each control ties to one or more verification activities, and each verification activity produces an artifact that goes into the submission.
- Postmarket monitoring plan. Drift, bias, and anomalous query patterns need monitoring rules and response playbooks, not just dashboards.
When a reviewer asks "how does your device address AI cybersecurity threats," the answer is not a paragraph. It is a traceable chain from threat to control to test to monitoring rule, for all seven categories.
Need help? Our team supports manufacturers with FDA cybersecurity submissions end-to-end. Explore our medical device cybersecurity services or book a discovery call.
How Blue Goat Cyber Approaches AI Threat Modeling
Our ThreatGoat workflow treats the seven FDA-listed AI threats as first-class entries in the threat model, not appendices. We map each one to STRIDE plus the relevant MITRE ATLAS techniques, generate the control list and test plan from that mapping, and produce the SPDF artifacts a 524B reviewer expects to see. Our AI penetration testing service then exercises the controls against the actual deployed model, so the evidence in your submission reflects how the system behaves under attack, not only how it is designed.
If you are preparing a 510(k), De Novo, or PMA for an AI-enabled device, the fastest path through a clean cybersecurity review is a threat model and SPDF that already speak the language of the Feb 3, 2026 guidance.
If you want one team to own the whole cybersecurity package, see our full-service FDA premarket cybersecurity submission support.
FAQ
Related Deep-Dives on the 7 AI Threats
Each of the threats above has a dedicated post with detection patterns, controls, and test evidence:
- Data Poisoning, Safeguarding Medical Device AI from Data Poisoning
- Model Inversion & Stealing, Medical Device AI Model Inversion: Threats and Solutions
- Model Evasion, Protecting Medical Devices from AI Model Evasion
- Data Leakage, AI Penetration Testing for Medical Devices (covers data-pipeline leakage paths)
- Overfitting, AI Overfitting Cybersecurity Threats in Medical Devices
- Model Bias, How to Respond to an FDA Cybersecurity AI Request (includes bias-monitoring evidence the FDA expects)
- Performance Drift, Understanding AI Performance Drift in Medical Devices
Supporting reading: Threat Modeling Connected and Implantable Devices · FDA Section 524B Subsections Explained
Sources & References
FDA and statute
- FDA, Cybersecurity in Medical Devices: Quality Management System Considerations and Content of Premarket Submissions (Final Guidance, February 3, 2026).
- FDA, Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations (Draft, January 2025).
- FDA, Predetermined Change Control Plans for Machine Learning-Enabled Device Software Functions (Final, December 2024).
- Section 524B of the Federal Food, Drug, and Cosmetic Act (added by the Consolidated Appropriations Act, 2023).
AI-attack taxonomies and testing frameworks
- NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2023).
- NIST, AI Risk Management Framework (AI RMF 1.0) and the Generative AI Profile (NIST AI 600-1).
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems) - tactic and technique catalog cited in the matrix above.
- OWASP Machine Learning Security Top 10.
- OWASP Top 10 for Large Language Model Applications.
Medical-device standards
- AAMI CR34971:2023, Application of ISO 14971 to Machine Learning in Artificial Intelligence. See our CR34971 explainer.
- AAMI TIR57:2016 (R2023), Principles for medical device security - Risk management.
- ANSI/AAMI SW96:2023, Standard for medical device security - Security risk management for device manufacturers.
- IEC 81001-5-1:2021, Health software and health IT systems safety, effectiveness and security - Part 5-1: Security - Activities in the product life cycle.
- ISO 14971:2019, Medical devices - Application of risk management to medical devices.
- ISO/IEC 23894:2023, Artificial intelligence - Guidance on risk management.
- ISO/IEC 27090 (in development), Cybersecurity - Artificial Intelligence - Guidance for addressing security threats to artificial intelligence systems.
Testing tooling referenced above
- IBM Adversarial Robustness Toolbox (ART), Foolbox, CleverHans, Microsoft Counterfit, NVIDIA Garak, Microsoft PyRIT, Giskard.
Need help building an SPDF and threat model that covers all seven of these AI threats? Talk to Blue Goat Cyber about a 524B-ready AI cybersecurity engagement, or start with our ThreatGoat threat-modeling workflow.
About the author, The Blue Goat Cyber team specializes in cybersecurity for FDA-regulated medical devices, including AI/ML-enabled systems subject to section 524B of the FD&C Act. We help manufacturers ship clean premarket submissions and run defensible postmarket programs.
Related on FDA Section 524B
- FDA Section 524B: complete cybersecurity requirements guide
- Does Section 524B apply to my auto-injector?
- Does Section 524B apply to legacy devices?
- Documenting update cadence for a 524B submission
About the author
Christian Espinosa, MBA · Founder & CEO, Blue Goat Cyber
U.S. Air Force Academy graduate and veteran with 30+ years in cybersecurity. Founded Alpine Security in 2014 (acquired 2020), then Blue Goat Cyber in 2022. Has supported 275+ medical devices, with no cybersecurity-related rejections to date. Author of three books including The Smartest Person in the Room. Ironman triathlete and mountaineer.
Sources & references
Primary sources cited in this article. Links open in a new tab.
- final premarket cybersecurity guidance dated February 3, 2026- U.S. FDA
- MITRE ATLAS- MITRE
- NIST AI 100-2 E2023, *Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations*- NIST
- OWASP LLM Top 10- OWASP
- *Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations*- U.S. FDA
- *Predetermined Change Control Plans for Machine Learning-Enabled Device Software Functions*- U.S. FDA
- *AI Risk Management Framework (AI RMF 1.0)*- NIST
- *Generative AI Profile (NIST AI 600-1)*- NIST