AI Training Data Security for Medical Devices: FDA Evidence

Part of our Medical device vulnerability, threat, and attack-technique catalogue. For the full overview, start with The Top 50 Cybersecurity Issues with Medical Devices.

Reviewed October 14, 2026

A model learns whatever its data teaches it. If someone can add, change, or relabel training records, they can change what the device does without touching a line of code. That kind of change is hard to spot in testing, because the model still passes most checks.

Most teams secure the device and the cloud well, then leave the training data in a shared bucket that many people can write to. This post covers the controls that close that gap and the evidence that shows reviewers you did. For the attack itself, see our post on data poisoning in medical device AI.

Why This Matters

The FDA's February 3, 2026 premarket cybersecurity guidance expects threat modeling and risk assessment to cover the whole system, including parts that are not on the device itself. For an AI device, the training pipeline is one of those parts. Under Section 524B, cyber device makers must show reasonable assurance of cybersecurity across the device and related systems.

The FDA's January 2025 draft AI-enabled device software functions guidance names data poisoning as an AI-specific threat and asks for detail on data collection and management. It is still a draft, but it shows what reviewers are reading. If you retrain under a PCCP, the final AI PCCP guidance expects a data management practice for each planned change.

What Can Go Wrong With Training Data?

• Poisoning. Bad records are added so the model behaves wrongly in some cases.
• Label tampering. Correct records get wrong labels, often through a labeling tool with weak access control.
• Silent swaps. A dataset is replaced or edited between validation and training, so the released model learned from data nobody reviewed.
• Leakage. Patient data in training sets is exposed, or the model can be queried to reveal it.
• Untracked sources. Data from a partner, public set, or scraped source enters the pipeline with no record of where it came from.

Each of these should appear in your threat model with a control and a test.

Which Controls Protect Training Data?

Provenance. Record the source, collection date, consent or agreement basis, and transformation steps for each record or batch. When a source turns out to be bad, you need to find every model it touched.

Access control. Write access to raw data, cleaned data, and labels should be limited to named roles, with multi-factor sign-in. Labeling tools need the same care as production systems.

Integrity. Freeze each training set as a versioned snapshot with a cryptographic hash. The training job should check the hash before it starts, and the model manifest should record which dataset version it used.

Separation of duties. The people and service accounts that train models should not be able to release them. A separate approver signs off on each release.

Monitoring. Watch for unusual data changes, large new batches from one source, and label shifts. Production drift monitoring can also catch the effects of slow poisoning.

Protection of patient data. De-identify where you can, encrypt at rest and in transit, and keep retention limits. Our data sovereignty post covers where that data is allowed to live.

What Evidence Should Go in the Submission?

• Architecture views that show data sources, stores, labeling tools, the training pipeline, and the release path, with trust boundaries.
• Threat model entries for poisoning, label tampering, swaps, and leakage, each tied to a hazard.
• A description of provenance, access, and integrity controls.
• Test results: attempts to write to data stores with the wrong role, hash checks that block an altered dataset, and a traced removal of a test batch.
• For PCCP devices, the security checks each retraining must pass before release.

Keep it short and specific. Reviewers prefer a one-page table that maps threats to controls and results over a long policy document.

What actually counts as data poisoning in a medical device?

See also: AI Model Integrity for Medical Devices: Signing and Updates, Threat Modeling AI-Enabled Medical Devices: Worked Example, and FDA PCCP Beyond AI: Cybersecurity & Firmware Uses.

Data poisoning is deliberate manipulation of the data a model learns from, so the model's behavior shifts in a way the attacker wants. It can happen at three points: in the original training set, in a vendor-supplied or licensed dataset, or in the stream of field data used for retraining after clearance. Each entry point needs a different control, because the trust boundary is different at each one.

A mislabeled subset of training images, a small number of shifted labels, or fabricated samples blended into a real dataset are all forms of the same attack. The device keeps functioning and keeps producing plausible-looking outputs, so the corruption is not obvious in a demo. It shows up later as a pattern of wrong classifications that nobody connects to a specific batch of training data until someone goes looking.

How does poisoning get in during retraining or continuous learning?

Retraining is the highest-risk path because it happens after the device is already in clinical use and the data source is field data rather than a curated dataset. A device that adapts to hospital-specific patterns, or that periodically retrains on aggregated field data, is trusting every contributing site and every device instance that fed that pipeline.

An attacker does not need to breach the device itself. Poisoning the data upstream, at a connected system, a labeling workflow, or a federated learning contributor, gets the same result with less effort. That is why the threat model for a continuously learning device has to extend past the device boundary to every system that touches the training data before it reaches the model.

What does an AIBOM add that an SBOM does not cover?

An SBOM inventories software components: libraries, frameworks, and their versions. It does not tell a reviewer what data trained the model, what version of the model is deployed, or which third-party or hosted models the device calls at inference time. An AIBOM, built in CycloneDX ML-BOM or the SPDX 3.0 AI profile, closes that gap by inventorying the model artifacts, their weights and versions, the training and evaluation datasets, and any external model dependencies.

That inventory matters directly for poisoning response. If a poisoning event is discovered in a specific dataset version, the AIBOM is what tells you which model versions used that dataset and need to be pulled or retrained. Our AIBOM guide walks through the fields reviewers expect and how to generate one from an existing training pipeline.

How Blue Goat Approaches This

We add the training pipeline to your threat model as its own set of components, then test it like any other attack surface. That includes trying to write or relabel data with roles that should not have access, altering a frozen dataset to see whether training stops, and checking that pipeline accounts cannot release models. Findings get CVSS and patient-harm ratings and trace back to the threat model. Retests are unlimited until findings close. See our AI/ML Medical Device Security service and the worked example in threat modeling AI-enabled devices.

FAQ

Is training data really in scope for a cybersecurity review?

Yes, when changes to it can change what the device does. The February 2026 premarket guidance expects the threat model to cover the whole system, and the FDA's draft AI guidance names data poisoning directly. Reviewers may not ask in the first round, but a pipeline left out of the threat model is an easy gap to find.

Do we need provenance for every single record?

Not always per record. Batch-level provenance works if each batch is small enough to remove without losing a lot of good data. What matters is that you can find and remove bad data from a known source and identify every model trained on it. Choose the level that makes that possible for your data volumes.

How does this relate to data poisoning defenses in the model?

Model-level defenses, such as outlier filtering during training, help but are not enough alone. Access control and integrity checks stop most poisoning before it reaches training. Monitoring catches what gets through. Use all three, and describe each in the threat model so reviewers see defense in depth.

What if we use a public or licensed dataset?

Record the source, version, license, and hash of the copy you used, and keep that copy frozen. Public sets can be changed or poisoned upstream, so do not pull the latest version during training. Treat the dataset as a third-party component, much as you treat software libraries in your SBOM.

Does a PCCP change what we need?

It adds to it. Each planned retraining should name the data management steps and security checks that must pass before release. If retraining uses new data sources, the plan should say how they are vetted. A retraining that breaks those bounds needs a new submission rather than a PCCP update.

CTA

Want to know whether your training pipeline would hold up to a reviewer or an attacker? Book a free 30-minute discovery call. We'll look at your data flow and send a fixed-fee proposal. Nothing is billed until you approve it.