AI Model Integrity for Medical Devices: Signing and Updates

Reviewed October 14, 2026

The model file is the product's clinical logic. If an attacker swaps it, the device still looks normal but makes different decisions. If a competitor copies it, years of work leave the building. Both risks are easy to miss because model files are often treated as data, not code.

This post covers how to protect the model file through its whole life: at rest, in transit, during loading, and when it is updated. For attacks that pull information out of a model through queries, see our post on model inversion.

Why This Matters

The FDA's February 3, 2026 premarket cybersecurity guidance asks for authenticity and integrity controls on software and updates, along with cryptography and a secure update process. A model file that changes device behavior falls under that expectation, even if your build system treats it as a data asset. Section 524B also requires a plan to provide updates and patches, which for AI devices includes model updates.

The FDA's January 2025 draft AI-enabled device software functions guidance lists model inversion and stealing among AI-specific threats. It is a draft, but it tells you what reviewers have in mind. If you plan to update the model after clearance, the final AI PCCP guidance expects an update procedure for each planned change.

How Can a Model Be Attacked?

• Swap. An attacker replaces the model file on the device, in the cloud, or in the registry with a modified one.
• Rollback. An older model with known weaknesses is loaded instead of the current one.
• Theft. The model is copied from device storage, an app package, or a cloud bucket.
• Extraction by query. Repeated API calls let an attacker build a close copy of the model.
• Tampered dependencies. A changed AI library or model format loader runs attacker code when the model loads. Some model formats can carry code that runs on load, so the format itself matters.

Which Controls Protect the Model?

Signing. Sign each released model and its manifest (version, dataset version, intended hardware). Use the same discipline as firmware signing.

Verification on load. The device or service checks the signature and compares the version to an allowed list before loading. Failures block the load and raise an alert. Rollback protection stops older versions from loading.

Key protection. Keep the production signing key in a hardware security module. Limit who can request a signature, and log every use. Use separate test keys so a test build can never pass as production.

Safe formats. Prefer model formats that store only weights and structure, and avoid formats that run code during loading.

Protection at rest. Encrypt models on the device and in storage, use secure storage where the hardware supports it, and avoid shipping full models inside app packages when the model can stay on a server.

Extraction limits. Rate limits, authentication, and output rounding on inference APIs make query-based copying slower and easier to spot.

See also: AI Training Data Security for Medical Devices: FDA Evidence, Threat Modeling AI-Enabled Medical Devices: Worked Example, and FDA PCCP Beyond AI: Cybersecurity & Firmware Uses.

SBOM. List the model, its version, and the AI runtimes and libraries it depends on. Our SBOM guide explains format and depth.

How Do You Test Model Integrity?

How Blue Goat Approaches This

We test model integrity the same way we test firmware updates: try to load unsigned, wrongly signed, altered, and older models on the device and in the cloud, look for model files in storage and app packages, and check whether the signing process can be abused. We add API extraction tests where the model is reachable over a network. Findings get CVSS and patient-harm ratings and trace back to your threat model. Retests are unlimited until findings close. See our AI/ML Medical Device Security page and the worked example in threat modeling AI-enabled devices.

FAQ

Is a model file software or data for FDA purposes?

For security, treat it as software. It controls device behavior, so a changed model can cause harm the same way changed code can. The February 2026 premarket guidance expects integrity and authenticity controls on software and updates. Reviewers will look for the same signing and verification you use for firmware, applied to model files too.

Do we need to sign models that run in the cloud?

Yes. A swapped cloud model affects every user at once, which can make it more dangerous than a swap on one device. The inference service should verify the signature and version before loading. Also limit who can write to the model registry, and log every promotion to production.

Can we stop a determined attacker from copying the model?

Not completely if the model runs on hardware the attacker controls. You can make it much harder with encryption, secure storage, and keeping the full model on a server. For cloud models, rate limits and monitoring make query-based copying slow and visible. Document what you did and what risk remains.

How does model signing fit with a PCCP?

Each planned model change in the PCCP should state that the new model is signed through the same release process and passes the integrity tests before going live. That shows reviewers the security controls stay in place across updates. A change that bypasses signing or uses a new release path needs its own review.

Should the model appear in our SBOM?

Yes. List the model and version along with AI runtimes and libraries. Hospitals increasingly ask for this, and it helps your own vulnerability monitoring when a library flaw is announced. If you use an outside model or service, list it too and note how you track its changes.

CTA

Shipping an AI model in a device or cloud service? Book a free 30-minute discovery call. We'll review how your model is signed, loaded, and updated, then send a fixed-fee proposal. Nothing is billed until you approve it.