Case study

Pen testing the AI inside a 510(k) SaMD

A cloud diagnostic platform. Three attack surfaces. 35 test cases. One annual test that served postmarket monitoring and a new 510(k).

At a glance

Why the AI layer matters

Clinicians make diagnostic decisions from what this platform shows them. If an attacker can pull patient records, tamper with an AI result, or extract the model, the harm lands on a patient.

The web application and APIs are familiar ground for any pen tester. The AI inference endpoints are not. They take patient data in and send clinical output back. They can leak sensitive data, mishandle output, or give up the model itself. Those risks need their own test cases.

Scope came from the threat model

Blue Goat did not start from an asset list. Scope came from the manufacturer's threat model. The test plan defined three test points:

1. The cloud web application
2. The API endpoints
3. The AI inference endpoints

Each in-scope interface was mapped to threat IDs before testing began. Out-of-scope interfaces were listed explicitly. Tester independence and expertise were documented in the plan, as FDA's 2026 guidance asks.

How Blue Goat attacked it

Testing was white box, with credentials for each user role, plus fuzz testing of inputs. Each surface was tested against its own checklist and against threats derived from STRIDE.

Inside the 35 test cases

Every test case is written before testing starts. Each one carries the same parts:

• Test point: which surface it targets
• Threats covered: the STRIDE categories it tests, such as information disclosure or elevation of privilege
• Validation resources: the tools used. Tool versions are recorded in the test plan, matching the tool detail FDA's 2026 guidance describes.
• Test steps: what the tester does, step by step
• Expected behavior: what a secure device should do

The register traces to the threat model through the test plan's threat mapping table. So every test answers a documented threat, and every threat has a test.

The 6 AI test cases are where this engagement went past standard web testing:

1. Training data poisoning review
2. Insecure output handling
3. Sensitive information disclosure
4. Model theft and extraction resistance
5. Improper file upload to the AI pipeline
6. AI system vulnerability assessment

What testing found, and what held

35 test cases produced 14 findings in the first round: 1 critical, 1 high, 9 medium, and 3 low.

Every finding got its own section: evidence, affected components, and remediation guidance.

The report also recorded what held. Authorization checks at the object, property, and function level found no vulnerabilities, and the report says so. A reviewer sees what was tested and passed, not only what failed.

Traceable from threat to finding

Every finding traces back to a documented threat. The chain runs in four links:

1. Threat model. The manufacturer's threat model identifies the threats.
2. Test plan. A threat mapping table ties each in-scope interface to its threat IDs.
3. Test cases. Each of the 35 cases names the STRIDE threats it covers.
4. Test report. Each finding traces to the test case that found it.

A reviewer can start at any threat and follow it to the test and the result. Or start at a finding and follow it back to the threat.

How risk was scored: exploitability and patient harm

FDA asks manufacturers to assess exploitability instead of probability, because no one can predict how often a human attacker will strike. Blue Goat scores two separate axes:

• Exploitability: scored with CVSS 4.0.
• Severity of patient harm: scored on a five-level scale consistent with the manufacturer's ISO 14971 risk management process, against the worst credible outcome of the hazardous situation. Severity is scored independently of how hard the vulnerability is to exploit.

Findings go into one of two matrices:

• Patient harm risk rating: for findings that can lead to patient harm. Exploitability against severity of the worst credible clinical outcome.
• Cybersecurity risk rating: for findings that cannot lead to patient harm, such as non-clinical data disclosure. Exploitability against impact.

Patient harm is the axis that sets priority. Two findings can be equally easy to exploit. If one can corrupt a diagnostic result and the other exposes a log entry, they are not the same risk.

How findings were closed

The report directed every finding into the manufacturer's risk management process. Acceptability is the manufacturer's call, made against the criteria in its own risk management plan. Security risk and safety risk are managed as separate, linked processes, as FDA's 2026 guidance describes. Any security risk that could lead to patient harm carries into the ISO 14971 safety risk file.

Blue Goat recommended a path for every finding:

• Critical and high: remediate and retest before submission.
• Medium and low: if not remediated, document a rationale tied to intended use and patient impact, name compensating controls, and get sign-off from the risk management authority. FDA's 2026 guidance asks for that rationale on any finding not fixed or deferred.
• False positives: disposition formally, with technical justification and reviewer sign-off. Never drop one silently.

Blue Goat retested after the fixes. Both the critical and the high finding were remediated and validated.

One program, two jobs

This manufacturer already had a device on the market and an annual postmarket program with Blue Goat. The program covers yearly penetration testing, remediation validation, SAST, SBOM monitoring, a coordinated vulnerability disclosure portal, and CISA KEV tracking.

A new 510(k) was coming. The annual test was built to serve both: postmarket assurance for the device in the field, and penetration testing evidence for the submission.

What the manufacturer received

• A test plan with scope traced to the threat model and a tester independence statement
• A 35-case test register, each case naming the threats it covers, its tools, its steps, and its expected behavior
• A test report carrying the five elements FDA's 2026 guidance lists: tester independence and expertise, scope, duration, methods, and results with findings and observations
• Risk distributions before and after remediation
• A recommendation for every finding, written to feed the manufacturer's security and safety risk files

What this means for your device

• If your device runs AI, test the AI. Web and API testing alone leave the inference endpoints unexamined.
• Scope from your threat model. Testing should trace to the threats you documented, so the report and the risk file tell one story.
• Plan one test for two jobs. If you have a device in the field and a new submission coming, an annual postmarket test can be scoped to produce premarket evidence too.

Frequently asked questions

275+ FDA submissions supported, zero cybersecurity rejections.