The Unique Risk Elements of ML enabled Medical Devices

Special Topics

Aug 3

Industry is getting a hang of things, 34971 was here to help, now 24971-2 is as well

Risk management for ML enabled medical devices

Machine learning whether standalone or embedded within a device, presents really unique challenges to manufacturers attempting to conduct risk management activities within the design and development system.

I have written this to give some clarity on where risk management activities actually need to change, and where they do not.

Thanks for reading The Other Consultants: MedTech Quality & Regulatory Insights! Subscribe for free to receive new posts and support my work.

The journey to where we are

Thanks to Edward Ball for letting me know its worth clarifying between 34971 and 24971-2. This article is intended as a kind of where we've come from. I always liked BS/AAMI 34971, and thought it was a great document for manufacturers to get a copy of to aid them in their risk management journey. A lot of what I exemplify here will be with reference to that standard as a foundation.

Very recently (June 2026) Guidance on the application of ISO 14971 - ISO 24971-2 was released, which takes 34971 further. However, in this article we're going to focus on the history of 34971 to where we are now, and we'll cover 24971-2 later, but it'll probably be updated by the time I get to that.

So what is so different about ML enabled medical device?

ISO 14971 has stabilised within industry. Most people are now comfortable with the chain from hazard to hazardous situation to harm, and with the link between that chain and post-market surveillance. If not, I will cover it elsewhere. So is the risk management process for these devices any different? What is unique about AI medical systems that creates particular difficulty?

Continuity. These systems can accumulate data to produce an output that could have a positive impact on patient health in any term defined by the intended purpose of the device, in vitro or otherwise, according to the relevant regulations.

Autonomy. These systems have the potential to modify processes and outputs in response to their learning, with reduced clinician oversight, or in some cases without it.

Explainability. Because of their computational ability, the statistics involved and the size and complexity of the datasets, the rationale for the output produced by such systems may not be easily understood by well-trained clinicians and other healthcare personnel, let alone by individuals without specialist knowledge.

Further, machine learning systems can introduce risks to safety by learning incorrectly, making wrong inferences, and then recommending or initiating actions that lead to harm.

If we compare this to a traditional sterilised medical device, a throwback to a simpler time, the differences are stark, and they change a few things:

It changes the way regulators perceive these devices from a safety-profile perspective.
Post-market surveillance needs to consider the performance metrics used in the initial evaluation, and design requirements must consider the capture of PMS data.
The manufacturer needs to consider change through an alternative route. The PCCP is a more dynamic change and PMS process. See Good Machine Learning Practices (GMLP).

The general process for ISO 14971 risk management does not change. The risk management plan, risk analysis, evaluation, control and reporting, with the benefit-risk profile considered throughout, all remain. What changes is the context. In the same way that a risk management file for an IVD differs from one for a medical device because of the way harm is caused, with medical devices more attributable to direct harm and IVDs to indirect, ML enabled medical device sits in its own context.

So let us go through the traditional elements of the risk management file and call out the areas I would recommend considering as part of the development of an ML enabled medical device.

Intended use and reasonably foreseeable misuse

As with any other device, the description of the device should be driven from the design and development plan. The device should not simply appear.

BS/AAMI 34971 provides a very useful list of the following that should be considered:

Does the software give diagnostic or treatment recommendations? If so, how strongly does that information influence the user? Does it inform the clinician, or does it drive how the patient is managed?
What is the target population? Is the patient’s condition non-serious, serious or critical?
How urgent is the information the device provides?
Does the algorithm offer options and rank how appropriate each one is?
Are errors detectable, for example where domain shift occurs, such as images taken with or without a contrast agent?
Does the system offer a non-ML version of the same function?
What autonomous functions does the system have, and at what level of autonomy? AAMI TIR 34971:2023 Annex D deals with the impact of autonomous functionality, and IEC/TR 60601-4-1 remains the reference point for degrees of autonomy.
Can the device learn and adjust its performance over time? If so, what are the limits on those adjustments? Which performance thresholds are acceptable, and which must it never fall below? What mitigations apply if a threshold is breached?
What are the potential off-label uses of the device? As users become familiar with a device they find uses beyond the intended one. Allow for reasonably foreseeable misuse and evaluate the use errors that could occur.
Are there contra-indications because certain patient conditions were restricted in the data used to train, test and validate the model? If so, is there a risk the user ignores those contra-indications?
Is the system intended to learn over time, and could that affect the intended use?

Identification of characteristics relating to safety

This is where we can leverage ISO/TR 24971-2 Annex A, which is where the questions from the old ISO 14971:2007 Annex C ended up. Who can remember anything anymore. AI is, ironically enough, probably to blame.

Joking aside, AAMI TIR 34971 is clear that the process for identifying characteristics relating to safety for ML enabled systems can and should follow the same methodology used for more traditional medical device systems. It's critical to mention 24971-2 here, as this is an area that the new standard significantly goes further within the Annex C - Identification of hazards and characteristics related to safety, which gives a really comprehensive list.

It is again worth noting that the document specifies that Annex C is not exhaustive and should not be used as a checklist.

The draft IMDRF document I recently led a consultation response on, N93 Technical Framework for Artificial Intelligence Life Cycle Management, groups risks more broadly into four categories:

Risks related to information, including inaccurate or misleading outputs and incomplete presentation of information, such as no explanation of the probability of error.
Risks related to the human-AI interaction, including over-reliance and automation bias, under-reliance and dismissal, workflow disruption, verification fatigue, alert fatigue and the de-learning of clinical knowledge over time.
Risks related to model training and data quality, including training data bias, data drift, incomplete data, annotation errors and out-of-distribution inputs.
Risks related to deployment and post-market monitoring, including interoperability failures, degradation caused by changes in third-party general-purpose models, version control failures and misaligned calibration where the context of use differs from the context of validation.

The first two get attention because they look like conventional usability and information hazards. The others require the manufacturer to admit that the device’s safety depends on data it did not generate and infrastructure it may not own.

Examples of ML enabled medical device hazards

In the original BSI/AAMI 34971, the following was covered:

Performance-related hazards:

Data quality, covering incorrect data, incorrect handling of outliers and incomplete data.

Bias, covering selection bias, implicit bias, group attribution bias and experimental bias.

Data storage, cybersecurity and privacy, covering privacy failures and inability to contact the patient.

Overtrust and over-reliance, covering overconfidence, perceived risk, user workload, self-confidence and variation in social trust.

24971-2 Annex B goes further into specific types of events and circumstances, in particular with number 8 - ML model training, and 9 functionality. There is also the table of example relationships that should be used as primary place for examples.

Example events and circumstances

Data quality - MLMD delivers insulin based on continuously measured input data. A loosely worn wearable medical device can interrupt the data flow, leading to incorrect conclusions by the decision-making MLMD.

Algorithm model: The MLMD does not learn enough training data and misses important features; underfitting; generalization error increases after training the ML model

ML enabled medical device and post-production activities

This is the other area where ML enabled medical device creates real difficulty for developers.

Take the EU MDR. Software under Rule 11 will, in almost all cases where it informs a diagnostic or therapeutic decision or monitors physiological processes, land at Class IIa or above, and with it comes the Periodic Safety Update Report under Article 86, annually for Class IIb and III and at least every two years for Class IIa. Great Britain has now followed, with the post-market surveillance amendments to the UK MDR 2002 introducing a broadly comparable reporting cadence.

Question time, and be honest.

Take away the regulatory requirement. If the sole purpose of post-market surveillance is to confirm that the information within your technical documentation is still valid, is it really enough to complete a PSUR every year, or every two years, when the device changes at the rate this one does?

The answer is no, and the reason is that the PSUR is a reporting artefact, not a surveillance activity. Article 86 tells you when to publish. Annex III tells you what to do. If your PMS plan says little more than that complaints will be reviewed and a PSUR produced every two years, you have no mechanism capable of detecting any of the failure modes described above.

N93 makes a distinction that I think is genuinely useful here, and one I would argue should be reflected in the structure of the PMS plan itself. It separates operations and monitoring, meaning the continuous and largely automated activity embedded in the QMS such as logging, alerting and incident handling, from real-world performance evaluation, meaning the structured analyses that use those data to answer specific questions about safety, effectiveness and clinical utility. One is a pipeline. The other is a study. Most PMS plans I see describe neither and instead describe a reporting schedule.

In practice that means a few things.

Define device performance indicators up front, framed by the intended purpose and, where feasible, using the same metrics you used to set acceptance criteria during verification and validation. If the pre-market and post-market metrics are not comparable, you cannot demonstrate that performance has been maintained, which is the whole point. This also has to tie into design requirements to ensure adequate capture of PMS data to prove performance.

Define those indicators for clinically relevant subgroups, not only in aggregate. Aggregate accuracy will hide differential performance across demographics, disease subtypes and deployment sites, and differential performance is precisely the failure mode these devices are prone to.

Pre-specify thresholds and trigger conditions, and state what happens when one trips. That is the link back into risk control, into corrective action, and into trend reporting under Article 88. A monitoring plan with no defined trigger is a data collection exercise, not surveillance.

Monitor the human-AI team, not just the model. Override rate is the one almost everybody misses. If clinicians are routinely overriding an output while your monitoring shows the model performing exactly to specification, your residual risk estimate for automation bias, under-reliance or workflow disruption is wrong. The model is fine and the device is not. No amount of model-level telemetry will surface that.

Monitor for drift explicitly, and distinguish it from expected output variability. For non-deterministic models this is not straightforward, and it needs to be designed into the infrastructure at the deployment stage rather than retrofitted when a signal appears.

Change, PCCPs and the file

Where a predetermined change control plan is implemented, the technical documentation will require updating and, at minimum, an impact assessment against the PMS documentation.

It is worth being precise about the jurisdictional position, because there is a lot of loose talk about PCCPs at the moment. The FDA has a defined route. The EU MDR does not currently have an equivalent mechanism, and a change that would otherwise trigger a significant change assessment still does; having anticipated it in a plan does not remove the notified body from the process. Great Britain has signalled an intention to introduce one.

Wherever you sit, treat a retraining event as an input to the risk management file rather than as a software release. The characteristics relating to safety may have changed. New hazards may have been introduced. The benefit-risk determination needs revisiting, and the clinical evaluation, PMS plan and transparency information provided to users all need an impact assessment. Appendix C of N93 sets out the labelling elements that ought to be revisited, and measured performance including subgroup performance sits squarely in that list.

Sunsetting

Now that we have a nice word for it from IMDRF, we can start to think about how to integrate it, or how it already integrates.

One thing N93 does that most risk management files do not is treat sunsetting as an explicit life cycle step. It is worth adopting. For cloud-hosted software, withdrawing a model version or terminating support is itself a risk-bearing event. Sites will have built workflow around the device, and removing it without a managed transition creates a hazardous situation that has nothing to do with the model’s performance and everything to do with the clinical process it has been absorbed into. Communicate the timeline, define what happens to retained data, and document the decision. It will be reviewed.

Where this leaves you

The process does not change.

What it requires is the same process applied honestly to a device whose characteristics relating to safety now include the data it was trained on, the people who labelled that data, the infrastructure it runs on, and the behaviour of the clinician using it.

Integrate these standards, guidance and regulations into the early phases of your design project and things will naturally flow from them.