Is the FDA's 510(k) Pathway Rigorous Enough for Adaptive AI?

The United States Food and Drug Administration's 510(k) pathway has been the dominant route through which artificial intelligence enabled medical devices reach the US market. The FDA's public list of AI-enabled medical devices shows that the great majority were authorised through 510(k) clearance rather than De Novo classification or Premarket Approval. The pathway was designed to assess whether a new device is "substantially equivalent" to a legally marketed predicate, rather than requiring developers to independently establish safety and effectiveness. It has been highly successful in accelerating innovation and market access. But the emergence of adaptive machine learning, generative AI and agentic AI raises a fundamental question.

Can a regulatory system designed for largely static technologies adequately govern technologies that learn, evolve, and increasingly exhibit autonomous behaviour after deployment?

This essay argues that while 510(k) remains appropriate for many traditional software-based medical devices, it is not sufficient by itself for adaptive, generative and agentic AI. The FDA has increasingly recognised this, moving toward a Total Product Lifecycle (TPLC) philosophy, Predetermined Change Control Plans (PCCPs) and post-market monitoring expectations for AI-enabled devices. Regulation is shifting from evaluating products at a single point in time toward evaluating real-world behaviour throughout deployment. That creates a strong case for independent intelligence infrastructure capable of observing AI after clearance.

1. Introduction

For decades, medical device regulation assumed a simple reality. A device is designed. It is tested. It is cleared. It remains substantially unchanged until the next regulatory submission.

The 510(k) framework emerged from that assumption. Under a premarket notification, manufacturers demonstrate that a device is substantially equivalent to a predicate. The governing regulation, 21 CFR 807.100(b), defines this as a device that "has the same intended use as the predicate device" and either "has the same technological characteristics as the predicate device" or has different characteristics where the information submitted "does not raise different questions of safety and effectiveness than the predicate device."

This works reasonably well for static devices. Adaptive AI systems are not static. They may retrain on new data, evolve through updates, perform differently across populations, degrade through model drift and exhibit behaviours not observed during validation.

2. Why 510(k) Worked Historically

The pathway was created to facilitate innovation without unnecessary duplication of evidence. Its limits were debated long before AI: the Institute of Medicine's 2011 review, Medical Devices and the Public's Health: The FDA 510(k) Clearance Process at 35 Years, concluded that the process was not designed to evaluate, and could not reliably assure, device safety and effectiveness. Even so, 510(k) has enabled many thousands of devices to reach patients efficiently.

Many early AI applications fitted comfortably within it: image segmentation, radiology triage, ECG interpretation support and diabetic retinopathy screening. These were generally "locked" algorithms whose behaviour did not change after deployment. For such systems, substantial equivalence is often a reasonable concept.

3. The Fundamental Problem: Adaptive AI Is Not Static

The FDA's January 2025 draft guidance, Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations, explicitly frames AI-enabled devices across the whole product lifecycle, including deployment, monitoring and modification. This is where the traditional model begins to struggle.

A radiology AI cleared today may perform differently six months later following retraining, in a new patient population, on different imaging hardware or within a different workflow. The system in use five years later may not be the system originally cleared. For traditional software that is unusual. For adaptive AI it is expected.

Real-world evidence already shows the danger. Finlayson and colleagues, writing in the New England Journal of Medicine (2021), described how dataset shift silently degrades clinical AI when populations, practices or infrastructure change. HealthAratus tracks this phenomenon publicly through Drift Watch.

4. The Predicate Problem

The strongest critique of AI regulation under 510(k) concerns predicate dependency. Muehlematter, Bluethgen and Vokinger, in The Lancet Digital Health (2023), mapped the predicate networks of FDA-cleared AI/ML devices. They found that more than a third of AI/ML devices originated from non-AI/ML predicate devices, that AI tasks evolved along predicate chains, particularly in radiology, and that this raised potential safety concerns. They recommended greater attention to the distinctive characteristics of AI/ML when determining substantial equivalence.

Consider a hypothetical chain. Device A is cleared in 2005. B claims equivalence to A, C to B, D to C, E to D. By the fifth generation, technological reality may differ substantially from the original predicate. With machine learning, behaviour depends on training data, architecture, optimisation and deployment environment, which may differ profoundly despite superficial functional similarity.

Equivalence of intended use is not equivalence of behaviour.

HealthAratus already links FDA regulatory history to each tool's record; see how this is explored in the Hidden Failure Modes of Clinical AI.

5. Generative AI Creates a Regulatory Discontinuity

Traditional clinical AI performs narrow tasks: detect a fracture, classify a lesion, predict sepsis risk. Generative systems produce novel outputs, synthesise information, communicate in natural language and interact dynamically with users.

The FDA has stated on its AI-enabled device list that it plans to identify and tag devices incorporating foundation models and large language model functionality. Its Digital Health Advisory Committee devoted its November 2024 meeting to generative AI-enabled devices. The reason is straightforward: substantial equivalence is much harder to define when outputs are probabilistic, open-ended and context-dependent. Two LLM-based systems may share a clinical purpose yet differ in architecture and training data, and produce different outputs for identical inputs.

6. Agentic AI Raises an Even Greater Challenge

Agentic systems plan actions, use tools, interact with software environments, make sequential decisions and initiate tasks autonomously. An agent handling care coordination, referrals, documentation, patient communication and scheduling may generate risks that cannot be observed in any premarket evaluation.

The regulatory challenge shifts from evaluating outputs to evaluating behaviour.

This is a profound departure from the historical scope of 510(k), and the subject of our White Paper on an independent intelligence layer for agentic AI.

7. FDA Recognition of the Problem

To its credit, the FDA has not ignored these challenges. Its 2019 discussion paper and the 2021 AI/ML-Based Software as a Medical Device Action Plan introduced a Total Product Lifecycle approach, and the FDA, Health Canada and the UK MHRA jointly published Good Machine Learning Practice guiding principles, including that deployed models should be monitored for performance. Premarket review alone is no longer treated as sufficient. Oversight must continue throughout real-world deployment.

8. Predetermined Change Control Plans (PCCPs)

In December 2024 the FDA issued final guidance, Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions. A PCCP sets out three elements in the original submission: a description of planned modifications, a modification protocol covering development, validation and implementation, and an impact assessment. Once authorised, those pre-specified changes can be implemented without a new submission for each one, supporting iterative improvement while maintaining safety and effectiveness.

The existence of PCCPs is itself evidence that the traditional 510(k) model was insufficient for continuously evolving AI. Yet a PCCP governs anticipated change. It does not, by design, reveal the unanticipated behaviour that emerges in deployment.

9. Real-World Examples Demonstrating the Regulatory Gap

Example 1: AI imaging systems

Radiology is by far the largest category on the FDA's AI-enabled device list, largely cleared through 510(k) on the strength of extensive predicate networks. Researchers have documented AI tasks evolving across predicate generations. Is similarity of intended use enough when learning systems differ in architecture, training data and behaviour? Explore the category through the HealthAratus Rankings.

Example 2: Sepsis prediction

Sepsis models are highly sensitive to populations, workflows and local practice. The external validation by Wong and colleagues in JAMA Internal Medicine (2021) of a widely implemented proprietary sepsis model reported an area under the curve of 0.63, substantially worse than the developer's reported performance, and found it missed around two thirds of sepsis cases. Sepsis prediction models of this kind have typically operated as clinical decision support outside device clearance, which sharpens the point: deployed performance must be observed, wherever authorisation sits.

The key risk is not approval-day performance. It is deployment-year-three performance. Premarket review captures only the former.

Example 3: Generative clinical documentation

Ambient scribes and documentation assistants increasingly rely on foundation models. Two systems may both produce clinical notes yet differ dramatically in error profiles, hallucination rates, omission patterns and workflow impact. Substantial equivalence offers little insight into these behavioural differences.

10. What Should Augment the Current Model?

The answer is not to eliminate 510(k). Adaptive AI requires augmentation through four pillars.

• Lifecycle regulation. The FDA's movement toward Total Product Lifecycle oversight should continue.
• Continuous real-world performance monitoring. Performance should be observed after deployment, not merely before clearance.
• Independent verification. Manufacturers should not be the sole source of evidence about post-deployment performance.
• Institutional memory. Healthcare lacks a longitudinal memory of AI behaviour, so failure modes are rediscovered rather than learned, as argued in Lessons from NASA.

11. Implications for Healthcare AI Governance

The FDA can evaluate the evidence available at submission. What it cannot continuously observe is how clinicians use a system, where trust changes, where performance drifts, which failures recur, how vendors respond and which populations benefit or suffer.

The central challenge is no longer premarket approval. It is post-market understanding.

These phenomena emerge only in real-world use. Governance therefore increasingly depends on intelligence systems operating beyond regulatory submissions, the subject of The Accountability Vacuum and The Chaordic Imperative.

Conclusion: Why HealthAratus Matters

The debate over whether 510(k) is rigorous enough for adaptive AI reveals a deeper issue. The challenge is no longer how to evaluate an AI system at one moment before market entry, but how to understand and govern its behaviour after deployment. TPLC and PCCPs acknowledge that static assessment is insufficient. Yet a gap remains.

Regulators review submissions. Manufacturers provide validation studies. Hospitals monitor local deployments. No stakeholder has a comprehensive view of how AI behaves across the wider healthcare ecosystem.

No regulator can continuously observe how thousands of AI systems perform across thousands of care settings.

No vendor can independently assess how its products are experienced across competing health systems. No individual hospital can see national patterns of drift, trust erosion, workflow disruption or recurring failure.

HealthAratus is not designed to replace FDA regulation. It addresses a problem regulation alone cannot solve: a continuously updated, real-world intelligence layer for healthcare AI. If the FDA evaluates whether an AI system is safe and effective enough to enter the market, HealthAratus is designed to understand what happens after it arrives. It addresses five challenges exposed by the limits of 510(k).

1. From static approval to continuous observation

510(k) evaluates evidence submitted at one point in time. HealthAratus continuously captures real-world signals about how AI behaves after deployment, through Drift Watch and Live Pulse. The question shifts from "was this system safe when it was cleared?" to "is it performing safely and effectively today?"

2. From local visibility to ecosystem intelligence

Degradation and emerging risk are distributed across institutions. A radiology model may underperform in one population but not another; a documentation system may disrupt certain specialties; a care-coordination agent may create downstream effects invisible to its developer. HealthAratus connects these observations through the Signal Network.

3. From predicate equivalence to real-world evidence

Instead of asking whether a new system resembles an older one, HealthAratus asks how it performs in practice: what clinicians experience, whether trust is rising or falling, whether failure modes recur and whether outcomes improve. This is observed behaviour rather than inferred equivalence, synthesised into the HealthAratus Score under a published methodology.

4. From periodic review to institutional memory

Hospitals often learn about AI failures in isolation, vendors may resolve issues without dissemination and regulators receive fragments. HealthAratus retains and structures deployment history, clinician experience, performance trends, recurring problems, vendor responses and verified outcomes as a longitudinal memory of the ecosystem.

5. From regulatory compliance to intelligent governance

Where is this AI deployed? How is it used? What is changing in its performance? Which organisations are experiencing problems? How have vendors responded? Which systems keep delivering value? These are intelligence questions rather than regulatory ones. HealthAratus exists to answer them. See how HealthAratus works.

Final reflection

The more important question is whether any premarket pathway, however rigorous, can fully govern technologies that keep learning, adapting and interacting with complex clinical environments long after clearance. The evidence increasingly suggests it cannot. Regulation remains essential, and the FDA's lifecycle approach is a major step forward. But regulation alone cannot provide continuous, ecosystem-wide awareness.

The critical question is no longer "was this AI safe on the day it was cleared?" It is "how is this AI behaving today, in real-world clinical practice?"

That is the role HealthAratus is designed to play: independent intelligence infrastructure that helps the healthcare system understand, remember and govern AI after deployment, when its most consequential behaviours emerge.

Frequently Asked Questions