Where the human-AI workflow in health care breaks down

U.S. health insurers process more than 5 billion payment claims each year, according to CMS and Kaiser Family Foundation data. At a large plan, an AI-assisted system may generate thousands of recommendations a day on claim denials, eligibility flags and encounter-data errors, each queued for human review with a confidence score and a suggested action. Reviewers focus on the most complex cases, check lower-risk recommendations for irregularities and approve the batch. The system records a name and a timestamp. On paper, a human was in the loop.

A human is in the loop. What could go wrong?

"Human in the loop" sounds like governance. Too often, at high volume, the signature at the end of a review queue is less oversight and more autopen, as CureIS Healthcare examines here.

A June 2026 piece in The Lancet observed that human in the loop often functions as symbolic reassurance rather than substantive protection. In October, a World Economic Forum article argued for a "human-in-command" model: Map where failures can occur, detect uncertainty, apply safeguards based on risk, escalate consequential cases with the right evidence and learn from outcomes. The American Medical Association made a similar point in its proposed framework for AI systems in clinical settings: Authority should depend on evidence, defined conditions of use and clear accountability.

A signature isn't the same as a decision

A signature at the end of a high-volume queue is not a decision. By the time a reviewer is asked to approve a recommendation, much of the decision has already been shaped by the plan's rules, policies, data and system configuration. On a practical level, the ethical and operational problem for health care occurs when the signature can't answer for what it authorized. The reviewer may be approving an outcome without enough time or context to independently test it.

For providers, clinical decision support tends to be high stakes, but lower volume, enabling a physician to examine the patient context, evidence and recommendation in detail. Risk-bearing plans face a different problem: Claims, eligibility and encounter decisions happen at massive scale, and the underlying data changes constantly.

A member's claim may be denied, or a provider's payment adjusted, on the basis of a recommendation a plan can no longer fully reconstruct. In that model, a human reviewer is asked to bless a conclusion without the time, context, or evidence to apply consistent judgment.

When a past decision is challenged, most plans can produce a log entry showing someone looked at the determination and can cite today's denial basis. Far fewer can show the exact data, rules, and recommendation that existed at the moment the decision was made. They must rely on an assumption that nothing material changed in between. That assumption is rarely provable.

Plans must be able to explain denials, payments and eligibility determinations. If they cannot show why a reviewer approved a recommendation in March because they cannot prove what the system knew in March, the failure is architectural, not personal.

What human in command means

Human in command puts people, not models, in control of the process. People set the policies and rules. They approve changes to those rules and make the judgment calls on exceptions the rules do not cover. Automation executes the known rules on clear cases and surfaces the rest. The data foundation keeps a record, available on demand any time later, of the exact state at decision time. The ethical and operational requirement is the same: the ability to show what the system knew when it acted.

Regulation is moving toward real human authority

Regulators are beginning to draw the same line. Arizona's HB 2175, effective July 2026, requires that before a health plan denies a claim or prior authorization on medical necessity grounds, a medical director must individually review the denial, exercise independent medical judgment and not rely solely on a recommendation from another source.

The law does not ban automation. It allows data to be gathered, rules to be applied, conflicts to be flagged and recommendations to be generated. But it makes clear that a decision affecting access, payment or coverage cannot rest on an automated recommendation alone. The organization must be able to show who made the decision, what information was available and why the outcome followed from the rules in place at the time.

That distinction sets a practical floor for trustworthy automation rather than a ceiling on technology: A system whose output cannot be explained should not be trusted to drive a consequential decision. A human can only be in command of a process they can inspect. To explain a decision, a plan must be able to recreate it.

Point-in-time reconstructability

Point-in-time reconstructability means being able to recreate the exact facts, rules and data versions that existed when a decision was made. That includes the member's eligibility status, the provider contract, the rate table, the applicable policy, the claim details, the rule version and any exception or recommendation the system generated.

This cannot be added reliably after the fact. A tool that sits on top of unstable or unversioned data can show what the system sees now. It may not be able to prove what the system saw then, leaving the organization to rebuild the decision instead of showing it.

What human in command looks like in claims

Human in command does not mean people decide every claim manually. At scale, that's unrealistic. It means people set the rules, approve changes to them and handle the exceptions that call for judgment.

Automation handles the clear, repeatable cases. It also flags what it shouldn't decide alone: missing data, conflicting evidence, unusual patterns and anything outside defined confidence or risk thresholds. When the outcome is consequential, the case goes to the right person along with the evidence needed to make a real decision.

That's where many claims AI tools fall short. They're probabilistic: They predict the most likely answer and send whatever is still uncertain to a review queue. The reviewer sees a recommendation and is asked to accept or reject it. That workflow is more like throughput management than governance.

In a human-in-command model, the reviewer can also see why the system reached that recommendation, what facts it relied on, which rules it applied, what uncertainty it detected and whether the case falls within the system's approved scope.

The process should also learn from every review. Overrides, repeated escalations, appeals, audit findings and corrected claims all point to something. If reviewers keep reversing the same type of recommendation, the rule, data source, model or workflow likely needs to change.

A practical model for human in command

Industry bodies are converging on the principle that clinical authority must be earned through evidence and maintained through performance. Before delegating authority to a tool, establish what it can reliably do, for whom and under what conditions; and then keep checking as conditions change.

Translated into an operating model for health plans, these principles start with the data layer and end with accountable decisions.

* Inventory consequential decisions. Identify where denials, payments and eligibility determinations are made and what evidence each decision relies on.

* Qualify before delegating. For each AI-assisted function, document its intended use, populations, limitations and failure modes. Define when the tool may act and when it must defer.

* Version what matters. Preserve point-in-time states for eligibility, rates, contracts, policies and rules. Make them searchable by decision ID and date.

* Design reviews that add value. Give reviewers the inputs, rules and reasoning the system used. Test whether the combined workflow improves accuracy, equity and consistency.

* Escalate with context. Route complex cases to the expert best equipped to judge them, with the facts needed to decide on first review.

* Continuously requalify. Monitor overrides, escalation rates, disagreements, error patterns and model or data drift. Adjust authority and checkpoints as evidence changes.

With reconstructable records and calibrated checkpoints, plans can fix a root cause once, propagate the correction and avoid repeating the same error across thousands of cases.

The human authority standard

AI may continue to move deeper into health plan operations because the volume of claims, eligibility checks and payment decisions is large, and automation makes it more manageable. The question is whether that automation is governed by a record strong enough to support human authority.

A signature is easy to collect. A defensible decision requires a system that can show the exact data, rules, policies and recommendations in place when the decision was made.

That is what lets a plan explain a denial to a member, defend a payment decision to an auditor and correct errors without rediscovering them one claim at a time.

Audits do not ask what the system knows today. Members do not ask what the rule says now. They ask why a decision was made then.

Human in the loop can answer, "Someone clicked approve." Human in command has to answer, "Here is what we knew, here is the rule we applied and here is why the decision followed."

This story was produced by CureIS Healthcare and reviewed and distributed by Stacker.

Distributed by Newsbank, inc.