Insurance’s AI ‘black box’: Can auditors trust the results?
As insurance companies expand their use of AI, increasingly complex systems could become difficult to explain, validate and audit, raising questions about how insurers can verify the technology's decisions.
Michelle Donahue, a managing director at Deloitte, raised those concerns Tuesday during the National Association of Insurance Commissioners' fall meeting in Dallas, where she outlined how the auditing profession is adapting to AI's growing role in insurance operations.
The risks extend to claims processing, underwriting and financial reporting, where AI systems may produce convincing results without providing a clear explanation of how they reached them.
"If we can't explain the models in what we call a black box, the challenge is the AI might not be able to be auditable," Donahue told the NAIC's Innovation, Cybersecurity, and Technology Committee. "It makes validation difficult and reliance on the AI even more of a challenge."
Inside AI's 'black box'
A "black box" describes an AI system whose internal decision-making process is difficult for people to understand, even when they can see the information entered and the results produced.
Donahue said the challenge is becoming more complicated as companies move beyond individual AI tools toward interconnected systems capable of completing entire business processes.
In insurance, that could mean several AI agents working together to process a claim, with one checking coverage, another evaluating fraud indicators and another reviewing decisions for potential bias.

Unlike traditional software, some AI systems may generate different responses to the same information, making their performance harder to predict and evaluate.
Donahue warned that AI can also produce fabricated information that appears accurate, a problem commonly known as hallucination.
"It's something that looks good, looks right, but it may not be good, and it may not look right," she said.
That creates challenges for employees and auditors who must determine whether AI-generated information can be trusted.
Who is responsible for oversight?
Donahue emphasized that companies need clear accountability for how AI is developed, deployed and monitored.
Boards of directors have overall oversight responsibility, while audit committees should understand how AI affects business operations and financial reporting.
Management must also identify where AI is being used, document its purpose and establish controls to monitor its performance.
One challenge, Donahue explained, is maintaining a complete inventory of AI applications across an organization.
Employees may use different AI tools for different tasks, making it difficult to determine which systems influence financial reporting or other important decisions.
Internal auditors, meanwhile, are responsible for independently assessing whether those controls are working and whether problems have been corrected.

Donahue said governance must evolve as companies adopt increasingly sophisticated AI systems.
She also described conversations with a company using multiple AI agents whose systems had become so complex that employees were concerned about their ability to explain how the models worked.
The concern extended to documenting those systems in a way that both internal and external reviewers could understand.
A changing regulatory landscape
The presentation also addressed the fragmented regulatory environment surrounding AI.
Donahue said companies face overlapping expectations from state and federal authorities, along with guidance from professional organizations and international standards bodies.
Despite those differences, she identified common regulatory priorities, including transparency, risk management, fairness, human oversight and data security.
Those considerations are particularly relevant to insurers using AI in claims processing, pricing and underwriting, where automated decisions can affect consumers.
The NAIC has been developing an AI Risk Evaluation Supplement to help state regulators assess insurers' use of AI. The supplement is designed to support existing financial examinations and market conduct reviews rather than establish new legal obligations.
Donahue said auditors face similar challenges because professional guidance is still evolving alongside the technology.
She pointed to work by accounting and auditing organizations to develop frameworks addressing AI governance, internal controls and auditing procedures.
AI changes the work of auditors
Despite the risks, Donahue said AI is also changing how Deloitte conducts audits.
The firm uses technology to analyze financial information, identify potential risks, prepare draft documents and reduce manual work.
That allows auditors to spend more time on complex matters requiring professional judgment, she said.
But the transition also raises questions about how younger auditors develop the experience needed to evaluate AI-generated results.
Donahue acknowledged some anxiety as technology takes over simpler assignments traditionally performed by less-experienced professionals.
She emphasized that auditors must continue developing the skills to challenge AI outputs rather than accept them without verification.
"They still need to do all that," Donahue said. "And I think it requires certain skills that, you know, maybe took them a few years to get before, but now they're getting to them more quickly in their career."
© Entire contents copyright 2026 by InsuranceNewsNet.com Inc. All rights reserved. No part of this article may be reprinted without the expressed written consent from InsuranceNewsNet.com.
Noah Lopez is editorial assistant for InsuranceNewsNet. Contact him at noah.lopez@innfeedback.com.



What an airspace shutdown tells us about risk
NAIC adopts credit rating agency framework despite industry concerns
Advisor News
- The Gen Z money paradox
- NAIFA applauds final SECURE 2.0 regulations
- What will Democrats in Congress do about healthcare and retirement?
- Election results cloud retirement legislation outlook, experts say
- Planning for the missing chapter
More Advisor NewsAnnuity News
- Best’s Market Segment Report: AM Best Maintains Stable Outlook on US Life/Annuity Insurance Segment
- AM Best Affirms Credit Ratings of Massachusetts Mutual Life Insurance Company and Its Subsidiaries
- Boomers want retirement protection, but their actions don’t match their mindset
- NAIC task force to address LTC riders
- ‘It’s really shocking:’ Regulators scrutinize assets backing annuity sellers
More Annuity NewsHealth/Employee Benefits News
- New York Life Announces Record $3.1 Billion in Policy Owner Dividends for 2027
- New Va. subsidies help lure people back to health insurance
- 3 ways Crowley can make Healthcare more affordable in Wisconsin
3 ways to make healthcare more affordable in Wisconsin | Coleman Drake (Copy)
- Vermont public schools expect the lowest health insurance increase in years. For districts, it’s only a small relief.
- Vt. public schools expect the lowest health insurance increase in years
More Health/Employee Benefits NewsLife Insurance News
- Nebraska widow’s big win against life insurer in legal limbo
- New York Life Announces Record $3.1 Billion in Policy Owner Dividends for 2027
- AM Best Affirms Credit Ratings of Massachusetts Mutual Life Insurance Company and Its Subsidiaries
- U-Haul Holding Company to Participate in Deutsche Bank Conference
- NAIC task force to address LTC riders
More Life Insurance News