The Delegated AI Oversight Questions
Fifteen questions for managing agents, syndicates, carriers and reinsurers to put to the MGAs on their paper. Use them in a binder review, a coverholder audit, or before a regulator asks first.
Written by Daniel Palacios, Co-Founder & CRO.
- 01AI is now part of the standard Lloyd's coverholder audit. The LMA has widened its common audit scope to cover artificial intelligence risks, and the change is being written into the Delegated Audit Manager, the platform the market's audits run on.
- 02Twelve US state regulators are piloting the NAIC's AI Systems Evaluation Tool through September 2026, inside the routine exams every insurer already receives. AI supplied by third parties is a core focus.
- 03Responsibility runs up the chain. Under the NAIC's Model Bulletin of December 2023, an insurer answers for the AI used on its paper, whoever built it and whoever runs it.
- 04The gap is already commercial. A managing agent with a governance framework carries a third-party AI risk the moment a coverholder beneath it has none.
- 05The fifteen questions are free to copy. Take them into your next binder review, whatever platforms your MGAs use, including ours.
AI enters the coverholder audit
For forty years the coverholder audit asked about segregation of duties, complaints logs and the whereabouts of client money. In 2026 it started asking about algorithms. The Lloyd's Market Association, the trade body for the market's underwriters, has quietly widened its standard audit scope so that information-security questions now cover artificial intelligence, and the change is being written into the Delegated Audit Manager, the platform on which the market's audits run. What was a special exercise is becoming a line on the checklist.
America is moving in parallel. Twelve state regulators, from California to Wisconsin, are piloting the NAIC's AI Systems Evaluation Tool — a structured questionnaire for examiners — inside the routine market-conduct and financial exams every insurer already receives. Behind the pilot sits the NAIC's Model Bulletin of December 2023, which is blunt about where responsibility lives: an insurer answers for the AI used on its paper, whoever built it and whoever runs it.
The two developments meet in an awkward place: the delegation boundary. A managing agent now carries a third-party AI risk the moment a coverholder beneath it has no AI governance of its own — a gap the trade press began reporting this summer as a commercial problem for binder relationships rather than a theoretical one. An MGA's AI is now, in the examiner's eyes, its capacity provider's problem.
Neither framework, however, tells a capacity provider what to actually ask the MGA across the table. The NAIC's tool is written for examiners; the LMA's scope is written for auditors; the law firms' client alerts are written for the defense. What follows is the practical version: fifteen questions in five groups, designed so that a competent MGA can answer them in writing in an afternoon, and so that the answers add up to most of what an examiner will eventually request. We suggest you send these before the review, not during it.
A final note: Lloyd's own doctrine for delegated oversight is proportionate oversight, and we believe our list of questions honors it. The underlying logic is older than any of the technology. In delegated authority, responsibility is the one thing that cannot be delegated.
The fifteen questions
A. Where AI touches the book
- 1.Where does AI touch a decision that affects what you bind on our paper?
- 2.Which systems are involved, and which of them are third party? Name them.
- 3.Which actions complete on their own, and which wait for a human? Draw the line for us.
The answer you want is a short, named inventory with the decision points marked. Vague talk of using AI for efficiency, with no list behind it, is a warning sign, because an MGA that cannot inventory its AI cannot really govern it. An inventory is also the first thing both the LMA's scope and the NAIC's tool ask for.
B. Authority and control
- 4.Can a machine action exceed a human’s authority in your shop?
- 5.Who sets the boundaries of autonomous action, and how fast can you change them?
- 6.What's the rollback when it gets something wrong? Has it ever been used?
Listen for permissions inherited from people, boundaries the MGA can change itself, and a rollback that has actually been used at least once. Reassurance that the model is very reliable is not a control, however sincerely it is offered.
C. Evidence and traceability
- 7.Show me the decision trail on this risk.
- 8.How long does producing that take you, and does it arrive with sources attached?
- 9.Are your audit records tamper evident, and how long are they kept?
The best answers are framed in minutes, not weeks, and every field will be traceable to its source page. Similarly, retention will be stated in years, and tamper evident simply means the record shows whether anyone has changed it. If producing a decision trail takes a week of digging through old inboxes, that tells you something as well.
D. Accuracy and data
- 10.How do you know your extraction is accurate, and how often do you check?
- 11.Is our data, or your insureds' data, ever used to train anyone's models?
- 12.Where does the data sit, and who else can see it?
Good practice measures accuracy continuously in production, states the sample and the period, and can point to a training exclusion in an actual contract clause. You also want to hear where the data sits and who can see it. A single percentage, measured once on implementation day, mostly tells you that nobody has measured since.
E. Accountability and governance
- 13.Who is accountable, by name, for the AI systems program?
- 14.What written program sits behind that name, and when was it last reviewed?
- 15.If we, our regulator, or Lloyd's asked tomorrow, what would you hand us?
You are listening for three things: a name, a dated document, and a pack that exists before anyone asks for it. An MGA that says its vendor handles compliance has given you none of the three.
If you're an MGA reading this
You'll meet these questions from your capacity providers whether or not they found them here. Section 08 of our buyer's guide covers the same ground from your seat, and the Delegated Authority AI Scorecard gives you the forty-question version to put to any vendor, including us. An operation that can answer all fifteen in the room is negotiating its binder from somewhere quite different.
Frequently asked questions
Are capacity providers responsible for AI used by their MGAs and coverholders?
Do Lloyd's coverholder audits now cover AI?
What should a binder review ask about an MGA's use of AI?
Sources
The claims above rest on:
- 1Lloyd's Market Association, Coverholder Audit Scope and Associated Guidance. The revised scope expands IT/Information Security to reflect cyber and artificial intelligence risks, with replication into the Delegated Audit Manager (LIMOSS) for the 2026 audit program. lmalloyds.com, 2026.
- 2NAIC AI Systems Evaluation Tool (v4.0) and the 12-state pilot, running through September 2026 (CA, CO, CT, FL, IA, LA, MD, PA, RI, VT, VA, WI), integrated with market conduct and financial examinations. Third-party risk is a core focus. content.naic.org, 2026.
- 3NAIC, Model Bulletin: Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023. Third-party AI expectations: due diligence, testing, documentation, audit rights, data practices, ongoing monitoring.
- 4Insurance Business (reporting Intersys), "AI adoption outpacing governance across MGA market," July 2026. The two-tier governance gap and its implications for coverholder relationships and delegated authority reviews.
- 5Lloyd's, Delegated Authority guidance and Oversight Framework; LMA Delegated Authority. Proportionate oversight framing. lloyds.com / lmalloyds.com, accessed August 2026.
- 6Regulation (EU) 2024/1689 (the AI Act) and the Digital Omnibus on AI. Current dates as maintained on our AI regulation dates page.
