Why Most AI Pilots Fail
And How to Make Yours Succeed

The conversation opened with a look at where insurers are actually experimenting. The range is broader than most assume: submissions intake, underwriting, policy administration, reporting, bordereaux, invoice reconciliation, legal admin. According to Sasha Haco, roughly forty percent of MGA employees spend their time on administrative work, and the figure is likely higher for brokers.

Kyle Ramsay noted that Hippo has been applying AI at production scale across claims, customer service, underwriting, and internal software development. The common thread across all of it is that the use cases with the most impact tend to involve repetitive, structured workflows where the cost of getting it wrong is high enough to demand rigor, but not so catastrophic that automation is off the table.

The build-versus-buy debate has become more complicated with the arrival of AI code generation tools. Kyle Ramsay described a third option — partnering with a vendor who works closely with the organization’s own data and systems, deploying in a “forward engineer” model rather than selling a packaged product.

Sasha Haco aligned with this framing. Things that are core to what an insurer does — assessing risk, proprietary underwriting logic — lean toward build. Processes that are simply painful and slow, stitched together across five systems that nobody loves, are good candidates for buying. A short-term vendor relationship can also be a way to demonstrate ROI quickly while a longer-term build strategy is developed in parallel.

Kyle Ramsay laid out the four failure modes he sees most consistently. The first is a lack of clear problem definition and a shared definition of success across the software team, the operations team, and the vendor. When those aren’t aligned from the start, it’s hard to know whether the pilot has succeeded or not.

The second is data quality. Demos almost always run on clean, structured data. Production data is messier, and the gap between sandbox performance and real-world performance is where many promising pilots fall apart.

The third is operational context. AI that works in isolation from the people who will actually use it tends to create work rather than remove it. End users need to be involved in the design, not handed a finished tool and asked to adapt to it.

The fourth is systems integration. If the AI can’t pull data from your policy admin system or write results into your underwriting notes, the workflow still requires a human to bridge the gap.

One of the sharpest observations in the webinar came from Sasha Haco: AI pilots tend to fail when the measure of success is accuracy rather than business outcome. A model that is ninety-seven percent accurate might still require a human to review every single output, which means it hasn’t actually changed the nature of the manual workload at all.

The practical solution, she explained, is to treat the accuracy problem differently depending on what you’re trying to achieve. If the goal is full automation, the answer is to use deterministic, rules-based software for every step where you can be certain of the output, and only deploy AI for the steps that genuinely require complex reasoning. That way, the human review is reserved for a small fraction of cases rather than everything.

Jeff Goldberg added that vendors sometimes compound this problem by leading with accuracy numbers in sales conversations, when what matters to an insurer is whether a process can actually be automated or only assisted. Those are very different outcomes, and they require completely different implementations.

Audience questions during the webinar surfaced a real concern: as the volume of data fed into an AI system grows, so does the risk of incorrect or fabricated outputs. Both panellists addressed this directly.

Sasha Haco described the approach Unitary uses: structuring the AI’s context carefully, and using a decision-tree model where AI is asked a series of specific questions in sequence rather than being handed all documents and all questions at once. This reduces the chance of context overload and keeps outputs more predictable.

Kyle Ramsay described a complementary approach: creating teams of specialised sub-agents rather than one large generalist agent. One agent might be tasked purely with finding the roof condition section in a document. A separate, more capable model then makes a judgment about what that information means. This keeps each agent operating within a narrow, well-defined scope where hallucinations are easier to catch and control.

The group also touched on the double standard applied to AI errors versus human errors. The panel’s view was that the higher bar for AI is appropriate in a regulated industry, where accountability needs to be traceable. But they also noted that AI has an advantage humans don’t: when an AI makes a mistake, the whole system can learn from it. Human error tends to stay contained to the individual.

Perhaps the most candid part of the conversation was about a failure mode that doesn’t get talked about enough: AI pilots that technically succeed but never get adopted. The technology works, the accuracy is fine, and then nothing changes.

Sasha Haco identified three reasons this happens. First, the business need was never clearly defined, or the end users who would actually interact with the tool were never consulted during design. When an operational team inherits something built by a vendor and an innovation team, it often creates work rather than removing it. Second, explainability is absent — if a claims handler can’t understand why the AI made a decision, they will override it every time, and the automation benefit disappears. Third, the pilot ran on synthetic or simplified data that didn’t represent real-world conditions, so the effort to move it into production exposes a gap between what was tested and what actually exists.

Kyle Ramsay added a structural point: pilots often de-risk the easy version of a use case — one line of business, low-dollar claims, clean data — and then struggle to extend to where the real value lies. Business stakeholders start asking whether the investment was worth it before the hard work of scaling has even begun. The answer is to think about step two and step three of the plan before you start step one.

Kyle Ramsay shared the story of Clara, Hippo’s AI claims agent built to handle first notice of loss for homeowners. The ambition from the start was to handle all FNOL at scale, not just a subset, with no wait times, better customer experience, and a clear escalation path to a human when needed.

Claims leadership was involved from day one, right down to choosing Clara’s voice and the structure of the data she captured. The team trained Clara the way they would train a new employee, reviewing her work and providing feedback before she went live. The result: adjusters handling thirty percent more volume, high customer satisfaction scores, and customers occasionally thanking Clara at the end of a call.

Sasha Haco described a project with an MGA whose internal team had been trying to automate their loss runs process for eighteen months. It was a complex workflow spanning multiple systems, but there was genuine buy-in across the organisation to get it done.

Unitary’s team deeply understood the workflow, built the automation, tested it on example data, and went live. The entire process took five weeks from start to live deployment. Within a few weeks, they were hitting ninety-nine percent automation on real data.

The headline results were a fifty percent cost reduction and a nineteen percent improvement in broker satisfaction scores, because what had previously taken days now happened in minutes. The MGA also went on to win two Stevie Awards. The outcomes included some that were expected and some that were not — which, as Sasha noted, is often the pattern when automation is done well.

Asked to distil their advice into a final takeaway, both panellists were direct.