Accountable AI inside a product workflow
Insurance software · customer-facing product · five-month engagement, from discovery through launch.
The problem
The client sold a platform to insurance brokers, whose staff spent their days extracting the same twenty fields from documents that arrived in a hundred inconsistent formats. The commercial case for automating that was obvious. The product risk was not: brokers work under professional liability, and software that quietly gets a figure wrong is worse to them than software that does nothing.
An earlier attempt had shipped as a black box with a confidence score. Adoption stalled, because a percentage is not an explanation.
What we did
We started from the reviewer's job rather than the model's output. The specification defined which judgements the system was allowed to make alone, which it had to propose for confirmation, and which it must never touch. That boundary was agreed with the client's own compliance function before any interface work began.
We then built extraction as a first-class part of the workflow instead of a feature bolted onto it. Every extracted value points back to the exact place in the source document it came from. Every correction a reviewer makes is captured as a correction — not silently overwritten — so the product learns where it is weak and the client can prove what a human decided and when.
Before launch we built an evaluation set from real historical documents and held the release to it, including the awkward categories: handwriting, scans of scans, and the formats that appear twice a year.
What we shipped
- A review surface where each automated value is traceable to its source, one click away.
- An explicit split between assisted fields, confirmation-required fields, and fields the system never fills.
- Correction capture that feeds evaluation and preserves a defensible record of human decisions.
- A regression suite over real document classes, run on every change to the extraction path.
- Straightforward degradation: when confidence or the document type falls outside the tested envelope, the workflow reverts to manual and says so.
The result
The feature shipped and stayed shipped. Reviewers use it because they can check it — the fastest path to trusting an automated figure turned out to be being able to see where it came from. Because the boundary between assisted and human-owned decisions was written down before the build, the client's compliance review was a short conversation rather than a renegotiation.
The evaluation suite outlived the engagement and is now the gate the client's own team ships through.
Client identity withheld by agreement.