Can you trust the route it recommends?
On 26 test claims, it sent 10 of 10 serious claims to an adjuster and none of the 16 simple ones. When it was unsure, it asked for another photo rather than guess. That's a good start, not proof: proof needs your own claims.
10 of 10
serious claims went to an adjuster
None was approved from photos. With so few claims, the real rate could be as low as 72%.
0 of 16
simple claims sent to an adjuster they didn't need
Each one would be an adjuster's time spent for nothing.
23 of 26
recommended the same route as our expert
The other 3 are explained below.
10 sec
and about 4 cents a claim
Runs in the background, so a reviewer rarely waits.
Not measured yet: whether the repair estimate matches what you'd actually pay. That needs your paid claims.
What we tested it on
Are they real claims? No. The photos are real photos of damaged cars, but the claim details (the customer, the policy, the car's value) are made up for testing. None come from an insurer.
Where are the photos from? 8 originals: four real crashes from Wikimedia Commons (a flood, a front-end crush, a van under a wall, a car into a tree), a press photo of a race-car crash, a dented Honda Civic and Toyota Camry I sourced, and one photo with no car in it.
Why 26? The other 18 are built from those 8 to test one thing each. 13 are harder versions of a photo (too dark, blurry, sideways, glare, close-ups): does it ask for a better photo instead of guessing? 5 reuse the Civic photo with a tricky claim detail (an injury, damage on the wrong side, a cheap car, two different cars): do the rules catch it?
Who decided the right answer? I did, as drafts. An estimator should review them before anyone relies on these numbers.
Did any answers change after testing? One. A test claim reuses the Civic photo on a car worth only $2,500, and I'd first labelled it a total loss. The repair is about $1,130, 45% of the car's value and under the 60% total-loss line, so an estimator would repair it. I corrected the label; sending it to an adjuster still counts as reasonable. With your claims, your estimators set the answers before testing.
10 should go to an adjuster, and only 7 should be approved from photos, so the next gap to fill is more simple claims. Every test claim is listed with its photo under More detail, at the bottom.
What we'd need from you to trust it
A few hundred past claimswith photos, where each one went, and what was finally paid
Two of your estimatorslabelling them separately, so we know how often experts agree
Today's numbershow often claims are escalated late, and how long a review takes
A test set we never tune onso every change is checked against it before it goes live
The 3 claims it didn't get exactly right
Every one erred on the safe side
A clean photo, uploaded sideways.Asked for a photo it didn't need
Expert said: Ready to approve · Recommended: Request more evidence
- What happened
- Sideways, the AI couldn't tell which car it was, so it asked the customer for another photo instead of guessing.
- What we'd change
- Straighten photos automatically before the AI looks at them. Today this costs the customer one extra photo; it never lets a serious claim through.
Front-corner damage priced close to the $2,500 approval limit.Different answer on repeat runs
Expert said: Ready to approve · Recommended: Ready to approve 3 times, Adjuster / total loss 1 time
- What happened
- The price sat right around the limit, so on repeat runs it went to an adjuster once and was approved three times. The expert would approve it, but an adjuster is a reasonable call too.
- What we'd change
- Claims whose price could run well past the limit now always go to an adjuster, so it gives the same answer every time. With your paid claims we'd set that threshold properly.
A blurry, forwarded copy of a photo.Reasonable, not the expert's first choice
Expert said: Ready to approve · Recommended: Request more evidence
- What happened
- It asked the customer for a better photo. The expert might have approved from it, but asking is reasonable.
- What we'd change
- Fine as it is. Your reviewers' decisions would tell us whether it asks too often.
More detail: every test claim with its photo, models compared, and how it was scored
Didn't guess when unsure 17 of 17Car identified right make 17 of 24, model 18 of 24, color 24 of 24Review flags raised 5 of 5Same route every run 25 of 26Slowest 1 in 20 14.0 sModel and prompt Claude Opus 5.5, extract-v2
Models and prompts side by side
Same 26 cases, same rules| Measure | Claude Opus 5.5, extract-v2 | Claude Sonnet 5.5, extract-v2 | Claude Opus 5.5, extract-v1 |
|---|---|---|---|
| Complex claims sent to an adjuster | 10 of 10 | 10 of 10 | 10 of 10 |
| Same route as the expert label | 23 of 26 | 23 of 26 | 22 of 26 |
| An acceptable route per the label | 25 of 26 | 25 of 26 | 23 of 26 |
| Simple claims sent to an adjuster | 0 of 16 | 0 of 16 | 1 of 16 |
| Didn't guess when unsure | 17 of 17 | 16 of 17 | 17 of 17 |
| Make / model / color | 17/24, 18/24, 24/24 | 17/24, 18/24, 23/24 | 17/24, 17/24, 24/24 |
| Typical time per claim | 9.7 s | 7.0 s | 9.6 s |
| Cost per case | $0.039 | $0.019 | $0.038 |
| AI failures | 0 | 0 | 0 |
Recommended route compared with the expert's
Each row is what the expert said; each column is what the AI and rules recommended. Bold is where they agree.| Expert said | Ready to approve | Request more evidence | Adjuster / total loss | Not assessed: manual triage |
|---|---|---|---|---|
| Ready to approve | 4 | 2 | 1 | 0 |
| Request more evidence | 0 | 8 | 0 | 0 |
| Adjuster / total loss | 0 | 0 | 11 | 0 |
Every case
Highlighted rows disagree with the expert label| Photo | Case | Expected | Recommended route | Why | Make · model · color | Estimate |
|---|---|---|---|---|---|---|
 | A_civic Core outputs on a clear photo | Ready to approve | Ready to approve | none | Honda ✓ Civic ✓ Silver ✓ | $950 to $1,300 under the limit Expert: $1,000 to $2,500 |
 | B_civic_closeup Doesn't guess the car from a close-up | Request more evidence | Request more evidence | E2, E3 | ? ✓ (didn't guess) ? ✓ (didn't guess) Silver ✓ | $1,050 to $1,500 provisional, not scored |
| B_civic_retake Customer's retake resolves the evidence request | Ready to approve | Ready to approve | none | Honda ✓ Civic ✓ Silver ✓ | $1,050 to $1,500 under the limit Expert: $1,000 to $2,500 |
 | C_f1 Non-road vehicle with structural damage | Adjuster / total loss needs an adjuster | Adjuster / total loss | S2, S4, P1, E2, E4, E6 | ? ✕ left blank ? ✓ (didn't guess) Orange and blue ✓ | none |
| 00a_camry Front-corner damage near the fast-path limit | Ready to approve | Adjuster / total loss | C4, R1 | Toyota ✓ Camry ✓ Grey ✓ | $2,500 to $3,450 straddles the limit Expert: unsure of the band |
 | 00b_camry_closeup Close-up of a different car | Request more evidence | Request more evidence | E2, E3, R1 | ? ✓ (didn't guess) ? ✓ (didn't guess) Dark grey ✓ | $800 to $1,150 provisional, not scored |
 | 01_low_light Too dark to judge the damage | Request more evidence | Request more evidence | E2, E4, R1 | ? ✕ left blank ? ✕ left blank ? ✓ (didn't guess) | $1,350 to $1,900 provisional, not scored |
 | 02_motion_blur Blurry photo | Request more evidence | Request more evidence | E2, E4, R1 | ? ✕ left blank ? ✕ left blank Grey ✓ | $500 to $750 provisional, not scored |
| 03_compressed Forwarded low-quality copy | Ready to approve | Request more evidence | E2, E4, R1 | ? ✕ left blank ? ✕ left blank Grey ✓ | $1,650 to $2,300 provisional, not scored Expert: unsure of the band |
 | 04_glare Glare hides the damage | Request more evidence | Request more evidence | E2, E4, R1 | ? ✕ left blank ? ✕ left blank Dark grey ✓ | $1,050 to $1,500 provisional, not scored |
| 05_rotated Sideways upload should match 00a | Ready to approve | Request more evidence | E2, R1 | ? ✕ left blank ? ✕ left blank Grey ✓ | $1,350 to $1,900 provisional, not scored Expert: unsure of the band |
 | 06_mirrored_duplicate Flipped copy of a past claim's photo | Adjuster / total loss needs an adjuster | Adjuster / total loss | I1, E2, R1 | ? ✕ left blank ? ✕ left blank Dark grey ✓ | $2,150 to $2,950 provisional, not scored |
 | 07_headlight_closeup Close-up in a sensor and lighting area | Request more evidence | Request more evidence | E2, E3, R1 | ? ✓ (didn't guess) ? ✓ (didn't guess) Grey ✓ | $1,350 to $1,900 provisional, not scored |
 | 08_wheel_closeup Wheel only; the damaged area isn't shown | Request more evidence | Request more evidence | E2, E3, R3 | ? ✓ (didn't guess) ? ✓ (didn't guess) Dark grey ✓ | $450 to $1,350 provisional, not scored |
 | 09_no_vehicle No vehicle in the photo | Request more evidence | Request more evidence | E1, E4 | ? ? ? | none |
 | 10_race_debris_crop Partial race-car crop | Adjuster / total loss needs an adjuster | Adjuster / total loss | S2, S4, P1, E2, E3, E4, E6 | ? ✓ (didn't guess) ? ✓ (didn't guess) Orange and black ✓ | none |
 | 11_race_greyscale Black-and-white photo: colour can't be known | Adjuster / total loss needs an adjuster | Adjuster / total loss | S2, S4, P1, E2, E4, E6 | ? ✓ (didn't guess) ? ✓ (didn't guess) ? ✓ (didn't guess) | none |
| V1_injury Injury on the claim overrides a simple photo | Adjuster / total loss needs an adjuster | Adjuster / total loss | S1 | Honda ✓ Civic ✓ Silver ✓ | $950 to $1,300 under the limit Expert: $1,000 to $2,500 |
| V2_wrong_side Damage doesn't match the customer's description | Ready to approve | Ready to approve | R3 | Honda ✓ Civic ✓ Silver ✓ | $950 to $1,300 under the limit Expert: $1,000 to $2,500 |
| V3_low_value Same damage on a car worth much less (repair under the total-loss line) | Ready to approve | Ready to approve | none | Honda ✓ Civic ✓ Silver ✓ | $950 to $1,300 under the limit Expert: $1,000 to $2,500 |
| V4_mixed_cars Folder with photos of two different cars | Adjuster / total loss needs an adjuster | Adjuster / total loss | I2, E2, R1 | ? ? ? | $2,500 to $3,400 provisional, not scored |
| V5_asked_twice Still not enough after two requests | Adjuster / total loss | Adjuster / total loss | E7, E2, E3 | ? ✓ (didn't guess) ? ✓ (didn't guess) Silver ✓ | $1,050 to $1,500 provisional, not scored |
 | D_nissan_front_crush Severe front crush on an ordinary road car | Adjuster / total loss needs an adjuster | Adjuster / total loss | S2, S4, C1, C2, R1 | Nissan ✓ Altima ✓ White ✓ | $3,250 to $4,450 over the limit Expert: total loss |
 | 12_flood Flood damage a photo can't price | Adjuster / total loss needs an adjuster | Adjuster / total loss | S4, S5, E3, E4, R4 | Mercedes-Benz ✓ 230 TE (W124 estate) ✓ Gold/beige ✓ | $400 to $600 provisional, not scored Expert: total loss |
 | 13_van_crushed Roof crushed by a collapsed wall; also a commercial-type van | Adjuster / total loss needs an adjuster | Adjuster / total loss | S2, S4, E3, E4 | Hyundai ✓ Grand Starex (H-1) ✓ Yellow ✓ | $8,850 to $12,000 provisional, not scored Expert: total loss |
 | 14_vw_tree Front corner into a tree; possible airbag and structural damage | Adjuster / total loss needs an adjuster | Adjuster / total loss | S2, S3, E3, E4, R1 | Volkswagen ✓ Golf ✓ Dark grey ✓ | $4,650 to $8,200 provisional, not scored Expert: total loss |