How OCR Bench scores field extraction: correct, wrong, missing, hallucinated
Diffing strings is a terrible way to grade an LLM. Here's the normalization-and-comparison engine OCR Bench uses to turn raw model output into a fair, per-field verdict — with dates, currency, and numbers handled properly.
Ask two models for an invoice total and one says $1,250.00 while the other says 1250. Both are right. A naive string comparison would mark one of them wrong — and your benchmark would be worthless.
Grading document extraction fairly is the whole game, and it's harder than it looks. This is how OCR Bench does it.
Step 1: snapshot the schema
When an evaluation starts, OCR Bench snapshots the field schema — every field's type and normalization rules — and stores it with the run. If you edit the dataset next week, last week's results don't silently change meaning. Reproducibility first.
Step 2: extract
For each document, the image and the field schema go to the model, which returns JSON: a value per field. That's the raw material. It is not the score.
Step 3: normalize both sides
Before anything is compared, both the expected value and the extracted value are run through the same normalization pipeline. Comparing apples to apples means first agreeing on what an apple is.
Normalization is a regex pre-step followed by a type-aware preset:
| Type | What it does |
|---|---|
text | trim, collapse whitespace, optional case-folding |
number | strip grouping separators, fix decimals |
currency | drop symbols and separators, compare the amount |
date | parse to a canonical form regardless of input format |
boolean | map yes/no/true/false/1/0 to one value |
enum | match against an allowed set |
So $1,250.00 and 1250 both normalize to the same number, 12/06/2026 and 2026-06-12 to the same date, and the comparison is fair.
Step 4: assign a verdict
After normalization, each field gets exactly one outcome:
correct— values match.wrong— model returned a value, but the wrong one.missing— there was a value to find; the model returned nothing.hallucinated— there was nothing to find; the model invented something.correct_empty— correctly blank.
That hallucinated vs missing distinction matters more than a single accuracy number ever could. A model that makes things up fails differently from one that gives up — and depending on your use case, one of those is far more dangerous than the other.
A grading engine, not a guesswork tool: we don't just diff strings, we score each field with the normalization a human reviewer would apply in their head.
Step 5: roll up the metrics
Field verdicts aggregate into the numbers you actually compare models on:
- Per-field accuracy — which fields each model struggles with.
- Document accuracy — how often a whole document comes out clean.
- Mean score, total cost, average latency — the headline trade-off.
Why it's pure Python
The entire engine — normalization, comparison, metrics — is plain Python with no Django and no network calls. That's a deliberate design choice: a scoring function with no I/O is deterministic, fast, and trivially unit-testable. Given the same (model, document, field), you get the same verdict every time.
That determinism is also what makes local development and testing possible without a single API key — the subject of the next post.
Curious how your models score on your documents? Run a benchmark for free.