How OCR Bench scores field extraction: correct, wrong, missing, hallucinated
Diffing raw strings is a terrible way to grade an LLM, and so is a grader that guesses. Here's the processing-and-comparison engine OCR Bench uses to turn raw model output into a fair, per-field verdict: what it cleans, what it deliberately leaves alone, and why.
On this page
Ask two models for an invoice date and one says 12/06/2026 while the other says 2026-06-12. Both are right. A naive string comparison would mark one of them wrong — and your benchmark would be worthless.
The opposite mistake is just as bad. A grader that decides 1,250 and 1.250 mean the same thing is guessing, and a guess that lands in a score quietly rewards whichever model happened to agree with it.
Grading document extraction fairly is the whole game, and it's harder than it looks. This is how OCR Bench does it.
Step 1: extract
For each document, the image goes to the model along with a prompt built from your field schema: every field's key, type, label and description. The model returns JSON, one value per field. That's the raw material. It is not the score.
Step 2: clean the answer, then compare
Every field carries its own list of processing steps, chosen by its type. They run in two stages:
- Extraction steps clean the model's answer only. Your expected values are entered clean, so they skip this stage.
- Comparison steps run on both the expected value and the cleaned answer, so values that only differ in form compare equal.
The rule of thumb: clean away what can't change the meaning, and leave everything else alone. Here is what a new field does by default, and what you can switch on:
| Type | Cleans the answer | Applies to both sides | Off by default |
|---|---|---|---|
text | trim spaces at both ends | collapse repeated spaces and line breaks, ignore case | ignore accents, keep a regex match |
number | trim, keep only digits, ,, . and - (drops currency symbols, spaces and units) | — | remove the thousands separator, read the decimal separator, keep a regex match |
date | trim | read the date in one of the accepted formats (by default YYYY-MM-DD and DD/MM/YYYY), then compare the day | keep a regex match |
presence | trim | ignore case, read true or false | keep a regex match |
So 12/06/2026 and 2026-06-12 are the same day, the answers $1,250.00 and 1,250.00 USD both clean to 1,250.00, and Acme Corp matches acme corp.
Why numbers keep their separators. 1,250 is one thousand two hundred and fifty on a US invoice and one and a quarter on a French one. Rather than pick a reading for you, OCR Bench compares the digits and separators as written: 1250.00 and 1250 count as different. If your documents follow one convention, turn on the separator steps in the field's settings and tell them which character is which: with . as the thousands separator and , as the decimal one, the answer 1.250,00 cleans to 1250.00. Those steps read the model's answer, so write your expected values with a dot. It's a setting you chose, not a guess.
Why presence is strict. The prompt asks the model to answer a presence field with the JSON literal true or false, and ground truth is stored the same way, so there's nothing to interpret. A blank answer counts as no answer, not as false.
Keeping part of an answer. Any field can keep only what a regex matches in the model's answer — say, INV-2024-17 out of Invoice INV-2024-17. When the regex matches nothing, the answer counts as empty.
One last rule keeps the verdicts honest: an answer that can't be read for its type (N/A on a number field, soon on a date field) is still an answer. It is compared as written and never matches a value that could be read, so it grades as wrong or hallucinated, never as a blank.
Each field's settings list its steps with a before-and-after example, so you can see exactly what will be cleaned before you run anything. After a run, you can open any graded answer and see each step it went through.
Step 3: assign a verdict
After cleaning, each field gets exactly one outcome:
correct— values match.wrong— model returned a value, but the wrong one.missing— there was a value to find; the model returned nothing.hallucinated— there was nothing to find; the model invented something.correct_empty— correctly blank.
That hallucinated vs missing distinction matters more than a single accuracy number ever could. A model that makes things up fails differently from one that gives up — and depending on your use case, one of those is far more dangerous than the other.
A grading engine, not a guessing one: it cleans what can't change the meaning, and anything that could is a setting you choose per field.
Step 4: roll up the metrics
Field verdicts aggregate into the numbers you actually compare models on:
- Per-field accuracy — which fields each model struggles with.
- Document accuracy — how often a whole document comes out clean.
- Mean score, total cost, average latency — the headline trade-off. To estimate the cost side before you run anything, use the cost-per-document calculator.
When you change the rules
Every run keeps the schema it was graded against and every raw answer the model returned. That makes two kinds of edit safe:
- Fixing the answer key — correcting an expected value, changing a field's cleaning steps, or excluding a document — re-grades the existing runs from their stored answers, once you confirm. No model is called and no credit is spent.
- Changing the question — adding or deleting a field, or editing one's key, name, type, description or required flag — changes what the models are asked to extract. On a dataset that already has a benchmark, that creates a new version of the dataset instead, so earlier results keep the question they answered.
Why it's pure Python
The entire engine — field processing, comparison, metrics — is plain Python with no Django and no network calls. That's a deliberate design choice: a scoring function with no I/O is deterministic, fast, and trivially unit-testable. Given the same expected value, the same answer and the same field settings, you get the same verdict every time.
Curious how your models score on your documents? Run a benchmark for free.