Introducing OCR Bench: stop rebuilding the same OCR benchmark
Every team that picks a document-extraction model builds the same throwaway benchmark: scripts, spreadsheets, hand-grading. OCR Bench is that work, done once. Bring your documents, get one scorecard for accuracy, cost, and latency.
On this page
Choosing a model to read your documents should take an afternoon. In practice it takes weeks, and the result is a benchmark nobody trusts and nobody can rerun.
We built OCR Bench because we kept watching the same work get redone, team after team. This post is about that work, and what it looks like when you stop doing it.
Every team rebuilds the same benchmark
Four chores that come before you learn anything about the models.
Glue code per provider
A different SDK, auth flow, and output shape for every model you want to try.
Hand-typed answer keys
Someone reads every document and types the right values into a spreadsheet.
Ad-hoc grading
Exact match here, fuzzy match there, and 1,240.00 vs 1240 argued case by case.
A result that can't be rerun
The script lives on one laptop. A new model arrives and the whole thing starts over.
Why this keeps happening
None of that work is specific to your company. The model glue, the grading rules, and the cost and latency bookkeeping (even a single number like what an LLM reads a document for) are identical from one team to the next. The only part that is truly yours is the documents and the answers you expect.
Yet that shared plumbing is where the weeks go, and it has a quiet side effect. When the benchmark is expensive to build, it gets built once. Then a cheaper model ships, or a provider retires one, and you're back to guessing.
Public leaderboards don't fix this. They score someone else's documents, so they can't tell you how a model handles your invoices, in your formats, with your edge cases.
What OCR Bench changes
OCR Bench does the repeatable part once, so you only supply the part that's yours.
From a project to a checkbox
Weeks of plumbing
- Write and maintain an integration per model
- Build the answer key by hand
- Invent grading rules, then defend them
- Rerun everything for each new model
One scorecard
- Every model sits behind one connection, so adding one is a checkbox
- Auto-fill drafts the answer key; you validate it
- One grading engine, the same rules for every model
- Add a model later without rerunning the rest
Three steps from documents to decision
Your documents in, a ranking out
Build a dataset
Upload documents, define the fields you need, and set the ground truth. Auto-fill drafts every value with Mistral OCR for you to accept or reject, and it's free.
Run a benchmark
Pick models yourself, or let the recommender read your schema and a sample document and shortlist them. Runs happen in the background.
Read the scorecard
Accuracy, cost, and latency side by side. Ship the winner on what matters to you.
A graded result, not a wall of output
You don't get raw model text to eyeball. Each extracted field is scored against your ground truth, per model, and rolled up into a single comparison.
Accuracy, cost, and latency in one view
The shape of what you get back. The numbers below are made up to show the layout.
- 1 Model A Accuracy94% Cost$$
- 2 Model B Accuracy91% Cost$
- 3 Model C Accuracy82% Cost$$$
The cheapest model is not always the worst, and the most accurate is not always worth its price. The scorecard puts both on one screen.
Every field lands in one of five outcomes
Matched the expected value after the field's cleaning steps.
Extracted something, but not the right value.
Should have been there; the model left it blank.
The model invented a value that shouldn't exist.
Correctly left blank.
Keeping missing and hallucinated apart matters. A blank is a visible gap you can route to review. An invented value looks like a real answer. Why a single similarity score hides that difference is covered in CER vs field-level accuracy.
Results stay trustworthy over time
A benchmark you can't reproduce is a benchmark you can't trust, so two rules protect your results:
- Changing the question creates a new version. Once a dataset has a benchmark, adding a field or editing a field's name, type, or description produces a new dataset version instead of rewriting the old one.
- Fixing the answer key re-grades, it doesn't rerun. Correct an expected value or a field's cleaning steps and every run is re-graded from its stored answers, with no model calls and no credits.
The grading engine itself, covering field processing, comparison, and metrics, is plain Python with no network or framework dependencies. That is why its results are deterministic and testable. For the details, read how the scoring engine turns model output into a verdict.
Do it once, reuse it forever
The point of OCR Bench isn't a prettier leaderboard. It's that the next time a model launches, or one gets retired, answering "should we switch?" takes a checkbox and a few minutes instead of a new project.
Start free, build a dataset, run one benchmark, and find out which model actually reads your documents.