Introducing OCR Bench: know which model reads your documents best
Stop picking document-extraction models by vibes. OCR Bench runs every model that matters over your own documents and ranks them on accuracy, cost, and latency — one scorecard, real numbers.
Picking a model to read your documents is usually a guess. You skim a pricing page, try one model on a couple of files, and ship it. Then the edge cases show up in production — a date in the wrong format, a total that's off by a decimal, a field the model confidently invented.
OCR Bench replaces the guess with a measurement. You bring your own documents and the answers you expect, pick the models you're considering, and get back a single scorecard ranking them on the three things that actually decide the call: accuracy, cost, and latency.
The core idea
A benchmark is only meaningful on your documents. So OCR Bench is built around data you control:
- A dataset is your documents plus a field schema — the fields you need extracted (
invoice_total,due_date,vendor_name, …) and the ground-truth value for each one. - An evaluation runs one model over the whole dataset.
- A benchmark runs several models over the same dataset and compares them side by side.
User → Project → Dataset ─┬─ Field key, label, type, normalization
├─ DocumentFile ── ExpectedValue (ground truth)
├─ Evaluation one model over the dataset
└─ Benchmark N evaluations, compared
Because the field schema is snapshotted when an evaluation runs, results stay reproducible even if you change the dataset later.
What you get back
Not a wall of raw model output — a graded result. Every extracted field is scored against your ground truth and lands in one of five buckets:
| Outcome | Meaning |
|---|---|
correct | matched the expected value (after normalization) |
wrong | extracted something, but not the right value |
missing | should have been there, model left it blank |
hallucinated | model invented a value that shouldn't exist |
correct_empty | correctly left blank |
Those roll up into per-field accuracy, per-document accuracy, a mean score, total cost, and average latency — for each model, in one matrix.
Three steps to a real answer
- Build a dataset. Upload your documents, define the fields you need, and set the right answers as ground truth.
- Run a benchmark. Pick the models to compare. Each one runs over the whole dataset in the background.
- Read the scorecard. Accuracy, cost, and latency side by side. Ship the model that wins on what matters to you — not on a leaderboard built from someone else's documents.
Built to extend
Under the hood, OCR Bench talks to many providers through one connection, so adding a model to a comparison is a checkbox, not an integration project. The grading engine — normalization, comparison, and metrics — is pure Python with no network or framework dependencies, which is exactly why its results are deterministic and testable.
We'll go deeper on both of those in upcoming posts: how the scoring engine turns model output into a verdict, and how you can develop and test the whole stack with zero API keys.
Want to see your own numbers? Start free — build a dataset, run one benchmark, and find out which model actually reads your documents.