OCRBench & OCRBench v2 Explained: Leaderboard Scores, How to Run It, and Why 94% Extraction Still Breaks in Production
A complete technical breakdown of OCRBench and OCRBench v2—featuring the latest official leaderboard scores, what the 8 metrics actually measure, why 94% extraction still breaks JSON pipelines, and how to benchmark your own documents.
On this page
- What is OCRBench (v1)?
- How OCRBench v1 is scored (and why it flatters models)
- What changed in OCRBench v2?
- The OCRBench v2 leaderboard (and what the numbers hide)
- Three takeaways every engineering team should notice
- OCRBench v2 vs. OmniDocBench vs. olmOCR-Bench
- How to run OCRBench locally
- Four reasons public OCRBench scores fail in production
- What the verdicts look like on one document
- Run your own private OCRBench on your documents
- FAQ
- What is the difference between OCRBench and OCRBench v2?
- Which model ranks #1 on the OCRBench v2 leaderboard?
- Does OCRBench include invoices and purchase orders?
- Why do some open models outrank GPT-5 on OCRBench v2?
- How can I benchmark OCR models on my own custom PDFs?
TL;DR (Quick Reference)
- What it is: OCRBench (v1, 2023) and its successor OCRBench v2 (2025) are text-reading benchmarks built by researchers at Huazhong University of Science and Technology (HUST) to measure text reading and visual reasoning in large multimodal models (LMMs), including vision-language models (VLMs).
- Scale: v1 has 1,000 question-answer pairs drawn from 29 datasets. v2 has 10,000 human-verified QA pairs across 23 tasks and 31 scenarios, in English and Chinese, plus a 1,500-image private test set that is never released.
- The catch: On the official OCRBench v2 private English leaderboard, 19 of the 26 models score 90% or higher on Relation Extraction, while Element Parsing ranges from 10.0 to 45.7. Neither version tests multi-page B2B invoices, and neither scores JSON schema validity directly.
- Benchmark your own docs: Want an OCRBench-style head-to-head test on your actual PDFs, schemas, and budget? Run a private benchmark on OCR Bench (ocrbench.app) →
What is OCRBench (v1)?
When vision-language models first started claiming they could "read any image," researchers at Huazhong University of Science and Technology (HUST) and Microsoft Research introduced OCRBench to test zero-shot optical character recognition without task-specific fine-tuning.
Rather than rely on a single clean scan dataset, OCRBench v1 combines 1,000 manually checked question-answer pairs from 29 existing datasets, split across five tasks:
- Text recognition: reading regular, irregular, occluded, artistic and handwritten text, plus Chinese text, digit strings and non-semantic character sequences.
- Scene text-centric VQA: answering questions about signs, labels and other text in natural photos.
- Document-oriented VQA: answering questions about scanned documents, infographics and charts.
- Key information extraction (KIE): pulling values such as company, date, address and total from receipts (SROIE), forms (FUNSD) and nutrition labels (POIE).
- Handwritten math expression recognition: converting handwritten formulas from the HME100K dataset into LaTeX.
How OCRBench v1 is scored (and why it flatters models)
Each question is worth 1 point, for a maximum score of 1,000. Instead of exact matching or ANLS, the paper uses one rule for every dataset: is the ground-truth answer present anywhere in the model's output? If the answer is "42.00" and the model replies "Based on the receipt image, the total amount paid is $42.00 USD," it gets full credit.
That rule works for testing whether a model can see text. It tells you almost nothing about whether the model will follow a strict JSON schema in production.
What changed in OCRBench v2?
As frontier models began saturating the original question set, the maintainers released OCRBench v2, a larger and harder benchmark built around localization, structured parsing and multi-step visual reasoning:
- 10x more QA pairs: 10,000 public, human-verified pairs, against 1,000 in v1, across English and Chinese.
- 4x more tasks: 23 tasks spread over 31 scenarios.
- A private test set: 1,500 manually annotated images that are not released, to limit contamination from web-scraped pretraining data. The maintainers say they update the leaderboard regularly, about quarterly.
- Normalized 0–100 scoring: each capability is scored on a 0–100 scale, and the overall rank is the unweighted average of the eight capability scores.
- Single-image by design: multi-image and multi-page document tasks are out of scope.
-
Eight capabilities:
- Text recognition: transcribing the text in an image.
- Text referring: identifying text inside a given bounding box.
- Text spotting: locating text and transcribing it, with coordinates.
- Relation extraction: extracting key-value pairs, mapping keys to values, and extracting handwritten content.
- Element parsing: converting tables, charts and formulas into structured HTML, LaTeX or Markdown.
- Mathematical calculation: performing arithmetic over numbers read from the image.
- Visual text understanding: answering comprehension questions over dense layouts.
- Knowledge reasoning: combining the text in an image with outside knowledge.
The OCRBench v2 leaderboard (and what the numbers hide)
To see why public OCR scores can mislead production teams, look at the official OCRBench v2 English private leaderboard, maintained by the benchmark authors. Instead of all eight sub-scores, the table below compares each model's overall average with the three capabilities that matter most for document pipelines: Relation Extraction, Element Parsing and Text Spotting.
| Rank | Model | Open Weights | Overall Avg | Relation Extraction | Element Parsing | Text Spotting |
|---|---|---|---|---|---|---|
| #1 | KDL Frontier (2026-10) | No | 76.9 | 94.4 | 45.7 | 78.4 |
| #2 | SEEU DOC | No | 76.2 | 93.8 | 42.4 | 80.6 |
| #3 | Yaochi-DTS-A2T-1.9 | No | 73.4 | 93.5 | 40.9 | 68.1 |
| #6 | Qwen3.6-27B | Yes (27B) | 68.3 | 92.8 | 26.4 | 74.3 |
| #8 | NVIDIA Nemotron 3.5 Super VL | Yes | 67.4 | 92.2 | 26.3 | 63.2 |
| #9 | NVIDIA Nemotron 3 Nano Omni | Yes (30B / A3B) | 65.8 | 93.3 | 26.3 | 54.2 |
| #13 | Gemini 3 Pro Preview | No | 63.4 | 93.9 | 27.1 | 80.8 |
| #19 | Gemini 2.5 Pro | No | 59.3 | 93.7 | 26.9 | 13.4 |
| #21 | Qwen3.5-9B | Yes (9B) | 58.7 | 91.1 | 18.7 | 41.8 |
| #25 | GPT-5 (2025-08-07) | No | 55.5 | 90.1 | 24.8 | 6.5 |
Source: official OCRBench v2 leaderboard, English private split, 2026.09 release. Ranked by the unweighted average of the eight capability scores.
Three takeaways every engineering team should notice
- Relation extraction looks easy; element parsing does not. 19 of the 26 models in the English private table score 90% or higher on Relation Extraction. On Element Parsing, where tables, charts and layouts must come back as structure, Gemini 3 Pro (27.1) and GPT-5 (24.8) fall about 65 points below their Relation Extraction scores.
- Text Spotting depends on output format. GPT-5 scores 90.1 on Relation Extraction but 6.5 on Text Spotting, which drops it to #25 overall. Spotting requires the model to return coordinates in a specific format, and the benchmark's prompts ask for coordinates normalized to a 0–1000 range. The maintainers report that rewriting the prompt to match a model's native coordinate format raised Qwen2.5-VL-7B's spotting score to 51.6 on public data. Part of the rank gap is about format, not about reading ability.
- The table has no cost, latency or hallucination data. Nemotron 3 Nano Omni (30B total, 3B active) scores 93.3 on Relation Extraction against 93.9 for Gemini 3 Pro. The leaderboard cannot tell you which one costs less per page, responds faster, or invents more fields on your invoices.
OCRBench v2 vs. OmniDocBench vs. olmOCR-Bench
Public OCR benchmarks measure fundamentally different tasks. Before trusting any benchmark score, check what it actually tests:
| Benchmark | Dataset Size | Primary Focus | Scoring Metric | Contains B2B Invoices? | Best For |
|---|---|---|---|---|---|
| OCRBench v1 | 1,000 QA pairs | General VLM text reading, scene text, VQA, KIE | Ground truth present in output (0–1,000) | No: receipts, forms and nutrition labels only | Quick zero-shot sanity checks |
| OCRBench v2 | 10,000 public + 1,500 private QA pairs | Bilingual (EN/ZH) single-image reading, localization and reasoning | Per-capability scores, macro-averaged (0–100) | No: single-image, no multi-page documents | Comparing reasoning, spotting and parsing |
| OmniDocBench | 1,651 PDF pages | Full-page layout, reading order, tables and formulas | Edit distance, TEDS, CDM | No | PDF-to-Markdown RAG ingestion |
| olmOCR-Bench | 1,403 PDFs (7,010 unit tests) | Pass/fail assertions on messy, old or multi-column PDFs | Binary pass rate (%) | No | Testing PDF linearization and header/footer stripping |
Notice the last column: none of these public benchmarks tests multi-page B2B invoices against a strict JSON schema.
How to run OCRBench locally
If you want to reproduce OCRBench v1 or v2 numbers against a local checkpoint, you don't need to write custom evaluation scripts. Both versions are available in lmms-eval under the task names ocrbench and ocrbench_v2:
python -m lmms_eval \
--model qwen2_5_vl \
--model_args pretrained="Qwen/Qwen2.5-VL-7B-Instruct" \
--tasks ocrbench,ocrbench_v2 \
--batch_size 1 \
--output_path ./logs/
For multi-GPU runs, wrap the same arguments in accelerate launch --num_processes=4 -m lmms_eval. Run python -m lmms_eval --tasks list first, since task names change between releases.
Plan for some setup friction. One user reported in lmms-eval issue #822 that ocrbench_v2 needed extra packages (ipdb, jieba, distance, editdistance, apted) and took more than seven hours on one machine. Check the install before starting a long run.
Public harnesses also cannot reproduce the private leaderboard, because the private images are never released. And while they are excellent for academic reproducibility, they need GPUs and dataset downloads, and they only test the benchmark's fixed prompts. They say nothing about your schema, your documents or your cost.
Four reasons public OCRBench scores fail in production
When teams move from public leaderboards to a real document extraction pipeline, four blind spots break their assumptions:
- No schema enforcement: Most public tasks check an answer or a format the prompt requests. They don't validate a nested JSON schema with typed fields, arrays of line items and no extra prose. OCRBench v2's relation-extraction tasks are scored as key-value F1 after parsing, not as schema validity.
- No normalization vs. hallucination tracking: If a document has no
purchase_order_number, a model that outputs"N/A"instead ofnullcreates a hallucinated value in your database. Conversely, penalizing"$1,250.00"against"1250.00"creates false negatives unless your evaluator separates formatting normalization from genuine extraction errors. Inclusion-based scoring has the opposite problem: it can reward a chatty answer that happens to contain the right string. - Domain and degradation blindness: Clean academic PDFs and street-view photos don't prepare models for rotated phone photos, faded dot-matrix receipts, overlapping stamp ink or 12-page freight bills where one table spans three page breaks.
- Missing unit economics (accuracy vs. cost vs. latency): Public leaderboards rank models by accuracy alone. Take two hypothetical models. Model A scores 96.2% at $14.00 per 1,000 pages with 7.4 s latency. Model B scores 95.8% at $0.90 per 1,000 pages with 1.1 s latency. A leaderboard puts A first. For many production pipelines, B is the better choice, and only a test on your own volume and budget will show which trade-off holds.
What the verdicts look like on one document
The same credit note, read by three models and graded field by field. The model outputs are scripted in OCR Bench's demo dataset to show each failure type, and they run through the same grading engine as a real evaluation.
| Field | Expected | Claude Sonnet 4.6 | GPT-4o mini | Mistral OCR 4.1 |
|---|---|---|---|---|
| Invoice number | CN-0042 | CN-0042Correct | CN-0042Correct | CN-0042Correct |
| PO reference | (empty) | PO-4004Hallucinated | (empty)Correct | (empty)Correct |
| Supplier | Umbrella Corp | Umbrella CorpCorrect | Umbrella CorpCorrect | Umbrella CorpCorrect |
| Issue date | 2024-06-30 | 2024-06-30Correct | 2024-06-30Correct | 2024-06-30Correct |
| Total | -250.00 | -250.00Correct | 250.00Wrong | (250.00)Wrong |
| VAT | (empty) | 41,67Hallucinated | (empty)Correct | (empty)Correct |
| Signed | true | (blank)Missing | unsureWrong | trueCorrect |
| Notes | Refund for damaged goods | Refund for damaged goodsCorrect | Refund for damaged goodsCorrect | Refund for damaged goods.Wrong |
| Correct | 5 / 8 | 6 / 8 | 6 / 8 |
Illustrative. The outputs are scripted to show each verdict and are not measured results from these models. Every verdict comes from the grading engine described above.
Claude Sonnet 4.6 invents a PO number and a VAT figure for a document that has neither, which is the hallucination that breaks a database. GPT-4o mini and Mistral OCR 4.1 never invent a value, but they get the sign on the total wrong. A single accuracy number would hide that difference.
Run your own private OCRBench on your documents
Public benchmarks tell you which model is best on their dataset. To know which model is best on your documents, you need to run the same controlled experiment on your own files.
That is why we built OCR Bench (ocrbench.app), a dedicated evaluation workbench for testing frontier and open-weights VLMs against your real documents and JSON schemas:
- Benchmark 20+ models in one go: Run models from OpenAI, Anthropic, Google, Qwen, Mistral, Meta, Amazon and others side by side on the same batch of PDFs or images, without writing glue code or managing API adapters.
- Deterministic field-level verdicts: Instead of fuzzy string matching, every extracted field is classified as Correct, Wrong, Missing or Hallucinated. Normalization rules for dates, numbers and whitespace are configurable per field and shown before each run.
- One scorecard for accuracy, cost and speed: Compare schema accuracy next to provider cost per 100 pages and average latency, so you can pick the model that is actually best for production.
- Publish a benchmark as a public page: Share a results page with a stable link so your team can see the comparison and the reasoning behind the model choice.
Run a free benchmark on your own documents at ocrbench.app →
FAQ
What is the difference between OCRBench and OCRBench v2?
OCRBench (v1) has 1,000 question-answer pairs across five tasks (text recognition, scene text VQA, document VQA, key information extraction and handwritten math) and is scored out of 1,000 by checking whether the ground-truth answer appears in the output. OCRBench v2 expands to 10,000 public pairs plus 1,500 private images, covering 8 capabilities made of 23 tasks across 31 scenarios, in English and Chinese, on a 0–100 scale.
Which model ranks #1 on the OCRBench v2 leaderboard?
On the official OCRBench v2 English private leaderboard (2026.09 release), KDL Frontier (2026-10) ranks #1 overall with an average of 76.9, followed by SEEU DOC (76.2) and Yaochi-DTS-A2T-1.9 (73.4). Among open-weights models, Qwen3.6-27B leads at 68.3.
Does OCRBench include invoices and purchase orders?
Not multi-page B2B invoices. OCRBench v1 covers receipts (SROIE), forms (FUNSD) and nutrition labels (POIE). OCRBench v2 is single-image by design. It includes receipts and a document-classification task whose categories include invoices, but it does not test extraction from multi-page invoices or custom ERP schemas.
Why do some open models outrank GPT-5 on OCRBench v2?
Mostly because of Text Spotting, which requires boxes in a specific coordinate format. GPT-5 scores 90.1 on Relation Extraction but 6.5 on Text Spotting. The benchmark's prompts ask for coordinates normalized to 0–1000, and the maintainers report that matching a model's native coordinate format changes its score (Qwen2.5-VL-7B reached 51.6 on public data with an adapted prompt). Part of the gap reflects format rather than reading ability, so compare the sub-scores you need, not only the overall rank.
How can I benchmark OCR models on my own custom PDFs?
Upload a sample of your real PDFs or images with the correct value for each field to ocrbench.app. You can then compare 20+ vision-language models side by side on field-level accuracy, hallucination rate, latency and cost per 100 pages.