OCR benchmarks compared: before and after vision-language models

For twenty years, OCR benchmarks tested one small job at a time: find the text, read the word, spot the table, fill in the field. Then vision-language models (VLMs), AI models that read images the way chatbots read text, started handling whole pages in one go. The old benchmarks weren't built for that, and the best models quickly maxed most of them out.

This guide explains what changed. It covers the classic benchmarks (IAM, ICDAR, FUNSD, SROIE, DocVQA), the ones that matter for today's AI models (OCRBench v2, CC-OCR, OmniDocBench, olmOCR-Bench, ExtractBench), and what a high score really tells you. A comparison table of the main OCR benchmarks closes the article.

The short version: old benchmarks graded one step of the OCR process at a time. New ones grade the whole page, and had to find fairer ways to score models that phrase things their own way and sometimes make things up. Neither kind tells you how a model will do on your documents.

What is an OCR benchmark?

An OCR benchmark is a set of document images with the correct answers written down, plus a rule for scoring. You run a model on the images, compare its output with the answers, and get a score you can put next to other models' scores.

That comparison only holds if everyone uses the same documents and the same scoring rule. Change either one and the numbers stop meaning the same thing, which is why benchmarks had to be rebuilt when the models changed.

Before VLMs: one benchmark per step (1999–2022)

Traditional OCR worked like an assembly line: find the text on the page, read it, work out the layout, then pull out the information you need. Each research benchmark tested one station on that line.

• Reading handwriting: the IAM Handwriting Database (1999) collects 1,539 handwritten pages from 657 writers. Models are scored on how many characters they get wrong.
• Reading text in photos: the ICDAR Robust Reading competitions (2013 and 2015) used street scenes and Google Glass snapshots, where text is small, blurred or tilted.
• Understanding layout and tables: DocLayNet (IBM, 2022) checks whether a model can find the titles, paragraphs, tables and pictures on 80,863 hand-labelled pages. PubTabNet (IBM, 2019) does the same for the inside of tables, with 568,000 examples.
• Pulling out fields: FUNSD (199 scanned forms) and SROIE (1,000 receipts, four fields: company, date, address, total) are the closest to real business use. They are also tiny.
• Answering questions: DocVQA (2020) asks 50,000 questions about 12,767 document images, such as "what is the date on this letter?".

Each benchmark had its own score: characters wrong for handwriting, regions found for layout, fields matched for receipts. That worked because each model did one thing and answered in one fixed format.

Why VLMs broke the classic benchmarks

From 2023, general-purpose models such as GPT-4V, Gemini, Claude and Qwen-VL learned to read well enough to replace the whole assembly line. Give them a page and they hand back clean text, a table or JSON. That broke the old benchmarks in four ways:

1. They do every step at once. There's no separate "find the text" output to grade, only the finished result.
2. They phrase things their own way. A date can come back as 2024-03-01 or 1 March 2024, a table as Markdown or HTML. Strict scoring counts these as mistakes when they aren't. Here's how OCR Bench handles it.
3. They can make things up. Old engines misread; AI models can also invent a plausible total or a line that isn't on the page. Old scores don't tell those two failures apart.
4. They've maxed out the tests. On DocVQA, Qwen2.5-VL-72B scores 96.4, against 98.1 for humans (Qwen2.5-VL report). Many of these datasets are also used to train the models, so a high score can partly reflect memory.

The VLM era: the benchmarks that matter now (2023–2026)

Testing reading skills: OCRBench v2 and CC-OCR

OCRBench (2023) was the first benchmark built for AI models that read: 1,000 questions covering printed text, documents, receipts and handwritten math. Models soon outgrew it, and OCRBench v2 (late 2024) raised the bar to 10,000 questions across 23 tasks in English and Chinese. At launch, 36 of the 38 models tested scored under 50 out of 100. A private test set and a leaderboard updated every quarter make it harder to train on the answers.

CC-OCR (Alibaba, 2024) splits reading into four tracks: text in everyday scenes, 10 languages, full documents and key fields, over 7,058 images. It's useful when you need to know where a model is weak, not just its average.

Turning pages into text: OmniDocBench and olmOCR-Bench

These are the two scores you'll see in almost every new OCR model announcement.

OmniDocBench (2024) gives a model a full PDF page, asks for the text back with its tables and formulas, and checks each part separately. It has grown from 981 pages to 1,651 in its 2026 version, and top scores climbed fast: from about 91 to 94.5 out of 100 between September 2025 and January 2026. The newest version deliberately adds harder pages.

olmOCR-Bench (Allen Institute for AI, 2025) takes a different approach. Instead of comparing text, it runs about 7,000 simple yes-or-no checks on about 1,400 pages: is this sentence there? Is the page number left out? Does this paragraph come before that one? That makes it fair to models that format things differently, and it catches text that shouldn't be there. Top scores went from about 77% at launch to the mid-80s in 2026, but old scanned documents still trip models up, with scores of roughly 30–50% on that category.

Pulling out the data you need: ExtractBench

For most businesses, the goal isn't a clean copy of the page. It's getting the invoice total, the due date and the line items into a database. ExtractBench (LlamaIndex, 2026) tests exactly that, on 370 business documents (4,869 pages) across 67 document types, each with a list of fields to extract. It checks whether the values are right, whether long lists come back complete, whether each value can be traced back to the page, and what it all costs. Its main finding: AI models do well on short documents but often cut long lists short. LlamaIndex's own product ranks first, so read it as you would any vendor benchmark.

A separate benchmark with the same name, ExtractBench by Contextual AI (2026), pairs 35 PDFs with 12,867 fields and scores a missing value differently from an invented one. On its largest form, a 369-field financial report, none of the leading models tested produced a valid result.

Other benchmarks you'll come across

You'll also see Fox and READoc (page to text), MMLongBench-Doc (long PDFs), KITAB-Bench (Arabic), the IDP Leaderboard and the Omni OCR Benchmark (data extraction), and 2026 newcomers such as ParseBench and MDPBench. Several are run by companies that also sell OCR tools. They're still useful, but worth reading with that in mind.

What public benchmarks can't tell you

A leaderboard answers "how well does this model read these documents?" Your real question is how well it reads yours, at your volume and your budget.

• Your documents aren't in there. Public benchmarks lean on research papers, receipts and forms, mostly in English and Chinese. French payslips or handwritten delivery notes are rare.
• Page accuracy isn't field accuracy. A model can copy a page almost perfectly and still put the wrong number in the total. We show why in CER vs field-level accuracy.
• Cost and speed are missing. A two-point lead can cost ten times more per page. Our cost calculator puts numbers on it.

Use public benchmarks to build a shortlist, then test on your own documents. Our guide to choosing an OCR model walks through it, and OCR Bench does it for you: upload your documents, set the correct values, and see every model's accuracy, cost and speed side by side.

FAQ

What is the best OCR benchmark in 2026?

It depends on what you need. For turning pages into text, OmniDocBench and olmOCR-Bench are the most cited. For general reading skills, look at OCRBench v2 and CC-OCR. For extracting fields into JSON, look at ExtractBench. None of them replaces a test on your own documents.

What is the difference between OCRBench and OmniDocBench?

OCRBench asks a model short, targeted questions (read this word, find this field) across many kinds of images. OmniDocBench hands it a whole page and grades the full text it returns. The first measures reading skills; the second measures how well a model converts documents.

How do I benchmark OCR models on my own documents?

Pick 20 to 100 real documents, including the messy ones, and write down the correct value of every field. Run each model on the same set with the same instructions, then compare accuracy per field, cost per document and speed. OCR Bench does these steps for you.

OCR benchmark comparison table

The main OCR benchmarks of each era, in release order. Sizes come from each benchmark's paper or official page, checked in October 2026.