K1Bench: Evaluating document parsing models on real-world K-1s
Introduction
A K-1 is a tax form that businesses issue to owners and investors to report their share of income, losses, deductions, and credits. The IRS processes over 40 million K-1s each year.
K1Bench evaluates document parsing models on their ability to accurately parse real-world K-1s.
K1Bench leaderboard
- Reducto r-1: 85.2%
- Extend Parse Performance 2.0: 76.0%
- Datalab Convert (Accurate mode): 69.9%
- LlamaIndex LlamaParse (Agentic Plus): 69.4%
- Mistral OCR 4.1: 61.7%
- Microsoft Azure Document Intelligence Layout: 39.8%
- Reductor-185.2%
- ExtendParse Performance 2.076.0%
- DatalabConvert (Accurate mode)69.9%
- LlamaIndexLlamaParse (Agentic Plus)69.4%
- MistralOCR 4.161.7%
- Microsoft AzureDocument Intelligence Layout39.8%
Overview
We evaluated Reducto’s r-1, Extend’s Parse Performance 2.0, Datalab’s Convert (Accurate mode), Mistral’s OCR 4.1, LlamaIndex’s LlamaParse (Agentic Plus), and Microsoft Azure’s Document Intelligence Layout on their ability to parse 20 real-world K-1s.
The documents were provided by a partner accounting firm and anonymized before parsing.
View example pages
All values are redacted here for privacy.
Methodology
Every provider parsed the same 20 anonymized K-1 packets. The length of each K-1 packet ranged from 2 pages to 40 pages, and totaled 196 pages across all packets.
We submitted each packet as a complete PDF to each provider’s parsing API. We selected each provider’s highest-quality parsing mode.
Results were adjudicated by human graders. The leaderboard measures the percentage of source pages parsed by the provider without a material error. Material errors affect the meaning of the extracted information, such as an incorrect amount, missing content, or a value associated with the wrong label. Harmless formatting differences were not adjudicated as material errors.
Results
Reducto leads K1Bench with 85.2% of pages parsed without material errors, followed by Extend at 76.0% and Datalab at 69.9%.
The rankings shift by error type. A material omission occurs when a parser leaves out information needed to correctly interpret the document, such as an amount, label, or selected checkbox. Extend has the fewest pages with material omissions at 1.5%, followed by LlamaIndex at 3.1% and Reducto at 3.6%.
Incorrect information occurs when a parser changes or invents content, such as misreading an amount or adding a value absent from the source. Reducto has the fewest pages with incorrect information at 3.6%, followed by Datalab at 9.2% and Mistral at 14.3%.
Material omissions
- Extend: 1.5% (3 of 196 pages)
- LlamaIndex: 3.1% (6 of 196 pages)
- Reducto: 3.6% (7 of 196 pages)
- Datalab: 5.1% (10 of 196 pages)
- Mistral: 20.9% (41 of 196 pages)
- Microsoft Azure: 23.5% (46 of 196 pages)
Incorrect information
- Reducto: 3.6% (7 of 196 pages)
- Datalab: 9.2% (18 of 196 pages)
- Mistral: 14.3% (28 of 196 pages)
- Extend: 15.8% (31 of 196 pages)
- LlamaIndex: 15.8% (31 of 196 pages)
- Microsoft Azure: 35.2% (69 of 196 pages)
Qualitative Analysis
To show where document parsing models still fall short, we’ve spotlighted material errors from each provider below.
Reducto r-1
Self-employment earnings (loss) A
The source lists codes A and C; Reducto omits the C row. Amounts are hidden on both sides.
| (d) Total (add columns (a) through (c)) |
|---|
| 12 |
The source amount field is blank; Reducto places the printed line number 12 in the Total column.
Extend Parse Performance 2.0
| (h) Tested loss | |
|---|---|
| A | ☐ |
| B | ☐ |
| C | ☐ |
The source has blank fields inside parentheses; Extend turns them into unchecked boxes.
Partner's share of: 1065, Line 22 [ ] PTP [ ] Aggregated [ ] SSTB 1065, Line 22 [ ] PTP [ ] Aggregated [ ] SSTB 1065, Line 22 [ ] PTP [ ] Aggregated [ ] SSTB
The source names a form and line in one column; Extend repeats it across all three.
Datalab Convert (Accurate mode)
| Yes | No | |
|---|---|---|
| 3 Does Part III apply? If "Yes," complete and attach Part III . . . . . | 3 | ✓ |
The source checks Yes; Datalab places the check under No.
| Check Applicable Box Estate Resident Nonresident Trust Grantor Trust |
The source has two boxes per row; Datalab adds a third and loses the link between Estate and Resident.
Mistral OCR 4.1
| G | General partner or LLC member-manager | Limited partner or other LLC member |
| H1 | Domestic partner | Foreign partner |
The source selects Limited partner and Domestic partner; Mistral drops both selections.
| Letter | Amount | |
|---|---|---|
| 24a | ☐ | |
| 24b | ☐ | |
| 24c | ☐ |
The source has empty Letter cells; Mistral inserts unchecked boxes.
LlamaIndex LlamaParse (Agentic Plus)
PTP: X
The source marks SSTB; LlamaIndex leaves SSTB blank and assigns the X to PTP.
| 13 | Other deductions | A [hidden], L [hidden] |
The source includes a * STMT row; LlamaIndex omits it. Amounts are hidden on both sides.
Microsoft Azure Document Intelligence Layout
$\frac { - 1 } { 6 }$ NO HANDWRITTEN ENTRIES ON FORMThe source contains a plain-text instruction; Azure inserts a fraction before it.
(e) Other (category code 1
The source leaves the category code blank; Azure inserts a 1.
Limitations
The sample was limited to 20 K-1s. Future editions should repeat the benchmark on a larger set.
Acknowledgements
We thank the partner accounting firm that supplied real client documents and helped define and adjudicate the benchmark.
Get in touch
Run your agent against the benchmark, get expert data, chat with the team