CostBasisBench: Evaluating frontier models on capital-gains tax preparation
CostBasisBench: Evaluating frontier models on capital-gains tax preparation
Introduction
When an accountant prepares a client’s annual income tax return, they need to calculate the gains or losses from any investments the client sold. This often requires reconciling transactions across brokerage statements and prior-year tax records, determining the correct cost basis, and applying tax rules that can change the reported gain or loss.
CostBasisBench evaluates frontier models on their ability to accurately calculate these gains and losses from real-world client documents.
CostBasisBench leaderboard
Claude Fable 5.1[max]88.5%
Kimi K3 US[max]88.0%
Gemini 3.8 Flash[high]88.0%
Muse Spark 1.3[xhigh]86.7%
Grok 4.6[xhigh]82.0%
GPT-5.6 Sol[max]81.0%
Overview
We built CostBasisBench from 20 real-world tax binders provided by a partner accounting firm. Each binder was anonymized and contained a real client’s prior-year tax return, current-year supporting documents, and professionally prepared current-year return.
Each binder was used to create one task in the benchmark. For each task, six frontier models were given the prior-year return and current-year supporting documents and asked to calculate the client’s capital gains and losses. The models were given tools to search and inspect client documents, consult IRS references, execute Python in a network-isolated sandbox, and maintain a persistent notepad. The prepared current-year return was withheld from the models and used, alongside the supporting documents, to construct a golden dataset that each model's calculations were scored against.
Methodology
Agent environment
Every model received the same prompt and tools. The models were run with a standardized harness that allowed them to call tools iteratively and, when ready, submit their final answers in the required schema through submit_schedule_d.
Prompt
Shared task instructions are shown below. Each task also included its case key, tax year, and the complete ordered output-field contract for that year.
Independently reconstruct the complete current-year Form 8949 / Schedule D benchmark interface from a neutral two-document client Binder and official tax-year references.
The supplied Binder contains taxpayer evidence. The supplied Reference Pack contains official tax-year materials and is not taxpayer evidence.
Begin with binder_list_documents. Use binder_document_overview in its 50-page cursor windows to understand the PYR and CYS structure. Treat deterministic page labels as fallible navigation metadata attached to stable page IDs, never as evidence or conclusions. Use labels to identify likely forms, statements, and better OCR search terms. Search OCR, read the complete relied-on page, then open its matching image when the fact can affect an output node. Cite stable page IDs, never label text. A fallback label, misleading label, or zero-match search never proves that evidence is absent. Apply the same workflow to Reference Pack labels, while keeping references distinct from taxpayer evidence.
OCR is provided for search and navigation and may contain transcription, layout, sign, checkbox, or column-assignment errors. Page images are the authoritative source. Verify consequential amounts, dates, tax-form lines, checkboxes, signs, codes, and table-column assignments against the corresponding page image before submitting.
Use reference_* tools for official instructions and forms. execute_code provides a persistent, network-disabled Python scratch workspace with no document access. update_notepad preserves working notes.
Determine the output from the supplied Binder and Reference Pack.
Every tax-year roster node must appear exactly once and in the exact order below. Use resolution computed with the declared value type and one of the exact literal enumValues when present. Follow signRule and blankZeroRule. Integer zero, computed blank, not_reached, not_required, and insufficient_information are distinct.
Cite Binder pageIds for taxpayer pages actually relied on and reference pageIds for official material actually relied on. Citation presence never changes whether the terminal schema is accepted.
Use insufficient_information when the supplied taxpayer evidence cannot determine a node. Reference one or more neutral openItemIds from the node, and define each Open Item once in the top-level openItems array with a generic reasonClass, a concise missingFact, and requestedInformation. Reuse an ID only for the same unresolved matter.
submit_schedule_d must be called alone when the complete ledger is ready.
Tools
| Tool | Description |
|---|---|
binder_list_documents | Start here. List the two neutral taxpayer-evidence documents, PYR-S001 and CYS-S001, with page counts. These stable IDs are valid only with binder_* tools. No held-out CYR document is available. |
binder_document_overview | Inspect at most 50 ordered pages of PYR-S001 or CYS-S001 from a zero-based cursor. Continue with next_cursor until null. Results carry stable (PYR|CYS)-S001-P#### IDs and fallible deterministic or fallback labels. Labels help locate likely forms or better search terms; they are never evidence or conclusions. |
binder_search_document_text | Search taxpayer-document OCR using distinct exact-literal, regex, or BM25 relevance semantics. Returns at most 20 page snippets with stable page IDs. OCR and snippets are navigation aids and can contain sign, checkbox, layout, or column errors; an empty result does not prove absence. |
binder_read_document_text | Read the complete OCR for one to ten selected one-based pages of one taxpayer document, optionally with Reducto layout blocks. Read the full relied-on page before use. OCR is a navigation aid; verify every consequential fact against the matching authoritative JPEG. |
binder_view_document_pages | Open one to five authoritative taxpayer-document JPEG page images by one-based page number. The result includes matching stable page IDs. Use these images to verify consequential amounts, dates, form lines, checkboxes, signs, codes, and table-column assignments. |
reference_list_sources | List official tax-year Reference Pack sources, with stable REF-S### IDs and page counts. References explain forms and law but never establish taxpayer facts. |
reference_source_overview | Inspect at most 50 ordered pages of one REF-S### source from a zero-based cursor. Continue with next_cursor until null. Stable REF-S###-P#### IDs and fallible labels are navigation hints, not authority or conclusions. |
reference_search_text | Search one official source's OCR with distinct exact-literal, regex, or any-keyword semantics. Returns at most 20 labeled page snippets. OCR is a navigation aid, images are authoritative, and zero matches do not prove a rule or form instruction is absent. |
reference_read_source_text | Read complete OCR for one to ten selected one-based pages of an official REF-S### source. Verify consequential form mechanics, signs, checkboxes, and table columns against the matching authoritative image. |
reference_view_source_pages | Open one to five authoritative official-reference JPEGs by one-based page number. Cite their stable REF-S###-P#### IDs only as reference evidence, never taxpayer evidence. |
execute_code | Run Python in a persistent, network-disabled scratch workspace for calculations only. It has no Binder, Reference Pack, database, or Storage access. |
update_notepad | Update the revision-fenced run-local working ledger, capped at 60,000 characters. Keep observed evidence, reference rules, calculations, inferences, and unresolved work distinct. |
submit_schedule_d | Terminal tool. Call alone with exactly one record for every node in the complete tax-year roster, preserving order, nulls, value types, optional evidence arrays, generic Open Items, and the computed versus insufficient-information variants. |
Submission schema
submit_schedule_d accepted one object containing the case key, tax year, an ordered list of output fields, and any unresolved information requests. A valid submission ended the task.
Each field contained either a computed value or an insufficient-information decision linked to an unresolved item. The tax-year contract specified the required fields, value types, and allowed values; validation checked these in addition to the JSON schema below.
{
"$defs": {
"ComputedNode": {
"additionalProperties": false,
"properties": {
"nodeId": {
"minLength": 1,
"title": "Nodeid",
"type": "string"
},
"resolution": {
"const": "computed",
"title": "Resolution",
"type": "string"
},
"value": {
"anyOf": [
{
"type": "integer"
},
{
"type": "boolean"
},
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Value"
},
"evidence": {
"items": {
"$ref": "#/$defs/EvidenceReference"
},
"title": "Evidence",
"type": "array"
},
"referenceEvidence": {
"items": {
"$ref": "#/$defs/ReferenceEvidence"
},
"title": "Referenceevidence",
"type": "array"
}
},
"required": [
"nodeId",
"resolution",
"value"
],
"title": "ComputedNode",
"type": "object"
},
"EvidenceReference": {
"additionalProperties": false,
"properties": {
"pageId": {
"pattern": "^(PYR|CYS)-S001-P\\d{4,}$",
"title": "Pageid",
"type": "string"
}
},
"required": [
"pageId"
],
"title": "EvidenceReference",
"type": "object"
},
"OpenNode": {
"additionalProperties": false,
"properties": {
"nodeId": {
"minLength": 1,
"title": "Nodeid",
"type": "string"
},
"resolution": {
"const": "insufficient_information",
"title": "Resolution",
"type": "string"
},
"openItemIds": {
"items": {
"pattern": "^[A-Za-z][A-Za-z0-9:_.-]{1,159}$",
"type": "string"
},
"minItems": 1,
"title": "Openitemids",
"type": "array"
},
"evidence": {
"items": {
"$ref": "#/$defs/EvidenceReference"
},
"title": "Evidence",
"type": "array"
},
"referenceEvidence": {
"items": {
"$ref": "#/$defs/ReferenceEvidence"
},
"title": "Referenceevidence",
"type": "array"
}
},
"required": [
"nodeId",
"resolution",
"openItemIds"
],
"title": "OpenNode",
"type": "object"
},
"ReferenceEvidence": {
"additionalProperties": false,
"properties": {
"pageId": {
"pattern": "^REF-S\\d{3}-P\\d{4,}$",
"title": "Pageid",
"type": "string"
}
},
"required": [
"pageId"
],
"title": "ReferenceEvidence",
"type": "object"
},
"SubmissionOpenItem": {
"additionalProperties": false,
"properties": {
"openItemId": {
"pattern": "^[A-Za-z][A-Za-z0-9:_.-]{1,159}$",
"title": "Openitemid",
"type": "string"
},
"reasonClass": {
"enum": [
"missing_fact",
"conflicting_evidence",
"unreadable_evidence",
"other"
],
"title": "Reasonclass",
"type": "string"
},
"missingFact": {
"maxLength": 500,
"minLength": 1,
"title": "Missingfact",
"type": "string"
},
"requestedInformation": {
"maxLength": 500,
"minLength": 1,
"title": "Requestedinformation",
"type": "string"
}
},
"required": [
"openItemId",
"reasonClass",
"missingFact",
"requestedInformation"
],
"title": "SubmissionOpenItem",
"type": "object"
}
},
"additionalProperties": false,
"properties": {
"caseKey": {
"pattern": "^[A-Za-z][A-Za-z0-9:_.-]{1,159}$",
"title": "Casekey",
"type": "string"
},
"taxYear": {
"enum": [
2024,
2025
],
"title": "Taxyear",
"type": "integer"
},
"nodes": {
"items": {
"discriminator": {
"mapping": {
"computed": "#/$defs/ComputedNode",
"insufficient_information": "#/$defs/OpenNode"
},
"propertyName": "resolution"
},
"oneOf": [
{
"$ref": "#/$defs/ComputedNode"
},
{
"$ref": "#/$defs/OpenNode"
}
]
},
"minItems": 1,
"title": "Nodes",
"type": "array"
},
"openItems": {
"items": {
"$ref": "#/$defs/SubmissionOpenItem"
},
"title": "Openitems",
"type": "array"
}
},
"required": [
"caseKey",
"taxYear",
"nodes",
"openItems"
],
"title": "ScheduleDSubmission",
"type": "object"
}Model Configuration
All six models ran with the highest reasoning effort and context window supported by their provider.
| Model | Reasoning | Context window | Output limit |
|---|---|---|---|
| Claude Fable 5.1 | max | 1,000,000 | 128,000 |
| Gemini 3.8 Flash | high | 1,048,576 | 65,536 |
| GPT-5.6 Sol | max | 1,050,000 | 128,000 |
| Grok 4.6 | xhigh | 500,000 | 32,768 |
| Kimi K3 US | max | 1,048,576 | 131,072 |
| Muse Spark 1.3 | xhigh | 1,048,576 | 32,768 |
Results
To compare the models on their ability to accurately calculate capital gains and losses, we measured field accuracy and fully correct tasks. A field is one required calculation, such as sale proceeds, cost basis, or a capital-loss carryover. Field accuracy is the percentage of calculations returned by the model that matched the ground truth. A task represents one client’s capital-gains tax preparation, which is a group of required calculations. A task counts as fully correct only when every output matches the ground truth.
Claude Fable 5.1 achieved the highest field accuracy at 88.5%, followed by Kimi K3 US at 88.0%. Claude Fable 5.1 was the only model to produce fully correct outputs, on 2 of 20 client tasks.
GPT-5.6 Sol had the shortest median execution time at 8.8 minutes per task. Kimi K3 US used the fewest tokens, averaging 4.03 million per task. Token usage includes recorded input and output across all model calls.
| Claude Fable 5.1 | 88.5% | 2/20 | 19.3 min | 7.52M |
|---|---|---|---|---|
| Kimi K3 US | 88.0% | 0/20 | 11.2 min | 4.03M |
| Gemini 3.8 Flash | 88.0% | 0/20 | 10.1 min | 10.24M |
| Muse Spark 1.3 | 86.7% | 0/20 | 66.2 min | 16.75M |
| Grok 4.6 | 82.0% | 0/20 | 22.8 min | 6.48M |
| GPT-5.6 Sol | 81.0% | 0/20 | 8.8 min | 6.84M |
Median execution time
- GPT-5.6 Sol[max]8.8 min
- Gemini 3.8 Flash[high]10.1 min
- Kimi K3 US[max]11.2 min
- Claude Fable 5.1[max]19.3 min
- Grok 4.6[xhigh]22.8 min
- Muse Spark 1.3[xhigh]66.2 min
Average tokens per task
- Kimi K3 US[max]
- Grok 4.6[xhigh]
- GPT-5.6 Sol[max]
- Claude Fable 5.1[max]
- Gemini 3.8 Flash[high]
- Muse Spark 1.3[xhigh]
Limitations
The benchmark covered 20 engagements from one accounting firm. Future benchmarks should include a larger set of returns from various firms.
Acknowledgements
We thank our partner accounting firm for providing the real-client documents and professionally prepared returns that made this benchmark possible.
Get in touch
Run your agent against the benchmark, get expert data, chat with the team