TaxRetrievalBench: Evaluating search APIs on US tax research
TaxRetrievalBench: Evaluating search APIs on US tax research
Introduction
TaxRetrievalBench evaluates search APIs on their ability to retrieve the authoritative sources and supporting passages needed to answer complex tax research questions.
TaxRetrievalBench leaderboard
- Exa Search · deep-reasoning: 74.9%
- Perplexity Search · default: 41.6%
- Parallel Search · advanced: 29.1%
- Tavily Search · advanced: 26.1%
- ExaSearch · deep-reasoning74.9%
- PerplexitySearch · default41.6%
- ParallelSearch · advanced29.1%
- TavilySearch · advanced26.1%
Overview
We evaluated the Search APIs from Exa (deep-reasoning), Parallel (advanced), Tavily (advanced), and Perplexity (default) on their ability to retrieve authoritative sources and relevant passages when answering tax research questions.
TaxRetrievalBench was built from 32 tax research questions curated by a partner accounting firm. Each question was based on a realistic client scenario and paired with a rubric specifying the authoritative sources and tax rules needed to answer it.
| Provider | Endpoint |
|---|---|
| Exa | https://api.exa.ai/search |
| Parallel | https://api.parallel.ai/v1/search |
| Tavily | https://api.tavily.com/search |
| Perplexity | https://api.perplexity.ai/search |
Methodology
We used each provider’s highest available search tier. We submitted every tax research question to each provider three times. All requests used the same source allowlist and requested ten results.
| Provider | Search mode | Results requested |
|---|---|---|
| Exa | deep-reasoning | 10 |
| Parallel | advanced | 10 |
| Perplexity | default | 10 |
| Tavily | advanced | 10 |
Example request and rubric
Find US federal tax authority applicable in 2025 governing the amount of gain deferred and the basis of replacement shares when an individual reinvests only part of the proceeds from a qualifying small-business-stock sale.
Jurisdiction: US federal
Tax years: 2025
Research cutoff: 2026-09-10
Facts:
- An individual sells qualifying original-issue shares acquired in 2022 during August 2025 for $700,000; adjusted basis is $100,000.
- Within 60 days, the individual purchases qualifying replacement shares for $650,000 and makes a timely rollover election.
Assumptions:
- The seller and both share issuances satisfy all other rollover conditions; replacement cost has not supported another rollover; no selling expenses or other basis adjustments exist.gold:
requirements:
- id: R1
proposition:
Recognition turns on amount realized exceeding unused qualifying replacement cost, not on gain
exceeding the reinvestment.
minimum_evidence:
The rule recognizing gain to the extent amount realized exceeds qualifying replacement cost. Timely
purchase, eligibility, and unused replacement cost are stipulated; their separate qualification clauses
need not be highlighted.
sufficient_passage_sets:
- - P1
accepted_alternatives:
Equivalent enacted statutory or applicable final-regulation passages are acceptable if
they establish every stated premise and qualification. Matching this example URL or exact excerpt boundary
is not required.
non_credit:
- A general claim that reinvesting the gain defers all tax.
- id: R2
proposition: Unrecognized gain reduces the basis of the qualifying replacement stock.
minimum_evidence: The basis-reduction rule connecting the deferred gain to the replacement stock’s cost basis.
sufficient_passage_sets:
- - P2
accepted_alternatives:
Equivalent enacted statutory or applicable final-regulation passages are acceptable if
they establish every stated premise and qualification. Matching this example URL or exact excerpt boundary
is not required.
non_credit:
- Treating the full purchase price as unchanged basis after a rollover.
authorities:
- id: A1
citation: 26 U.S.C. §1045, (a), (b)(2)–(3)
url: https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title26-section1045&num=0&edition=prelim
alternative_urls: []
authority_type: Federal statute
applicability: Qualifying 2025 sale with partial proceeds reinvestment and a valid election.
identity:
kind: usc
identifier: 26 USC 1045
version:
Enacted Code text applicable to the stated tax year through 2026-09-10; later codifications acceptable
only if the targeted provisions are substantively unchanged.
target_provision: (a), (b)(2)–(3)
passages:
- id: P1
quote: |-
In the case of any sale of qualified small business stock held by a taxpayer other than a corporation for more than 6 months and with respect to which such taxpayer elects the application of this section, gain from such sale shall be recognized only to the extent that the amount realized on such sale exceeds-
(1) the cost of any qualified small business stock purchased by the taxpayer during the 60-day period beginning on the date of such sale, reduced by
(2) any portion of such cost previously taken into account under this section.
This section shall not apply to any gain which is treated as ordinary income for purposes of this title.
locator: "§1045(a); saved text: ../sources/TRB-001/S1.txt, line 54"
- id: P2
quote: |-
(2) Purchase
A taxpayer shall be treated as having purchased any property if, but for paragraph (3), the unadjusted basis of such property in the hands of the taxpayer would be its cost (within the meaning of section 1012).
(3) Basis adjustments
If gain from any sale is not recognized by reason of subsection (a), such gain shall be applied to reduce (in the order acquired) the basis for determining gain or loss of any qualified small business stock which is purchased by the taxpayer during the 60-day period described in subsection (a).
locator: "§1045(b)(2)–(3); saved text: ../sources/TRB-001/S1.txt, line 62"
notes:
"Legal key: Recognized gain is $50,000, deferred gain is $550,000, and replacement basis is $100,000. Evidence\
\ must establish proceeds-based recognition and reduction of replacement basis by deferred gain, not reproduce\
\ these computations.\nResearch difficulty: Reinvesting more than the economic gain can still leave recognized\
\ gain; the deferred portion also alters the next investment’s basis. Intended difficulty only; no Exa/Gemini\
\ experiment performed. Each authority is registered once per question; sufficient sets are examples, not exclusive\
\ retrieval paths. "
Source allowlist
uscode.house.gov
ecfr.gov
federalregister.gov
govinfo.gov
law.cornell.edu
ustaxcourt.gov
supremecourt.gov
uscourts.gov
courtlistener.com
storage.courtlistener.com
irs.govScoring
Before the eval, CPAs at a partner accounting firm curated a rubric for each tax research question. Each rubric contained two types of criteria: authority coverage, which checks whether the response retrieved the appropriate authoritative sources, and evidence coverage, which checks whether the returned passages establish the tax rules needed to answer the question. An LLM judge graded each response against its rubric, awarding one point for each criterion satisfied.
Results
We measured the percentage of rubric criteria each provider met across all three runs, with authority coverage and evidence coverage reported separately. Across both authority and evidence criteria, Exa scored highest at 74.9%, followed by Perplexity at 41.6%, Parallel at 29.1%, and Tavily at 26.1%.
| Provider | Supported criteria | Authority coverage | Evidence coverage | Mean latency | Search cost |
|---|---|---|---|---|---|
| 74.9% | 83.5% | 67.4% | 13.69 s | $1.43 | |
| 41.6% | 52.3% | 35.4% | 0.84 s | $0.48 | |
| 29.1% | 47.2% | 15.6% | 2.11 s | $0.48 | |
| 26.1% | 41.9% | 13.5% | 2.33 s | $1.54 |
Limitations
Future editions should repeat the benchmark on a larger and more varied set of tax research questions.
Acknowledgements
We thank the partner accounting firm that curated the tax research questions and rubrics.
Get in touch
Run your agent against the benchmark, get expert data, chat with the team