TaxRetrievalBench: Evaluating search APIs on US tax research

TaxRetrievalBench: Evaluating search APIs on US tax research

Introduction

TaxRetrievalBench evaluates search APIs on their ability to retrieve the authoritative sources and supporting passages needed to answer complex tax research questions.

TaxRetrievalBench leaderboard

  1. Exa Search · deep-reasoning: 74.9%
  2. Perplexity Search · default: 41.6%
  3. Parallel Search · advanced: 29.1%
  4. Tavily Search · advanced: 26.1%
  1. ExaSearch · deep-reasoning74.9%
  2. PerplexitySearch · default41.6%
  3. ParallelSearch · advanced29.1%
  4. TavilySearch · advanced26.1%

Overview

We evaluated the Search APIs from Exa (deep-reasoning), Parallel (advanced), Tavily (advanced), and Perplexity (default) on their ability to retrieve authoritative sources and relevant passages when answering tax research questions.

TaxRetrievalBench was built from 32 tax research questions curated by a partner accounting firm. Each question was based on a realistic client scenario and paired with a rubric specifying the authoritative sources and tax rules needed to answer it.

Search API endpoints used in the evaluation
ProviderEndpoint
Exahttps://api.exa.ai/search
Parallelhttps://api.parallel.ai/v1/search
Tavilyhttps://api.tavily.com/search
Perplexityhttps://api.perplexity.ai/search

Methodology

We used each provider’s highest available search tier. We submitted every tax research question to each provider three times. All requests used the same source allowlist and requested ten results.

Search modes and requested results per provider
ProviderSearch modeResults requested
Exadeep-reasoning10
Paralleladvanced10
Perplexitydefault10
Tavilyadvanced10
Example request and rubric
Request
Find US federal tax authority applicable in 2025 governing the amount of gain deferred and the basis of replacement shares when an individual reinvests only part of the proceeds from a qualifying small-business-stock sale.

Jurisdiction: US federal

Tax years: 2025

Research cutoff: 2026-09-10

Facts:
- An individual sells qualifying original-issue shares acquired in 2022 during August 2025 for $700,000; adjusted basis is $100,000.
- Within 60 days, the individual purchases qualifying replacement shares for $650,000 and makes a timely rollover election.

Assumptions:
- The seller and both share issuances satisfy all other rollover conditions; replacement cost has not supported another rollover; no selling expenses or other basis adjustments exist.
Grading rubric
gold:
  requirements:
    - id: R1
      proposition:
        Recognition turns on amount realized exceeding unused qualifying replacement cost, not on gain
        exceeding the reinvestment.
      minimum_evidence:
        The rule recognizing gain to the extent amount realized exceeds qualifying replacement cost. Timely
        purchase, eligibility, and unused replacement cost are stipulated; their separate qualification clauses
        need not be highlighted.
      sufficient_passage_sets:
        - - P1
      accepted_alternatives:
        Equivalent enacted statutory or applicable final-regulation passages are acceptable if
        they establish every stated premise and qualification. Matching this example URL or exact excerpt boundary
        is not required.
      non_credit:
        - A general claim that reinvesting the gain defers all tax.
    - id: R2
      proposition: Unrecognized gain reduces the basis of the qualifying replacement stock.
      minimum_evidence: The basis-reduction rule connecting the deferred gain to the replacement stock’s cost basis.
      sufficient_passage_sets:
        - - P2
      accepted_alternatives:
        Equivalent enacted statutory or applicable final-regulation passages are acceptable if
        they establish every stated premise and qualification. Matching this example URL or exact excerpt boundary
        is not required.
      non_credit:
        - Treating the full purchase price as unchanged basis after a rollover.
  authorities:
    - id: A1
      citation: 26 U.S.C. §1045, (a), (b)(2)–(3)
      url: https://uscode.house.gov/view.xhtml?req=granuleid:USC-prelim-title26-section1045&num=0&edition=prelim
      alternative_urls: []
      authority_type: Federal statute
      applicability: Qualifying 2025 sale with partial proceeds reinvestment and a valid election.
      identity:
        kind: usc
        identifier: 26 USC 1045
        version:
          Enacted Code text applicable to the stated tax year through 2026-09-10; later codifications acceptable
          only if the targeted provisions are substantively unchanged.
        target_provision: (a), (b)(2)–(3)
      passages:
        - id: P1
          quote: |-
            In the case of any sale of qualified small business stock held by a taxpayer other than a corporation for more than 6 months and with respect to which such taxpayer elects the application of this section, gain from such sale shall be recognized only to the extent that the amount realized on such sale exceeds-
            (1) the cost of any qualified small business stock purchased by the taxpayer during the 60-day period beginning on the date of such sale, reduced by
            (2) any portion of such cost previously taken into account under this section.
            This section shall not apply to any gain which is treated as ordinary income for purposes of this title.
          locator: "§1045(a); saved text: ../sources/TRB-001/S1.txt, line 54"
        - id: P2
          quote: |-
            (2) Purchase
            A taxpayer shall be treated as having purchased any property if, but for paragraph (3), the unadjusted basis of such property in the hands of the taxpayer would be its cost (within the meaning of section 1012).
            (3) Basis adjustments
            If gain from any sale is not recognized by reason of subsection (a), such gain shall be applied to reduce (in the order acquired) the basis for determining gain or loss of any qualified small business stock which is purchased by the taxpayer during the 60-day period described in subsection (a).
          locator: "§1045(b)(2)–(3); saved text: ../sources/TRB-001/S1.txt, line 62"
  notes:
    "Legal key: Recognized gain is $50,000, deferred gain is $550,000, and replacement basis is $100,000. Evidence\
    \ must establish proceeds-based recognition and reduction of replacement basis by deferred gain, not reproduce\
    \ these computations.\nResearch difficulty: Reinvesting more than the economic gain can still leave recognized\
    \ gain; the deferred portion also alters the next investment’s basis. Intended difficulty only; no Exa/Gemini\
    \ experiment performed. Each authority is registered once per question; sufficient sets are examples, not exclusive\
    \ retrieval paths. "
Source allowlist
Allowed source domains
uscode.house.gov
ecfr.gov
federalregister.gov
govinfo.gov
law.cornell.edu
ustaxcourt.gov
supremecourt.gov
uscourts.gov
courtlistener.com
storage.courtlistener.com
irs.gov

Scoring

Before the eval, CPAs at a partner accounting firm curated a rubric for each tax research question. Each rubric contained two types of criteria: authority coverage, which checks whether the response retrieved the appropriate authoritative sources, and evidence coverage, which checks whether the returned passages establish the tax rules needed to answer the question. An LLM judge graded each response against its rubric, awarding one point for each criterion satisfied.

Results

We measured the percentage of rubric criteria each provider met across all three runs, with authority coverage and evidence coverage reported separately. Across both authority and evidence criteria, Exa scored highest at 74.9%, followed by Perplexity at 41.6%, Parallel at 29.1%, and Tavily at 26.1%.

TaxRetrievalBench provider results
ProviderSupported criteriaAuthority coverageEvidence coverageMean latencySearch cost
Exa74.9%83.5%67.4%13.69 s$1.43
Perplexity41.6%52.3%35.4%0.84 s$0.48
Parallel29.1%47.2%15.6%2.11 s$0.48
Tavily26.1%41.9%13.5%2.33 s$1.54

Limitations

Future editions should repeat the benchmark on a larger and more varied set of tax research questions.

Acknowledgements

We thank the partner accounting firm that curated the tax research questions and rubrics.

Get in touch

Run your agent against the benchmark, get expert data, chat with the team