What is the AI equity research benchmark?
The AI equity research benchmark described in this note starts from a simple observation. Every quarter, more AI tools claim to read filings, summarize earnings calls, and generate investment theses, yet almost none publish evaluation resembling what an analyst actually does.
Generic LLM benchmarks test broad capability, not the high-stakes skill of reading a 10-K accurately, reconciling numbers across periods, and forming a defensible view. This note proposes a nine-task benchmark covering earnings analysis through hallucination resistance, separating base-model from agentic performance and pairing automated scoring with human judgment on materiality.
Measuring what matters: from document retrieval to defensible investment theses
1. Executive Summary An AI equity research benchmark must test the composite analyst workflow, not single-turn trivia, because being wrong here misallocates capital. FinanceBench found a leading model with retrieval incorrectly answered or refused 81% of questions. FinanceQA shows ~60% failure on professional tasks. Published evidence shows AI is useful for scoped extraction yet unreliable as an autonomous analyst on reasoning-heavy tasks. Brane commits to measuring its own system publicly on this scale.
2. Why an AI Equity Research Benchmark Is Needed Equity research is a discipline where being wrong has a price. A mis-extracted revenue figure or thesis built on a stale filing misallocates real capital. For India, the gap is wider: NSE and BSE filings, SEBI disclosures, and concall transcripts follow conventions materially different from SEC filings most benchmarks are built from. Without a shared scale, vendor claims cannot be compared.
3. What an AI System Should Be Able to Do: Nine Tasks Together Read and extract: earnings analysis, filings analysis, financial data extraction with period reconciliation. Research and compare: company research on model and risks, peer screening against stated criteria. Reason and defend: thesis generation from evidence, thesis challenge identifying invalidation, source accuracy and hallucination resistance throughout. An AI good at only the first two is a document-reading tool, not an associate.
4. What the Evidence Already Shows FinanceBench: 16 configs, 10,231 triplets, 40 companies — 81% incorrect or refused with retrieval; hallucinations in all models. FinanceQA: professional tasks incl. hand-spreading — ~60% failure; weakest on defensible assumptions. FinQA / ConvFinQA: numerical reasoning over tables — arithmetic backbone nontrivial before retrieval. Independent reviews observe the same ceiling: strong at extraction, sharply weaker on forecasting and multi-step reasoning.
5. Hallucination, Retrieval Depth, and Citation Integrity PHANTOM, built around filings past 30,000 tokens, found hallucination detection degrades as documents lengthen and facts sit deeper — where material disclosure usually lives. Retrieval failure corrupts every later step. Citation correctness (source says the claim) differs from citation faithfulness (source actually grounded the claim). A meaningful share pass correctness while failing faithfulness.
6. The Brane Benchmark Framework Nine tasks: earnings analysis, filings analysis, company research, financial data extraction, company screening, investment thesis generation, thesis challenge, source accuracy, and hallucination resistance. Tasks are composite — e.g. three most material changes across two annual reports, quantified and cited — not single-figure lookups. Base-model runs are reported separately from agentic, tool-augmented runs, with model version and date on every run.
7. What Existing Benchmarks Miss: Five Gaps That Matter Single-turn QA instead of multi-step research. Retrieval treated as solved when it drives downstream error. No thesis interrogation — bull cases are easy, company-specific bear cases are the skill. Citation faithfulness almost never measured. Almost no India-regime or non-US filing coverage.
8. Human Judgment, Materiality, and What Investors Should Ask Two summaries can both be factually correct while only one identifies the investment-relevant signal. Ask any vendor: what benchmark and task mix, what model version and date, was retrieval involved and how scoped, and can you show where the system fails with examples and error rates.
9. Five Documented Weaknesses 1. 81% failure or refusal on grounded filing questions — FinanceBench, 10,231 triplets. 2. ~60% failure on professional-grade tasks — FinanceQA hedge-fund / PE / IB workflows. 3. Long-context detection decay — PHANTOM, 30,000-token filings. 4. Unfaithful citations that look credible — correctness without faithfulness. 5. No thesis invalidation or India-regime testing in most suites.
10. Five Concrete Improvements 1. Test composite analyst tasks end to end with quantification and citations. 2. Score retrieval separately from reasoning by document depth. 3. Mandate thesis-challenge tasks scored by analysts. 4. Audit citation faithfulness, penalize post-hoc citations. 5. Add Indian-filing and versioned agentic tracks with NSE, BSE, SEBI-regime documents.
11. Conclusion: What This Means for Institutional Research Current models are strong at narrow retrieval and meaningfully weaker at reasoning, judgment, and evidentiary discipline. Deploy AI for scoped extraction with verification, keep thesis judgment human, and demand benchmarks that test composite work with faithful citations.
Where Brane Fits: Brane proposes this benchmark as a commitment device — methodology first, then public results including where its own system underperforms, updated as models change.
FAQ Q1: What is the AI equity research benchmark proposed by Brane? Nine tasks covering earnings, filings, company research, extraction, screening, thesis generation, thesis challenge, source accuracy, and hallucination resistance, with versioned runs. Q2: How do current models score? FinanceBench 81% incorrect or refused with retrieval; FinanceQA ~60% failure. Strong extraction, weak multi-step reasoning and assumptions. Q3: Why separate base-model and agentic performance? A standalone model and a system with live retrieval and databases are not comparable. Separation reveals whether gains come from model or tooling. Q4: What is citation faithfulness? Correctness asks if the source states the claim; faithfulness asks if it actually produced it. Unfaithful citations mask post-hoc justification. Q5: How should investors evaluate a vendor? Ask for benchmark mix, model version and date, retrieval scope, and disclosed failures with error rates. Prefer versioned reruns over selected wins.
Sources: FinQA; ConvFinQA; FinanceBench (10,231 triplets, 40 companies); FinanceQA; PHANTOM (30,000+ tokens); MMLU (as non-diagnostic reference).
This article is educational and does not make a recommendation about any security. Evidence cutoff September 2026.