HELM is the most systematic attempt to date to define what a complete evaluation of a language model should look like. Where earlier benchmarks measured a single capability in isolation — accuracy on a knowledge test, a coding score — HELM insists that no single number is adequate. The paper evaluates 42 models across 16 scenarios using 7 simultaneous metric families: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. The result forces a honest reckoning with the trade-offs that isolated leaderboards obscure.

The core methodological contribution is the separation of what is being measured (scenarios) from how it is measured (metrics). This means the same model can be compared on the same task using different success criteria simultaneously — a much closer approximation of how real decisions are actually made. The framework also standardises the way shots (zero-shot vs few-shot), prompts, and decoding parameters are reported, which reduces the silent variability that makes benchmark comparisons unreliable.

HELM also surfaces a result that has since become widely cited: models that top one metric often perform poorly on others. A model that is highly accurate can also be poorly calibrated, or produce outputs with significantly higher toxicity rates than a lower-accuracy competitor. This tension is not visible from accuracy-only leaderboards — HELM makes it structural and unavoidable.

Practical read: HELM's framework is the right mental model for enterprise AI procurement. Before choosing a model based on a single leaderboard score, ask: accurate at what cost? Under what distribution shift? For which user groups? Build an internal evaluation stack that tests at least accuracy, calibration, and task-relevant robustness together — not accuracy alone.

This paper introduced two ideas that have since become infrastructure for the entire evaluation field. First, MT-Bench: a set of 80 multi-turn, open-ended questions spanning reasoning, math, coding, writing, and roleplay — specifically chosen to catch the gaps that fill-in-the-blank benchmarks miss. Second, and more far-reaching: the use of a strong LLM (GPT-4) as the evaluator, replacing or supplementing costly human judgment at scale. The combination became the methodological foundation for Chatbot Arena.

The paper tackles a hard problem directly: how do you evaluate subjective qualities like helpfulness, coherence, and instruction-following in multi-turn conversations without requiring humans to read every response? The answer is to use a capable LLM judge, but to also rigorously quantify when that judge's scores agree with human assessors — and when they diverge. The paper reports over 80% agreement between GPT-4 judgments and human expert judgments on MT-Bench, establishing a credible upper bound for automated evaluation quality.

Critically, the paper also documents failure modes: position bias (LLM judges tend to prefer whichever answer appears first), verbosity bias (longer answers receive higher scores regardless of quality), and self-enhancement bias (a model tends to rate its own outputs higher). These are not minor footnotes — each is a systematic distortion that invalidates evaluation results if left uncorrected. The paper proposes mitigation strategies that have become standard practice in teams running LLM-as-judge pipelines.

Practical read: If you are building an internal automated eval system — a regression suite, a red-teaming loop, an output quality gate — this paper is the required reading before you write a single line of code. Understand the biases, build swap-position checks, and never trust a single-pass judge score without a calibration baseline.

Chatbot Arena is the paper that turned the crowd into an evaluation instrument. Rather than designing a fixed test set — which gets gamed as models are trained on its distribution — the platform lets real users pose any question they want, see two anonymous model responses side by side, and vote for the one they prefer. The resulting Elo scores, computed from more than one million pairwise battles, represent the most robust human-preference leaderboard currently in existence.

The methodological contribution is a serious treatment of Elo rating uncertainty. Unlike many leaderboards that report a single score, Chatbot Arena reports confidence intervals and explicitly tracks when Elo gaps between neighbouring models are statistically meaningful versus noise. The paper demonstrates that small gaps at the top of the leaderboard (say, 10–20 Elo points) are often statistically insignificant — a point that is almost universally ignored by press coverage of model releases.

The paper also studies the coverage bias inherent in open-ended crowdsourcing: users don't ask uniformly distributed questions. They cluster around coding, writing, and creative tasks, which means the Arena ranking reflects user population preferences rather than a theoretically balanced capability test. The authors are transparent about this limitation — which makes the paper more credible than benchmarks that obscure their selection process.

Practical read: Arena Elo is the best available proxy for "does a typical user prefer this model's outputs?" — but only for consumer-facing or general-purpose applications. For structured B2B workflows, you still need domain-specific evals. Treat Arena rank as the first filter, not the final answer. And whenever a vendor announcement cites a small Elo improvement, check the confidence intervals before treating it as a meaningful capability gap.

MMLU remains one of the most consequential benchmark papers in the history of NLP — not because it is technically the best designed, but because it arrived at the right moment and asked the right question at scale. Covering 57 subjects from elementary school to professional certification level (law, medicine, economics, computer science, STEM, ethics), it was the first benchmark to convincingly demonstrate that language models had absorbed a substantial fraction of the world's textbook knowledge. At the time of its publication, GPT-3 achieved 43% — just below random chance for a 4-choice test if you account for the right baseline. Within three years, frontier models exceeded 90%.

The benchmark's design choices are worth studying. Questions are drawn from real, human-authored exams and textbooks, which means they reflect genuine expert judgment about what matters in each domain — rather than synthetic problems constructed by researchers. The range of difficulty is intentional: elementary questions anchor the bottom of the distribution, while professional-level questions (bar exam, USMLE medical licensing) anchor the top. This gradient proved essential for tracking capability jumps as models scaled.

MMLU's saturation by 2024 is itself informative. The fact that models can now exceed 90% on an evaluation that covers 57 subjects at professional level is not a failure of the benchmark — it is a genuine measurement of an unprecedented capability shift. It is also a reminder that once a static benchmark is saturated, it stops differentiating and needs to be replaced by harder successors like GPQA Diamond and Humanity's Last Exam.

Practical read: MMLU scores are still the most widely reported reference point in model cards and research papers. Understand the baseline (random chance = 25%, GPT-3-era = ~43%, current frontier = 88–90%) so you can contextualise vendor claims accurately. A model citing 80% MMLU in 2026 is not impressive — it is below the frontier. If MMLU is the only benchmark a vendor reports, treat that as a red flag.

This paper is the intellectual foundation for the entire ARC-AGI programme and one of the most important philosophical contributions to AI evaluation. Chollet's core argument is that the field has been measuring the wrong thing: benchmark performance measures skill, but skill is not the same as intelligence. A system can achieve expert-level performance on every known task by memorising the training distribution — and that would tell us nothing about whether it can adapt to a genuinely novel situation it has never encountered. Intelligence, properly defined, is the ability to efficiently acquire new skills across a wide range of novel situations — not the degree to which prior skills have been cached.

The paper draws on a long tradition in cognitive psychology, particularly Catell's distinction between fluid intelligence (reasoning from first principles in new situations) and crystallised intelligence (the accumulated store of learned knowledge and skills). Chollet argues that current AI — including large language models — is almost entirely crystallised: it is extraordinarily good at recombining and retrieving what it has seen, but poor at solving problems that require genuinely novel rule induction with minimal data. ARC-AGI is his proposed operational test for the fluid side.

The paper also proposes a formal definition of intelligence as "skill-acquisition efficiency" — the rate at which a system gains performance on novel tasks per unit of prior knowledge and experience. This framing makes explicit that a system which required 10 trillion tokens of training to solve a task should not be classified as more intelligent than a human who solves the same task after reading two sentences, even if the final scores are equal.

Practical read: The skill-vs-intelligence distinction is practically important. When a vendor demonstrates a model performing well on a demo task, ask: how much of that performance is adaptation to a genuinely new problem, and how much is pattern matching to similar tasks in training data? The answer changes how you evaluate reliability in production, particularly for edge cases, novel workflows, and customer queries your training data did not cover.

SWE-bench changed what it means to evaluate a coding model seriously. Rather than asking models to complete isolated functions or solve algorithmic puzzles, it presents 2,294 real issue–pull-request pairs drawn from 12 popular open-source Python repositories — including Django, Flask, NumPy, and scikit-learn. A model passes a task only if its generated code patch resolves the issue and passes the repository's existing test suite. There is no partial credit, no self-reported score, and no proxy metric — the test suite is the ground truth.

At launch, the performance numbers were striking: the best available model at the time resolved roughly 1.96% of tasks. The framework immediately revealed something that coding benchmark scores had hidden: the ability to write correct code from a clean specification is a fundamentally different skill from the ability to read an existing, undocumented codebase, understand where a reported bug originates, and make a targeted fix without breaking adjacent functionality. These are the skills that real engineering teams actually need.

SWE-bench Verified, a cleaned subset with verified human solutions, has since become the standard for agentic coding agent comparisons. By 2025, leading systems were resolving 40–55% of tasks — a dramatic improvement that correlates closely with the growth of extended-context windows and better tool-use capabilities. The benchmark's design — real repositories, real tests, automated verification — makes it extremely difficult to game without genuinely improving software engineering capability.

Practical read: If you are evaluating a coding agent for internal use — code review, bug fixing, refactoring, or developer productivity — SWE-bench Verified is the single most task-representative public benchmark available. When comparing vendors, verify the score is in autonomous mode (no human hints or guided steps) and that it was measured on the full Verified set, not a curated subset. A 35% autonomous score is materially better than a 55% assisted score for a hands-off deployment.

BIG-bench is the most ambitious collaborative benchmark construction effort in NLP history. 450 researchers across 130 institutions contributed 204 diverse tasks — ranging from linguistic structure and algorithmic reasoning to social knowledge, creativity, and logic — with the explicit goal of finding what frontier language models cannot do. The benchmark was designed as a living document of the capability frontier: tasks that models currently fail on, which become the most informative evaluation targets precisely because they have not yet been absorbed into the training distribution.

One of BIG-bench's most important published findings concerns emergent abilities: certain tasks showed near-zero performance across a wide range of model sizes, then jumped sharply to reasonable performance once models crossed a scale threshold — without any obvious intermediate improvement. This pattern, described as "emergent" behaviour, generated substantial academic debate about whether scaling laws are smooth or discontinuous, and whether large models exhibit qualitatively new capabilities or simply better-calibrated versions of existing ones. The debate is not fully resolved and has direct implications for anyone trying to predict model capability from benchmarks.

BIG-Bench Hard, a curated subset of the 23 tasks that most consistently stumped early models, has become the standard reference for multi-step reasoning evaluation. Its inclusion of tasks like date arithmetic, causal judgment, formal fallacy detection, and object tracking under natural language transformation makes it one of the most practically informative reasoning benchmarks for teams deploying models in knowledge-work applications.

Practical read: BIG-bench Hard is a useful scan for whether a model's reasoning capability is genuine or narrow. If your use case involves multi-step inference, causal analysis, or following complex conditional instructions — common in legal, financial, and compliance workflows — test against the BIG-bench Hard task categories most relevant to your domain. A model that scores well on MMLU but poorly on BIG-bench Hard likely has broader knowledge than it has reliable reasoning.

This survey is the most comprehensive single-document reference for the AI evaluation landscape, covering more than 100 benchmarks and evaluation frameworks across a unified taxonomy. It organises evaluations into three top-level branches: knowledge and capability evaluations (covering reasoning, coding, science, and domain knowledge), alignment evaluations (covering safety, ethics, honesty, and instruction-following), and safety evaluations (covering adversarial robustness, factuality, and potential for harm). Each category is mapped to the benchmarks most commonly used to probe it.

The paper's taxonomic contribution is its practical value. Teams building evaluation stacks often start by collecting individual benchmark results without a framework for deciding what has and has not been covered. This survey provides that framework: if you have tested knowledge breadth and reasoning quality but not robustness or calibration, you have a partial picture that could fail in deployment in predictable ways. The survey makes those gaps explicit and suggests what to measure next.

The treatment of alignment evaluation is particularly useful in the context of AI governance and the Australian regulatory environment. The paper covers a range of evaluation approaches for model honesty, value alignment, and susceptibility to jailbreak attacks — topics that are increasingly relevant as enterprises face internal AI governance requirements and early signs of external regulatory scrutiny. The coverage is balanced and avoids both overclaiming safety guarantees and dismissing them.

Practical read: Use this survey as a starting point for building your internal evaluation checklist. Map your deployment scenario to the survey's taxonomy: What knowledge domains does the model touch? What alignment risks are relevant to your use case? What robustness assumptions are you making? A one-page evaluation coverage matrix derived from this taxonomy is a useful artefact for any enterprise AI risk review.

TruthfulQA targets a failure mode that is easy to overlook: the tendency of language models to reproduce false beliefs that are popular among humans, because those false beliefs are well-represented in training data. The benchmark's 817 questions span 38 categories including health, law, finance, politics, conspiracy theories, and common misconceptions — all designed so that a model trained to predict the most statistically likely next token will give a wrong but plausible-sounding answer. GPT-3 at launch was truthful on approximately 58% of questions; the human baseline was 94%.

The paper's finding that larger models are sometimes less truthful than smaller ones was counterintuitive and broadly cited. The explanation is coherent: larger models are better at capturing the statistical distribution of human text, which includes more confident expression of widespread falsehoods. A model that has absorbed more internet text has also absorbed more confident misinformation. This result directly challenges naive scaling narratives and has influenced how safety researchers think about the relationship between capability and alignment.

For evaluation methodology, TruthfulQA introduced a two-axis scoring framework: a model can be evaluated on both truthfulness (does it state true things?) and informativeness (does it say anything substantive?). A model that answers "I don't know" to every question is highly truthful but useless. The paper proposed scoring models on both dimensions simultaneously — a practice now standard in safety-oriented evaluation suites.

Practical read: If your application surfaces model outputs to users in domains where factual accuracy matters — health information, legal guidance, financial advice, regulatory compliance — TruthfulQA-style evaluation is not optional. The specific failure mode (confident reproduction of widespread misconceptions) is exactly the one that generates reputational and liability risk in professional settings. Test explicitly for it on your domain's known misconceptions, not just on the public benchmark set.

GAIA evaluates AI systems on tasks that require the full stack of general assistant capabilities: reading and interpreting files, browsing the web, using tools, chaining multi-step reasoning, and arriving at a specific, verifiable answer. The questions are deliberately written to look simple on the surface ("What was the exchange rate for USD to AUD on 14 June 2019?") but require an agent to actually retrieve, process, and synthesise real information rather than recall a training-data answer. At launch, GPT-4 with plugins answered approximately 15% of Level 1 tasks — the easiest tier. Humans answered the same tasks at ~92%.

GAIA's three difficulty levels are calibrated to the complexity of the tool-use chain required. Level 1 requires a single external lookup and a direct answer. Level 2 requires multiple lookups, format conversions, and intermediate reasoning steps. Level 3 requires navigating ambiguous instructions, cross-referencing multiple sources, and synthesising a non-obvious conclusion. The gap in AI performance across the three levels is steep — systems that perform adequately at Level 1 typically collapse at Level 3 — which makes the benchmark unusually revealing about where real-world capability limits currently sit.

The benchmark is particularly important in the context of enterprise AI deployment because it directly measures the failure modes that appear most often when AI assistants are given semi-autonomous research, analysis, and data-retrieval tasks. The gap between what these systems claim to be capable of and what they can reliably accomplish on GAIA-style tasks is a useful calibration tool for setting appropriate scope for internal pilots.

Practical read: GAIA is the most relevant public benchmark for evaluating AI assistants in knowledge-work contexts: research, document analysis, compliance checking, and data retrieval. Before deploying an agentic AI workflow in your organisation, run a GAIA-style evaluation on your specific task types. The Level 2 and Level 3 task patterns almost always reveal failure modes that the vendor's demo never showed you.

ARC-AGI-2 is the successor to Chollet's original fluid intelligence benchmark, released after the first edition became solvable by hybrid retrieve-and-match systems that circumvented genuine in-context rule learning. The new edition introduces significantly harder task classes, novel visual transformation patterns, and three times the task diversity — specifically designed to close the loopholes that allowed purpose-built ARC-1 systems to score competitively without demonstrating the abstract generalisation the benchmark was meant to probe.

Each ARC-AGI-2 task presents a handful of input-output grid demonstrations. The solver must induce the underlying transformation rule and apply it to a novel test grid — with no training examples from the task distribution, no hint about which concept class is being tested, and no way to brute-force solutions using ARC-1 program synthesis shortcuts. The paper reports that frontier reasoning models without fine-tuning score in low single-digit percentages on harder tiers, re-establishing the benchmark as a genuinely hard generalisation challenge.

Four major AI labs — Anthropic, Google DeepMind, OpenAI, and xAI — all referenced ARC-AGI performance in their 2025 model cards. The benchmark attracted a $1 million Kaggle competition with 1,455 teams; the top public submission reached 24% on the private evaluation set, compared to humans averaging around 65%. This gap is the most honest public signal available of the frontier AI–human reasoning gap in 2025.

Practical read: ARC-AGI-2 performance is the clearest available proxy for a model's genuine task-adaptive generalisation — its ability to handle structurally novel problems it could not have memorised. For deployments involving edge-case reasoning, unusual data formats, or genuinely non-routine analytical tasks, treat ARC-AGI-2 scores as the leading indicator of whether you are buying real reasoning or sophisticated pattern retrieval.

This paper is the most important methodological critique published in 2025 on how AI reasoning benchmarks are actually run. The authors conduct a large-scale empirical study demonstrating that reported improvements on mathematical reasoning benchmarks are highly sensitive to implementation choices — decoding parameters, random seeds, prompt formatting, and even hardware and software configurations — that are routinely underreported. Gains that seem decisive in one setup frequently evaporate under controlled replication.

The central result: most reinforcement learning approaches that claimed large gains on AIME’24 and similar benchmarks produced only modest improvements when re-evaluated under a standardised protocol with proper variance reporting. Supervised fine-tuning methods showed consistently stronger generalisation. The paper interprets this as evidence that many RL-based reasoning claims are overfitted to specific evaluation artefacts of a small benchmark rather than demonstrating genuine capability improvement.

The authors release a standardised evaluation framework with detailed best-practice specifications: how seed variance should be reported, what prompt formats should be held constant, how decoding temperature interacts with benchmark scores, and what statistical tests are minimally required to claim meaningful improvement. Accepted at COLM 2025 — the premier venue for rigorous evaluation methodology — this is now the reference standard for reproducible reasoning benchmark evaluation.

Practical read: When a vendor or internal team reports an improvement on AIME, GSM8K, or similar reasoning benchmarks, ask: what seed variance was reported, what prompt template was used, and was the result replicated on a held-out variant set? Gains of under 5 percentage points on these benchmarks are almost certainly within noise if these checks are not performed. Apply this scepticism to any model adoption decision anchored on a single benchmark number.

BrowseComp consists of 1,266 questions that require genuine multi-step web research to answer. Each has an objectively verifiable correct answer that exists somewhere on the public internet but cannot be found through simple keyword lookup. Questions require navigating across multiple sources, resolving ambiguities, cross-referencing dates and names, and arriving at a specific conclusion that the web does not present anywhere as a pre-formed answer. Construction ensures that even a competent human researcher would need ten or more minutes per question.

Performance results established the difficulty ceiling clearly. GPT-4o with browsing answered roughly 1.4% of questions at initial evaluation; o3 (OpenAI’s then-most-capable reasoning model with web access) scored around 51% — a dramatically wide gap mapping directly to the difference between shallow web lookup and genuine research capability. The tiered difficulty structure allows benchmark users to identify exactly at which complexity tier a given system stops being useful for production knowledge work.

The paper’s methodological contribution is its protocol for constructing “unanswerable-by-memorisation” questions — a framework any team can apply to generate domain-specific evaluation questions that are guaranteed to require active retrieval rather than training-data recall. This protocol transfers to internal benchmarks for competitive intelligence, regulatory monitoring, or other research-intensive workflows.

Practical read: If your deployment involves AI assistants used for web research — competitive intelligence, due diligence, regulatory monitoring, literature review — BrowseComp is the right benchmark to anchor capability claims. Tool access and research capability are not the same thing. A model that fails Level 2 multi-hop BrowseComp tasks will fail structurally similar real research tasks. Test before deploying in knowledge-work roles where factual mistakes have real consequences.

PaperBench asks AI agents to replicate 20 full ML papers published at ICML 2024 — each with a full code submission, experimental setup, and results to reproduce from scratch. The agent receives the paper PDF and must implement and run the experiments autonomously. Performance is measured against a rubric developed by the original paper authors: partial credit is awarded per subtask (data pipeline, model implementation, training code, evaluation code, result reproduction), making it possible to identify exactly which steps agents can and cannot reliably complete.

The leading AI agent scored approximately 26% on average across papers and rubric dimensions at publication. Simple retrieval-augmented approaches struggle to maintain consistency across the full code pipeline, while stronger agents that interact with their own execution environment and read error traces perform significantly better on incremental subtasks. The partial-credit rubric is the key methodological contribution: it turns a pass/fail task into a calibrated measurement of where in the research workflow capability currently falls short.

PaperBench establishes the clearest available measurement of how close AI agents are to being genuinely useful collaborators in AI research itself. The failure pattern — agents succeeding on data loading and model definition but failing on evaluation code and result verification — reveals that current AI research agents are better at imitation than at the verification steps that distinguish rigorous science from plausible-looking results.

Practical read: PaperBench is directly relevant to any team evaluating AI coding agents for scientific or engineering research workflows. The partial-credit rubric maps to real workflows: an agent that succeeds on data pipelines but fails on result verification will produce misleading experimental outputs. Before deploying AI agents in research assistance roles, run PaperBench-style replication tests on representative tasks from your own domain to understand where in the workflow agent failures cluster.

RewardBench 2 is the second-generation revision of AllenAI’s reward model evaluation suite, updated to address shortcomings identified after the original RewardBench (2024) became gamed and saturated. The revision expands coverage to structured reasoning tasks, code evaluation scenarios, and multi-turn preference scenarios — categories underrepresented in the first edition that matter for fine-tuning pipelines now used by all major frontier model developers. A public leaderboard on HuggingFace tracks over 100 reward models against the updated benchmark.

The paper’s primary finding: reward models that score well on the original RewardBench frequently underperform on reasoning-intensive and code-related preference tasks, revealing a category-specific gap invisible in aggregate scores. Conversely, models fine-tuned specifically on reasoning tasks transfer poorly back to general preference tasks. The revised benchmark’s category structure exposes these trade-offs, making it possible to select reward models matched to a specific RLHF application domain rather than optimising for aggregate leaderboard position.

RewardBench 2 matters beyond academic interest because reward models now sit at the centre of nearly every major alignment and instruction-following training pipeline. An underperforming reward model corrupts training signal silently — the resulting models may appear fluent and helpful while being misaligned with intended behaviour in precisely the categories the reward model scored poorly on. This benchmark provides the infrastructure to catch these failures before they propagate into production models.

Practical read: If you are building or fine-tuning models using RLHF or preference optimisation, the reward model you choose is the ceiling on alignment quality. Use RewardBench 2 to evaluate reward model performance on the specific task categories in your training distribution — not just the aggregate leaderboard score. A reward model that scores 80% overall but 55% on your task category is a miscalibrated training signal for your pipeline.

MathArena addresses benchmark contamination in mathematical reasoning with a simple but powerful insight: evaluate models on math competition problems as soon as they are released — before any training data cutoff can include them. By running evaluations on AIME, CMIMC, and IMO problems within hours of their public release, the benchmark guarantees zero contamination by construction, making it the most reliable source of unbiased mathematical reasoning scores available for frontier models.

The paper reports strong evidence of training data contamination in AIME 2024: models score meaningfully higher on AIME 2024 problems than on structurally similar problems from competitions released after their training cutoffs. On harder, genuinely uncontaminated competitions like CMIMC 2025, performance drops substantially — the gap is the contamination measurement, and it is large enough to materially change interpretations of which models are strongest. The paper also first evaluates proof-writing at scale: on IMO 2025, top models achieved slightly less than 40%, showing both notable progress and the distance remaining to human expert level.

MathArena demonstrates a contamination-testing methodology with wide applicability. Any static benchmark available online long enough for models to have been trained on it is vulnerable to contamination masquerading as genuine capability. The live evaluation protocol — test immediately after problem release — is the only complete defence, and MathArena provides the first working longitudinal implementation of it at competition scale.

Practical read: Since most public math benchmarks are now saturated or contaminated, MathArena scores from recent competitions are the most reliable signals of genuine mathematical capability. When evaluating models for mathematical or analytical tasks, prefer the most recently released benchmark variant over historical problem sets whose internet availability may have corrupted older models’ scores. This is also the template for building internal contamination-resistant evaluation: test on genuinely new problems.

PHYBench provides 500 original physics problems spanning high school through Physics Olympiad difficulty, all created by the benchmark’s authors to guarantee zero contamination. Problems cover classical mechanics, electromagnetism, thermodynamics, optics, and modern physics — requiring chains of quantitative reasoning that draw on both physical intuition and algebraic manipulation. Even the best-performing model at publication (Gemini 2.5 Pro) achieved only 36.9% accuracy compared to human expert performance of 61.9%, establishing a meaningful capability gap across the current frontier.

PHYBench introduces the Expression Edit Distance (EED) Score as a novel evaluation metric for mathematical outputs. Rather than binary correct/incorrect scoring, EED measures structural similarity between a model’s symbolic expression and the correct answer, improving sample efficiency by 204% over binary scoring by capturing partial credit at the algebraic level. This metric transfers directly to any benchmark requiring symbolic or algebraic outputs, where conventional exact-match scoring discards meaningful signal about how close a model’s reasoning is to the correct approach.

Comparative analysis shows PHYBench elicits more discriminative model evaluation than AIME 2024 or OlympiadBench on the same models — the score distribution is wider and relative rankings differ in ways that reveal previously hidden capability differences. PHYBench activates more extended reasoning chains per problem, confirming it probes deeper inference processes rather than just different factual content.

Practical read: PHYBench is the right evaluation for any deployment involving applied physics reasoning — engineering simulation, product design, STEM tutoring, or scientific computing assistance. The 37% vs 62% gap between frontier AI and human experts confirms that even the best models are not reliably expert-level in applied physics. The EED scoring framework is worth adapting for internal benchmarks in any domain with structured symbolic or algebraic outputs.

DeepSeek-Prover-V2 is the 2025 frontier result in formal theorem proving, achieving 88.9% pass ratio on MiniF2F-test and solving 49 out of 658 problems on PutnamBench — both state-of-the-art results at publication. The model is trained via a pipeline that prompts a general-purpose LLM to decompose complex theorems into subgoals, synthesises proofs of those subgoals into chain-of-thought training examples, and then cold-starts reinforcement learning from these. The contribution is integrating informal and formal mathematical reasoning into a unified training signal rather than training formal reasoning in isolation.

The paper also introduces ProverBench, a dataset of 325 formalised problems including 15 carefully selected problems from the 2024–2025 AIME competitions — the first publicly released formal evaluation set bridging competition-level informal mathematics and Lean 4 formalisation. ProverBench provides the evaluation infrastructure to track progress in the critical gap between informal mathematical capability and formal proof verification capability.

The comparison on the 15 AIME problems is methodologically pointed: DeepSeek-Prover-V2 solves 6 in formal Lean 4, while DeepSeek-V3 (the informal version) solves 8 with majority voting. This 2-problem gap is the most concrete published quantification of how close formal and informal mathematical AI capabilities currently are — a measurement that matters because formal verification eliminates the class of plausible-but-wrong mathematical outputs that informal models produce.

Practical read: Formal theorem proving capability is the gold standard for mathematical reliability — a formally verified proof cannot be wrong. ProverBench provides the right evaluation framework for quantifying the gap between a model’s informal mathematical fluency and its ability to produce verifiably correct formal reasoning. For high-stakes quantitative domains — financial risk modelling, engineering verification, scientific derivation — understanding this gap is the difference between useful AI assistance and costly mathematical errors.

The ARC Prize 2025 Technical Report surveys the results of the $1 million Kaggle competition targeting ARC-AGI-2, attracting 1,455 teams and 15,154 submissions. The top public score reached 24% on the private evaluation set — significant progress from near-zero at ARC-AGI-2’s release in May 2025, though still far below the ~65% human baseline. The report is the most comprehensive analysis available of what architectural and algorithmic approaches move performance on abstract visual reasoning benchmarks.

The defining theme of competition performance was the emergence of the refinement loop — iterative per-task program optimisation guided by a feedback signal rather than single-pass generation. Top teams used variants of three strategies: evolutionary program synthesis (generating and selecting code transformations), application-layer feedback loops built on commercial AI systems, and zero-pretraining deep learning methods achieving competitive performance with remarkably small networks (7M parameters). The report’s taxonomy of refinement loop architectures is the main technical contribution and maps directly onto the emerging agentic AI paradigm.

The report documents a critical finding about benchmark contamination: current frontier AI reasoning performance remains fundamentally constrained by knowledge coverage rather than genuine abstraction. When tasks require transformations that do not surface in training data, performance degrades in ways that suggest knowledge-based interpolation rather than rule induction. This is the most precisely documented instance of knowledge-dependent overfitting in high-stakes benchmarking published in 2025, and shapes the design of ARC-AGI-3.

Practical read: The refinement loop architecture described in this report — iterative per-task optimisation guided by feedback — drove top competition performance and is the design pattern now being adopted broadly in agentic AI systems. If you are building AI systems for structured problem-solving tasks, this report provides the most current evidence-based framework for deciding whether a single-pass approach is adequate or whether iterative correction loops are necessary for your reliability requirements.

Deep Research Bench evaluates AI web research agents on tasks requiring multi-step internet research and accurate, well-sourced research report generation — the exact workflow underpinning commercial “deep research” products from OpenAI, Google, Perplexity, and others that became widely deployed in 2025. The benchmark covers factual accuracy, source quality, coverage breadth, and claim-level citation accuracy across open-ended research tasks with expert-verified ground truth. It is the first evaluation standard designed specifically for the “deep research agent” capability class.

Results across commercially available deep research systems show meaningful differentiation in factual accuracy and source attribution quality — the two dimensions that matter most for knowledge-work deployments. Systems optimised for impressive report structure frequently underperform on factual accuracy when evaluated against expert ground truth, while systems with smaller output volume but tighter source verification perform better on the dimensions that actually matter. Report length was negatively correlated with accuracy in several categories.

Deep Research Bench addresses the specific accountability gap that emerged in 2025 as deep research products proliferated without standardised evaluation protocols. Organisations deploying these products for professional research — legal research, due diligence, policy analysis, academic literature review — were unable to compare performance on a common standard. This benchmark provides the first such standard and reveals performance differences of up to 30 percentage points on factual accuracy between leading commercial systems.

Practical read: If you are evaluating deep research agents for professional knowledge work, apply Deep Research Bench-style evaluation on your specific research domains before committing to a vendor. Report quality and factual accuracy frequently diverge — a well-formatted report with confidently stated inaccuracies is the most dangerous failure mode in high-stakes research workflows. Require factual accuracy scores side by side with fluency scores when requesting any deep research AI product evaluation.