Abc Bench
ABC-Bench
Software Engineering
Backend Service EngineeringEnvironment Configuration And Service Deployment
- Interaction
- None · static
- Scale
- 92 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Benchmarks
What each benchmark measures, how it is structured, and which ctxdex models meet its recorded requirements. Applicability says a model meets the requirements. It is not a result: no scores, ranks or leaderboard values appear here.
- Definitions
- 196
- Families
- 84
- Models
- 94
196 of 196 definitions
View
Sorted by name. Nothing here is ranked.
Agnostics
Ag-LiveCodeBench-X
Code generation
- Interaction
- None · static
- Scale
- 499 problems
- Metric
- Pass@k
- Structure
- Single set
AgentBench
AgentBench (original / v0.2 protocol)
General Agent
Heterogeneous Interactive Agent Execution
- Interaction
- None · static
- Scale
- 269 cases
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →v0.2 protocol
AgentBench
AgentBench FC
General Agent
Heterogeneous Interactive Agent ExecutionTool Selection And InvocationStateful Multi Turn Tool Use
- Interaction
- None · static
- Scale
- 5 environments
- Metric
- Task Success Rate
- Structure
- Single set
No family
AGIEval
Human exams
A human-centric benchmark built from 20 public admission, qualification, civil-service, law, and mathematics examination tasks.
- Interaction
- None · static
- Scale
- 20 tasks
- Metric
- Accuracy
- Structure
- 1 component
AI2 Reasoning Challenge
AI2 Reasoning Challenge (ARC)
Science reasoning
A 7,787-question grade-school science benchmark partitioned into Easy and Challenge sets.
Grade-school science multiple-choice Q&A
- Interaction
- None · static
- Scale
- 7,787 questions
- Metric
- Accuracy
- Structure
- 2 components
Aider Polyglot Benchmark
Aider Polyglot Benchmark
Code editing
- Interaction
- None · static
- Scale
- 225 problems
- Metric
- Pass@k
- Structure
- 6 components
AndroidWorld
AndroidWorld
Mobile Agent
Computer UseMobile Gui Interaction
- Interaction
- Mobile GUI
- Scale
- 20 applications
- Metric
- Task Success Rate
- Structure
- 2 scenarios
API-Bank
API-Bank
Tool Use
Api CallingApi Retrieval And CallingApi Planning Retrieval And Calling
- Interaction
- None · static
- Scale
- 753 apis
- Metric
- Accuracy
- Structure
- 3 scenarios
APPS
APPS
Code generation
- Interaction
- None · static
- Scale
- 10,000 problems
- Metric
- Pass@k
- Structure
- 3 components
AppWorld
AppWorld
Tool Use
Multi App Interactive Task Execution
- Interaction
- None · static
- Scale
- 457 apis
- Metric
- Task Goal Completion
- Structure
- Single set
AppWorld
AppWorld-UL
Tool Use
User In The Loop Tool UseRequirement ClarificationInfeasibility HandlingUser Confirmation Seeking
- Interaction
- None · static
- Scale
- 210 tasks
- Metric
- Task Goal Completion
- Structure
- 4 scenarios
ARC-AGI
ARC-AGI-1
Abstract reasoning
A grid-transformation abstraction benchmark requiring rule induction from a small number of demonstrations.
- Interaction
- None · static
- Scale
- 400 tasks
- Metric
- Pass@2
- Structure
- Single set
ARC-AGI
ARC-AGI-2
Abstract reasoning
The second ARC-AGI benchmark generation, with calibrated harder evaluation tasks designed for modern reasoning systems.
- Interaction
- None · static
- Scale
- 1,000 tasks
- Metric
- Pass@2
- Structure
- Single set
AI2 Reasoning Challenge
ARC-Challenge
Science reasoning
The hard ARC partition, containing science questions that defeated two simple baseline systems during construction.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
AI2 Reasoning Challenge
ARC-DA
Science reasoning
A direct-answer variant derived from ARC that removes the multiple-choice answer selection format.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Exact Match
- Structure
- Single set
AssistantBench
AssistantBench
Web Agent
Web Information Retrieval
- Interaction
- Web browser
- Scale
- 214 tasks
- Metric
- Accuracy
- Structure
- Single set
Berkeley Function Calling Leaderboard (BFCL)
BFCL V1
Tool Use
Tool Selection And InvocationParallel Tool InvocationTool Relevance Detection
- Interaction
- None · static
- Scale
- 200 chatting non leaderboard
- Metric
- Accuracy
- Structure
- Single set
Berkeley Function Calling Leaderboard (BFCL)
BFCL V2 Live
Tool Use
Tool Selection And InvocationParallel Tool InvocationTool Relevance Detection
- Interaction
- None · static
- Scale
- 875 irrelevance
- Metric
- Accuracy
- Structure
- Single set
Berkeley Function Calling Leaderboard (BFCL)
BFCL V3
Tool Use
Stateful Multi Turn Tool UseTool Selection And Invocation
- Interaction
- None · static
- Scale
- 200 base
- Metric
- Accuracy
- Structure
- Single set
Berkeley Function Calling Leaderboard (BFCL)
BFCL V4 Agentic
Tool Use
Stateful Multi Turn Tool UseWeb Search Tool UseMemory Tool UseTool Call Format Robustness
- Interaction
- None · static
- Scale
- 26 format configurations
- Metric
- Accuracy
- Structure
- Single set
BIG-bench
BIG-bench
General reasoning
A collaborative benchmark suite with more than 200 diverse tasks for probing language-model capabilities.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
BIG-bench
BIG-Bench Extra Hard (BBEH)
General reasoning
A newer extra-hard reasoning suite that replaces BBH tasks with novel, substantially harder tasks probing similar capabilities.
BBEH reasoning task suite
- Interaction
- None · static
- Scale
- 4,520 examples
- Metric
- Accuracy
- Structure
- 2 components
BIG-bench
BIG-Bench Hard (BBH)
General reasoning
A suite of 23 challenging BIG-bench tasks selected because prior evaluated language models did not outperform average human raters.
BBH task suite
- Interaction
- None · static
- Scale
- 23 tasks
- Metric
- Accuracy
- Structure
- 1 component
BigCodeBench
BigCodeBench
Code generation
- Interaction
- None · static
- Scale
- 1,140 problems
- Metric
- Pass@k
- Structure
- 2 scenarios
BigCodeBench
BigCodeBench-Hard
Code generation
- Interaction
- None · static
- Scale
- 148 problems
- Metric
- Pass@k
- Structure
- 2 scenarios
CanItEdit
CanItEdit
Code editing
- Interaction
- None · static
- Scale
- 105 problems
- Metric
- Pass@k
- Structure
- 2 scenarios
Ccbench
CCBench
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 187 tasks
- Metric
- Resolved Rate
- Structure
- Single set
OSWorld
ChainWorld
Computer Use
Computer UseDesktop Gui InteractionMulti Application WorkflowLong Horizon Computer Use
- Interaction
- Desktop GUI
- Scale
- 347 chains
- Metric
- Task Success Rate
- Structure
- 2 scenarios
Codeassistbench
CodeAssistBench
Software Engineering
Project Grounded Code Assistance
- Interaction
- None · static
- Scale
- 7 languages
- Metric
- Requirement Satisfaction Rate
- Structure
- Single set
Codeclash
CodeClash
Software Engineering
Goal Oriented Software Development
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Elo Rating
- Structure
- Single set
No family
CommonsenseQA
Commonsense
A multiple-choice commonsense question-answering benchmark constructed from ConceptNet relations.
- Interaction
- None · static
- Scale
- 12,247 questions
- Metric
- Accuracy
- Structure
- Single set
CrossCodeEval
CrossCodeEval
Code completion
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Not recorded
- Structure
- 4 components, 1 scenario
CRUXEval
CRUXEval
Code reasoning
CRUXEval-I — Input PredictionCRUXEval-O — Output Prediction
- Interaction
- None · static
- Scale
- 800 examples
- Metric
- Pass@k
- Structure
- Single set
No family
DROP
Reading comprehension
A reading-comprehension benchmark requiring discrete reasoning such as addition, counting, sorting, and reference resolution over paragraphs.
- Interaction
- None · static
- Scale
- 96,000 questions
- Metric
- Exact Match, F1
- Structure
- Single set
DS-1000
DS-1000
Code generation
- Interaction
- None · static
- Scale
- 1,000 problems
- Metric
- Pass@k
- Structure
- 7 components
EvalPlus Benchmarks
EvalPerf
Code generation
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Differential Performance Score
- Structure
- Single set
Fea Bench
FEA-Bench Lite v1.0
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 200 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Fea Bench
FEA-Bench v1.0
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 83 repositories
- Metric
- Resolved Rate
- Structure
- 2 scenarios
Featbench
FeatBench v1.0
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 25 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Featurebench
FeatureBench
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 24 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Featurebench
FeatureBench Fast
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 100 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Featurebench
FeatureBench Lite
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 30 tasks
- Metric
- Resolved Rate
- Structure
- Single set
FrontierMath
FrontierMath Erdős
Formal reasoning
A 68-conjecture benchmark of significant open Erdős problems formalized in Lean.
- Interaction
- None · static
- Scale
- 68 conjectures
- Metric
- Formal Proof Acceptance
- Structure
- Single set
FrontierMath
FrontierMath Tier 4
Mathematics
The v2 43-problem FrontierMath expansion focused on exceptionally difficult research-level mathematics.
- Interaction
- None · static
- Scale
- 43 problems
- Metric
- Accuracy
- Structure
- Single set
13applicable models →v2 (2026-06-12)
FrontierMath
FrontierMath Tier 4
Mathematics
The historical v1 FrontierMath Tier 4 expansion containing 50 exceptionally difficult research-level mathematics problems.
- Interaction
- None · static
- Scale
- 50 problems
- Metric
- Accuracy
- Structure
- Single set
FrontierMath
FrontierMath Tiers 1–3
Mathematics
The v2 base FrontierMath set of 295 expert-written problems spanning advanced undergraduate through early-career research mathematics.
- Interaction
- None · static
- Scale
- 295 problems
- Metric
- Accuracy
- Structure
- Single set
13applicable models →v2 (2026-06-12)
FrontierMath
FrontierMath Tiers 1–3
Mathematics
The historical v1 core FrontierMath set of 300 expert-written advanced mathematics problems.
- Interaction
- None · static
- Scale
- 300 problems
- Metric
- Accuracy
- Structure
- Single set
FrontierMath
FrontierMath Tiers 1–4
Mathematics
The FrontierMath Tiers 1–4 benchmark line for highly challenging expert-written mathematics.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Not recorded
- Structure
- Single set
GAIA
GAIA
General Assistant
Open World General Assistance
- Interaction
- None · static
- Scale
- 301 current test
- Metric
- Accuracy
- Structure
- Single set
GPQA
GPQA
Knowledge
A graduate-level, four-option science Q&A benchmark designed to resist simple web lookup.
Graduate-level science multiple-choice Q&A
- Interaction
- None · static
- Scale
- 448 questions
- Metric
- Accuracy
- Structure
- 3 components
GPQA
GPQA Diamond
Knowledge
The 198-question highest-quality/hard GPQA subset widely used for frontier-model reasoning evaluation.
Graduate-level science multiple-choice Q&A
- Interaction
- None · static
- Scale
- 198 questions
- Metric
- Accuracy
- Structure
- Single set
GSM8K and multilingual derivatives
GSM8K
Mathematics
A dataset of grade-school mathematical word problems designed to evaluate multi-step arithmetic reasoning.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Exact Match
- Structure
- Single set
Gso
GSO
Software Engineering
Repository Performance Optimization
- Interaction
- None · static
- Scale
- 10 codebases
- Metric
- Opt At K
- Structure
- Single set
No family
HellaSwag
Commonsense
An adversarially filtered commonsense inference benchmark based on selecting plausible continuations to grounded situations.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
HumanEval
HumanEval
Code generation
- Interaction
- None · static
- Scale
- 164 problems
- Metric
- Pass@k
- Structure
- Single set
EvalPlus Benchmarks
HumanEval+
Code generation
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Pass@k
- Structure
- Single set
No family
Humanity's Last Exam
Knowledge
A broad, expert-authored multimodal academic benchmark at the frontier of human knowledge.
- Interaction
- None · static
- Scale
- 2,500 questions
- Metric
- Accuracy
- Structure
- 3 components
Kotlin Benchmark
Kotlin Benchmark — First Public Iteration
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 8 repositories
- Metric
- Resolved Rate
- Structure
- Single set
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2026-06-25.
AMPS HardCode GenerationConnectionsConsecutive EventsJavaScriptParaphraseTheory of MindCode CompletionIntegrals with GamePlot UnscramblingSimplifyTable JoinTypeScriptZebra PuzzleMath CompetitionPythonSpatialStory GenerationTable ReformatTyposLogic with NavigationOlympiadSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 7 components
30applicable models →2026-06-25
LiveBench
LiveBench
Rolling evaluation
A rolling benchmark refreshed with objective, verifiable questions across six categories to reduce contamination and avoid LLM-as-judge scoring.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2026-01-08.
AMPS HardCode GenerationConnectionsConsecutive EventsJavaScriptParaphraseTheory of MindCode CompletionIntegrals with GamePlot UnscramblingSimplifyTable JoinTypeScriptZebra PuzzleMath CompetitionPythonSpatialStory GenerationTable ReformatTyposLogic with NavigationOlympiadSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 7 components
30applicable models →2026-01-08
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2025-12-23.
AMPS HardCode GenerationConnectionsJavaScriptParaphraseTable JoinTheory of MindCode CompletionMath CompetitionPlot UnscramblingSimplifyTable ReformatTypeScriptZebra PuzzleOlympiadPythonSpatialStory GenerationTyposLogic with NavigationSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 7 components
30applicable models →2025-12-23
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2025-11-25.
AMPS HardCode GenerationConnectionsJavaScriptParaphraseTable JoinTheory of MindCode CompletionMath CompetitionPlot UnscramblingSimplifyTable ReformatTypeScriptZebra PuzzleOlympiadPythonSpatialStory GenerationTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 7 components
30applicable models →2025-11-25
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2025-05-30.
AMPS HardCode GenerationConnectionsJavaScriptParaphraseTable JoinWeb of Lies v3Code CompletionMath CompetitionPlot UnscramblingSimplifyTable ReformatTypeScriptZebra PuzzleOlympiadPythonSpatialStory GenerationTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 7 components
30applicable models →2025-05-30
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2025-04-25.
AMPS HardCode GenerationConnectionsParaphraseTable JoinWeb of Lies v3Code CompletionMath CompetitionPlot UnscramblingSimplifyTable ReformatZebra PuzzleOlympiadSpatialStory GenerationTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 6 components
30applicable models →2025-04-25
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2025-04-02.
AMPS HardConnectionsCTACode Generation (LCB)ParaphraseWeb of Lies v3Coding CompletionMath CompetitionPlot UnscramblingSimplifyTable JoinZebra PuzzleOlympiadSpatialStory GenerationTable ReformatTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 6 components
30applicable models →2025-04-02
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2024-11-25.
AMPS HardConnectionsCTACode Generation (LCB)ParaphraseWeb of Lies v2Coding CompletionMath CompetitionPlot UnscramblingSimplifyTable JoinZebra PuzzleOlympiadSpatialStory GenerationTable ReformatTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 6 components
30applicable models →2024-11-25
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2024-08-31.
AMPS HardConnectionsCTACode Generation (LCB)ParaphraseWeb of Lies v2Coding CompletionMath CompetitionPlot UnscramblingSimplifyTable JoinZebra PuzzleOlympiadSpatialStory GenerationTable ReformatTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 6 components
30applicable models →2024-08-31
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2024-07-26.
AMPS HardConnectionsCTACode Generation (LCB)ParaphraseWeb of Lies v2Coding CompletionMath CompetitionPlot UnscramblingSimplifyTable JoinZebra PuzzleOlympiadSpatialStory GenerationTable ReformatTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 6 components
30applicable models →2024-07-26
LiveBench
LiveBench
Rolling evaluation
Official LiveBench release dated 2024-06-24.
AMPS HardConnectionsCTACode Generation (LCB)ParaphraseWeb of Lies v2Coding CompletionMath CompetitionPlot UnscramblingSimplifyTable JoinZebra PuzzleOlympiadStory GenerationTable ReformatTyposSummarize
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- 6 components
30applicable models →2024-06-24
LiveCodeBench
LiveCodeBench
Code generation
- Interaction
- None · static
- Scale
- 400 problems
- Metric
- Pass@k
- Structure
- 4 scenarios
3applicable models →release_v1
LiveCodeBench
LiveCodeBench
Code generation
- Interaction
- None · static
- Scale
- 511 problems
- Metric
- Pass@k
- Structure
- 4 scenarios
3applicable models →release_v2
LiveCodeBench
LiveCodeBench
Code generation
- Interaction
- None · static
- Scale
- 612 problems
- Metric
- Pass@k
- Structure
- 4 scenarios
3applicable models →release_v3
LiveCodeBench
LiveCodeBench
Code generation
- Interaction
- None · static
- Scale
- 713 problems
- Metric
- Pass@k
- Structure
- 4 scenarios
3applicable models →release_v4
LiveCodeBench
LiveCodeBench
Code generation
- Interaction
- None · static
- Scale
- 880 problems
- Metric
- Pass@k
- Structure
- 4 scenarios
3applicable models →release_v5
LiveCodeBench
LiveCodeBench
Code generation
- Interaction
- None · static
- Scale
- 1,055 problems
- Metric
- Pass@k
- Structure
- 4 scenarios
3applicable models →release_v6
No family
MATH
Mathematics
A 12,500-problem competition mathematics benchmark with full step-by-step solutions.
- Interaction
- None · static
- Scale
- 12,500 problems
- Metric
- Exact Match
- Structure
- Single set
MathArena ArXivLean
MathArena ArXivLean
Formal reasoning
MathArena ArXivLean 06/2026, containing 46 problems.
- Interaction
- None · static
- Scale
- 46 problems
- Metric
- Formal Proof Acceptance
- Structure
- Single set
MathArena ArXivLean
MathArena ArXivLean
Formal reasoning
MathArena ArXivLean 03/2026, containing 41 problems.
- Interaction
- None · static
- Scale
- 41 problems
- Metric
- Formal Proof Acceptance
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 05/2026, containing 40 problems.
- Interaction
- None · static
- Scale
- 40 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 06/2026, containing 48 problems.
- Interaction
- None · static
- Scale
- 48 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 08/2026, containing 57 problems.
- Interaction
- None · static
- Scale
- 57 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 12/2025, containing 17 problems.
- Interaction
- None · static
- Scale
- 17 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 01/2026, containing 23 problems.
- Interaction
- None · static
- Scale
- 23 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 02/2026, containing 32 problems.
- Interaction
- None · static
- Scale
- 32 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 03/2026, containing 30 problems.
- Interaction
- None · static
- Scale
- 30 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena ArXivMath
MathArena ArXivMath
Research mathematics
MathArena ArXivMath 04/2026, containing 40 problems.
- Interaction
- None · static
- Scale
- 40 problems
- Metric
- Accuracy
- Structure
- Single set
MathArena BrokenArXiv
MathArena BrokenArXiv
Research mathematics
MathArena BrokenArXiv 05/2026, containing 50 problems.
- Interaction
- None · static
- Scale
- 50 problems
- Metric
- Mean judged score
- Structure
- Single set
MathArena BrokenArXiv
MathArena BrokenArXiv
Research mathematics
MathArena BrokenArXiv 06/2026, containing 54 problems.
- Interaction
- None · static
- Scale
- 54 problems
- Metric
- Mean judged score
- Structure
- Single set
MathArena BrokenArXiv
MathArena BrokenArXiv
Research mathematics
MathArena BrokenArXiv 08/2026, containing 56 problems.
- Interaction
- None · static
- Scale
- 56 problems
- Metric
- Mean judged score
- Structure
- Single set
MathArena BrokenArXiv
MathArena BrokenArXiv
Research mathematics
MathArena BrokenArXiv 02/2026, containing 31 problems.
- Interaction
- None · static
- Scale
- 31 problems
- Metric
- Mean judged score
- Structure
- Single set
MathArena BrokenArXiv
MathArena BrokenArXiv
Research mathematics
MathArena BrokenArXiv 03/2026, containing 56 problems.
- Interaction
- None · static
- Scale
- 56 problems
- Metric
- Mean judged score
- Structure
- Single set
MathArena BrokenArXiv
MathArena BrokenArXiv
Research mathematics
MathArena BrokenArXiv 04/2026, containing 61 problems.
- Interaction
- None · static
- Scale
- 61 problems
- Metric
- Mean judged score
- Structure
- Single set
MBPP
MBPP
Code generation
- Interaction
- None · static
- Scale
- 974 problems
- Metric
- Pass@k
- Structure
- Single set
EvalPlus Benchmarks
MBPP+
Code generation
- Interaction
- None · static
- Scale
- 378 problems
- Metric
- Pass@k
- Structure
- Single set
GSM8K and multilingual derivatives
MGSM
Mathematics
A multilingual grade-school math benchmark created by manually translating 250 GSM8K problems into 10 languages.
- Interaction
- None · static
- Scale
- 250 problems
- Metric
- Exact Match
- Structure
- Single set
Mind2Web
Mind2Web
Web Agent
Web NavigationWeb State Modification
- Interaction
- None · static
- Scale
- 912 domains
- Metric
- Accuracy
- Structure
- 4 components
MiniWoB
MiniWoB
Web Agent
Web NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 100 tasks
- Metric
- Task Success Rate
- Structure
- Single set
MiniWoB
MiniWoB++
Web Agent
Web NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 125 tasks
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →maintained
MMLU
MMLU
Knowledge
A 57-task multiple-choice benchmark covering broad academic and professional knowledge.
Multiple-choice academic/professional question answering
- Interaction
- None · static
- Scale
- 57 tasks
- Metric
- Accuracy
- Structure
- 1 component
MMLU
MMLU-Pro
Knowledge
A harder, more reasoning-focused successor/derivative of MMLU with 10 answer choices across 14 domains.
Ten-option academic question answering
- Interaction
- None · static
- Scale
- 12,032 questions
- Metric
- Accuracy
- Structure
- 1 component
MiniWoB
MobileMiniWoB++
Mobile Agent
Computer UseMobile Gui Interaction
- Interaction
- Mobile GUI
- Scale
- 62 standard evaluation subset
- Metric
- Task Success Rate
- Structure
- 1 component
51applicable models →2024 mobile adaptation
Mtac Ifbench
MTAC-IFBench
Software Engineering
Multi Turn Software Instruction Following
- Interaction
- None · static
- Scale
- 91.33 instances
- Metric
- Constraint Following Rate
- Structure
- Single set
Mtac Ifbench
MTAC-IFBench Lite
Software Engineering
Multi Turn Software Instruction Following
- Interaction
- None · static
- Scale
- 20 tasks
- Metric
- Constraint Following Rate
- Structure
- Single set
Multi Swe Bench
Multi-SWE-bench
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 2,132 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Multi Swe Bench
Multi-SWE-bench Flash
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 300 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Multi Swe Bench
Multi-SWE-bench Mini
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 50 languages
- Metric
- Resolved Rate
- Structure
- Single set
MultiPL-E
MultiPL-E HumanEval
Code generation
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Pass@k
- Structure
- Single set
MultiPL-E
MultiPL-E MBPP
Code generation
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Pass@k
- Structure
- Single set
No family
MuSR
General reasoning
A multistep soft-reasoning benchmark built from natural-language narratives such as murder mysteries and other realistic reasoning domains.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
No family
OlympiadBench
Mathematics
A bilingual multimodal benchmark of 8,476 olympiad-level mathematics and physics problems.
- Interaction
- None · static
- Scale
- 8,476 problems
- Metric
- Accuracy
- Structure
- 2 components
Mind2Web
Online-Mind2Web
Web Agent
Web NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 300 tasks
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →2025 rolling
No family
OpenBookQA
Science reasoning
An elementary-science open-book question-answering benchmark requiring a core science fact plus additional common knowledge.
- Interaction
- None · static
- Scale
- 5,957 questions
- Metric
- Accuracy
- Structure
- Single set
Os Bench
OS-bench
Software Engineering
Cross Platform Software Migration
- Interaction
- None · static
- Scale
- 140 tasks
- Metric
- Resolved Rate
- Structure
- Single set
OSUniverse
OSUniverse
Computer Use
Computer UseDesktop Gui InteractionMulti Application Workflow
- Interaction
- Desktop GUI
- Scale
- 9 categories
- Metric
- Task Success Rate
- Structure
- Single set
OSWorld
OSWorld
Computer Use
Computer UseDesktop Gui InteractionMulti Application Workflow
- Interaction
- Desktop GUI
- Scale
- 43 tasks
- Metric
- Task Success Rate
- Structure
- 3 scenarios
OSWorld
OSWorld 2.0 — 2026.06.24
Computer Use
Computer UseDesktop Gui InteractionMulti Application WorkflowLong Horizon Computer Use
- Interaction
- Desktop GUI
- Scale
- 27.25 checkpoints
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →2026.06.24
OSWorld
OSWorld 2.0 — 2026.08.08
Computer Use
Computer UseDesktop Gui InteractionMulti Application WorkflowLong Horizon Computer Use
- Interaction
- Desktop GUI
- Scale
- 31 tasks
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →2026.08.08
OSWorld
OSWorld V2.1
Computer Use
Computer UseDesktop Gui InteractionMulti Application WorkflowLong Horizon Computer Use
- Interaction
- Desktop GUI
- Scale
- 108 tasks
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →V2.1 / 2026-09-16
OSWorld
OSWorld-Pro
Computer Use
Computer UseDesktop Gui InteractionLong Horizon Computer Use
- Interaction
- Desktop GUI
- Scale
- 67,264 annotations
- Metric
- Task Success Rate
- Structure
- Single set
OSWorld
OSWorld-Verified
Computer Use
Computer UseDesktop Gui InteractionMulti Application Workflow
- Interaction
- Desktop GUI
- Scale
- 361 no google drive subset
- Metric
- Task Success Rate
- Structure
- 1 component
No family
PIQA
Commonsense
A binary-choice benchmark for physical commonsense reasoning in natural language.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
Programbench
ProgramBench
Software Engineering
Black Box Program Reimplementation
- Interaction
- None · static
- Scale
- 200 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Race Bench
RACE-Bench
Software Engineering
Feature Implementation
- Interaction
- None · static
- Scale
- 12 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Recode H
RECODE-H
Software Engineering
Research Code Implementation
- Interaction
- None · static
- Scale
- 102 tasks
- Metric
- Test Pass Rate
- Structure
- 7 scenarios
Refactorbench
RefactorBench
Software Engineering
Code Refactoring
- Interaction
- None · static
- Scale
- 9 repositories
- Metric
- Resolved Rate
- Structure
- 3 scenarios
RepoBench
RepoBench
Code completion
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Exact Match
- Structure
- 2 components, 3 scenarios
RepoBench
RepoBench
Code completion
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Not recorded
- Structure
- Single set
Reporeason
RepoReason
Software Engineering
Repository Abductive Assertion Verification
- Interaction
- None · static
- Scale
- 7 repositories
- Metric
- Accuracy
- Structure
- Single set
Rubench
RuBench 1.0
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 5 repositories
- Metric
- Resolved Rate
- Structure
- Single set
No family
SimpleBench
Commonsense
A text multiple-choice benchmark of over 200 everyday reasoning questions where non-specialized human performance was intended to remain stronger than frontier models at introduction.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
No family
SimpleQA
Factuality
A 4,326-question benchmark for short-form factuality on stable, fact-seeking questions with a single verifiable answer.
- Interaction
- None · static
- Scale
- 4,326 questions
- Metric
- Accuracy
- Structure
- Single set
WebArena
ST-WebAgentBench
Web Agent
Web NavigationWeb State ModificationPolicy Constrained Web Interaction
- Interaction
- Web browser
- Scale
- 222 historical population
- Metric
- Completion Under Policy
- Structure
- Single set
51applicable models →ICLR 2026
ToolBench
StableToolBench
Tool Use
Open Domain Tool Selection And InvocationMulti Tool CompositionTool Retrieval
- Interaction
- None · static
- Scale
- 153 g1 category
- Metric
- Task Success Rate
- Structure
- Single set
No family
StrategyQA
General reasoning
A 2,780-example question-answering benchmark where the multi-step reasoning strategy is implicit rather than spelled out in the question.
- Interaction
- None · static
- Scale
- 2,780 examples
- Metric
- Accuracy
- Structure
- Single set
Swe Refactor Bench
SWE Refactor Bench
Software Engineering
Whole Repository Stack Migration
- Interaction
- None · static
- Scale
- 20 tasks
- Metric
- Migration Composite Score
- Structure
- Single set
Swe Bench
SWE-bench
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 12 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench
SWE-bench Lite
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 23 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench
SWE-bench Multilingual
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 42 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Multimodal
SWE-bench Multimodal
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 102 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Multimodal
SWE-bench Multimodal v2
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 100 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Pro
SWE-bench Pro
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 41 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Pro
SWE-Bench Pro Verified
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 102 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Promax
SWE-Bench ProMax v1
Software Engineering
Code Refactoring
- Interaction
- None · static
- Scale
- 70 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Science
SWE-bench Science
Software Engineering
Issue ResolutionScientific Software Diagnosis
- Interaction
- None · static
- Scale
- 96 default unrestricted
- Metric
- Resolved Rate
- Structure
- 5 scenarios
Swe Bench
SWE-bench Verified
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 500 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Live
SWE-bench-Live
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 1,888 full
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Live
SWE-bench-Live Lite
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 300 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Live
SWE-bench-Live Verified
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 500 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Live
SWE-bench-Live/MultiLang
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 431 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Bench Live
SWE-bench-Live/Windows
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 9 languages
- Metric
- Resolved Rate
- Structure
- Single set
Swe Explore
SWE-Explore
Software Engineering
Repository Exploration And Localization
- Interaction
- None · static
- Scale
- 848 issues
- Metric
- Precision
- Structure
- Single set
Swe Fficiency
SWE-fficiency
Software Engineering
Repository Performance Optimization
- Interaction
- None · static
- Scale
- 9 repositories
- Metric
- Speedup Ratio
- Structure
- Single set
Swe Gate
SWE-Gate
Software Engineering
Issue ResolutionReview Constraint Compliance
- Interaction
- None · static
- Scale
- 75 repositories
- Metric
- Resolved Rate
- Structure
- 2 scenarios
Swe Marathon
SWE-Marathon v1.0
Software Engineering
Project Scale Technical Engineering
- Interaction
- None · static
- Scale
- 20 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Marathon
SWE-Marathon v1.1
Software Engineering
Project Scale Technical Engineering
- Interaction
- None · static
- Scale
- 20 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Perf
SWE-Perf
Software Engineering
Repository Performance Optimization
- Interaction
- None · static
- Scale
- 9 repositories
- Metric
- Patch Apply Rate
- Structure
- 2 scenarios
Swe Polybench
SWE-PolyBench
Software Engineering
Bug FixFeature ImplementationCode Refactoring
- Interaction
- None · static
- Scale
- 21 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Polybench
SWE-PolyBench Verified
Software Engineering
Bug FixFeature ImplementationCode Refactoring
- Interaction
- None · static
- Scale
- 382 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Polybench
SWE-PolyBench500
Software Engineering
Bug FixFeature ImplementationCode Refactoring
- Interaction
- None · static
- Scale
- 500 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Swe Polyvision
SWE-PolyVision v1.0
Software Engineering
Issue ResolutionCross Image Repository Abductive Reasoning
- Interaction
- None · static
- Scale
- 92 tasks
- Metric
- Resolved Rate
- Structure
- 3 scenarios
Swe Prometheus
SWE-Prometheus v1.0
Software Engineering
Repository Governance Retrofit
- Interaction
- None · static
- Scale
- 6 governance dimensions
- Metric
- Normalized Governance Improvement
- Structure
- Single set
Swe Proof
SWE-Proof
Software Engineering
Issue Resolution
- Interaction
- None · static
- Scale
- 500 tasks
- Metric
- Formally Verified Resolution Rate
- Structure
- 3 scenarios
Swe Refactor
SWE-Refactor
Software Engineering
Code Refactoring
- Interaction
- None · static
- Scale
- 922 atomic
- Metric
- Refactoring Success Rate
- Structure
- Single set
Swe Rpg
SWE-RPG
Software Engineering
Issue ResolutionRequirement ClarificationRepository Implementation Planning
- Interaction
- None · static
- Scale
- 31 repositories
- Metric
- Resolved Rate
- Structure
- Single set
Swe Together
SWE-Together
Software Engineering
Interactive Repository Software Engineering
- Interaction
- None · static
- Scale
- 109 tasks
- Metric
- Resolved Rate
- Structure
- Single set
Terminal-Bench
Terminal-Bench 1.0
Terminal Agent
Terminal Use
- Interaction
- Terminal
- Scale
- 80 tasks
- Metric
- Accuracy
- Structure
- Single set
Terminal-Bench
Terminal-Bench 2.0
Terminal Agent
Terminal Use
- Interaction
- Terminal
- Scale
- 89 tasks
- Metric
- Accuracy
- Structure
- Single set
Terminal-Bench
Terminal-Bench 2.1
Terminal Agent
Terminal Use
- Interaction
- Terminal
- Scale
- 89 tasks
- Metric
- Accuracy
- Structure
- Single set
Terminal-Bench
Terminal-Bench 3.0
Terminal Agent
Terminal Use
- Interaction
- Terminal
- Scale
- 74 tasks
- Metric
- Accuracy
- Structure
- Single set
Terminal-Bench
Terminal-Bench 4.0
Terminal Agent
Terminal Use
- Interaction
- Terminal
- Scale
- 66 tasks
- Metric
- Accuracy
- Structure
- Single set
Terminal-Bench-Science
Terminal-Bench-Science 0.1
Terminal Agent
Terminal Use
- Interaction
- Terminal
- Scale
- 70 tasks
- Metric
- Accuracy
- Structure
- Single set
TheAgentCompany
TheAgentCompany
Knowledge Work
Computer UseMulti Application WorkflowWorkplace Collaboration
- Interaction
- Mixed
- Scale
- 167 tasks
- Metric
- Task Success Rate
- Structure
- 8 components
ToolBench
ToolBench
Tool Use
Open Domain Tool Selection And InvocationMulti Tool CompositionTool Retrieval
- Interaction
- None · static
- Scale
- 469,585 apis
- Metric
- Task Success Rate
- Structure
- Single set
ToolSandbox
ToolSandbox
Tool Use
Stateful Conversational Tool UseTool State Dependency ReasoningTool Input CanonicalizationInsufficient Information Handling
- Interaction
- None · static
- Scale
- 3.8 tools
- Metric
- Milestone Similarity
- Structure
- 4 scenarios
TruthfulQA
TruthfulQA
Truthfulness
An 817-question benchmark designed to test whether language models repeat common human misconceptions and falsehoods.
- Interaction
- None · static
- Scale
- 817 questions
- Metric
- Truthful Rate
- Structure
- Single set
TruthfulQA
TruthfulQA Improved Multiple Choice
Truthfulness
The January 2025 improved multiple-choice TruthfulQA variant using the best correct and best incorrect answer for each question.
- Interaction
- None · static
- Scale
- Not recorded
- Metric
- Accuracy
- Structure
- Single set
TUA-Bench
TUA-Bench
Terminal Agent
Terminal UseComputer Use
- Interaction
- Terminal
- Scale
- 100 tasks
- Metric
- Task Success Rate
- Structure
- 7 components
WebArena
VisualWebArena
Web Agent
Web Information RetrievalWeb NavigationWeb State ModificationVisually Grounded Web Interaction
- Interaction
- Web browser
- Scale
- 3 sites
- Metric
- Task Success Rate
- Structure
- 3 scenarios
WebArena
WebArena
Web Agent
Web Information RetrievalWeb NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 241 intent templates
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →named benchmark
WebArena
WebArena-Pro
Web Agent
Web Information RetrievalWeb NavigationWeb State ModificationLong Horizon Computer Use
- Interaction
- Web browser
- Scale
- 20 applications
- Metric
- Task Success Rate
- Structure
- Single set
WebArena
WebArena-Verified
Web Agent
Web Information RetrievalWeb NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 812 tasks
- Metric
- Task Success Rate
- Structure
- Single set
WebArena
WebArena-Verified Hard
Web Agent
Web NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 137 tasks
- Metric
- Task Success Rate
- Structure
- Single set
51applicable models →current hard subset
WebArena
WebChoreArena
Web Agent
Web Information RetrievalWeb NavigationWeb State ModificationLong Horizon Computer Use
- Interaction
- Web browser
- Scale
- 532 tasks
- Metric
- Task Success Rate
- Structure
- Single set
WebVoyager
WebVoyager
Web Agent
Web Information RetrievalWeb NavigationWeb State ModificationVisually Grounded Web Interaction
- Interaction
- Web browser
- Scale
- 15 sites
- Metric
- Task Success Rate
- Structure
- Single set
No family
WinoGrande
Commonsense
A large adversarial Winograd-style benchmark for commonsense reasoning and referent disambiguation.
- Interaction
- None · static
- Scale
- 44,000 problems
- Metric
- Accuracy
- Structure
- Single set
WorkArena
WorkArena L1
Web Agent
Web NavigationWeb State Modification
- Interaction
- Web browser
- Scale
- 33 tasks
- Metric
- Task Success Rate
- Structure
- Single set
WorkArena
WorkArena++
Web Agent
Web NavigationWeb State ModificationInfeasibility Handling
- Interaction
- Web browser
- Scale
- 341 tasks
- Metric
- Task Success Rate
- Structure
- 2 components
τ-bench
τ-bench
Tool Use
Policy Constrained Conversational Tool Use
- Interaction
- None · static
- Scale
- 50 airline
- Metric
- Pass K Reliability
- Structure
- Single set
τ-bench
τ-Knowledge / τ-Banking
Tool Use
Knowledge Grounded Conversational Tool Use
- Interaction
- None · static
- Scale
- 51 tools
- Metric
- Pass K Reliability
- Structure
- Single set
τ-bench
τ-Voice
Tool Use
Full Duplex Voice Task ExecutionPolicy Constrained Conversational Tool Use
- Interaction
- None · static
- Scale
- 50 airline
- Metric
- Pass K Reliability
- Structure
- Single set
τ-bench
τ^τ-bench / Hyper-τ
Agent Construction
End To End Agent ConstructionRequirement Clarification
- Interaction
- None · static
- Scale
- 6 airline plus
- Metric
- Budget Adjusted Reward
- Structure
- Single set
τ-bench
τ²-bench
Tool Use
Policy Constrained Conversational Tool UseDual Control Shared State Coordination
- Interaction
- None · static
- Scale
- 50 airline
- Metric
- Pass K Reliability
- Structure
- Single set
τ-bench
τ³-bench v1.0.0
Tool Use
Policy Constrained Conversational Tool UseDual Control Shared State CoordinationKnowledge Grounded Conversational Tool Use
- Interaction
- None · static
- Scale
- 50 airline
- Metric
- Pass K Reliability
- Structure
- Single set
τ-bench
τ³-bench v1.0.1
Tool Use
Policy Constrained Conversational Tool UseDual Control Shared State CoordinationKnowledge Grounded Conversational Tool Use
- Interaction
- None · static
- Scale
- 50 airline
- Metric
- Pass K Reliability
- Structure
- Single set
Applicable models counts current models whose recorded capabilities meet every required condition, including those that qualify for only some components or scenarios. A benchmark that records no requirements shows 0: its models are not recorded, not refused. Families group versions of one benchmark; a family is not evaluated itself.