Benchmark definition

Humanity's Last Exam

ActiveEstablishedCommon useHLE

A broad, expert-authored multimodal academic benchmark at the frontier of human knowledge.

Applicability does not mean evaluated; benchmark result/model score data is outside this UI contract.

Compare with others

About

Description: HLE contains multiple-choice and short-answer questions across mathematics, humanities, natural sciences, and many other subjects, authored and reviewed by subject-matter experts.
What it measures: Expert-level closed-ended academic knowledge and reasoning across broad subjects, including multimodal items.
Comparison guidance: Not published
Intended difficulty: Not published

Not recorded for this benchmark: what it does not measure, why it matters, motivation, gap addressed.

Classification

Categories: KnowledgeGeneral reasoningMultimodal
Domains: General academicMathematicsHumanitiesScience
Evaluation style: Model judgedStatic Q&A

Structure

3 components. Applicability can differ between them.

Components

All questionsSectionAll finalized HLE questions.
Text-onlySubsetQuestions answerable from text alone.
Image-dependentSubsetQuestions outside the text-only split and therefore requiring image/figure understanding in addition to text.

Tasks

None recorded

Metrics

What a result on this benchmark would be reported as. ctxdex does not carry results.

Metric	Role	Direction	Unit
Accuracy	Primary	Higher is better	Percent
Calibration Error	Secondary	Lower is better	Percent

Methodology

Official HLE simple evaluationThis definition

Official repository uses temperature 0, large completion budgets for reasoning models, and a judge step; reported metrics include accuracy and calibration error.

Evaluation environment

What the benchmark's harness provides and allows. These are properties of the environment, not model requirements, and applicability does not test for them.

Interaction: None. Static prompts and answers.
Evaluation style: Model judged, Static Q&A
Environment summary: Not published

Model requirements

Required conditions decide applicability. Relevant ones describe what the benchmark exercises and decide nothing.

Required · decides applicability

Whole benchmark

Text (output)Complex reasoning

Component: Image-dependent

Image (input)Text (input)Text (output)

Component: All questions

Text (input)Text (output)

Component: Text-only

Text (input)Text (output)

Relevant · informational

Multimodal understandingExplicit reasoningGeneral purpose modelsReasoning models

CTXDEX models

13 of the 58 active, preview, experimental models meet this benchmark’s recorded requirements, at some scope. Applicable means the requirements are met, not that the model was evaluated.

View all 13 applicable models →

Applicable · full: 7
Applicable · components: 6
Applicable · scenarios: 0
Conditionally applicable: 0
Not recorded: 45

Counted over 58 models in the default lifecycle scope (active, preview, experimental).

Every required condition, for the whole benchmark, is recorded as supported. Each line is one recorded condition and why it counts.

Ctxdex Efficient 3
Ctxdex Efficient
Full benchmark
Every one of the 9 required conditions is recorded as supported.
- MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output) — benchmark, All questions, Text-only, Image-dependent
- MODEL_TASK:COMPLEX_REASONINGthe catalogue records SUPPORTED for COMPLEX_REASONING
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input) — All questions, Text-only, Image-dependent
- MODALITY:INPUT:IMAGEthe catalogue records SUPPORTED for IMAGE (input) — Image-dependent
Ctxdex Enterprise 4
Ctxdex Enterprise
Full benchmark
Every one of the 9 required conditions is recorded as supported.
- MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output) — benchmark, All questions, Text-only, Image-dependent
- MODEL_TASK:COMPLEX_REASONINGthe catalogue records SUPPORTED for COMPLEX_REASONING
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input) — All questions, Text-only, Image-dependent
- MODALITY:INPUT:IMAGEthe catalogue records SUPPORTED for IMAGE (input) — Image-dependent

The whole benchmark is not resolved for these models, but one or more components are. The benchmark state is not a summary of its components, so both are shown.

Ctxdex Chintana 1 Large
Ctxdex Efficient Reasoning
Components only
The full benchmark is not resolved for this subject, and 2 component(s) are.
Applies to All questions, Text-only. Not recorded for Image-dependent.
- MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output) — benchmark, All questions, Text-only, Image-dependent
- MODEL_TASK:COMPLEX_REASONINGthe catalogue records SUPPORTED for COMPLEX_REASONING
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input) — All questions, Text-only, Image-dependent
Ctxdex Chintana 1 Mini
Ctxdex Efficient Reasoning
Components only
The full benchmark is not resolved for this subject, and 2 component(s) are.
Applies to All questions, Text-only. Not recorded for Image-dependent.
- MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output) — benchmark, All questions, Text-only, Image-dependent
- MODEL_TASK:COMPLEX_REASONINGthe catalogue records SUPPORTED for COMPLEX_REASONING
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input) — All questions, Text-only, Image-dependent

Preview holds 5 of 6. View all →

Applicability/relevance ordering only; never a performance ranking.

Scale

342
Question
Image-dependent · Derived exact
2,500
Question
All questions · Exact
2,158
Question
Text-only · Exact
2,500
Question
Finalized benchmark · Exact

Data and access

Dataset: Open
Evaluation code: Open source
Unknown: Leaderboard, Peer review, Maintenance

Coverage

Languages, regions: None recorded
Coverage summary: Not published

Limitations

None recorded

Lineage

Family: None recorded
Related: None recorded
Changes: 2025-04-03HLE was finalized at 2,500 questions. Questions flagged through the bug-bounty process and searchable questions were removed and replaced.

Sources

Humanity's Last Exam official site ↗Official website · Benchmark official
Official HLE repository ↗Official repository · Benchmark official
HLE environment README ↗Documentation · Benchmark official
HLE paper ↗Research paper · Original creator

Freshness

Content version: 1.0
Last reviewed: 2026-09-22
Refresh model: Static
Time sensitivity: Low
Benchmark data: c40cccac878af5d0

13 contract fields are empty for every benchmark in this release and are not shown: applicability groups, conditional requirements, feature requirements, model relevance domains, required capabilities, openness summary, reference artifacts, interaction protocols, tool protocols, benchmark last major refresh at, short name, target expertise level, target population.