ctxdexSynthetic model data for learning

Benchmark definition

MathArena BrokenArXiv

03/2026
DeprecatedEstablishedSpecialized useMATHARENA_BROKENARXIV_2026_03

MathArena BrokenArXiv 03/2026, containing 56 problems.

Applicability does not mean evaluated; benchmark result/model score data is outside this UI contract.

Compare with others

About

Description
Plausible but false research-mathematics statements derived from recent arXiv papers; models are rewarded for recognizing the false premise.
What it measures
Plausible but false research-mathematics statements derived from recent arXiv papers; models are rewarded for recognizing the false premise.
Why it matters
The release is separately identified by MathArena and must not be merged with other monthly competition sets.
Comparison guidance
Not published
Intended difficulty
Not published

Not recorded for this benchmark: what it does not measure, motivation, gap addressed.

Classification

Categories
Research mathematicsRolling evaluationGeneral reasoning
Domains
Research mathematics
Evaluation style
Model judged

Structure

One undivided item set. No components or scenarios recorded.

Tasks

None recorded

Metrics

What a result on this benchmark would be reported as. ctxdex does not carry results.

MetricRoleDirectionUnit
Mean judged score
PrimaryHigher is betterScore

Methodology

MathArena BrokenArXiv 03/2026 methodologyThis definition

Release-specific MathArena methodology; preserve this monthly identity rather than applying one timeless family methodology.

Evaluation environment

What the benchmark's harness provides and allows. These are properties of the environment, not model requirements, and applicability does not test for them.

Interaction

None. Static prompts and answers.

Evaluation style
Model judged
Environment summary
Not published

Model requirements

Required conditions decide applicability. Relevant ones describe what the benchmark exercises and decide nothing.

Required · decides applicability

Whole benchmark
Text (input)Text (output)Complex reasoning

Relevant · informational

Explicit reasoningGeneral purpose modelsReasoning models

CTXDEX models

13 of the 58 active, preview, experimental models meet this benchmark’s recorded requirements, at some scope. Applicable means the requirements are met, not that the model was evaluated.

View all 13 applicable models →
Applicable · full
13
Applicable · components
0
Applicable · scenarios
0
Conditionally applicable
0
Not recorded
45

Counted over 58 models in the default lifecycle scope (active, preview, experimental).

Every required condition, for the whole benchmark, is recorded as supported. Each line is one recorded condition and why it counts.

  • Ctxdex Chintana 1 Large
    Ctxdex Efficient Reasoning
    Full benchmark

    Every one of the 3 required conditions is recorded as supported.

    • MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
    • MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output)
    • MODEL_TASK:COMPLEX_REASONINGthe catalogue records SUPPORTED for COMPLEX_REASONING
  • Ctxdex Chintana 1 Mini
    Ctxdex Efficient Reasoning
    Full benchmark

    Every one of the 3 required conditions is recorded as supported.

    • MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
    • MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output)
    • MODEL_TASK:COMPLEX_REASONINGthe catalogue records SUPPORTED for COMPLEX_REASONING
Preview holds 12 of 13. View all →

Applicability/relevance ordering only; never a performance ranking.

Scale

  • 56
    Problem
    Total · Exact

Data and access

Dataset
Open
Evaluation code
Open source
Peer review
Technical report
Unknown
Leaderboard, Maintenance

Coverage

Languages, regions

None recorded

Coverage summary
Not published

Limitations

None recorded

Lineage

Family
FamilyMathArena BrokenArXiv
Related
Inverse of successor of BenchmarkMathArena BrokenArXiv
Successor of BenchmarkMathArena BrokenArXiv
Other definitions in family
MathArena BrokenArXiv 02/2026 DeprecatedMathArena BrokenArXiv 04/2026 DeprecatedMathArena BrokenArXiv 05/2026 ActiveMathArena BrokenArXiv 06/2026 ActiveMathArena BrokenArXiv 08/2026 Active
Changes

None recorded

Sources

Freshness

Content version
1.0
Last reviewed
2026-09-22
Refresh model
Periodic
Time sensitivity
High
Benchmark data
c40cccac878af5d0

13 contract fields are empty for every benchmark in this release and are not shown: applicability groups, conditional requirements, feature requirements, model relevance domains, required capabilities, openness summary, reference artifacts, interaction protocols, tool protocols, benchmark last major refresh at, short name, target expertise level, target population.