Benchmark definition

LiveCodeBench

release_v1

ActiveLIVECODEBENCH_RELEASE_V1

Applicability does not mean evaluated; benchmark result/model score data is outside this UI contract.

Compare with others

About

Description: LiveCodeBench
What it measures: LiveCodeBench
Comparison guidance: Not published
Intended difficulty: Not published

Not recorded for this benchmark: what it does not measure, why it matters, motivation, gap addressed.

Classification

Categories: Code generationRolling evaluation
Domains: Computer science
Evaluation style: None recorded

Structure

4 scenarios. Applicability can differ between them.

Scenarios

Code Generation
Self Repair
Test Output Prediction
Code Execution

Tasks

None recorded

Metrics

What a result on this benchmark would be reported as. ctxdex does not carry results.

Metric	Role	Direction	Unit
Pass@k Probability/estimate that at least one of k generated solutions passes the benchmark correctness tests.	Primary	Higher is better	Percent
Pass@k Probability/estimate that at least one of k generated solutions passes the benchmark correctness tests.	Secondary	Higher is better	Percent

Methodology

Canonical methodology noteThis definition

Release version and scenario are independent evaluator dimensions.

Source lane: L1

Canonical methodology noteThis definition

release_latest is a floating alias, not a BenchmarkDefinition.

Source lane: L1

Canonical methodology noteThis definition

Fine-grained selectors such as v1_v3/v4_v5 are dataset windows/selectors unless independently versioned.

Source lane: L1

Evaluation environment

What the benchmark's harness provides and allows. These are properties of the environment, not model requirements, and applicability does not test for them.

Interaction: None. Static prompts and answers.
Evaluation style: Not recorded
Environment summary: Not published

Model requirements

Required conditions decide applicability. Relevant ones describe what the benchmark exercises and decide nothing.

Required · decides applicability

Whole benchmark

Text (input)Text (output)Source code (output)

Scenario: Self Repair

Code editing

Scenario: Test Output Prediction

Code execution reasoning

Scenario: Code Execution

Code execution reasoning

Scenario: Code Generation

Code generation

Relevant · informational

Coding models

CTXDEX models

3 of the 58 active, preview, experimental models meet this benchmark’s recorded requirements, at some scope. Applicable means the requirements are met, not that the model was evaluated.

View all 3 applicable models →

Applicable · full: 0
Applicable · components: 0
Applicable · scenarios: 3
Conditionally applicable: 0
Not recorded: 55

Counted over 58 models in the default lifecycle scope (active, preview, experimental).

The whole benchmark is not resolved for these models, but one or more scenarios are.

Ctxdex Builder 1
Ctxdex Builder
Scenarios only
The full benchmark is not resolved for this subject, and 2 scenario(s) are.
Applies to Code Generation, Self Repair. Not recorded for Code Execution, Test Output Prediction.
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
- MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output)
- IO_REPRESENTATION:OUTPUT:SOURCE_CODEthe catalogue records SUPPORTED for SOURCE_CODE (output)
- MODEL_TASK:CODE_GENERATIONthe catalogue records SUPPORTED for CODE_GENERATION — Code Generation
- MODEL_TASK:CODE_EDITINGthe catalogue records SUPPORTED for CODE_EDITING — Self Repair
Ctxdex Code 2
Ctxdex Code
Scenarios only
The full benchmark is not resolved for this subject, and 2 scenario(s) are.
Applies to Code Generation, Self Repair. Not recorded for Code Execution, Test Output Prediction.
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
- MODALITY:OUTPUT:TEXTthe catalogue records SUPPORTED for TEXT (output)
- IO_REPRESENTATION:OUTPUT:SOURCE_CODEthe catalogue records SUPPORTED for SOURCE_CODE (output)
- MODEL_TASK:CODE_GENERATIONthe catalogue records SUPPORTED for CODE_GENERATION — Code Generation
- MODEL_TASK:CODE_EDITINGthe catalogue records SUPPORTED for CODE_EDITING — Self Repair

Applicability/relevance ordering only; never a performance ranking.

Scale

400
Problem
Total · Exact

Data and access

UnknownDataset, evaluation code, leaderboard, peer review and maintenance status are all unrecorded for this benchmark.

Coverage

Languages, regions: None recorded
Coverage summary: Not published

Limitations

None recorded

Lineage

Family: FamilyLiveCodeBench
Related: Inverse of successor of BenchmarkLiveCodeBench
Other definitions in family: LiveCodeBench release_v2 ActiveLiveCodeBench release_v3 ActiveLiveCodeBench release_v4 ActiveLiveCodeBench release_v5 ActiveLiveCodeBench release_v6 Active
Changes: None recorded

Sources

https://github.com/LiveCodeBench/LiveCodeBench ↗Official repository · Benchmark official

Freshness

Content version: B2-1.0
Last reviewed: 2026-09-23
Refresh model: Unknown
Time sensitivity: Unknown
Benchmark data: c40cccac878af5d0

13 contract fields are empty for every benchmark in this release and are not shown: applicability groups, conditional requirements, feature requirements, model relevance domains, required capabilities, openness summary, reference artifacts, interaction protocols, tool protocols, benchmark last major refresh at, short name, target expertise level, target population.