Benchmark definition

TUA-Bench

2026

ActiveTUA_BENCH_2026

Applicability does not mean evaluated; benchmark result/model score data is outside this UI contract.

Compare with others

About

Description: TUA-Bench
What it measures: TUA-Bench
Comparison guidance: Not published
Intended difficulty: Not published

Not recorded for this benchmark: what it does not measure, why it matters, motivation, gap addressed.

Classification

Categories: Terminal AgentAgenticKnowledge Work
Domains: None recorded
Evaluation style: None recorded

Structure

7 components. Applicability can differ between them.

Components

Office & ProductivityTask family
Web & InformationTask family
System & Software OperationsTask family
Scientific & EngineeringTask family
Multimedia & DesignTask family
Everyday DigitalConstruction trackScoped OSWorld-derived lineage applies here.
Professional Scientific & EngineeringConstruction trackSeparately authored with expert/domain input.

Tasks

Terminal UseComputer Use

Metrics

What a result on this benchmark would be reported as. ctxdex does not carry results.

Metric	Role	Direction	Unit
Task Success Rate Canonical Batch-2 metric code TASK_SUCCESS_RATE; benchmark-specific formula/parameters remain on benchmark_metric and methodology records.	Primary	Higher is better	Percent
Pass@k Probability/estimate that at least one of k generated solutions passes the benchmark correctness tests.	Secondary	Higher is better	Percent
Pass K Reliability Fraction/probability of tasks succeeding in all k independent trials.	Secondary	Higher is better	Percent

Methodology

InteractionThis definition

General computer-use workflows reformulated for terminal-first execution; deterministic task packages and execution-based scoring.

Source lane: L4

Evaluation environment

What the benchmark's harness provides and allows. These are properties of the environment, not model requirements, and applicability does not test for them.

Terminal · Container

Actions: Terminal commandPTY keystroke
Observations: Terminal text
Session: Both exec and PTY
Reset: Recreate
Network: External allowed
State persists per: Episode

Shell/PTY wrapper is harness-specific unless a benchmark scenario says otherwise.

Evaluation style: Not recorded
Environment summary: Not published

Model requirements

Required conditions decide applicability. Relevant ones describe what the benchmark exercises and decide nothing.

Required · decides applicability

Whole benchmark

Text (input)

Relevant · informational

Complex reasoningCoding modelsGeneral purpose models

CTXDEX models

51 of the 58 active, preview, experimental models meet this benchmark’s recorded requirements, at some scope. Applicable means the requirements are met, not that the model was evaluated.

View all 51 applicable models →

Applicable · full: 51
Applicable · components: 0
Applicable · scenarios: 0
Conditionally applicable: 0
Not recorded: 7

Counted over 58 models in the default lifecycle scope (active, preview, experimental).

Every required condition, for the whole benchmark, is recorded as supported. Each line is one recorded condition and why it counts.

Ctxdex Chintana 1 Large
Ctxdex Efficient Reasoning
Full benchmark
Every one of the 1 required conditions is recorded as supported.
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
Ctxdex Chintana 1 Mini
Ctxdex Efficient Reasoning
Full benchmark
Every one of the 1 required conditions is recorded as supported.
- MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)

Preview holds 12 of 51. View all →

Applicability/relevance ordering only; never a performance ranking.

Scale

100
Task
Total · Exact
20
Task
Total · Exact
20
Subcategories
Total · Exact
5
Task
Total · Exact
120
Task
Total · Exact
100
Task
Everyday Digital · Exact
20
Task
Professional Scientific & Engineering · Exact

Data and access

UnknownDataset, evaluation code, leaderboard, peer review and maintenance status are all unrecorded for this benchmark.

Coverage

Languages, regions: None recorded
Coverage summary: Not published

Limitations

None recorded

Lineage

Family: FamilyTUA-Bench
Related: Derived from BenchmarkOSWorld
Changes: None recorded

Sources

https://arxiv.org/abs/2606.28480 ↗Research paper · Original creator
https://github.com/facebookresearch/TUA-Bench ↗Official repository · Benchmark official

Freshness

Content version: B2-1.0
Last reviewed: 2026-09-23
Refresh model: Unknown
Time sensitivity: Unknown
Benchmark data: c40cccac878af5d0

13 contract fields are empty for every benchmark in this release and are not shown: applicability groups, conditional requirements, feature requirements, model relevance domains, required capabilities, openness summary, reference artifacts, interaction protocols, tool protocols, benchmark last major refresh at, short name, target expertise level, target population.