Benchmark definition
OSWorld V2.1
V2.1 / 2026-09-16Applicability does not mean evaluated; benchmark result/model score data is outside this UI contract.
About
- Description
- OSWorld V2.1
- What it measures
- OSWorld V2.1
- Comparison guidance
- Not published
- Intended difficulty
- Not published
Not recorded for this benchmark: what it does not measure, why it matters, motivation, gap addressed.
Classification
- Categories
- Computer UseAgenticMultimodalLong Horizon
- Domains
None recorded
- Evaluation style
None recorded
Structure
One undivided item set. No components or scenarios recorded.
Tasks
Metrics
What a result on this benchmark would be reported as. ctxdex does not carry results.
| Metric | Role | Direction | Unit |
|---|---|---|---|
Task Success Rate Canonical Batch-2 metric code TASK_SUCCESS_RATE; benchmark-specific formula/parameters remain on benchmark_metric and methodology records. | Primary | Higher is better | Percent |
Partial Completion Score Benchmark-defined normalized partial progress/checkpoint score. Exact source formula MUST be preserved in metric parameters/methodology. | Secondary | Higher is better | Score |
Methodology
Exact release pins code, task snapshots, assets, websites and environment/provider images as the comparability boundary.
- Source lane
- L4
Evaluation environment
What the benchmark's harness provides and allows. These are properties of the environment, not model requirements, and applicability does not test for them.
- Actions
- Coordinate pointer actionElement actionKeyboard inputProgrammatic environment action
- Observations
- ScreenshotAccessibility treeStructured environment state
- Session
- Turn based actions
- Reset
- Snapshot restore
- Network
- External allowed
- State persists per
- Episode
- Grounding
- Mixed
Exact environment artifacts/versioning remain benchmark methodology/reference-artifact provenance.
- Evaluation style
- Not recorded
- Environment summary
- Not published
Model requirements
Required conditions decide applicability. Relevant ones describe what the benchmark exercises and decide nothing.
Required · decides applicability
Relevant · informational
CTXDEX models
51 of the 58 active, preview, experimental models meet this benchmark’s recorded requirements, at some scope. Applicable means the requirements are met, not that the model was evaluated.
View all 51 applicable models →- Applicable · full
- 51
- Applicable · components
- 0
- Applicable · scenarios
- 0
- Conditionally applicable
- 0
- Not recorded
- 7
Counted over 58 models in the default lifecycle scope (active, preview, experimental).
Every required condition, for the whole benchmark, is recorded as supported. Each line is one recorded condition and why it counts.
Every one of the 1 required conditions is recorded as supported.
MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
Every one of the 1 required conditions is recorded as supported.
MODALITY:INPUT:TEXTthe catalogue records SUPPORTED for TEXT (input)
Applicability/relevance ordering only; never a performance ranking.
Scale
- 108TaskTotal · Exact
Data and access
Coverage
- Languages, regions
None recorded
- Coverage summary
- Not published
Limitations
None recorded
Lineage
- Family
- FamilyOSWorld
- Related
- Other definitions in family
- ChainWorld 2026 ActiveOSWorld 2024 SupersededOSWorld 2.0 — 2026.06.24 2026.06.24 SupersededOSWorld 2.0 — 2026.08.08 2026.08.08 SupersededOSWorld-Pro 2026 ActiveOSWorld-Verified Verified Superseded
- Changes
None recorded
Sources
- https://github.com/xlang-ai/OSWorld-V2 ↗Official repository · Benchmark official
- https://github.com/xlang-ai/OSWorld-V2/releases ↗Official repository · Benchmark official
Freshness
- Content version
- B2-1.0
- Last reviewed
- 2026-09-23
- Refresh model
- Unknown
- Time sensitivity
- Unknown
- Benchmark data
- c40cccac878af5d0
13 contract fields are empty for every benchmark in this release and are not shown: applicability groups, conditional requirements, feature requirements, model relevance domains, required capabilities, openness summary, reference artifacts, interaction protocols, tool protocols, benchmark last major refresh at, short name, target expertise level, target population.