Skip to content

Terminal-Bench 4.1 subset

4.1 results · Archived 4.0 results · Official Terminal-Bench 4.1 leaderboard

The 4.1 view uses the same 19 tasks and five slots per task as the original 4.0 subset. We ran the pinned 4.1 verifiers on the recorded outputs of the published agent attempts. We did not rerun the agents, change their outputs, or choose a different set of attempts. The original trace bundles, model costs, replacement lineage and 4.0 scores remain available.

What changed

The task instructions and expected artifact lists did not change for the subset. The verifier-only rescores use the 4.1 task packages. The historical rescore batch used Hugging Face Jobs and Docker for its follow-ups. We determined that wal-recovery-ordering could not be graded reliably in those environments and used Modal instead; the final 4.1 scores do not fall back to 4.0 WAL verdicts. The newer rescore batch runs all 19 tasks' verifiers on Modal.

The denominator remains 95. If a source trial scored zero and produced no collected artifact to regrade, it still contributes zero; we do not present that as a new verifier verdict. Integrity decisions apply to the original agent execution. If a disqualified attempt passes the new verifier, it remains disqualified.

What the page shows

First-party run pages link to the original execution and to the separate 4.1 verifier evidence bundles. Trace scan summaries and excerpts describe the original execution and use the original verifier labels; the score and task cells use 4.1 verdicts. Model costs and tokens are from the original runs and do not include verifier infrastructure.

External comparisons use the same comparator entries as the 4.0 view, matched to their 4.1 Harbor rows. Their subset scores use the recorded 4.1 verifier outcomes for those 19 tasks, and run pages still show the published full-suite 4.1 score. The 4.0 methodology and calibration figures document the original task selection, not a new selection exercise.