Skip to content

Ledger prototype

The benchmarks ledger rendered in Python at build time (docs/benchmark_ledger.py) instead of in the browser. Hover a task column for its native tooltip; the filters need a ~1 KB script, and everything else works with JavaScript off.

Terminal-Bench 2.1

Model · harness89 tasks × 5 attemptseasiest → hardest across all runsScoreRun cost
OpenAIOpenAI GPT-5.64 of our runs
OpenAIGPT-5.6 Sol · highOur runfast-agent 0.9.24Provisionalatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 4 of 5 attempts passed agent timeout atif-scan: 2 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passdna-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 55% of attempts passextract-elf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: no result for 1 attempt All runs: 44% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 1 of 5 attempts passed agent timeout, agent timeout atif-scan: no result for 1 attempt All runs: 9% of attempts pass88.3%393/445 · ±2.5~$216OpenAIGPT-5.6 Luna · maxOur runfast-agent 0.10.126h timeoutatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) atif-scan: no result for 1 attempt All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 4 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 4 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 2 of 5 attempts passed atif-scan: no result for 1 attempt All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 88% of attempts passcount-dataset-tokens 3 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: no result for 1 attempt All runs: 86% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 84% of attempts passcancel-async-tasks 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 82% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 71% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 65% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 4 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 56% of attempts passdna-assembly 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 55% of attempts passextract-elf 0 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 52% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 48% of attempts passgcode-to-text 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 44% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 9% of attempts pass82.2%366/445 · ±3.1~$29.54OpenAIGPT-5.6 Sol · mediumOur runfast-agent 0.9.21Provisionalatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passpytorch-model-recovery 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 65% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 56% of attempts passdna-assembly 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 55% of attempts passextract-elf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 3 of 5 attempts passed agent timeout, agent timeout atif-scan: no result for 1 attempt All runs: 44% of attempts passtorch-pipeline-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 9% of attempts pass82.0%365/445 · ±3.2~$169OpenAIGPT-5.6 Terra · highOur runfast-agent 0.9.15Provisionalatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 4 of 5 attempts passed agent timeout atif-scan: no result for 1 attempt All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsparql-university 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passpassword-recovery 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 4 of 5 attempts passed agent timeout atif-scan: no result for 1 attempt All runs: 88% of attempts passfinancial-document-processor 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 65% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 55% of attempts passextract-elf 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: no result for 3 attempts All runs: 42% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 4 of 5 attempts passed agent timeout atif-scan: no result for 1 attempt All runs: 9% of attempts pass77.7%346/445 · ±3.6~$134
OpenAIGPT-5.6 Sol · highVendor claimOpenAI · GPT-5.6 launch postNo trial data published · score only84.7%reported~$388
OpenAIGPT-5.6 Sol · mediumVendor claimOpenAI · GPT-5.6 launch postNo trial data published · score only81.8%reported~$317
OpenAIGPT-5.6 Terra · maxLeaderboardCodex 0.144.xnot scanneddistribution-search 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passfix-code-vulnerability 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: this run hasn't been scanned All runs: 99% of attempts passgit-leak-recovery 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passcrack-7z-hash 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 98% of attempts passhf-model-inference 2 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 95% of attempts passfeal-linear-cryptanalysis 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 95% of attempts passportfolio-optimization 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 94% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 94% of attempts passcustom-memory-heap-crash 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passlarge-scale-text-editing 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passpypi-server 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passqemu-alpine-ssh 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 92% of attempts passwrite-compressor 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 91% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed agent timeout atif-scan: this run hasn't been scanned All runs: 90% of attempts passkv-store-grpc 0 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 88% of attempts passfinancial-document-processor 3 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 88% of attempts passheadless-terminal 2 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 88% of attempts passcount-dataset-tokens 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 86% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 84% of attempts passfix-ocaml-gc 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 84% of attempts passregex-chess 4 of 5 attempts passed error atif-scan: this run hasn't been scanned All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 82% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 80% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 79% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 76% of attempts passquery-optimize 2 of 5 attempts passed error, agent timeout atif-scan: this run hasn't been scanned All runs: 75% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 66% of attempts passpytorch-model-cli 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 65% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 62% of attempts passsanitize-git-repo 0 of 5 attempts passed agent timeout atif-scan: this run hasn't been scanned All runs: 57% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 56% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 55% of attempts passextract-elf 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 52% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 50% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 48% of attempts passgcode-to-text 0 of 5 attempts passed error, error atif-scan: this run hasn't been scanned All runs: 46% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: this run hasn't been scanned All runs: 44% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 42% of attempts passtrain-fasttext 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: this run hasn't been scanned All runs: 42% of attempts passvideo-processing 4 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 34% of attempts passraman-fitting 5 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 30% of attempts passdna-insert 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: this run hasn't been scanned All runs: 9% of attempts passmake-doom-for-mips 1 of 5 attempts passed agent timeout atif-scan: this run hasn't been scanned All runs: 9% of attempts pass78.4%349/445 · ±3.3$421
OpenAIGPT-5.6 Terra · highVendor claimOpenAI · GPT-5.6 launch postNo trial data published · score only76.6%reported$280
xAIxAI Grok3 of our runs
xAIGrok 4.7 · mediumOur runfast-agent 0.10.426h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passfix-ocaml-gc 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 79% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 75% of attempts passmake-mips-interpreter 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 57% of attempts passdb-wal-recovery 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 56% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 55% of attempts passextract-elf 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 2 attempts with a medium finding All runs: 52% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 50% of attempts passprotein-assembly 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 48% of attempts passgcode-to-text 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passtrain-fasttext 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 34% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 30% of attempts passdna-insert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts pass88.5%394/445 · ±2.8recorded 90.3%≥$523xAIGrok 4.6 · mediumOur runfast-agent 0.10.10Provisionalatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 82% of attempts passsam-cell-seg 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 79% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 2 of 5 attempts passed agent timeout atif-scan: 2 attempts with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passdna-assembly 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 55% of attempts passextract-elf 1 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 52% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 2 of 5 attempts passed error, error, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 42% of attempts passtrain-fasttext 4 of 5 attempts passed agent timeout atif-scan: 1 attempt with a medium finding All runs: 42% of attempts passvideo-processing 0 of 5 attempts passed error atif-scan: 3 attempts with a medium finding All runs: 34% of attempts passraman-fitting 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding All runs: 9% of attempts pass87.6%390/445 · ±2.9$193xAIGrok 4.6 · highOur runfast-agent 0.10.9Provisionalatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 4 of 5 attempts passed agent timeout atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 82% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 56% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 55% of attempts passextract-elf 1 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 52% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 50% of attempts passprotein-assembly 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 1 of 5 attempts passed error, error, error, error atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passtrain-fasttext 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 0 of 5 attempts passed error, error atif-scan: 3 attempts with a medium finding All runs: 34% of attempts passraman-fitting 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding All runs: 9% of attempts pass87.1%388/445 · ±2.9$238
Z.aiZ.ai GLM-5.31 of our runs
Z.aiGLM-5.3-Flash · maxOur runfast-agent 0.10.116h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passcaffe-cifar-10 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 88% of attempts passcount-dataset-tokens 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 3 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 1 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 66% of attempts passpytorch-model-cli 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 57% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passdna-assembly 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 55% of attempts passextract-elf 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 52% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 50% of attempts passprotein-assembly 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 48% of attempts passgcode-to-text 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 9% of attempts pass82.4%367/445 · ±3.0recorded 84.4%~$77.53
DeepSeekDeepSeek V43 of our runs
DeepSeekDeepSeek V4.1 Flash · maxOur runfast-agent 0.10.246h timeoutatif-scan basic + imagesrevieweddistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 89% of attempts passcompile-compcert 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 88% of attempts passfinancial-document-processor 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 88% of attempts passcount-dataset-tokens 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 84% of attempts passpytorch-model-recovery 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 4 attempts with a medium finding All runs: 82% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 79% of attempts passchess-best-move 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 66% of attempts passmteb-retrieve 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 66% of attempts passpytorch-model-cli 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 62% of attempts passsanitize-git-repo 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 57% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 55% of attempts passextract-elf 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 52% of attempts passconfigure-git-webserver 5 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 1 attempt with a medium finding All runs: 50% of attempts passprotein-assembly 2 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 48% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 42% of attempts passtrain-fasttext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 34% of attempts passraman-fitting 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 30% of attempts passdna-insert 1 of 5 attempts passed pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review), pass, disqualified (leaderboard judge or our review) atif-scan: 4 attempts with a high or critical finding (4 passed) All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts pass84.2%375/445 · ±3.3recorded 95.0%≥$31.20DeepSeekDeepSeek V4 Flash Vision · maxOur runfast-agent 0.10.136h timeoutatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 4 of 5 attempts passed error atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 5 attempts with a high or critical finding (5 passed) All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 4 of 5 attempts passed error atif-scan: 3 attempts with a medium finding All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: 2 attempts with a high or critical finding (2 passed) atif-scan: 1 attempt with a medium finding All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 82% of attempts passsam-cell-seg 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 2 of 5 attempts passed error atif-scan: 3 attempts with a medium finding All runs: 79% of attempts passchess-best-move 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 4 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 66% of attempts passpytorch-model-cli 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 3 of 5 attempts passed error atif-scan: 1 attempt with a medium finding All runs: 62% of attempts passsanitize-git-repo 4 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 3 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passdna-assembly 0 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 55% of attempts passextract-elf 3 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 4 attempts with a medium finding All runs: 52% of attempts passconfigure-git-webserver 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 2 of 5 attempts passed error, error atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 2 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout, error, agent timeout, error atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 1 of 5 attempts passed error atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 30% of attempts passdna-insert 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 28% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 9% of attempts pass83.1%370/445 · ±2.9~$53.68DeepSeekDeepSeek V4 Flash · maxOur runfast-agent 0.9.30Provisionalatif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 4 attempts with a high or critical finding (4 passed) atif-scan: 1 attempt with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 96% of attempts passpassword-recovery 4 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 1 of 5 attempts passed agent timeout, agent timeout, error, error atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 3 of 5 attempts passed agent timeout, agent timeout atif-scan: 4 attempts with a medium finding All runs: 91% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed agent timeout atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 4 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 88% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a high or critical finding (0 passed) atif-scan: 3 attempts with a medium finding All runs: 86% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 84% of attempts passwinning-avg-corewars 4 of 5 attempts passed agent timeout atif-scan: 1 attempt with a medium finding All runs: 82% of attempts passsam-cell-seg 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 80% of attempts passtorch-tensor-parallelism 3 of 5 attempts passed agent timeout atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 2 attempts with a medium finding All runs: 79% of attempts passchess-best-move 2 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 5 attempts with a medium finding All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed agent timeout, agent timeout atif-scan: 3 attempts with a medium finding All runs: 66% of attempts passmteb-retrieve 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 66% of attempts passpytorch-model-cli 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding atif-scan: no result for 1 attempt All runs: 62% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 2 of 5 attempts passed agent timeout, agent timeout atif-scan: 1 attempt with a high or critical finding (1 passed) atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passdna-assembly 1 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: 2 attempts with a medium finding atif-scan: no result for 1 attempt All runs: 55% of attempts passextract-elf 5 of 5 attempts passed atif-scan: 3 attempts with a high or critical finding (3 passed) atif-scan: 1 attempt with a medium finding All runs: 52% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 3 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 48% of attempts passgcode-to-text 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, error, error atif-scan: no result for 4 attempts All runs: 44% of attempts passtorch-pipeline-parallelism 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 3 attempts with a medium finding All runs: 42% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: no result for 4 attempts All runs: 42% of attempts passvideo-processing 0 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 34% of attempts passraman-fitting 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 9% of attempts pass75.5%336/445 · ±3.6$8.74
AnthropicAnthropic Claudecomparators only
AnthropicFable 5 · xhighLeaderboardClaude Code 2.1.167atif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed 4 attempts ran on another model (fallback) All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed 3 attempts ran on another model (fallback) All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 86% of attempts passoverfull-hbox 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passpytorch-model-recovery 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 82% of attempts passsam-cell-seg 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 2 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 4 of 5 attempts passed 1 attempt ran on another model (fallback) All runs: 66% of attempts passmteb-retrieve 3 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 66% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 64% of attempts passgpt2-codegolf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 2 of 5 attempts passed agent timeout 5 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 56% of attempts passdna-assembly 4 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 1 attempt with a medium finding All runs: 55% of attempts passextract-elf 1 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 52% of attempts passconfigure-git-webserver 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 48% of attempts passgcode-to-text 5 of 5 attempts passed 1 attempt ran on another model (fallback) All runs: 46% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 2 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed 2 attempts ran on another model (fallback) All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout 2 attempts ran on another model (fallback) All runs: 9% of attempts pass83.8%373/445 · ±3.2$553AnthropicFable 5 · highLeaderboardTerminus 2 2.0.0atif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed 4 attempts ran on another model (fallback) All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 92% of attempts passwrite-compressor 3 of 5 attempts passed agent timeout, agent timeout 5 attempts ran on another model (fallback) All runs: 92% of attempts passadaptive-rejection-sampler 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed 5 attempts ran on another model (fallback) atif-scan: 3 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 90% of attempts passqemu-startup 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout 5 attempts ran on another model (fallback) All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passpytorch-model-recovery 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 82% of attempts passsam-cell-seg 5 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 64% of attempts passgpt2-codegolf 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 2 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passdna-assembly 2 of 5 attempts passed agent timeout, agent timeout 5 attempts ran on another model (fallback) All runs: 55% of attempts passextract-elf 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 52% of attempts passconfigure-git-webserver 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 3 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 48% of attempts passgcode-to-text 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 44% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 1 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 0 of 5 attempts passed 5 attempts ran on another model (fallback) All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 9% of attempts pass80.4%358/445 · ±3.5$439AnthropicOpus 4.8 · highLeaderboardClaude Code 2.1.205atif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 98% of attempts passsparql-university 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passpassword-recovery 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passcount-dataset-tokens 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passcancel-async-tasks 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 84% of attempts passwinning-avg-corewars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 56% of attempts passdna-assembly 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 55% of attempts passextract-elf 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passconfigure-git-webserver 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 50% of attempts passprotein-assembly 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding All runs: 44% of attempts passtorch-pipeline-parallelism 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 0 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 34% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 1 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 9% of attempts pass78.8%351/445 · ±3.5$287AnthropicSonnet 5 · highLeaderboardClaude Code 2.1.205atif-scan basicdistribution-search 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfeal-differential-cryptanalysis 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passfix-git 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passmodernize-scientific-stack 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passpolyglot-c-py 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passprove-plus-comm 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passregex-log 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 100% of attempts passbuild-pmars 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcircuit-fibsqrt 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passcode-from-image 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passconstraints-scheduling 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passfix-code-vulnerability 5 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 99% of attempts passgit-leak-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passmerge-diff-arc-agi-task 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passreshard-c4-data 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passsqlite-db-truncate 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 99% of attempts passbuild-cython-ext 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passcrack-7z-hash 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 98% of attempts passgit-multibranch 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passlog-summary-date-ranges 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmailman 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passmulti-source-data-merger 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passnginx-request-logging 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passopenssl-selfsigned-cert 4 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 98% of attempts passsparql-university 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passsqlite-with-gcov 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 98% of attempts passhf-model-inference 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 96% of attempts passllm-inference-batching-scheduler 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 96% of attempts passmteb-leaderboard 5 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 96% of attempts passpassword-recovery 4 of 5 attempts passed agent timeout atif-scan: 5 attempts with a medium finding All runs: 96% of attempts passtune-mjcf 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 96% of attempts passcobol-modernization 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passfeal-linear-cryptanalysis 2 of 5 attempts passed agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 95% of attempts passpolyglot-rust-c 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 95% of attempts passportfolio-optimization 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 95% of attempts passbn-fit-modify 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passlargest-eigenval 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passrstan-to-pystan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 94% of attempts passschemelike-metacircular-eval 3 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 94% of attempts passcustom-memory-heap-crash 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passlarge-scale-text-editing 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passpypi-server 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passqemu-alpine-ssh 4 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 92% of attempts passvulnerable-secret 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passwrite-compressor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 92% of attempts passadaptive-rejection-sampler 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passmcmc-sampling-stan 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 91% of attempts passpath-tracing 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 91% of attempts passbreak-filter-js-from-html 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 5 attempts with a medium finding All runs: 90% of attempts passcaffe-cifar-10 4 of 5 attempts passed agent timeout atif-scan: 2 attempts with a medium finding All runs: 90% of attempts passqemu-startup 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 90% of attempts passkv-store-grpc 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 89% of attempts passcompile-compcert 3 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 88% of attempts passfinancial-document-processor 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 88% of attempts passheadless-terminal 4 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 88% of attempts passcount-dataset-tokens 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passpath-tracing-reverse 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 86% of attempts passoverfull-hbox 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 85% of attempts passbuild-pov-ray 0 of 5 attempts passed atif-scan: 1 attempt with a high or critical finding (0 passed) All runs: 84% of attempts passcancel-async-tasks 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passfix-ocaml-gc 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 3 attempts with a medium finding All runs: 84% of attempts passpytorch-model-recovery 5 of 5 attempts passed atif-scan: nothing above low priority All runs: 84% of attempts passregex-chess 4 of 5 attempts passed pass, disqualified (leaderboard judge or our review) atif-scan: 1 attempt with a high or critical finding (1 passed) All runs: 84% of attempts passwinning-avg-corewars 2 of 5 attempts passed agent timeout, agent timeout atif-scan: nothing above low priority All runs: 82% of attempts passsam-cell-seg 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 80% of attempts passtorch-tensor-parallelism 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 79% of attempts passchess-best-move 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 76% of attempts passquery-optimize 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 75% of attempts passmake-mips-interpreter 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 71% of attempts passinstall-windows-3.11 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passmteb-retrieve 2 of 5 attempts passed atif-scan: nothing above low priority All runs: 66% of attempts passpytorch-model-cli 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 65% of attempts passmodel-extraction-relu-logits 1 of 5 attempts passed agent timeout atif-scan: nothing above low priority All runs: 64% of attempts passgpt2-codegolf 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 62% of attempts passsanitize-git-repo 3 of 5 attempts passed atif-scan: 5 attempts with a medium finding All runs: 57% of attempts passdb-wal-recovery 0 of 5 attempts passed atif-scan: 2 attempts with a medium finding All runs: 56% of attempts passdna-assembly 5 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 55% of attempts passextract-elf 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 52% of attempts passconfigure-git-webserver 4 of 5 attempts passed atif-scan: 3 attempts with a medium finding All runs: 50% of attempts passprotein-assembly 0 of 5 attempts passed error, error, error, error, error atif-scan: nothing above low priority All runs: 48% of attempts passgcode-to-text 4 of 5 attempts passed atif-scan: nothing above low priority All runs: 46% of attempts passextract-moves-from-video 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: 1 attempt with a medium finding atif-scan: no result for 1 attempt All runs: 44% of attempts passtorch-pipeline-parallelism 3 of 5 attempts passed atif-scan: nothing above low priority All runs: 42% of attempts passtrain-fasttext 1 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 42% of attempts passvideo-processing 1 of 5 attempts passed atif-scan: 1 attempt with a medium finding All runs: 34% of attempts passraman-fitting 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 30% of attempts passdna-insert 0 of 5 attempts passed error atif-scan: nothing above low priority All runs: 28% of attempts passfilter-js-from-html 0 of 5 attempts passed atif-scan: nothing above low priority All runs: 9% of attempts passmake-doom-for-mips 0 of 5 attempts passed agent timeout, agent timeout, agent timeout, agent timeout, agent timeout atif-scan: nothing above low priority All runs: 9% of attempts pass74.6%332/445 · ±3.3$288
our passcomparator passfailtimeouterrorjudge DQatif-scan: high or critical findingtask has a flagged attemptran on another modelnot scannedvendor claim, no trials