DeepSWE v1.1

How each maker reported it

• DeepSeek-V4.1-Flash Table 3, p.33

  All models at max reasoning effort. Code-agent benchmarks use DeepSeek Harness Minimal mode with 1M context; DeepSWE uses mini-SWE; visual agent tasks use the Claude Code harness with 512K context. HLE for some models is the text-only subset. The Hugging Face card lists NL2Repo-Bench as 64.0; the report and change log say 65.4.

• GPT-6 Astra Launch post (openai.com/index/gpt-6-astra); system card §6, §10.1; GPT-6.1 Sol addendum §5, §9

  Coding and reasoning rows come from the GPT-6 Astra launch post (3 Sep 2026); OpenAI's own comparison table, settings as published there. The system card reports safety and Preparedness evaluations, not general capability benchmarks. Cyber and bio scores are at maximum reasoning effort; bio scores are refusal-adjusted. HealthBench scores are length-adjusted. GPT-6.1 Sol and some GPT-5.6 Sol values come from the GPT-6.1 Sol addendum (29 Sep 2026).

• Gemini 3.8 Flash p.5

  Results as of September 2026, as published in Google's model card. Prices in the card list $0.75 / $3.75 with $1.50 / $7.50 as the regular rate. LVBench value for 3.8 Flash is the agentic setting (static: 87.1%).

• Grok 4.7 §2–6, p.5–16

  Grok 4.7 at xhigh effort except DeepSWE, SWE-Marathon, CADGenBench and CyberGym (high). Grok 4.6 at high except FrontierSWE, EEBench and HealthBench (xhigh). Most results come from third-party evaluators, each running its own harness; competitor results use max effort (CADGenBench GPT-5.6 Sol at xhigh). CyberGym is run without production safeguards.

• Claude Opus 5 Table 8.1.A, p.148–149

  Claude Opus 5 results use adaptive thinking at max effort, averaged over five trials (ARC-AGI-3 at high effort). GPT-5.6 Sol's FrontierBench figure uses Codex and its ARC-AGI-1 figure is at xhigh. The Fable 5 HealthBench Professional figure carries a footnote reading "Mythos 5". Competitor figures come from the developers' own published results or leaderboards.

• Mistral Large 4 launch post

  Preliminary, vendor-reported preview results. Many charts in the post give no numbers; only figures stated in text are listed. Blind human coding eval run by Surge AI on a 1–5 scale.