SWE-bench Verified

Fixing real issues in real open-source repositories; a patch counts only if the project's own tests pass. Variants differ in task set and languages.

How each maker reported it

Gemini 3.1 Pro p.4
Results as of February 2026, Gemini at Thinking (High); competitors at their highest reported settings, as listed by Google.
Mistral Medium 3.5 HF model card, launch post
Vendor-reported. Comparisons with other models appear only in charts without numbers. LEXam-hard is asterisked on the card.
Muse Glimmer 30B Benchmarks
Meta's own results. Muse Glimmer at high reasoning; Gemma4-31B and Qwen3.6-27B in thinking mode. GDPVal-AA v2 is an Elo-style score.
gpt-oss-120b Figures 1, 2, 4; Table 2
OpenAI's own pass@1 results at reasoning level high. AIME and HLE with tools; Codeforces with a terminal tool. o4-mini and o3 values read from OpenAI's charts.
gpt-oss-20b Figures 1, 2, 4; Table 2
OpenAI's own pass@1 results at reasoning level high. AIME and HLE with tools; Codeforces with a terminal tool. o3-mini was run without tools on AIME, HLE and Codeforces, so those cells are left empty. Baseline values read from OpenAI's charts.

Only each model's own maker's number is shown; scores a maker reports for rival models are left out.