HumanEval: 2026 AI Leaderboard
164 Python programming problems: does the generated code pass unit tests?
What it tests
HumanEval is 164 handwritten Python programming problems with hidden unit tests. A model sees the function signature plus docstring and must generate a body that passes every test.
How it is scored
pass@1 -- the percentage of problems solved on the first attempt. Frontier models in 2026 sit in the 94-99% range, so this benchmark is effectively saturated for top-tier LLMs.
Why it matters
Still useful as a floor-check: any serious coding model should clear 90% here. For real-world discrimination, SWE-bench Verified and LiveCodeBench are the benchmarks that still separate the field.
Leaderboard (13 models)
Sorted by HumanEvalscore. Tier column shows the tool's overall AIToolTier rank, which blends this benchmark with pricing, features, and real-world usability.
| # | Model | Tier | HumanEval score | Variant | Overall |
|---|---|---|---|---|---|
| 1 | Codex (OpenAI) GPT-5.2-Codex (HISTORICAL -- this model was RETIRED 2026-07-23; scores retained for trend context only. Current Codex default is GPT-5.6 Sol, whose first-party Codex benchmarks are pending) | A | 95% | HumanEval | 8.3/10 |
| 2 | ChatGPT GPT-5.5 (launched 2026-04-23; scores below are the GPT-5.4 baseline -- GPT-5.5 launch benchmarks per OpenAI are logged in Known Issues, pending third-party verification) | A | 95% | HumanEval | 8.8/10 |
| 3 | Claude (Anthropic) Claude Opus 5 (launched 2026-07-24) is the current flagship and the default on Max -- Anthropic released COMPARATIVE claims only, no absolute scores: more than doubles Opus 4.8 on Frontier-Bench, within 0.5% of Fable 5 on CursorBench 3.2 at half the cost, 3x the next-best model on ARC-AGI 3, ~1.5x on Zapier AutomationBench, and beats Fable 5's best OSWorld 2.0 result at just over a third of the cost. Fable 5 (2026-06-09) holds vendor SWE-Bench Pro 80.3% (vs GPT-5.5 58.6%), #1 LMArena Elo 1510 and #1 Artificial Analysis Index 65 as of 6/11. Sonnet 5 (2026-06-30): OSWorld-Verified 78.5%. Legacy Opus-line reasoning-suite scores shown below as baseline pending third-party suites for Opus 5 | A | 94% | HumanEval | 8.5/10 |
| 4 | Qwen (Alibaba) Qwen3.5-397B MoE | A | 92.5% | HumanEval | 8.8/10 |
| 5 | Mistral AI Mistral Medium 3.5 (vendor-published; third-party verification pending) | B | 92% | HumanEval | 7.5/10 |
| 6 | DeepSeek DeepSeek V4-Pro (SWE-bench + Arena Elo third-party verified post-launch; knowledge rows are V3.x baseline pending V4 figures). V4.1-Flash (2026-09-10) vendor-only so far: GPQA Diamond 90.9, DeepSWE 74.2, Terminal-Bench 2.1 90.6 -- third-party verification pending | A | 91.5% | HumanEval | 8.0/10 |
| 7 | Muse Spark (Meta) Muse Spark | A | 91% | HumanEval | 8.8/10 |
| 8 | Grok Grok 4.20 (baseline -- Grok 4.5 launched 2026-07-08; vendor-reported 4.5 scores in Known Issues pending third-party verification) | B | 90% | HumanEval | 7.5/10 |
| 9 | Nemotron (Nvidia) Llama-Nemotron Ultra 253B (prior gen -- Nemotron 3 Ultra 550B third-party scores pending) | B | 89.6% | HumanEval | 7.8/10 |
| 10 | GLM / Z.ai (Zhipu AI) GLM-5.2 (~753B MoE, launched 2026-06-13) -- vendor-published; third-party verification still settling | A | 89.1% | HumanEval | 8.0/10 |
| 11 | Llama 4 (Meta) Llama 4 Maverick (17B/400B MoE) | B | 88% | HumanEval | 7.9/10 |
| 12 | Gemma 4 (Google) Gemma 4 31B | A | 85% | HumanEval | 8.3/10 |
| 13 | Falcon (TII) Falcon 3 10B | B | 73.8% | HumanEval | 7.1/10 |
About HumanEval
- Creator
- OpenAI, 2021
- Unit
- % (max 100)
- Official source
- https://arxiv.org/abs/2107.03374