HumanEval: 2026 AI Leaderboard
164 Python programming problems: does the generated code pass unit tests?
What it tests
HumanEval is 164 handwritten Python programming problems with hidden unit tests. A model sees the function signature plus docstring and must generate a body that passes every test.
How it is scored
pass@1 -- the percentage of problems solved on the first attempt. Frontier models in 2026 sit in the 94-99% range, so this benchmark is effectively saturated for top-tier LLMs.
Why it matters
Still useful as a floor-check: any serious coding model should clear 90% here. For real-world discrimination, SWE-bench Verified and LiveCodeBench are the benchmarks that still separate the field.
Leaderboard (13 models)
Sorted by HumanEvalscore. Tier column shows the tool's overall AIToolTier rank, which blends this benchmark with pricing, features, and real-world usability.
| # | Model | Tier | HumanEval score | Variant | Overall |
|---|---|---|---|---|---|
| 1 | Codex (OpenAI) GPT-5.2-Codex (launched 2026-04-23 -- SOTA on SWE-Bench Pro and Terminal-Bench 2.0; first-party scores below pending detailed third-party verification) | A | 95% | HumanEval | 8.3/10 |
| 2 | ChatGPT GPT-5.5 (launched 2026-04-23; scores below are the GPT-5.4 baseline -- GPT-5.5 launch benchmarks per OpenAI are logged in Known Issues, pending third-party verification) | A | 95% | HumanEval | 8.8/10 |
| 3 | Claude (Anthropic) Claude Fable 5 (launched 2026-06-09; suspended 2026-06-12 by US gov order; RESTORED globally 2026-07-01 after controls lifted 6/30) -- vendor SWE-Bench Pro 80.3% (vs GPT-5.5 58.6%); #1 LMArena Elo 1510 and #1 Artificial Analysis Index 65 as of 6/11. New default Sonnet 5 (2026-06-30): OSWorld-Verified 78.5%. Legacy Opus-line reasoning-suite scores shown below as baseline pending full third-party suites | A | 94% | HumanEval | 8.5/10 |
| 4 | Qwen (Alibaba) Qwen3.5-397B MoE | A | 92.5% | HumanEval | 8.8/10 |
| 5 | Mistral AI Mistral Medium 3.5 (vendor-published; third-party verification pending) | B | 92% | HumanEval | 7.5/10 |
| 6 | DeepSeek DeepSeek V4-Pro (SWE-bench + Arena Elo third-party verified post-launch; knowledge rows are V3.x baseline pending V4 figures) | A | 91.5% | HumanEval | 8.0/10 |
| 7 | Muse Spark (Meta) Muse Spark | A | 91% | HumanEval | 8.8/10 |
| 8 | Grok Grok 4.20 (baseline -- Grok 4.5 launched 2026-07-08; vendor-reported 4.5 scores in Known Issues pending third-party verification) | B | 90% | HumanEval | 7.5/10 |
| 9 | Nemotron (Nvidia) Llama-Nemotron Ultra 253B (prior gen -- Nemotron 3 Ultra 550B third-party scores pending) | B | 89.6% | HumanEval | 7.8/10 |
| 10 | GLM / Z.ai (Zhipu AI) GLM-5.2 (~753B MoE, launched 2026-06-13) -- vendor-published; third-party verification still settling | A | 89.1% | HumanEval | 8.0/10 |
| 11 | Llama 4 (Meta) Llama 4 Maverick (17B/400B MoE) | B | 88% | HumanEval | 7.9/10 |
| 12 | Gemma 4 (Google) Gemma 4 31B | A | 85% | HumanEval | 8.3/10 |
| 13 | Falcon (TII) Falcon 3 10B | B | 73.8% | HumanEval | 7.1/10 |
About HumanEval
- Creator
- OpenAI, 2021
- Unit
- % (max 100)
- Official source
- https://arxiv.org/abs/2107.03374