HumanEval: 2026 AI Leaderboard
164 Python programming problems: does the generated code pass unit tests?
What it tests
HumanEval is 164 handwritten Python programming problems with hidden unit tests. A model sees the function signature plus docstring and must generate a body that passes every test.
How it is scored
pass@1 -- the percentage of problems solved on the first attempt. Frontier models in 2026 sit in the 94-99% range, so this benchmark is effectively saturated for top-tier LLMs.
Why it matters
Still useful as a floor-check: any serious coding model should clear 90% here. For real-world discrimination, SWE-bench Verified and LiveCodeBench are the benchmarks that still separate the field.
Leaderboard (10 models)
Sorted by HumanEvalscore. Tier column shows the tool's overall AIToolTier rank, which blends this benchmark with pricing, features, and real-world usability.
| # | Model | Tier | HumanEval score | Variant | Overall |
|---|---|---|---|---|---|
| 1 | Codex (OpenAI) GPT-5.2-Codex (HISTORICAL -- this model was RETIRED 2026-07-23; scores retained for trend context only). GPT-6 Sol and Luna joined Codex 2026-09-22: OpenAI reports DeepSWE v1.1 68.8% (Sol, max effort) and 66.6% (Luna, max) -- first-party, third-party verification pending; see Known Issues | A | 95% | HumanEval | 8.3/10 |
| 2 | ChatGPT GPT-5.5 (launched 2026-04-23; scores below are the GPT-5.4 baseline -- GPT-5.5 launch benchmarks per OpenAI are logged in Known Issues, pending third-party verification) | A | 95% | HumanEval | 8.8/10 |
| 3 | Qwen (Alibaba) Qwen3.5-397B MoE | A | 92.5% | HumanEval | 8.8/10 |
| 4 | DeepSeek DeepSeek V4-Pro (SWE-bench + Arena Elo third-party verified post-launch; knowledge rows are V3.x baseline pending V4 figures). V4.1-Flash (2026-09-10) vendor-only so far: GPQA Diamond 90.9, DeepSWE 74.2, Terminal-Bench 2.1 90.6 -- third-party verification pending | A | 91.5% | HumanEval | 8.0/10 |
| 5 | Grok Grok 4.20 (baseline -- Grok 4.5 launched 2026-07-08; vendor-reported 4.5 scores in Known Issues pending third-party verification) | B | 90% | HumanEval | 7.5/10 |
| 6 | Nemotron (Nvidia) Llama-Nemotron Ultra 253B (prior gen -- Nemotron 3 Ultra 550B third-party scores pending) | B | 89.6% | HumanEval | 7.8/10 |
| 7 | GLM / Z.ai (Zhipu AI) GLM-5.2 (~753B MoE, launched 2026-06-13) -- vendor-published; third-party verification still settling | A | 89.1% | HumanEval | 8.0/10 |
| 8 | Llama 4 (Meta) Llama 4 Maverick (17B/400B MoE) | B | 88% | HumanEval | 7.9/10 |
| 9 | Gemma 4 (Google) Gemma 4 31B | A | 85% | HumanEval | 8.3/10 |
| 10 | Falcon (TII) Falcon 3 10B | B | 73.8% | HumanEval | 7.1/10 |
About HumanEval
- Creator
- OpenAI, 2021
- Unit
- % (max 100)
- Official source
- https://arxiv.org/abs/2107.03374