Computer Science and Programming
HumanEval
OpenAI's canonical 164 Python coding problems scored by unit-test pass@1. Largely saturated but still cited as a baseline.
HumanEval leaderboard
| # | Model | Provider | Score | $/1M out | Score / $ |
|---|---|---|---|---|---|
| 1 | Claude Fable 5 | Anthropic | 99.4 | $50.00 | 2.0 |
| 2 | GPT-5.5 Pro | OpenAI | 99.1 | $180.00 | 0.6 |
| 3 | Gemini 3.1 Pro | 98.8 | $12.00 | 8.2 | |
| 4 | Claude Opus 4.8 | Anthropic | 97.6 | $25.00 | 3.9 |
| 5 | DeepSeek V4 Pro | DeepSeek | 96.9 | $2.20 | 44.0 |
| 6 | o3 | OpenAI | 96.2 | $8.00 | 12.0 |
About HumanEval
OpenAI's canonical 164 Python coding problems scored by unit-test pass@1. Largely saturated but still cited as a baseline.
Source: https://arxiv.org/abs/2107.03374. Back to all benchmarks.
Last updated: · Source: Pricing dataset · Methodology
Frequently asked questions
How are token costs calculated?
Multiply input tokens by the input rate and output tokens by the output rate, add the two amounts and divide by 1,000,000. A standard request example uses 1,000 input and 500 output tokens. Monthly budgets multiply the result by request volume; taxes, hidden reasoning tokens, tools and tier surcharges can add costs.
Which recorded model has the lowest request cost?
LFM 2.2 6b by Liquid AI costs $0.00002 for the standard request, with $0.01 input and $0.02 output per million tokens. This price ranking does not establish quality equivalence; verify task success and context requirements.
Can cached and batch discounts be combined?
Only recorded rates are used. Cached input can have cache-write or storage charges, while batch rates apply only to eligible asynchronous jobs. We do not automatically stack the discounts or treat an absent rate as zero. Check the linked provider source before budgeting.
How current is the public dataset?
The live pricing table synchronizes stored model records daily; public file snapshots update when the site is rebuilt. Synchronization is not independent provider verification. It contains 1,026 priced models across 124 providers, last updated 2026-10-31. Directory entries are counted separately. A synchronization date is not proof of independent verification.