Foundation LLM Leaderboard & Benchmarks
Real-time Chatbot Arena Elo, GPQA Diamond, SWE-bench Verified, and AIME 2025 reasoning scores paired with live throughput telemetry and pricing.
Claude 3.7 Sonnet (Thinking)
Foundation Model Benchmark Matrix
Click any column header to re-sort rankings in real-time.
# Rank | Model & Provider | Arena Elo 🔥 | SWE-bench | GPQA PhD | AIME 2025 | Speed ⚡ | Cost / 1M | Weekly Vol | Action |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude 3.7 Sonnet (Thinking)200k Anthropic | 1392 | 70.3% | 92.4% | 88.4% | 72 t/s | $3.00 / $15.00 | 6.45T | Explore |
| 2 | OpenAI o3-mini (high reasoning)200k OpenAI | 1386 | 76.8% | 93.6% | 87.3% | 88 t/s | $1.10 / $4.40 | 5.90T | Explore |
| 3 | DeepSeek-R1 (Reasoning)128k DeepSeek | 1364 | 73.1% | 82.4% | 93.1% | 42 t/s | $0.55 / $2.19 | 5.80T | Explore |
| #4 | Google Gemini 2.5 Pro (Experimental)2M Google | 1388 | 74.2% | 91.5% | 89.2% | 95 t/s | $1.25 / $5.00 | 4.85T | Explore |
| #5 | OpenAI GPT-4.5 / o1 Preview128k OpenAI | 1374 | 81.2% | 94.2% | 84.5% | 38 t/s | $15.00 / $60.00 | 4.10T | Explore |
| #6 | Claude 3.5 Sonnet v2200k Anthropic | 1352 | 79.6% | 89.9% | 78.0% | 82 t/s | $3.00 / $15.00 | 3.90T | Explore |
| #7 | DeepSeek V3 (MoE 671B)128k DeepSeek | 1348 | 79.0% | 88.1% | 78.5% | 65 t/s | $0.14 / $0.28 | 3.75T | Explore |
| #8 | Google Gemini 2.0 Flash1M Google | 1356 | 65.2% | 84.0% | 74.2% | 145 t/s | $0.10 / $0.40 | 3.50T | Explore |
| #9 | Qwen 2.5 Coder 32B Instruct128k Alibaba | 1338 | 68.4% | 81.2% | 76.0% | 98 t/s | $0.20 / $0.20 | 3.10T | Source |
| #10 | Grok 3 (Reasoning / Frontier)1M xAI | 1390 | 78.5% | 92.0% | 93.5% | 60 t/s | $3.00 / $15.00 | 2.80T | Source |
| #11 | Llama 3.3 70B Instruct128k Meta | 1318 | 61.2% | 83.5% | 71.0% | 110 t/s | $0.18 / $0.18 | 2.40T | Source |
| #12 | Mistral Large 2 / Codestral128k Mistral AI | 1312 | 64.8% | 81.0% | 71.5% | 85 t/s | $2.00 / $6.00 | 1.95T | Source |
| #13 | Claude 3.5 Haiku200k Anthropic | 1284 | 40.6% | 75.2% | 62.0% | 120 t/s | $0.80 / $4.00 | 1.70T | Explore |
| #14 | GLM 5.2 / MiniMax M3128k Zhipu AI | 1320 | 67.2% | 91.2% | 79.0% | 75 t/s | $0.60 / $2.00 | 1.55T | Source |
| #15 | Kimi K2.6 (Long Context)256k Moonshot AI | 1325 | 66.5% | 86.8% | 81.2% | 50 t/s | $0.80 / $2.40 | 1.35T | Source |
Human blind preference rating aggregating >100,000 randomized A/B matchups on LMSYS. Measures general conversational IQ and instruction following.
Autonomous resolution of real GitHub issues across 500+ production repositories, requiring bug localization, patch synthesis, and passing unit tests.
Google-Proof Graduate-Level QA written by PhD experts in biology, physics, and chemistry. Tests deep multi-disciplinary reasoning.
American Invitational Mathematics Examination problems assessing test-time compute, mathematical proof search, and combinatorial problem solving.
Frontier Trends in LLM Architecture & Reasoning
The Dominance of Test-Time Compute: Models now scale reasoning at inference time via chain-of-thought exploration, verification, and backtracking—dramatically outperforming traditional fixed-token generation on math and code.
Open-Weights Parity: Open-weights models like DeepSeek-R1, DeepSeek-V3, and Qwen 2.5 Coder 32B match or exceed proprietary models in math and coding benchmarks at a fraction of API costs.
Massive Multi-Million Context Windows: Frontier models like Google Gemini 2.5 Pro offer up to 2,000,000 token context windows, enabling in-context search across entire codebases and books without RAG degradation.
Academic Reasoning & Hard-Benchmark Methodologies
Evaluating foundation models requires metrics that resist simple web-crawled memorization. Standard benchmarks like GSM8k and HumanEval have saturated. To measure genuine intelligence and problem solving, we track four gold-standard benchmarks: LMSYS Chatbot Arena Elo, GPQA Diamond, SWE-bench Verified, and AIME 2025. GPQA Diamond consists of PhD-level questions in biology, physics, and chemistry designed by experts to be un-searchable on Google. SWE-bench Verified evaluates autonomous resolution of real GitHub issues by writing verified patches. AIME 2025 tests olympiad-level mathematical reasoning requiring multi-step proofs and algorithmic calculation.
- Chatbot Arena Elo: Live human blind preference rankings aggregating tens of thousands of pairwise head-to-head model battles.
- GPQA Diamond PhD Validation: Rigorous double-blind verified questions assessing deep scientific reasoning without internet memorization.
- AIME 2025 Competition Math: Olympiad-level algebra, combinatorics, and number theory testing test-time reasoning and verification.
Real-Time Telemetry & Throughput Scaling Laws
Throughput and token volume provide empirical insight into real-world developer adoption. Telemetry from OpenRouter and public model gateways tracks weekly prompt and completion tokens across millions of requests. Reasoning models (like Claude 3.7 Sonnet Thinking, o3-mini, and DeepSeek-R1) generate internal thinking chains before answering, balancing higher completion token counts with superior problem-solving accuracy.
- Live Token Telemetry: Tracking weekly trillions of tokens to monitor commercial developer adoption and platform shifts.
- Price-to-Performance Optimization: Analyzing input/output cost per million tokens alongside context windows (up to 2M tokens on Gemini).
- Throughput Speed (tokens/sec): Measuring inference velocity for low-latency streaming and real-time interactive user interfaces.
Frequently Asked Questions
What is Chatbot Arena Elo and why does it matter?
Chatbot Arena Elo is calculated from blind, randomized head-to-head A/B battles judged by real human users on LMSYS. It provides an unbiased measure of general conversational quality, instruction following, and helpfulness.
What is GPQA Diamond and why is it Google-Proof?
GPQA Diamond is a curated subset of graduate-level STEM questions designed by PhD experts so that even subject-matter experts without domain specialization cannot find answers via Google search. It measures deep conceptual synthesis.