Official benchmarks don't tell you anything. Bench maxing does.
Every model launch comes with a bar chart claiming it changed everything. The thesis — validated live, testing every new model on launch day — is different: the number that matters is the one that comes out of YOUR operation, measured in delivery and cost, not the lab's slide.
Four findings from testing every launch live
01 · Official benchmarks are marketing with a number attached
Labs train their LLMs to score high on benchmarks — and the number doesn't say how the model behaves in YOUR business. The case that proved it: a model took 1st place in 7 of 8 benchmarks while the official announcement itself admitted "a modest, but tangible improvement."
02 · What matters is bench maxing
Testing on your own operation, with your own numbers. Every benchmark only becomes relevant once you run the test inside your own application — one-shot, side by side, measuring what came out and what it cost.
03 · The strongest model doesn't run everything
Bench maxing means mixing: the right agent + the right model + the right CLI per task. Scout on a fast, cheap model, builder on a model with its own separate limit, orchestrator on the strong model — each CLI has its own per-model limits, and ignoring that burns the expensive model's limit on mechanical tasks.
04 · The market validated the thesis
When the world's biggest lab shipped parallel agent orchestration — an orchestrator splitting activities and delegating to builders with review — it was shipping what Overclock had already been doing for months. The harness doctrine stopped being a bet.
“Benchmarks are barely useful for anything — it's just a number to convince you. You have to be using it on your own operation and running your own benchmark, with your own numbers.” — said in the thesis video
How we measure: 17 models, 6 vendors, 9 real tests
It's not opinion or vibes: it's a battery of 153 runs — each model pushed to its max effort ceiling, judged by machine. The battery is closed and the scoreboard is right below — no number showed up here before the measurement was done.
01 · Tasks come from data, not guesswork
We classified 7,426 real vibe-coder prompts to extract what people actually do: fix a bug (8.8%), analyze data (3.8%), build a feature (3.1%), review code (2.9%), tweak CSS (2.9%). Two tracks: build from scratch (3D solar system, site clone, physics, FPS) and work in existing code — each test with a fixture mini-repo and the problem planted.
02 · The pure rig: isolated model on the official CLI
Each model runs on its own vendor's official CLI — the stack the vibe coder actually uses — but in base mode, in a clean environment: no skills, no MCP, no subagents (explicitly blocked). In one round, a model spawned a subagent to review its own work — it was disqualified. The rig measures the model, not the orchestration around it.
03 · One shot, verbatim prompt
The canonical prompt goes byte for byte (with a logged hash), no prefix and no help. One attempt per model. Usage and cost come from each CLI's own primary source and get recorded on the run.
04 · Machine judgment, against an answer key
No human judge, no model judge — both can be gamed. Runs are anonymized, the artifact is measured in the browser against a functional answer key, and the verdict comes from what the app DOES: shipped, shipped with defects, or broke.
05 · The score that decides the harness
Consolidated score: 0.5 quality + 0.25 price + 0.25 speed — all derived from measurement. Golden rule: no invented numbers; when in doubt, it's a fail. The result feeds straight into the harness: which model goes into which slot.
The scoreboard: who ships, with defects, or breaks — and at what cost
Verdicts come from what the app DOES in the browser, against an answer key — shipped, shipped with defects, or broke. Filter by the activity that IS your day to day: the right model changes with the request.
Consolidated score across the 9 tests: 0.5 quality + 0.25 price + 0.25 speed — all derived from measurement.
| # | Model | Score | Shipped | Defects | Broke | Battery cost |
|---|---|---|---|---|---|---|
| 01 | Grok Composer 2.5 Fast | 0,936 | 7 | 2 | 0 | US$ 0,28 |
| 02 | Grok 4.5 | 0,927 | 8 | 1 | 0 | US$ 1,40 |
| 03 | GPT-5.3 Codex Spark* | 0,887 | 6 | 1 | 1 | US$ 0,95 |
| 04 | GLM 5 Turbo | 0,840 | 7 | 1 | 1 | US$ 0,56 |
| 05 | Sonnet 4.6 | 0,831 | 7 | 2 | 0 | US$ 3,49 |
| 06 | GPT-5.6 Luna | 0,831 | 6 | 3 | 0 | US$ 1,20 |
| 07 | Haiku 4.5 | 0,826 | 6 | 1 | 2 | US$ 0,53 |
| 08 | Opus 4.8 | 0,792 | 7 | 2 | 0 | US$ 3,86 |
| 09 | GLM 5.2 | 0,785 | 6 | 3 | 0 | US$ 1,21 |
| 10 | GPT-5.6 Terra | 0,776 | 7 | 2 | 0 | US$ 6,22 |
| 11 | Sonnet 5 | 0,765 | 6 | 3 | 0 | US$ 3,22 |
| 12 | Fable 5 | 0,754 | 8 | 1 | 0 | US$ 7,43 |
| 13 | GPT-5.4 | 0,751 | 6 | 3 | 0 | US$ 3,64 |
| 14 | GPT-5.4 Mini | 0,700 | 7 | 1 | 1 | US$ 1,23 |
| 15 | GPT-5.6 Sol | 0,640 | 6 | 3 | 0 | US$ 14,50 |
| 16 | GPT-5.5 | 0,635 | 5 | 4 | 0 | US$ 13,40 |
* judged on 8 of 9 tests so far — the missing run lands in the study's next update. This scoreboard is an editorial cut: the full study, test by test, with transcript, cost and evidence for every run, is at bench.overclock.sh.
Watch the full breakdown
The “that changed everything” launch, dissected: benchmarks vs. the official announcement, the launch timing, and bench maxing in practice.
watch on youtubeThe cockpit is your bench-maxing bench
Panes side by side, each with its own model and CLI, tokens and cost updating in real time — a new model drops, you run your own test in minutes and decide with your own numbers. The category comparison shows it: no one else gives you this bench.