Agent Leaderboards Measure Score. We Added Price.

Agent benchmarks tell us who can solve a task. We are building the missing layer: what each verified result cost.

Every leaderboard answers only half the question

Agent benchmarks are everywhere now: SWE-bench, Terminal-Bench, OSWorld, GAIA. Each has a leaderboard. Each crowns a weekly champion.

They mostly answer one question:

Who scores highest?

But before hiring anyone — human or machine — the practical question is different:

What does it cost to get the task done?

Some of that data already exists. Princeton's HAL publishes recorded run costs alongside scores. Terminal-Bench 2.0 submissions include a result.json for individual task attempts, including dollar cost and pass/fail outcome.

What has been missing is a shared, task-level view that connects:

agent configuration → benchmark task → verified pass → recorded cost

We started assembling that layer as part of the Agent Taxonomy Graph.

What we ingested

• HAL (Princeton): 46 evaluation runs across 9 benchmarks, with reported run-level cost.
• Terminal-Bench 2.0: 75 public submissions and **32,803 per-task result.json files**, each carrying a recorded cost and pass/fail result.
• Curated benchmark observations: merged into a dataset of 131 evaluation runs and 29 model configurations with both a score and a price attached.

Every run keeps its source URL and verification status. This is intended as an evidence layer, not a scraped leaderboard table.

Scope: run-level comparisons are directional, not a normalized ranking. Benchmark mix, harness design, retry policy, context limits, and cost accounting differ across public evaluations. The cleaner comparison is at the level of the same benchmark task and the same verifier pass condition.

What the money says

The spread is the story.

Across the evaluated configurations, recorded cost per benchmark run ranges from roughly $0.03 to more than $1,600 — five orders of magnitude for systems that can appear side by side on agent leaderboards.

The current cost × score Pareto frontier, as of July 2026:

Read it bottom-up:

A configuration costing around $0.06 already clears 60%.

Moving from roughly 62% to 87% score costs around 10–30× more in this sample.

That does not prove that cheap agents plus retries always beat a more expensive single pass. Reliability, verifier cost, retry budgets, and failure modes still matter.

But it does establish the question that should be measured:

What is the cost per accepted result, not just the cost per attempt?

For tasks with cheap verification and room for retries, lower-cost configurations may be much more competitive than leaderboard rankings alone suggest.

Per-task is where it gets interesting

Aggregate scores hide the strongest signal.

When the benchmark task and its verifier are held fixed, the pricing spread becomes hard to ignore:

• pytorch-model-cli — passed by 47 successful priced runs, for $0.001 to $1.47. Same benchmark task, same verified pass condition, 1,109× price difference.
• qemu-startup — 48 successful priced runs, with an 810× cost spread.
• log-summary-date-ranges — 58 successful priced runs; the cheapest, Terminus 2 + DeepSeek-V3.2, cost roughly half a cent.

These counts are individual successful runs, not deduplicated configurations — the same agent × model pair can appear at several prices, because every trial takes its own trajectory: steps, retries, tokens.

They are not identical agent trajectories. But they are comparable verified completions of the same externally checked task.

We have not found a common public price layer for that result.

A reader can see that one configuration passed. They can see that another scored higher overall. What they usually cannot see is that both passed the same task — and one paid three orders of magnitude less.

That is the gap.

Method, in plain terms

The current dataset joins public evaluation artifacts into a common graph:

• Unit of analysis: a recorded evaluation run or task attempt
• Cost: reported API spend in the source artifact
• Success: benchmark verifier pass
• Coverage: 131 evaluation runs, 29 model configurations, and 31,913 extracted per-task price rows
• Evidence: source URLs and verification status retained per run
• Not yet normalized for: infrastructure cost, harness differences, token accounting differences, retry policy, context limits, or reliability across repeated runs

The purpose is not to declare a universal "best agent."

It is to make the cost of a verified outcome visible enough to compare.

The price graph is live

The 31,913 extracted task-level price rows are no longer just feedstock — the task-anchored price view now ships on the ATG dashboard as a dedicated screen.

For each of the 87 (out of 90) Terminal-Bench 2.0 tasks with at least one solved, priced run, it shows:

• the cheapest and median cost of a verified pass,
• the full min→max price range on a shared log scale,
• the solve rate across all recorded attempts (the hardest priced task, make-doom-for-mips, was solved by only 12 of 358 attempts),
• the 20 cheapest successful runs, one click away,
• and a scatter of cheapest-solve price versus how many priced runs solved the task at all.

The next step is publishing the same structure as a knowledge graph: each canonical task a node, every priced verified pass an edge weighted by recorded cost and linked back to its evidence.

The result is not just another leaderboard.

It is a public price layer for the agent market:

Which configurations can complete this verified task — and what did each one pay?

The full map is live on the ATG dashboard: 133 agents, verified capabilities, protocols — and now per-task prices.

Feedback: @maximl