Back to TokenSize
Routing evaluation protocol

Benchmarks that survive contact with reality.

TokenSize measures the complete routing outcome: which model was selected, what it produced, how an independent judge graded it, how much it cost and how long the routing layer took.

01
Grade outputs, not selections

Agreement with another router is diagnostic. The real outcome is whether the selected model produced the better answer for the prompt.

02
Blind the judge

Answer order and router identity are hidden from graders. Position is randomized and ties remain valid outcomes.

03
Keep the economics attached

Quality, model cost, time-to-first-token and routing overhead are reported together. A cheaper loss is still a loss.

04
Publish prompt-level evidence

Aggregate metrics are accompanied by prompts, selected models, answer references and grading rationale so disagreements can be audited.

End-to-end test flow

From prompt to verdict

  1. 01
    Freeze the test manifest

    Prompts, candidate models, objectives, model parameters and router versions are recorded before execution.

  2. 02
    Route independently

    TokenSize and the comparison service receive the same prompt and candidate pool without seeing each other's decision.

  3. 03
    Execute the selected models

    Each selected upstream model produces a full answer under matching generation settings. Failures and timeouts remain in the dataset.

  4. 04
    Grade blind and aggregate

    Judges compare correctness, instruction following and usefulness. Results are paired with cost and latency before summary statistics are generated.

Metric definitions

Win rate

The share of non-tied prompts where one router's selected model produced the preferred answer.

Selection agreement

How often two routers choose the same model. Useful for explaining behavior, but not proof of answer quality.

Quality-adjusted cost

Total model spend considered alongside wins, losses and ties so savings cannot hide degraded answers.

Routing overhead

Time consumed by selection before provider inference, reported as percentiles rather than a single best-case number.

Measured value — July 21, 2026

~67–93% lower estimated cost

Balanced routing over 25 representative coding tasks, using the production router and its own cost estimates: ~67% cheaper than a mid-tier (Sonnet-class) default, ~88% than a premium default, ~93% than a frontier default. Chosen models held 94.5% of the frontier's predicted quality with every choice inside the quality-safe set. Cost and quality are catalog estimates, not measured answer quality.

Quality measured, not asserted

On six coding tasks graded by running the code, every model the quality-safe set admits — Opus, Sonnet, grok — passed 6/6, matching the strongest model. The cheapest free model, which the safe set excludes, passed 0/6. That threshold is why TokenSize will not route everything to the cheapest option. In quality mode 100% of requests stay on the strongest model — asking for quality can cost more, never less.

3.5× more workload shared

With real discovery on a machine whose primary subscription sat at 5% remaining, allowance-aware routing moved 58% of a 24-task batch onto healthier subscriptions the developer already owns, versus 17% when the primary was healthy. A real inspect task executed on an unmetered local model at zero API cost.

Reproducible

Raw results and the exact scripts live in the repository under benchmark/value and tools/e2e. Numbers vary with your catalog, installed tools, authentication, and quotas.

Claim policy

What we will not do

  • Present synthetic selection agreement as model quality
  • Remove provider failures from the denominator
  • Publish a comparative headline without a dated artifact
  • Call estimates measurements without labeling them

Don't trust the summary. Inspect the trace.

The launch benchmark will include the full prompt-level decision report and the exact harness configuration used to produce it.

Run the public router