Constraining outputs to a JSON schema lifts schema-valid rate by 9–22 points across all eleven models. Once you do that, a 24B model matches a 70B one on exact-match. Latency, not accuracy, becomes the differentiator.
Every agent framework we have deployed for clients this year failed in the same, boring way: not because the model reasoned badly, but because it emitted a tool call the runtime could not parse. A missing quote. A stray markdown fence. An enum value that almost matched. We wanted a number on how often this happens, per model, under a protocol we could rerun in six months.
So we built a harness. This article is the first published run. We will re-run it quarterly and keep the raw .parquet files in the repo.
Protocol
- Models
- 11 open-weight, 7B–70B, instruct variants
- Hardware
- 2× H100 80GB · vLLM 0.9.2 · bf16
- Dataset
- toolcall-v2 · 200 tasks × 11 models
- Metrics
- exact-match · schema-valid · p95 latency
- Decoding
- temp 0.0 · seed 42 · max 512 tok
- Repo
- github.com/ExakisNelite/bench-toolcall @
a3f9c1e
Each task gives the model a system prompt, 2–6 tool definitions in OpenAI function format, and a user request that requires exactly one call. We score the first emitted call. Exact match means the tool name and every argument equal the reference. Schema-valid means the call parses and validates against the tool’s JSON schema, regardless of correctness. Each model runs twice: once in free-form mode (the model writes the call as text), once in constrained mode (vLLM guided decoding against the schema).
Results
Toggle the mode and the metric. The chart is a small React island; everything around it is static HTML.
| Model | Params | Exact (free) | Exact (constr.) | Valid (free) | Valid (constr.) | p95 |
|---|---|---|---|---|---|---|
| qwen3-32b | 32B | 0.792 | 0.871 | 0.824 | 0.941 | 1.84 s |
| llama-4-scout | 17B×16 | 0.761 | 0.842 | 0.780 | 0.902 | 1.21 s |
| mistral-small-3.2 | 24B | 0.723 | 0.815 | 0.741 | 0.927 | 1.05 s |
| llama-3.3-70b | 70B | 0.768 | 0.809 | 0.802 | 0.915 | 3.10 s |
| gemma-3-27b | 27B | 0.688 | 0.779 | 0.712 | 0.868 | 1.62 s |
| qwen3-8b | 8B | 0.641 | 0.752 | 0.655 | 0.889 | 0.62 s |
| phi-4 | 14B | 0.602 | 0.718 | 0.630 | 0.851 | 0.88 s |
Why schema strictness wins
Two effects stack. The obvious one: constrained decoding removes the parse failures, which were 4–11 % of all free-form calls. The less obvious one is that the model’s choice of tool improves too. With a grammar in place, the first token has to be one of the allowed tool names, which prunes the “let me explain what I’m about to do” preamble that derails smaller models.
from principia.bench import Harness
h = Harness(models=MODELS, seed=42, temperature=0.0)
h.run(tasks="toolcall-v2", n=200, mode="constrained")
h.run(tasks="toolcall-v2", n=200, mode="free")
h.report() # → results/2026-09-02/*.parquetThe gap between 32B and 70B collapses once you constrain. If your budget is a single node, this is the practical finding: spend the memory on context and concurrency, not parameters.
Variance across seeds
We re-ran the top five models with seeds 1–5 at temperature 0.3, which is closer to what people deploy. This is where the story gets uncomfortable, and where we stopped trusting single-run numbers. Three of the five models swap places between runs, and the
Get the full article, the analysis and the raw data
We'll unlock the rest of this article and send you the occasional Principia note — new benchmarks, new demos. No spam.
Limits
Locked — 4 paragraphs.
Reproduce it
Locked — commands, Docker image, expected runtime.