Four separate bugs in my grader produced plausible, wrong numbers, and none of them raised an error. Fixing them took one headline result from −20 points to −3.2. If you measure local LLMs, the grader is where the danger is.
I measured twenty-seven models on a Strix Halo with a power meter running, on 190 small tasks with checkable answers and 100 questions of GPQA Diamond for eleven of them. The earlier posts in this series cover the results. This one covers how I measured them, how much of the data is noise, and where the conclusions stop.
Grader bias cancels in a ratio and does not cancel in a ranking. Re-grading left every within-model comparison alone and moved the cross-model ranking four separate times.
Audit your grader before you trust a ranking
The bias wasn't random. One comparator quirk cost the reasoning models up to thirty answers each and cost llama and phi nothing. Another cost llama and phi twenty-eight and twenty-seven answers, and the reasoning models next to nothing. Each bug favored a different group of models, and each looked like a legitimate difference between them.
If you repeat this kind of work, compare a model against itself across conditions first, since that comparison survived every grader fix. Treat a cross-model ranking as provisional until you have read a sample of the answers the grader marked wrong.
Budget for four percent noise
I ran trinity-mini twice under identical settings by accident, which gave me a free reproducibility check. Nothing is sampled and temperature is zero, so the runs should match exactly. They don't, because four concurrent requests batch differently from run to run.
| trinity-mini, same weights, same settings | run 1 | run 2 | spread |
|---|---|---|---|
| pass | 90.5% | 86.3% | 4 tasks |
| tokens per answer | 1,535 | 1,594 | 3.8% |
| J per answer | 2,081 | 2,166 | 4.1% |
Cost reproduces to about 4%, and accuracy to about four tasks in ninety-five. That is the resolution of the easy-suite numbers: a 4% cost gap is noise, and so is a four-point accuracy gap. The hard questions come from a hundred-question sample and are noisier still.
Cost reproduces to about 4% and accuracy to about four tasks in ninety-five, which sets the resolution of every easy-suite number in this series.
The four-task spread is the net, not the number of tasks that moved. Twenty-two of the 190 tasks flipped between the two runs. Eight of them were in the terse half, four each way, which cancelled out and left the terse score identical at 86 of 95 both times. The other fourteen were in the free half and didn't cancel. Short answers land on the same result more often. Long generations under concurrency drift, and some of that drift hides inside a total that looks stable.
Related in this series: the headline cost results this noise floor applies to. [CTA: link to Blog 0 once published]
Build on the metric that already exists
The metric is not mine. Accuracy-adjusted energy accounting is established work. TokenPowerBench benchmarks inference power directly, and there is a good argument for energy per token as a standard reporting unit.
The verbosity problem is also well covered. OckBench coined the "Overthinking Tax" and found same-size models at matched accuracy differing 3.3x in tokens. Decomposing Reasoning Efficiency separated accuracy from token cost across 25 models and found the two rankings pulling apart at 0.63. Mine is the same kind of divergence between a different pair of numbers, energy per token against energy per correct answer, and it pulls apart further, at +0.27.
Record the measurement setup
Everything is Q4_K_M, with four concurrent requests unless noted. Energy comes from 1Hz wall-power telemetry, integrated across each run, then split between the two prompt conditions by time share. Prices use my actual California summer consumer rates: 34 cents per kilowatt-hour off-peak and 58 cents on-peak. Hard-task energy figures are derived from wall time and each model's own measured power instead of being metered directly, so treat the hard-task dollars as approximate and the ratios between them as the solid part.
Read the limits before you generalize
This is one box, one suite and one quantization. Prompts are a few hundred tokens, so the work is short-task work and says nothing about long context.
qwen3.6-35b-a3b is where that bites. The other reason I run it is its hybrid attention, and no task in this suite goes anywhere near enough context to exercise it. Read its row as task fit instead of model quality, and treat it as the standing warning against generalizing from any one benchmark, including this one.
The suite saturates, as the fourth post in this series describes. The GPQA numbers that address this cover eleven of the twenty-seven models at 100 questions each, so they carry roughly five points of standard error and cannot resolve gaps smaller than about thirteen points. devstral-2-123b isn't among the eleven, because at 2.1 tokens a second it would have taken twelve hours, so the biggest model here has no hard-task score at all.
Two of the twenty-seven active-parameter counts come from model cards instead of being read from the file, because I deleted those weights during a disk crunch.
phi-4-reasoning's accuracy would be a lower bound if it were budget-limited, so I measured that directly. At four times the cap its accuracy did not move and its cost went up nine times. The cost number was never in doubt, since truncated tokens get paid for either way.
Take the tables across this series as a shortlist and as a list of things worth measuring, then run your own work against the top few.
[CTA: one line pointing to a single live ciq.com page]




