Across the eleven local models I ran on GPQA Diamond, the rank correlation between active parameter count and score is negative: −0.50. Two of the three models with the most active parameters sit near the bottom of the list. The model with the most active parameters of all comes fourth.
I measured twenty-seven models on a Strix Halo with a power meter running. Each answered 190 small tasks of my own design, and eleven of them also answered the first 100 questions of GPQA Diamond: four-way multiple-choice physics, chemistry and biology written by domain experts and deliberately Google-proof. All models are Q4_K_M. The first post in this series covers what those hard questions do to cost. This one covers which model characteristics predict the score.
Across two quite different task types, parameter count never once predicted which model to pick off a mixed list. The one shortcut that held is narrow: inside a single family from a single vendor, more total parameters won.
Swap in harder questions when your suite saturates
Every easy-question number I measured comes from my own 190 tasks, and they cannot tell a good model from a great one. gpt-oss-120b answered all 190 correctly in both prompt conditions, so the suite had no headroom left to show that anything was better than anything else. Eight models finished within three tasks of each other, which is inside my measured run-to-run noise.
So I re-ran eleven of them on GPQA Diamond, and the two benchmarks disagree about nearly everything:
| model | my suite | GPQA Diamond |
|---|---|---|
| qwen3.6-35b-a3b | 89.5% | 79.0% |
| qwen3-coder-next | 97.9% | 79.0% |
| ornith-35b | 96.8% | 78.0% |
| gemma-4-31b (dense, reasoning off) | 100.0% | 70.0% |
| gemma-4-26b-a4b (reasoning off) | 100.0% | 67.0% |
| gpt-oss-120b | 100.0% | 64.0% |
| qwen3-30b-a3b-instruct | 97.9% | 63.0% |
| gemma-4-12b (dense, reasoning off) | 100.0% | 61.0% |
| mistral-small-24b | 98.9% | 50.0% |
| gemma-3-27b | 97.9% | 44.0% |
| nemotron-3-super-120b | 98.9% | 40.0%* |
*nemotron-3-super-120b returned nothing at all on 40 of its 100 questions. Its reasoning ran into the token budget and the answer field came back empty, which the scorer counts as wrong. Over the sixty it answered it scores 66.7%, so its true number sits somewhere in that range, and closing the gap needs a re-run at a budget it can't exhaust. No other model had more than three empties.
GPQA spreads these models over 39 points. My suite fits them into ten, in roughly the reverse order, at a rank correlation of −0.32. mistral-small-24b and gemma-3-27b come third and seventh out of twenty-seven on my suite, and ninth and tenth out of eleven on the hard questions.
The cause is mechanical. When every competent model gets the questions right, accuracy can only vary on whether a model finishes inside the token budget. Careful models get cut off and fast ones score 100%. A saturated benchmark ranks models by how quickly they stop talking, so treat its accuracy column as a filter for "does this model work at all," never as a ranking.
On Diamond, PhDs in the field scored 81% and skilled non-experts with half an hour of web access scored 22%. Both numbers are flattered by how the subset was picked: Diamond keeps the questions experts got right and non-experts got wrong. On the unfiltered 546-question set the figures are 65% and 34%.
Ignore parameter count when you pick for hard questions
I sorted the eleven models by score and added what I know about each:
| model | released | arch | active params | GPQA |
|---|---|---|---|---|
| qwen3.6-35b-a3b | 2026 | MoE | 3.45B | 79% |
| qwen3-coder-next | 2026 | MoE | 3.87B | 79% |
| ornith-35b | 2026-06 | MoE | 3.45B | 78% |
| gemma-4-31b | 2026-04 | dense | 30.70B | 70% |
| gemma-4-26b-a4b | 2026-04 | MoE | 3.82B | 67% |
| gpt-oss-120b | 2025-08 | MoE | 5.71B | 64% |
| qwen3-30b-a3b-instruct | 2025-07 | MoE | 3.35B | 63% |
| gemma-4-12b | 2026-04 | dense | 11.91B | 61% |
| mistral-small-24b | 2025-06 | dense | 23.57B | 50% |
| gemma-3-27b | 2025-03 | dense | 27.01B | 44% |
| nemotron-3-super-120b | 2026-03 | MoE | 12.77B | 40%, or 67% over the 60 it answered |
Every correlation in this series is a rank correlation, so here is what one means if you haven't met them. Sort the models by one thing, sort them again by the other, and compare the two orderings. +1 means the two lists are in the same order, −1 means one is the other reversed, and 0 means a model's place in one list tells you nothing about its place in the other. It looks only at order and ignores how big the gaps are, which suits the question here, because "which model do I pick" is a question about order. The Spearman rank correlation is the standard name.
Between active parameters and GPQA score the correlation is −0.50. gemma-3-27b at 27B active scores 44%, and mistral-small-24b at 23.6B scores 50%. Both sit near the bottom. The single largest model by active count, gemma-4-31b at 30.7B, comes fourth on 70%. More parameters don't reliably buy you anything here, and they don't reliably cost you anything either.
"Largest" needs a definition. gpt-oss-120b is 116.8B parameters on disk but only 5.71B of them run on any given token, because 124 of its 128 experts sit out each time. gemma-4-31b is a quarter the size on disk and does five times the work per token, because it is dense. Active count is what you pay for per token, and total size mostly tells you whether the model fits in memory.
Related in this series: Tokens per second won't tell you what an answer costs on a Strix Halo
Treat sparsity and recency as odds, not rules
Size alone didn't order the list, so I tested two more tidy explanations. Neither held.
Sparsity. As a rule of thumb it holds up. The mixture-of-experts (MoE) models average 67% and the dense ones 56%, an eleven-point gap, and the top three are all sparse. If you were picking blind off a list, you would do better betting on sparse.
It does not survive a controlled test. Google shipped three Gemma 4 models in the same month off the same recipe, and I have hard scores for all three:
| model | total params | active params | my GPQA | Google's GPQA | Google's MMLU-Pro |
|---|---|---|---|---|---|
| gemma-4-12b | 11.9B | 11.9B | 61% | 78.8 | 77.2 |
| gemma-4-26b-a4b | 25.2B | 3.8B | 67% | 82.3 | 82.6 |
| gemma-4-31b | 30.7B | 30.7B | 70% | 84.3 | 85.2 |
Inside that family, total size orders the models. Architecture does not, and neither does active size. The sparse model lands exactly where its 25B of total weights says it should, between the 12B and the 31B, while doing a third of the per-token work of either. All three columns agree on that order. Google's two are run on the original weights instead of my Q4_K_M copies, which is a useful check that quantizing didn't invent the ranking. My numbers sit below theirs mostly because I ran these with reasoning off, a choice the second post in this series explains.
gemma-4-26b-a4b reads 3.8B parameters per token and comes out six points above a same-generation sibling that reads 11.9B. Six points is inside my error bars at 100 questions, so read it as "no worse" and not as "better." Google's runs put the same two models three and five points apart in the same direction, and three measurements agreeing beat a coin flip. Either way, three times the work per token changed nothing I can measure, so architecture alone is not doing the work. Sparsity is a reasonable indicator and a poor explanation.
The indicator comes from the older releases, as far as I can tell. Among the 2025 models the MoEs average sixteen points above the dense ones, and among the 2026 models that shrinks to three. The dates on those four 2025 models show the sparse ones came out in July and August and the dense ones in March and June, so within that year sparse and recent are the same set. That gap can't separate the two. The Gemma 4 table above is the only comparison here that holds everything else still.
Recency. Sort by release date and the 2025 models cluster low, 64% and below, every one of them. The 2026 group runs from 79% down to 40%. gemma-4-12b is a 2026 model at 61%, sitting among the 2025 scores, and nemotron-3-super-120b is 2026 and last of everything. Newer is a decent bet and not a rule.
So for hard questions on this box, all three shortcuts are betting odds. The one that held cleanly is the narrowest version of size: inside a single family from a single vendor, more total parameters won on every measurement I could find, mine and both of Google's. Step outside the family and it tells you nothing.
Know where the claim stops
My hard task is graduate science reasoning and nothing else. Big dense models may earn their size on work I didn't measure. What I can say is that across two quite different task types, parameter count never predicted which model to pick off a mixed list.
GPQA covers eleven of the twenty-seven models at 100 questions each, which gives roughly five points of standard error and a resolution of about thirteen points between models. devstral-2-123b, the biggest model in the set, isn't among the eleven. At 2.1 tokens a second it would have taken twelve hours, so it has no hard-task score at all.
The practical version: run your own hardest representative questions against the top few models on a shortlist, and ignore the number in the model's name when you build it.
The last post in this series covers how I measured all of this, including the grader bugs that moved the rankings four separate times.
