Reasoning-Model Tuning Asymmetry#

Practitioners assume “better prompt = better output”. For one model class, that assumption is correct. For the other, the same prompt makes things measurably worse. This article documents the asymmetry, names the dividing line, and gives you a 4-cell test to confirm it on your own canary before you commit to a prompt.

The asymmetry is empirical, not theoretical. It shows up cleanly across four independent OFAT (one-factor-at-a-time) matrices run between 2026-05-18 and 2026-05-20: sonnet POC, grok matrix v1+v2, deepseek matrix v1, kimi matrix v1.

Decision-first: Evaluate on the agent loop (read/edit/test/push), not one-shot patches. Use a multi-file execution-stamina task as your discriminator, tune OFAT at N≥3, and distinguish turn-ceiling vs token-ceiling vs capability-ceiling — only the last is unfixable by config.

Scope & freshness: Methodology is durable; the named results are 2026-05 snapshots — re-run the harness for current models.

Why public leaderboard scores mislead#

SWE-bench-style and chat leaderboards measure something adjacent to, but not the same as, autonomous tool-using coding. A model can score well on one-shot patch generation and still fail as an agent because the agent loop demands sustained, multi-turn behavior: read files, edit several, run tests, react to failures, and push — without giving up, looping, or declaring “done” early. Evaluate on the loop you’ll actually run.