The Self-Ask Trap#

Practitioners ask the LLM about itself as a research shortcut: “What are your common quirks? What temperature should I use? Do you need reasoning_content echoed in multi-turn?” The output looks plausible, often cites specific behaviors, sometimes includes API parameter names. It is often wrong.

The 2026-05-20 kimi-k2.6 tuning research surfaced a clean example. Self-ask said one thing. Documentation, partner adapter source, GitHub issues, and direct API probes said the opposite. The model is provably wrong about itself, and the failure mode is structural — not specific to kimi.

xAI Grok Operational Quirks#

xAI’s Grok API is OpenAI-compatible on paper. In practice it has more wire-format edge cases than any other provider in production: error responses change shape, rate-limit pages come back as HTML, assistant turns reject missing fields with HTTP 422, and the two flagship models (grok-4.3 and grok-4.20-reasoning) have incompatible parameter sets. Wrap it carelessly and the adapter crashes the conversation mid-turn.

This page is the production-confirmed quirks list, each as Symptom → Cause → Fix → Verify. Numbers come from two OFAT matrix runs (15 cells × N=3 baseline, 3 cells × N=5 validation) on api.x.ai and the heavy-tier POC. Full synthesis: ~/.claude/projects/-Users-mstather/memory/project_xai_adapter_wireerror_bug_2026_05_19.md and project_grok_matrix_v1_2026_05_19.md.

The Five-Agent Research Pattern#

Adopting a new LLM provider for a coding-agent role looks easy from the docs. Read the model card, copy the partner adapter’s defaults, ship. A week later you find out the provider rejects tool_choice=required in thinking mode, the docs lied about reasoning_content echoing, and your retry loop multiplies the per-turn timeout by 3x because the rate-limit response isn’t JSON.

The docs miss what was patched after release. The community catches what the docs miss. Partner adapters encode lived defaults nobody published. Your own adapter has bugs you can’t see from inside it. Reading any one of these in isolation gets you to “I think I understand this provider.” Reading all five in parallel gets you a knob list, an open-contradictions list, and a list of bugs to fix before the matrix runs. The pattern: spawn 5 parallel research sub-agents, one per angle, then synthesize.

Tiered-LLM Tooling: Local by Default, Escalate to Frontier#

When you build a chat or ops interface backed by an LLM, paying a frontier model for every interaction is wasteful — most interactions are cheap lookups, summaries, and routing. A tiered design serves the high-frequency majority with a small local model (e.g. an Ollama-served model on a GPU you already have) and escalates to a frontier model (e.g. Claude) only for the hard minority.

Decision-first: Follow this order and you’ll have a deployable model + tuned config in days, not weeks: (1) scope the hardware, (2) shortlist by active params, (3) per-model OFAT matrix, (4) run serially with an OOM guard (smoke first), (5) write a finding card per model, (6) decide. The expensive mistakes are skipping the smoke step, sweeping more than one factor at once, and trusting a single run.

Scope & freshness: Process is model/hardware-independent; the worked numbers are from a 2026-05 effort on a GB10 (128 GB) + an Apple-Silicon Mac, evaluating local MoE models vs cloud baselines for agentic coding. Re-validate the findings, not the workflow.

Decision-first: Write research docs so a downstream agent can act in 30 seconds and verify cheaply. Lead with the recommendation + its biggest caveat; attach a one-line verification recipe to every load-bearing claim; put what didn’t work where it can’t be missed. Descriptive “how X works” prose is the least valuable part.

Scope & freshness: Format conventions, version-independent. Authored 2026-05-25 from a local-LLM benchmarking effort; the examples are LLM/GPU but the format applies to any research/benchmarking knowledge.

Decision-first: Evaluate on the agent loop (read/edit/test/push), not one-shot patches. Use a multi-file execution-stamina task as your discriminator, tune OFAT at N≥3, and distinguish turn-ceiling vs token-ceiling vs capability-ceiling — only the last is unfixable by config.

Scope & freshness: Methodology is durable; the named results are 2026-05 snapshots — re-run the harness for current models.

Why public leaderboard scores mislead#

SWE-bench-style and chat leaderboards measure something adjacent to, but not the same as, autonomous tool-using coding. A model can score well on one-shot patch generation and still fail as an agent because the agent loop demands sustained, multi-turn behavior: read files, edit several, run tests, react to failures, and push — without giving up, looping, or declaring “done” early. Evaluate on the loop you’ll actually run.

Decision-first: Per new model, sweep temperature (don’t assume 0.3), try reasoning off for builders, test echo_reasoning both ways, and on budget_exceeded check turns-vs-tokens before changing either. The right config is model-specific — assume nothing.

Scope & freshness: Local + cloud models for agentic coding, 2026-05. Findings are per-model (see the specific models named); treat them as examples of shape, not universal constants — re-sweep for any new model.

An agent is dispatched on a task it cannot complete. The spec is broken. The dependency is missing. The credentials are wrong. What happens next determines whether you have an autonomous fleet or a fleet that quietly fails.

The most common answer — instructing the agent in its prompt to “ask for help if stuck” — does not survive contact with production. Agents either keep grinding and produce broken work, or output text that looks like a question but never reaches a human, or politely “complete” the task by writing nothing and reporting success. None of these failure modes are visible from the outside until the dashboards have been lying for hours.

You need to know whether model-X is worth deploying for your real workload. The benchmarks suggest yes, but benchmarks are static and your workload is not. The standard answer — build an eval harness — runs into two structural problems: harnesses are expensive to build well, and they tend to over-fit to the inputs you remembered to include in the corpus, missing the real production failure modes you discover only later.