Crux in sessions: the benchmark that forced a result-format redesign

This is the technical companion to Crux in real sessions: 90% correct, 47% cheaper per answer. It covers the session-shaped methodology, the complete numbers, and the part we care most about: the token blowup the benchmark exposed on sympy, and the Crux 0.6.2 redesign it forced.

Method

• Sessions, not questions. Each run gives the agent six code-navigation questions in one continuous conversation — definition location, callers, reference files — drawn from the same verified question bank as the single-question benchmark. Fixed costs amortize; context grows the way real sessions do.
• Corpus: django (2,572 files, 344k lines) and sympy (1,555 files, 770k lines), pinned commits. Four sessions per repo per arm.
• Arms: vanilla (grep only) and Crux (index registered, one routing sentence in the prompt). Same agent (Codex CLI, gpt-5.6-sol, xhigh), metered through a local passthrough proxy for identical token accounting. Compression off — this write-up isolates the index; the full-stack measurement is separate.
• Scoring: definitions match on file plus line (±2); set answers score F1 after normalization; “correct” means F1 = 1.0.
• Binaries: every number below is the shipped Crux 0.6.2 unless explicitly labeled “before.”

Headline result

Suite totals: Crux 43/48 (90%) at 1,360,699 tokens; vanilla 28/48 (58%) at 1,663,040. That is 18% fewer raw tokens and 47% less per correct answer — on sympy alone, 59% less.

Note the shape, not just the totals: every Crux session on both corpora ran 13–14 turns with exactly 4 index calls. The variance collapsed along with the cost.

The blowup this benchmark exposed

The first sympy session runs looked nothing like that table. Crux-arm sessions ran 29–38 turns, made 7–10 index calls, and burned 474k–649k tokens each — 2,247,104 total for 16/24. Accuracy still beat grep by 17 points; cost was 2.3× worse. Three compounding causes, all invisible on django:

1. Uncapped ambiguity dumps. A bare name like solve matches dozens of definitions in sympy. scip_map returned the entire candidate list — no cap, no ranking, no reference data — and closed with “specify the qualified name.” Every ambiguous name triggered a 3–4 call drill-down loop: map → candidate dump → find → map again with a qualified name.

2. Truncation that re-bills. With 400 references and a display limit of 20, the truncation tail said “raise ref_limit.” The obedient agent re-issued the call at the maximum, re-transmitting the 20 lines it already had plus 180 more — each line carrying a source snippet read from disk. There was no way to request the rest; only everything, again.

3. Alphabetical reference ordering. Truncation kept whatever sorted first by path, so a cut list was an arbitrary sample. Agents learned to distrust it and page.

The 0.6.2 redesign

Five changes to the result surface, none to the index itself:

• Ambiguity is ranked, capped at 10, grouped by file — and when one candidate clearly dominates (exact display-name match, uniquely top-ranked), Crux auto-resolves it and returns the normal answer block with other candidates: N noted. The common case collapsed from four calls to one.
• Grouped references above 30 sites. Per-line source snippets give way to file.py: 12, 40, 91 — the same locations at a fraction of the tokens. Snippets remain for small result sets, where they carry real signal.
• Real pagination. Every truncated section says … 50 more (pass offset=20). Offsets thread through find, map, references, and outline — which previously had a hard 38-line cap with no parameter at all, silently pushing agents to read whole files.
• Relevance ordering. References sort definition-file first, then same top-level package, then the rest — so a truncated list is the useful prefix rather than the alphabetical accident.
• A global line cap (250 lines) bounds the worst case across a multi-name call, and the compact callers:/files: summary lines are emitted only when they are complete, never as a truncated list that reads as complete.

Measured on a real sympy symbol with 70 references: the default response went from ~70 snippet lines plus duplicate summaries to 30 compact lines. The complete answer is one ref_limit=200 call away at 84 lines.

One hardening change shipped alongside, bought with a full wasted bench run: a crashed indexer had left a 91-byte stub index that answered a plausible-looking “no symbol named X” to every query, and two sessions measured garbage before we caught it. Crux 0.6.2 treats a zero-document index as an explicit error — “index is empty (0 documents), likely a crashed indexer; run scip_index to rebuild” — for every query tool, and crux check warns on it. Related operational note: SCIP indexes are content-addressed to the commit, so a healthy index built in one checkout can be copied to another checkout of the same commit — which is also how we recovered when scip-python itself crashed on sympy under two different Node majors.

Every session

sympy, Crux 0.6.2 (before-fix range for comparison: 474k–649k tokens, 29–38 turns):

django, Crux 0.6.2:

Vanilla baselines:

Amortization

In matched sessions, the per-question premium over vanilla melts as context builds: +29% (question 1), +32% (2), +7% (3), +5% (4), +6% (5), +0.3% (6). The first two questions pay for MCP setup and index warm-up; from question three the index answers at grep prices.

Honest limits

Four sessions per cell, one run each; the 3.3× before/after token gap on sympy is far outside observed between-run variance, single-digit accuracy deltas are not. One model configuration; Python-only corpus; six-question sessions of navigation questions — not edit tasks. Scoring does not bill the downstream cost of acting on a wrong answer, which favors the cheap-but-wrong baseline.

Takeaway

The index was never the expensive part — the conversation about the index’s output was. Result shaping is not cosmetics for an agent-facing tool: “raise the limit” versus “pass offset=20” is the difference between 2.3× grep and 41% under it on the same repo, same questions, same index. Crux 0.6.2 is out today, open source, MIT: github.com/pedr0v/crux.