This is the technical companion to One of Halv’s tools made AI coding agents 45% more accurate. It covers how we benchmarked Crux — our open-source SCIP-backed code-navigation MCP server — against plain grep, the full numbers, and the redesign of Crux the numbers forced.
Method
- Corpus: django (2,572 files, 344k lines) and sympy (1,555 files, 770k lines), pinned commits. Both exceed the activation threshold Halv uses for Crux (200 files or 100k lines per language).
- Questions: 50, three types — definition location (18), callers (16), reference files (16). Generated from the SCIP index, then a seeded 20% sample verified with an independent pure-text check so the index never grades itself. Caller questions cap at 10 callers per symbol.
- Arms: the same agent (Codex CLI,
gpt-5.6-sol,xhigh), one question per session, both arms metered through a local passthrough proxy for identical token accounting. The Crux arm registers the MCP server, gets a pre-built index (built once per repo, ~60s, off the clock), and one routing sentence in the prompt. - Scoring: definition answers match on file plus line (±2). Set answers (callers, reference files) score F1 after symbol-format normalization; “correct” means F1 = 1.0.
- Compression and command filtering are off. This benchmark isolates the index. The session-shaped benchmark that measures the full stack is separate.
Headline result — vanilla vs Crux 0.6 enhanced
| Question type | n | Vanilla correct | Crux correct | Vanilla tokens/q | Crux tokens/q | Crux index calls/q |
|---|---|---|---|---|---|---|
| Definition location | 18 | 18/18 | 18/18 | 46,585 | 45,824 | 0.00 |
| Callers | 16 | 11/16 | 15/16 | 90,997 | 82,088 | 1.00 |
| Reference files | 16 | 4/16 | 15/16 | 50,123 | 79,836 | 1.00 |
| Total | 50 | 33 (66%) | 48 (96%) | — | — | — |
Totals: Crux 3,415,633 tokens vs vanilla 3,096,478 (+10.3%) for 15 more correct answers. Tokens per correct answer: 71,159 vs 93,832 (−24.2%). On definitions the routed agent skips the index entirely and still comes out slightly cheaper than vanilla. On callers it is cheaper AND more correct. The only premium — reference files, +59% per question — buys accuracy from 25% to 94%.
Against the 0.5.1 baseline leg (same questions, same accuracy): total Crux-arm spend fell 25% (4.55M → 3.42M). The redesign kept every correct answer and dropped a quarter of the cost.
What the wrong answers cost
Splitting each cell by outcome exposes the real economics — wrong answers are not discounted:
| Arm · type | Correct: n, mean tokens | Wrong: n, mean tokens |
|---|---|---|
| vanilla · references | 4 × 45,749 | 12 × 51,582 |
| vanilla · callers | 11 × 89,214 | 5 × 94,920 |
| crux · references | 15 × 79,379 | 1 × 86,689 |
| crux · callers | 15 × 81,690 | 1 × 88,054 |
Vanilla’s wrong answers cost more than its correct ones — full exploration, wrong conclusion. In total, vanilla spent 1,093,584 tokens (35% of its entire budget) producing wrong answers. Crux wasted 174,743 (5.1%). The 24% tokens-per-correct advantage is mostly this: not cheaper answers, but almost no spend on garbage.
Restricting to the 33 questions BOTH arms answered correctly: callers −12.8% for Crux (n=11), definitions −1.6% (n=18, the skip working as designed), references +80.3% but n=4 — the only references vanilla got right were the easy ones. Where both are right, Crux is cheaper or even; the premium exists only where vanilla is usually wrong.
The tuning trail
We ran this benchmark four times, and each run changed the tool. That is the point of owning a harness.
| Iteration | Design | Result | Lesson |
|---|---|---|---|
| 0.5.1 | 8 tools, verbose schemas, blanket “use the index” prompt | 96% vs 66% accuracy; tokens/correct equal (~94k both) | The index converts spend into correctness — but ~988 tokens of schema rode every request, and easy questions paid the tax for nothing |
| 0.6 slim | 4 consolidated tools (~400-token schemas), find→map disambiguation, scip_expand for AI-selected surface |
Flat ~70k/question, 1.0 calls — but find→map double-called on unambiguous names | Mandatory disambiguation taxes the common case; 6-run sample was enough to see it |
| 0.6 + hedged prompt | Auto-resolve in scip_map; prompt: “text search is cheaper for simple definitions” |
Definitions fixed (0 calls, grep parity) — but callers and references regressed to 125–156k/question | A cost-hedge in the prompt made the model grep first and index second, paying for both. Doubt is expensive |
| 0.6 enhanced (final) | Same binary; prompt routes by question type: “go straight to the index for callers/references/impact; only simple definitions are cheaper by search” | Definitions at grep parity with 0 calls; callers cheaper than vanilla; references one flat call; final: 96% accuracy, −24.2% tokens/correct vs vanilla | Route by question shape, positively. Never make the model second-guess a category it should own |
Two more mechanisms landed in 0.6 along the way: scip_expand lets the agent reveal the full narrow toolset mid-session via tools/list_changed (verified working in Codex CLI), and --profile full restores the legacy 8-tool surface for clients that want it advertised up front.
Five cases from the 0.5.1 baseline leg
These are from the first full run (old surface, blanket prompt) — they motivated the redesign.
Where the index saved tokens. sympy-17, callers of TR0: 161,642 tokens by grep, 97,932 by index — 39% less, both correct. django-05, callers of max_y: 155,154 vs 97,639.
Where it fixed the answer AND saved tokens. sympy-11, callers of sylvester: vanilla spent 123,610 and missed half the callers (F1 0.43); Crux spent 97,316 for the full set.
Where it fixed what grep cannot fix. Fifteen wrong-to-right flips. django-09, files referencing get_page: vanilla returned five files — the definition, three docs, one real reference. Four of five were textual mentions. The index returned exactly the one referencing file. Grep cannot tell a mention from a reference.
Where it cost more. sympy-20, callers of _as_int_ceiling: vanilla correct for 64,646; Crux correct for 229,139 across eight calls chasing an ambiguous name. This case directly produced 0.6’s one-call disambiguation and auto-resolve.
Where both failed. Two caller questions with heavily overloaded names beat both arms. They stay in the suite; a benchmark with a 100% ceiling teaches nothing.
What did not work: organic adoption
We measured organic adoption — the model reaching for the index with no prompt text about it — three ways in Codex CLI: server instructions on 0.5.1 wording (0 of 40 sessions), server instructions rewritten trigger-first for 0.6 (0 of 10), and trigger-worded tool descriptions, the text the model reads at decision time (0 of 10). Accuracy without the index fell to grep level. The conclusion is structural, not a wording problem: Codex initiates tool use from prompt-space text only; MCP metadata shapes how a tool is called, never whether the model reaches for it. Claude Code surfaces server instructions far more prominently. If you ship an MCP server for codex users, something must put the tools into the prompt itself — which is why crux setup codex installs a marker-delimited note in AGENTS.md, the one channel that measurably works.
Single-question sessions understate the index
Every run here is one question in one fresh session, so each answer carries the full fixed cost alone: MCP session setup, the tool-call round trip, and the schema bytes on every request of that session. That structure maximizes the index’s overhead and minimizes its amortization. Multi-question sessions — the shape of real agent work — spread those fixed costs across every question in the conversation, so the per-question gap should widen in the index’s favor. We built a separate session-shaped benchmark to measure exactly that; its results get their own write-up.
Honest limits
One repeat per cell on the enhanced leg; one model configuration; Python-only corpus; the index pre-built off the clock (60s once per repo, reused across all sessions — the amortization a real project gets, but a first session pays it). Single-shot scoring does not bill the downstream cost of acting on a wrong answer, which understates the index’s value.
Crux is MIT and fully local: github.com/pedr0v/crux.