Code mode vs tool mode: the agentic retrieval harness matters
Same retriever and model, different harness: 65% higher recall, 51% fewer prompt tokens, 44% lower cost.
Jo Kristian Bergum6 min read
The agentic retrieval harness can change retrieval quality even when the retriever does not change. We tested that by holding the retriever, corpus, model, and session limits constant, then changing the harness mode: one tool call per model turn or one bounded Python program per model turn. This follow-up to code mode for agentic retrieval compares tool mode and code mode on the BrowseComp-Plus benchmark.
The two modes are:
On 100 BrowseComp-Plus questions with gpt-5.4-mini, code mode raised GoldRecall from 0.265 to 0.437, a 65% relative gain. It cut prompt tokens by 51% and lowered LLM cost by 44%. GoldRecall is the fraction of labeled gold documents placed in the final curated set, macro-averaged across questions. It measures retrieval quality, not end-to-end answer accuracy.
Same retriever and gpt-5.4-mini, different harness.
The evaluation used BrowseComp-Plus , a fixed-corpus benchmark for deep-research agents. Each question is associated with human-labeled evidence and gold documents from a corpus of 100,195 web pages, so sessions can be scored against fixed relevance judgments.
Both modes queried one Hornet deployment:
| Component | Setting |
|---|---|
| Corpus | BrowseComp-Plus, 100,195 web documents |
| Questions | First 100 numeric question IDs |
| Retriever implementation | Hornet Query API, lexical-only |
| Scoring profile | BM25 over text, k1=10, b=1 |
| Model | gpt-5.4-mini |
| Per-question limits | 35 model turns, 60 total searches, 30 curated documents |
This fixed the retriever, corpus, model, scoring, and session limits. The harness mode was the experimental variable: tool mode allowed one search, read, verify, or curate call per turn, while code mode allowed one bounded Python program that could make several of those calls before the next turn.
Same runtime, different harness.
Tool mode follows the Harness-1 pattern : the model sees a compact render of session memory and chooses one tool call. Query refinement, document reads, verification, and curation each cost a turn, so six query variants mean six turns.
Code mode keeps the same session state but lets the model search, read, scan, verify, and curate several results before the next prompt. The model writes Python against a read-only Hornet SDK. In the BrowseComp-Plus app, the SDK exposes search, search_many, get_document, get_documents, raw_query, and describe; the harness adds scan_candidates, scan_docs, curate_many, verify, and end_search.
Within one Python program, search_many runs concurrent queries and get_documents performs concurrent document fetches.
For each code-mode turn, the model emits a Python snippet against the app-specific Hornet SDK. The harness validates it, runs it in a constrained sandbox, captures the output, and folds searched candidates, scanned documents, curated evidence, and verification results into the same session state used by tool mode. The code handles query fanout, filtering, document scans, and curation inside one turn, but it cannot write to the corpus or deployment.
The sandbox stays narrow: a read-only retrieval SDK, an AST allowlist, output caps, timeouts, and search limits for each program. Every observed result enters the same candidate pool used by tool mode.
Expressing the agentic retrieval loop as code improved recall and reduced spend at the same time: GoldRecall rose from 0.265 to 0.437, while LLM cost per question fell from $0.155 to $0.086. The savings came from fewer prompt tokens, 95,675 versus 196,173, even though code mode ran more Hornet searches per question, 44.6 versus 28.4, in fewer model turns, 12.5 versus 33.1.
| Metric | tool mode | code mode |
|---|---|---|
| GoldRecall | 0.265 | 0.437 |
| Model turns per question | 33.1 | 12.5 |
| Runtime searches per question | 28.4 | 44.6 |
| Searches per model turn | 0.86 | 3.57 |
| Prompt tokens per question | 196,173 | 95,675 |
| Completion tokens per question | 1,833 | 3,258 |
| LLM cost per question | $0.155 | $0.086 |
Cost uses the gpt-5.4-mini prices recorded for this run: $0.75/M input tokens and $4.50/M output tokens. Verifier calls are included in the LLM token totals.
Code mode emitted 78% more completion tokens because the model wrote code. Output tokens cost six times as much as input, but the penalty was still smaller than the prompt-token savings from rendering session memory fewer times: 51% fewer prompt tokens and 44% lower LLM cost per question.
Code mode did 4.2 times as many searches per model turn while using 62% fewer turns overall. It used 74% of the shared 60-search limit, compared with 47% in tool mode. The code-mode harness let the model use more of the available searches without paying for another session-memory render after every query, read, or curation step.
The comparison is deliberately narrow: one model, one corpus, and one lexical retrieval method. That narrow setup is the point. With the retriever, model, and session limits fixed, switching the harness from tool mode to code mode changed both GoldRecall and cost.
Deep research agents need a harness that lets them run the retrieval loop without spending a turn on every query formulation.
Tool mode made the model pay for each query, read, and curation decision in context. Code mode let one bounded Python program fan out, inspect candidates, scan documents, and return only the evidence worth carrying forward.
The broader pattern is that harness design matters for agentic retrieval too. Harness-Bench measured harness effects across 5,194 agent trajectories and concluded that capability should be reported for a model-harness pairing, not the model alone.
Code mode does not remove context limits. It lets the model decide where to spend them by running queries concurrently, fetching selected documents, scanning relevant regions, and curating evidence before the next turn. The traces suggest two explanations worth testing across more models and retrieval methods:
For agents, retrieval quality depends on the model, retriever, and harness together. The retriever returns candidates; the harness determines whether the agent can turn them into evidence before a turn, search, or context limit stops the loop.
We're building Hornet for teams working on agentic retrieval. For new posts, benchmarks, and early product notes, .