---
title: "Code mode vs tool mode: the agentic retrieval harness matters"
description: "Same retriever and model, different harness: 65% higher recall, 51% fewer prompt tokens, 44% lower cost."
excerpt: "Same retriever and model, different harness: 65% higher recall, 51% fewer prompt tokens, 44% lower cost."
date: "2026-07-14"
authors: ["bergum"]
tags: ["Agentic retrieval"]
category: "Proof"
slug: "same-retriever-fewer-tokens-better-recall"
heroImage: "hero.webp"
---

The agentic retrieval harness can change retrieval quality even when the retriever does not change. We tested that by holding the retriever, corpus, model, and session limits constant, then changing the harness mode: one tool call per model turn or one bounded Python program per model turn. This follow-up to [code mode for agentic retrieval](/blog/code-mode-for-agentic-retrieval) compares tool mode and code mode on the [BrowseComp-Plus](https://arxiv.org/abs/2508.06600) benchmark.

The two modes are:

- **tool mode:** one structured search, read, verify, or curate call per model turn
- **code mode:** one bounded Python program against a read-only Hornet SDK, so a single turn can fan out queries, inspect candidates, fetch documents, scan text, and curate evidence before the next turn

On 100 BrowseComp-Plus questions with `gpt-5.4-mini`, code mode raised GoldRecall from 0.265 to 0.437, a 65% relative gain. It cut prompt tokens by 51% and lowered LLM cost by 44%. GoldRecall is the fraction of labeled gold documents placed in the final curated set, macro-averaged across questions. It measures retrieval quality, not end-to-end answer accuracy.

![Bar chart comparing tool mode and code mode with the same retriever and gpt-5.4-mini. Code mode improves GoldRecall by 65%, cuts prompt tokens by 51%, and lowers LLM cost by 44%.](./cost-recall-summary.webp)

*Same retriever and `gpt-5.4-mini`, different harness.*

## The experiment isolates the harness

The evaluation used [BrowseComp-Plus](https://arxiv.org/abs/2508.06600), a fixed-corpus benchmark for [deep-research agents](/blog/deep-research-is-a-retrieval-problem). Each question is associated with human-labeled evidence and gold documents from a corpus of 100,195 web pages, so sessions can be scored against fixed relevance judgments.

Both modes queried one Hornet deployment:

| Component | Setting |
| --- | --- |
| Corpus | BrowseComp-Plus, 100,195 web documents |
| Questions | First 100 numeric question IDs |
| Retriever implementation | Hornet Query API, lexical-only |
| Scoring profile | BM25 over `text`, `k1=10`, `b=1` |
| Model | `gpt-5.4-mini` |
| Per-question limits | 35 model turns, 60 total searches, 30 curated documents |

This fixed the retriever, corpus, model, scoring, and session limits. The harness mode was the experimental variable: tool mode allowed one search, read, verify, or curate call per turn, while code mode allowed one bounded Python program that could make several of those calls before the next turn.

![Diagram contrasting tool mode and code mode over the same Hornet runtime. Tool mode spends one model turn per search, read, or curate call. Code mode groups search, read, and curate calls inside one bounded Python program per turn.](./fixed-retriever-different-harness.webp)

*Same runtime, different harness.*

## Tool mode serializes the loop

Tool mode follows the [Harness-1 pattern](https://arxiv.org/abs/2606.02373): the model sees a compact render of session memory and chooses one tool call. Query refinement, document reads, verification, and curation each cost a turn, so six query variants mean six turns.

## Code mode combines retrieval work within a turn

Code mode keeps the same session state but lets the model search, read, scan, verify, and curate several results before the next prompt. The model writes Python against a read-only Hornet SDK. In the BrowseComp-Plus app, the SDK exposes `search`, `search_many`, `get_document`, `get_documents`, `raw_query`, and `describe`; the harness adds `scan_candidates`, `scan_docs`, `curate_many`, `verify`, and `end_search`.

Within one Python program, `search_many` runs concurrent queries and `get_documents` performs concurrent document fetches.

For each code-mode turn, the model emits a Python snippet against the app-specific Hornet SDK. The harness validates it, runs it in a constrained sandbox, captures the output, and folds searched candidates, scanned documents, curated evidence, and verification results into the same session state used by tool mode. The code handles query fanout, filtering, document scans, and curation inside one turn, but it cannot write to the corpus or deployment.

The sandbox stays narrow: a read-only retrieval SDK, an AST allowlist, output caps, timeouts, and search limits for each program. Every observed result enters the same candidate pool used by tool mode.

## Code mode makes the agentic retrieval loop cheaper and more effective

Expressing the agentic retrieval loop as code improved recall and reduced spend at the same time: GoldRecall rose from 0.265 to 0.437, while LLM cost per question fell from $0.155 to $0.086. The savings came from [fewer prompt tokens](/blog/the-context-window-is-not-your-database), 95,675 versus 196,173, even though code mode ran more Hornet searches per question, 44.6 versus 28.4, in fewer model turns, 12.5 versus 33.1.

| Metric | tool mode | code mode |
| --- | ---: | ---: |
| GoldRecall | 0.265 | 0.437 |
| Model turns per question | 33.1 | 12.5 |
| Runtime searches per question | 28.4 | 44.6 |
| Searches per model turn | 0.86 | 3.57 |
| Prompt tokens per question | 196,173 | 95,675 |
| Completion tokens per question | 1,833 | 3,258 |
| LLM cost per question | $0.155 | $0.086 |

Cost uses the `gpt-5.4-mini` prices recorded for this run: $0.75/M input tokens and $4.50/M output tokens. Verifier calls are included in the LLM token totals.

Code mode emitted 78% more completion tokens because the model wrote code. Output tokens cost six times as much as input, but the penalty was still smaller than the prompt-token savings from rendering session memory fewer times: 51% fewer prompt tokens and [44% lower LLM cost](/blog/agentic-query-workloads-change-retrieval-cost) per question.

Code mode did 4.2 times as many searches per model turn while using 62% fewer turns overall. It used 74% of the shared 60-search limit, compared with 47% in tool mode. The code-mode harness let the model use more of the available searches without paying for another session-memory render after every query, read, or curation step.

The comparison is deliberately narrow: one model, one corpus, and one lexical retrieval method. That narrow setup is the point. With the retriever, model, and session limits fixed, switching the harness from tool mode to code mode changed both GoldRecall and cost.

## The harness is part of agentic retrieval

Deep research agents need [a harness that lets them run the retrieval loop](/blog/how-we-build-a-retrieval-engine-for-agents) without spending a turn on every query formulation.

Tool mode made the model pay for each query, read, and curation decision in context. Code mode let one bounded Python program fan out, inspect candidates, scan documents, and return only the evidence worth carrying forward.

The broader pattern is that harness design matters for agentic retrieval too. [Harness-Bench](https://arxiv.org/abs/2605.27922) measured harness effects across 5,194 agent trajectories and concluded that capability should be reported for a model-harness pairing, not the model alone.

Code mode does not remove context limits. It lets the model decide where to spend them by running queries concurrently, fetching selected documents, scanning relevant regions, and curating evidence before the next turn. The traces suggest two explanations worth testing across more models and retrieval methods:

- Code gave this model a compact way to express query fanout, filtering, and selection.
- Selective document scans let it inspect more text without repeatedly rendering whole documents into the prompt.

For agents, retrieval quality depends on the model, retriever, and harness together. The retriever returns candidates; the harness determines whether the agent can turn them into evidence before a turn, search, or context limit stops the loop.

*We're building Hornet for teams working on agentic retrieval. For new posts, benchmarks, and early product notes, {% signup-link %}join our user community list{% /signup-link %}.*