Story 2 · Tuning the demo

Giving the answer the rest of the point

Giving the demo's answer the passages either side of each one it chose let it answer all five questions whose answer runs across two passages, where it had answered four. It also made every question 60% more expensive, past the limit I set before measuring. I switched it on anyway, because the better answers are worth it.

Widening shipped, though it missed the bar: it went live switched off, and I switched it on the same day, for better answers at about two-thirds of a cent more a question.

The problem

The demo cuts each PDF into passages of about 900 characters, ending at a sentence. A point that takes two sentences, a rule and its exception or a term and its definition, is often split between two passages. The reranker picks the passages that share the question's words, which is usually only the first half. The answer model then gets half the point and answers with half, or says nothing.

The book in my eval set has a clean example. Asked what sycophancy is and where it comes from, the demo found the passage that defines it, which ends “It is not politeness and it is not indecision.” Where it comes from is in the next passage: “agreeable continuations were reinforced during training because human raters preferred them.” In both runs without widening the answer gave the definition but not that cause.

What changed

The idea comes from Howard's Lantern, whose pipeline calls it widening: once the reranker has chosen a law, the rest of its section is added after it, under a budget. A PDF has no sections, so here the unit is the neighbouring passage. Each chosen passage is joined with the one before and the one after it in the same file. Neighbours that touch become one passage, so nothing is sent twice, and the additions stop at 10,800 characters, which at most doubles what the answer reads. The check that refuses an unsupported figure reads the joined text, because that is what the answer was written from.

Before running anything I wrote five new questions into the golden set, each with its answer in the passage after the one its words point to, and I set the bar widening had to clear: pass more of those five, lose no question that passed both times without it, and cost no more than half again as much a question, and take no more than half again as long, both at the median. Story 1 explains how the harness measures.

What we measured

Two runs without widening and two with it, all 30 questions each time, on the live models. The models vary from run to run, so each side ran twice.

On answers, widening did what it was for. The sycophancy question failed both times without it and passed both times with it, giving the definition and the cause. It also fixed a question from story 1: asked why the book's worked example was blocked, the demo had given one of the two reasons, and with widening it gave both, in both runs. No question that passed in both runs without widening failed in both runs with it. One got worse: asked which review lenses the book uses, it named all six in one run without widening, and in both runs with it named five and said a sixth comes in a later chapter. That is a partial answer, not a wrong one.

On cost, it failed. The answer model read about 6,900 tokens a question instead of about 3,900, and the median question cost $0.0176 against $0.0110: 60% more, past the 50% I allowed. Time barely moved, 4.4 seconds against 4.3 at the median. The four runs cost $1.66.

Missing the bar, and switching it on

By the rule I wrote before the runs, widening failed, and it went live switched off. Then I looked at what the rule was protecting. The extra 60% is about two-thirds of a cent a question. What it buys is answers that give the whole point instead of half of it, and that is the thing the demo exists to show. So I switched it on the same day.

The bar did its job anyway. It made the trade visible: a gain I can name, at a price I can name, decided by a person rather than slipping in because the answers looked better. Had I tuned without it, I would have shipped widening without knowing what it cost. And the rule stays as written, so the runs and the decision sit side by side on this page.

What this means for your project

Widening is a good idea that costs real money on every question, and whether it is worth it depends on what a missed half-answer costs you. The useful part is not the idea, which is old, but knowing the price and the gain before deciding, from questions written before the change was measured. That is how the grounding and eval review works on an existing system, and how a retrieval pipeline build is tuned from the start, as part of AI built on your company's data.