Story 5 · Tuning the demo

The typewriter effect, with a fact-checker in the way

The demo used to show nothing until the whole answer was written and checked. Now it checks each sentence as the model writes it and shows it the moment it passes, so the first sentence is on the page a median 1.1 seconds before the last, and no sentence is ever shown before it has passed the check. The first version of this threw away 8 of the 26 answers the demo had been giving, and the eval set caught it in ten minutes, before anyone saw it.

Shipped: every sentence shown has passed the same checks as before, the first one arrives sooner, and no question that passed before was lost.

The problem

Ask the demo a question and, until this week, you looked at the word "Thinking…" for about five seconds. In that time it searched your documents, had a model choose the best passages, had a second model write the whole answer, ran the answer through the check, and only then drew it. Every chat product you have used shows the words as they are written, and the wait felt long by comparison.

The reason I had not done the same is the demo's whole point. An answer citing a passage it was never given, or stating a figure that no cited passage states, is never shown: a program checks every answer before the page gets it. Streaming the model's words to the page as they arrive, the way a chat product does, would show text that the check then refuses. Shown and then taken back is worse than a wait.

What changed

The model now writes plain text with the ids of the passages each sentence rests on in square brackets at the end of the sentence, and a list as one line per item, each ending the same way. The Worker reads the model's reply as a stream, the words as the model writes them rather than the reply when it is whole, and cuts it at sentence ends, by the same rule the old check used, where a list number is not the end of a sentence. Each complete sentence is checked on its own: every id must be a passage the model was given; every figure in it must appear in a passage it cites or in the question; and it may not be a fifth sentence of prose or a thirteenth item of a list. The citation markers are taken out of the sentence before the figures are read, because the ids contain digits. A sentence that passes goes to the page at once. The first that fails ends the answer.

flowchart TD
  M["The model's reply, as it arrives"] --> S["A sentence is complete"]
  S --> C{"Does it cite a passage?"}
  C -- "no" --> H["Held for the next sentence"]
  H --> S
  C -- "yes" --> G{"Every id a given passage, every figure in one, within the caps?"}
  G -- "no" --> X["The answer stops here"]
  G -- "yes" --> P["Shown, with any held sentence"]
  P --> S

The page shows which stage it is at, searching the documents, choosing passages, writing, and then each sentence as it passes, a bullet at a time for a list, with the sources drawn once at the end. When a later sentence fails, the sentences already on the page stay, since each passed its own check, a line says the answer stopped because its next sentence did not check out, and the question counts as one of the ten. When the first sentence fails, or the model writes nothing, the visitor sees the same fallback line as before and nothing of the model's words.

There is one pipeline, not two. The ask produces a sequence of events: the stage, each sentence, a stop, the end. A browser gets them as a stream, one event at a time. The eval harness, the admin page and the history after a reload get the same events drained into the one JSON body they always read. A second implementation for the stream would have been a second place for the check to go wrong.

What the evals caught

Before changing anything I ran the 30-question eval set on the old code, so I would know what the change cost. Story 1 explains how the harness measures. It passed 26 of 30.

The first streamed version passed 19. Eight questions the demo had been answering now fell back, and every one for the same reason: a sentence with no citation. The harness keeps the model's actual reply, and reading it showed the pattern. Asked whether caching a prompt used once is worth it, the model opened with "No, it is a small loss." and cited nothing until the next sentence. Asked what jobs a résumé lists, it wrote "He has held these jobs:" with no ids, then a bullet per job, each cited. The rule I had written, every sentence cites or the answer ends, threw whole answers away over their first five words.

I tried telling the model. The instructions said every sentence without exception, with examples, and a re-run of the eight questions got five back; the three left were the openers. Tightened again with an example opener in the instructions, one of three. The model opens with a short uncited sentence however it is asked.

So the rule changed. A sentence without its own citation is held: not shown, and not fatal. It is released with the next sentence that has one, and checked against that sentence's passages, figures included, so "No, it is a small loss." is shown with the sentence that says why and cites where. A reply that ends on a held sentence stops before it, since nothing came to vouch for it. Nothing unchecked is ever shown; in those answers the first sentence arrives one sentence later. Two full runs on that rule passed 26 and 25 of 30, and no question that passed before failed.

Then the number this was all for looked wrong. In every run the first sentence was arriving within ten milliseconds of the last, as if the whole answer still came at once. It did not. The harness counts the tokens of every model call by reading a copy of the reply, and for a streamed reply its copy held the real one back until the reply was whole, so the harness was measuring itself. Fixed to read the reply as it passes, one last full run gave the real figure.

In the final run the whole answer took a median 5.4 seconds and the first checked sentence was on the page at a median 4.6 seconds; on each answered question the first sentence came a median 1.1 seconds before the last, and at most 2.3 seconds. The two questions that moved from pass to review in that run are questions with a false premise, which the demo is expected to decline; it answered them instead, correctly, by saying the premise is wrong. Cost went up about five percent a question, for a longer set of instructions. The eight runs, three of them partial, cost $2.90.

What it bought, honestly

Most of the wait is before the first word: searching, the reranker, and the model's own thinking before it writes, which streams nothing. Streaming cannot touch that, and the first sentence still takes about four and a half seconds. What it buys is the second half of the wait: the answer reads as it is written instead of landing whole, and the page says what it is doing until then.

The real return was not the second. It was that the first version silently lost a third of the answers, and I knew within ten minutes, from questions written before the change, for about fifty cents. Without the eval set I would have shipped it, read the first few answers, found them fine, and never known what it was doing to the rest.

The code is public: the RAG pipeline for joelstevick.com/demo, with the per-sentence check in demo/src/check.ts, the stream in demo/src/app.ts, and these runs in evals/runs/.

What this means for your project

Streaming is the first thing every chat product does and the first thing that breaks a source check. The way to have both is to check the unit you show, not the unit the model writes, and to measure the change against questions written before it, so a regression shows up as a number before it shows up to a customer. That is what the eval harness and golden set gives an existing system, and how a custom build is tuned from the start, as part of AI built on your company's data.