What a golden set found in my book's chatbot

Last night I put a chatbot in front of my book. Ask the book runs the same pipeline as the demo: keyword search (BM25) and a vector index find passages, and Claude answers from them. It is meant to talk, not cite: a few sentences, code when you ask for code, and "The book doesn't cover that" when it doesn't.

It worked when I tried it. This morning I ran it against a golden set: 32 questions with the facts a right answer must carry, written down before the run. Fifteen the book answers, five that ask for code, two that ask where the book says something, and ten it does not cover. Every number below is from those runs.

The index had never been built

The run starts by building the book's vector index in full. It failed on the first call:

216,100 tokens but model supports only 60,000

The book is 629 passages, 88 of them code listings, and the code appendix has whole files. I was embedding 50 passages a call. With long listings in the batch, one call came to 216,100 tokens against the embedding model's 60,000.

In production the same thing had been happening since the first deploy. The indexing step caught the error, logged it and moved on, so Ask the book quietly answered from keyword search alone. The only sign was a line in the panel, "The book is still being indexed", which I had read as patience.

My first fix capped a call at about 40,000 characters in all. It failed again, at 92,000 tokens. The embedding model pads every text in a call to the length of the longest one, so what a call costs is the number of texts times the longest, not their sum:

The fix that held: the number of passages times the longest stays under 120,000 characters, and each passage is embedded from its first 6,000. The index built: 629 passages.

Code questions never got their code

"Show me the first call to the model in TypeScript" came back with "The passages I have don't include the code of that first call", a description of the call, and a pointer to Chapter 1. The listing is in the book, in Chapter 1. Retrieval found the paragraphs around it and not the listing itself. A file of TypeScript shares almost no words with "show me the first call", and its embedding sits far from an English question.

The fix was to bring code along with the prose that introduces it: a chosen passage brings the listing that follows it in its section.

"Give me the code for BM25 scoring" still failed, and each failure taught me something about BM25:

1. Length counts against a passage. The BM25 listing is a 5,667-character file. BM25 divides by length relative to the average, so a short paragraph that mentions bm25 beats the whole file. I ranked listings among listings only.
2. The test file said "bm25" more often. It is shorter, and every test calls the function. The code appendix also repeats every chapter's file, so duplicates took two of the places.
3. So I ranked a card instead of the code: each listing once, as its section's title and the names it defines (function bm25, const k1). That still lost, to sections titled "How to read the code" and "A reply your code can read". The word "code" in "give me the code" matched them. The words that ask for code say nothing about which code, so they are now left out of that ranking.

flowchart TB
  Q["Give me the code for BM25 scoring"] --> F["BM25 and the vector index, fused by rank: 12 passages"]
  Q --> C["It asks for code: rank one card per listing (section title and the names it defines), leaving out give, code, show"]
  F --> N["Each chosen passage brings the listing that follows it"]
  C --> T["The 2 best listings"]
  N --> M["Claude writes the answer"]
  T --> M

Then the answer arrived and stopped halfway through the file, its code fence never closed. A whole listing is near 2,000 tokens, and the answer model thinks before it writes, inside the same 2,000-token limit. The limit is 4,000 now.

None of these were found by reading the answer. Each was found by printing the passages the question had been given.

My scorer was older than my rules

The full run scored 24 of 32. Five of the eight failures were questions the book does not cover, and they had not said only "The book doesn't cover that."

I had changed that rule the night before. I asked for a code snippet calling Vectorize, and it said the book didn't cover it. It does cover it in part: Chapter 5 shows the index and the script that fills it. So now a question the book covers in part gets that part, and a sentence on what the book does not show. The five answers did exactly that. Asked how to deploy on AWS Lambda, it said "The book doesn't show a Lambda deployment" and gave the function that builds the pipeline from whatever environment it finds. Asked the capital of France, it said the book doesn't answer that, then that the book uses that very question as its example of something out of scope, with a test whose scripted model answers "Paris" and passes only if the reader never sees the word.

The scorer was out of date, not the answers. It now marks an answer that opens by saying the book doesn't cover it as "review", for a person to read whether the rest stays on the book. I scored the saved answers again with no model calls: 23 pass, 6 review, 3 fail.

The three failures were six sentences in two paragraphs against a rule of two to four. The model was adding background nobody asked for. The rule now says at most four sentences, in one paragraph, and to stop once the question is answered.

I ran the whole set twice more, because one run of a model is one sample:

The one failure in the third run is six sentences again, so the length rule holds most of the time, not every time. Every review answer I read stayed on the book. Asked about LangChain's agent executor, it said the book doesn't mention it and gave the book's own definition of an agent.

What it cost

Under two cents a question. The two cases that cost the most, at four cents each, were the ones that returned whole files.

What I would tell myself last night

• A step that catches its own error needs a number someone looks at. "Still being indexed" was that number, and I did not look.
• Measure the cost the way the service counts it. Mine was padding to the longest text.
• When retrieval misses, print what it was given before you change the prompt.
• Keep the scorer in step with the rules. A rule changed on Tuesday made Wednesday's run look worse than it was.
• Run it more than once. The first run said three answers ran long; the next two said none, then one.