Story 1 · Tuning the demo

“The passages don't list his job history.”

That is what my own demo chatbot said about my own résumé, on the morning of October 21. By mid-afternoon the bug was fixed with a test that proves it, and I had an eval harness that measured the pipeline on the live models and told me which of my tuning ideas were right. Total spend on the models: $1.96.

The question that failed

My site has a free demo: you upload your own PDFs and ask a chatbot questions about them. My two-page résumé went into it, with the most obvious question there is: what jobs has he held. It answered that the passages it had didn't list my job history, and cited two passages from page 1.

Asked What role did he hold at Zonar Systems?, it answered correctly, from page 2, with dates. So the text was there. The broad question just never reached it.

Was it chunking?

That was my first guess, and it was wrong. My résumé becomes 9 chunks of about 900 characters, and the seven employers sit in 5 of them, across both pages. Three things lined up:

1. The answer only ever saw 5 chunks. Seven jobs in five chunks across two pages, plus a summary, could never fit.
2. The words of the question matched the wrong chunks. “jobs” and “held” appear in my summary (“while waiting to land my next engineering job”) and my outbound-calls paragraph, not in “Team Lead - EVIR Web”. Keyword search put those first.
3. The model said the information was absent, and my check let it through. The check rejects an answer that cites a passage it wasn't given or states a figure no cited passage has. “The passages don't list his job history” did neither.

Chunking did have a separate bug, which I found on the way: the page sends each line of a PDF on its own line, and the chunker only recognised “. ” as the end of a sentence, so real PDFs were being cut mid-sentence. It split my bullets, not the employer headings, so it wasn't the cause. I fixed it anyway.

A test first, failing for the right reason

Before changing anything, I wanted a test that failed the way the demo had. It uploads a fictional two-page résumé (seven jobs, two on page 1 and five on page 2, after a summary full of the question's words) through the real demo page, and asks exactly “what jobs has he held”. The models are scripted, and the scripted answer can only list jobs that appear in the passages it is actually given.

On the old code it named five jobs. Team Lead at Corvane Freight and Senior Developer at Quillon Systems were missing, because their chunk never reached the answer. That is the bug, reproduced on purpose.

The fix: the answer is written from up to 12 passages, and the reranker now puts them in order instead of cutting them. A small upload that fits is read whole, and only then does the demo say “I couldn't find that in your documents”. Otherwise it says it searched only part of them. Sources are now grouped one line per page, with every passage still a click away.

Then I broke it back on purpose: with the limit set back to 5, the same assertion goes red. A second agent, in a fresh context that hadn't built the fix, checked the work and found a hole. You could delete the “read it whole” rule and every test stayed green. So I added a test with the vector index down, where reading it whole is the only way to reach page 2. With the rule removed, that test reproduces the original failure exactly.

That shipped at 12:44. Two changes I asked for while I was in there shipped by 13:20: the demo now takes a whole book (300 pages), and a demo's documents are deleted after 10 days without use instead of 14 days after it starts.

Why the fix wasn't the end

A scripted test proves the pipeline does what I designed. It says nothing about whether the real models, given the real context, answer well. For that I needed an evaluation that runs the real thing.

• 25 questions, not 200. I chose a golden set of 25 to keep my costs down: four on my résumé and 21 on my book, Agentic Workflows, all 267 pages of it. Question one is the one that failed.
• Hard on purpose. The same résumé question with the book uploaded too, so the résumé competes with 501 other chunks. “What role did he hold at Microsoft?”, a trap, since Microsoft appears only as the Microsoft Store that sells one of my apps. Three topics the book never covers. Three questions that assume something false about the book. Three that ask for a whole list spread across chapters.
• Every expected fact comes from the PDF, not from a model. Each case carries a quote from the page that supports it, and a program checked all of them, character for character, against the text the demo extracts.
• A spending cap the harness enforces. $2 a run and $5 in all, which I later raised by 50% to $3 and $7.50. The harness stops when it hits one, and a case it never reached is recorded as not run, never as a pass.

Each run uploads both PDFs through the demo's real upload route, asks every question through its real ask route with the real models (Claude Sonnet 5.5 answers, Claude Haiku 5.5 reranks), and records for each answer: whether the right passage was among the passages the reranker read, whether it made the 12 the answer was written from, whether the answer cites it, which required facts it contains, how long it took and what it cost. A full run takes ten to fifteen minutes and costs about 27 cents.

The baseline: 21 of 25

The good news first: my question now passed on the live models. Counting the cases I read by hand (all three false-premise questions were answered by saying what the book actually says), the pipeline scored 21 of 25. Four findings taught me more than the passes:

• The reranker was quietly broken. Haiku 5.5 thinks before it answers, and its thinking used up its 256-token limit before it wrote the ranking on 4 of 25 questions. Each time the pipeline fell back to keyword-and-vector order without a word. No test of mine could have seen it, because my tests script the models.
• The answer model was a few tokens from breaking. Two answers used 587 and 588 of their 600 tokens. Any longer and the reply is cut off mid-JSON, and the prospect sees “try again”. In the next run, with the limit raised, two answers took 616 and 607 tokens, past the old limit.
• A perfect answer was refused by my own check. Asked for the chapters, the model listed all thirteen. The check refused it as too long, because its sentence counter read “1. Talking to a Model; 2. Giving the Model Hands” as fifteen sentences.
• One was simply retrieval. Asked for the book's six review lenses, the model saw passages naming three of them.

Seven runs, one change at a time

Round 1 is the one I'd point to if someone asked why you measure. Turning the reranker's thinking off did make it finish: the résumé-plus-book question went from 4 of 7 employers to 7 of 7. But the score went down, because a reranker that doesn't think ranks worse: on the chapters question it left out the contents page that holds the answer. One number would have told me round 1 failed. The per-question view told me which of its three changes helped and which hurt. Round 2 kept the thinking and gave it room, and fixed both problems at once.

Round 3 is a trade, not a free win. The wider pool fixed two questions and broke one that had passed with 24 passages: with more to choose from, the reranker picks a different twelve, and one of the two reasons in a worked example no longer makes the cut. Net, one better. I kept it, and the broken one is now a case every future change is measured against.

Round 4 is the other kind of lesson: an idea that sounded right (give the answer more passages) that bought nothing for 17% more cost. It was reverted.

I ran the final settings three times because the models are not deterministic: one question passed once in three runs, and I didn't want to tune toward a lucky run. Across the three: 23, 24 and 23 of 25, against 21 before.

What the harness, and I, got wrong

Before the tuning shipped, a second agent checked my notes against the run files, and it caught me overclaiming twice. I had written that the reranker ran out of room on six questions; the files say four. And I had written that the broken question above failed because its passage was never retrieved; the files say it passed in two of the three runs with the smaller pool, answering from other passages. It also found that my fix to the sentence counter was too generous: it would have let “The total was 100. The tax was 20. …” through as one sentence. That now has a test of its own. If I had published the story from my notes, two of its numbers would have been wrong.

The harness needed fixing too. An edit to it stopped halfway once, and a helper used before it was defined crashed a run after it had spent money the spending ledger never recorded. The ledger is now written after every question, and the lost spend is in it as a $0.02 estimate, inside the $1.96 total. One quote in my golden set was split across two chunks, so the harness under-counted a passage that was really there, and it now counts either half of a quote. These are the same kind of quiet failure as the reranker's, in the thing measuring it.

What it still gets wrong

• One question, the price of round 3: asked why a change in the book's worked example was blocked, it gives one of the two reasons. It gave both in two of three runs with the smaller pool.
• One question, two times in three: the six review lenses are spread over three chapters, and twelve passages chosen by a reranker usually hold only some of them; when it fails, it names two or three.

Both are partial answers, not wrong ones, and both are now cases that any future change has to be measured against.

What shipped

• The reranker thinks at low effort with room to finish, reads 48 passages, and has 8 seconds to do it (the slowest took 4.3 against the old 5).
• The answer has 2,000 tokens of room.
• The check counts “1. Talking to a Model” as part of a sentence, not the end of one, and still counts “The fee was 5.” as a sentence.
• Each of those, except the 8-second timeout, has a test that fails if it is put back, run on every change without calling a model.
• The harness, the 25 cases and every run's results live in the repository, next to the code they measure.

A note on who did what. I directed all of this; Claude Code, my coding agent, wrote the code, the tests and the harness and ran the evals; a second agent, in a fresh context, checked each change before it shipped; and a merge gate ran the whole test suite first. The decisions were mine: what to spend, how many questions, what to keep. The eval is what told me which ideas were right.

What this means for your chatbot

If you have a document chatbot, it almost certainly has a version of each of these: a question it answers well when asked narrowly and badly when asked broadly, a component failing quietly and falling back to something that mostly works, and a check that refuses a good answer. You only find them by running questions with known answers through the real pipeline, on the real models. The grounding and eval review does that for an existing chatbot, and the chatbot and retrieval pipeline engagements are built this way from the first day. Or try the demo on your own PDFs: it is the same pipeline, as tuned here.