Where the layers stub, and what each can catch

The deployed pipeline has four kinds of test, and what makes them four rather than one is where each one stops being real.

Unit tests run one function at a time over the committed corpus, with the model clients faked, and they are where the corpus is held to its sources: every chunk verbatim in its section file, every manifest recomputed, every rule of the prompt present by a distinctive phrase. That last one is a test file of its own in the deployed pipeline, learner-prompt.test.ts, and it holds each of the rules from Chapter 2 by name, in both languages, so that a rewrite that lost one fails before the eval has to find it.

Handler tests run the request handler in the worker’s own runtime against the real database schema, with the model scripted. They are where the things around the pipeline live: that the question is scrubbed of phone numbers before the model or the row sees it, that the challenge is asked on the first turn only, that the ceiling is counted on the learner’s own table, that the section and schedule joins read what the push wrote. The handler also claims the turn in its table before the model is asked, so a double submit cannot ask twice, and refunds a paid question’s token on an outage and not on an honest no. The token design document names the refund and its withholding as guards 26 and 27; the claim is tested on the partner box’s handler, whose shape the learner’s follows.

Browser tests run the built app in a real browser, and this is the layer readers get wrong. The wire to the worker is stubbed at the network edge, as you would expect. What answers the stubbed request is not a canned response. It is the pipeline itself, answerLearner, run in-process over the real corpus file for the state the page sent, with the real retrieval and the real checks and only the model faked. The spec’s header comment says why. A guard on retrieval or on a check “goes red here, through /#ask, and not only in workerd: a retrieval that reached Florida’s laws for an Ohio learner would draw Florida’s figure on this screen, which is the failure the whole plan is shaped to prevent.”

The measurement is Chapter 11, by hand.

One file in the deployed pipeline carries the same lesson, and its header comment is worth quoting in full because it is the whole chapter in a paragraph. The file is cite.ts, the citation module of the site’s partner box, a second ask box with a pipeline of its own; the function that decides what a citation may say lives there, in a module of its own with no dependency on the model client, for one reason:

the Playwright fixture has to serve exactly what the worker serves, and the way it did that was by retyping these two regexes. That copy agreed with itself forever – it reproduced the single-section truncation below rather than exposing it, so the bug shipped past a test written to catch precisely this. A test importing the real function cannot make that mistake, and importing ask.ts would drag the Anthropic SDK into a browser test.

A test that reimplements the thing it tests is testing its own understanding of the thing. The browser suite imports the pipeline; the handler suite imports the pipeline; your pipeline.test.ts imports the pipeline. None of them has a copy to agree with.