Measuring the Whole Thing

This chapter builds the one program in the pipeline that calls the live model on purpose, spends money doing it, and is not part of the test suite. It answers the question nothing in npm test can: does the box answer well?

Every test so far has described the model rather than called it. The checker is tested with replies you wrote; retrieval is tested with ids you supplied; the pipeline, in the next chapter, is tested with a model that answers from a script. That is deliberate, and it leaves a hole exactly the shape of the model. Whether the real model, given the hydrant law and the rules, writes 15 feet and cites MD-S01, is a fact about the model, and the only way to learn it is to ask.

Asking has two properties the tests do not. It costs money, a fraction of a cent to a few cents per question depending on how much law is in the prompt. And it is not deterministic: non-determinism is the property that the same input produces different output on different runs, and a model has it. The same question, twice, can give a right answer and a wrong one. A program that measures the model has to be run as a measurement, by hand, with its result read by a person, rather than as a gate that runs on every commit.