A test is a mutation pair

Most of the deployed pipeline’s design documents carry a section headed How this fails silently, and every entry in it has the same shape: a way the pipeline could be wrong without anyone noticing, the test that would notice, and the mutation that proves the test notices. A mutation is a deliberate change to the code that introduces the failure the test is for; a test that stays green under its mutation is a test of nothing.

From the first road-law design document, four of the nine:

• Another state’s laws answer. Test: on Ohio, the hydrant question shows Ohio’s 10 feet and page 30, not Florida’s 15 and page 66. Mutation: the law loader ignores the state code and reads Florida. Red on the Ohio figure.
• The citation comes from the model, or not at all. Test: the Florida answer shows a citation with p.66 and the manual’s quote. Mutation: drop citations from the response. Red on the citation.
• A guess on a law nobody retrieved. Test: an uncovered question, and the model, scripted, writes a figure anyway with no source; the not-covered line is shown, with no digit in it. Mutation: checkLawAnswer accepts empty sources. Red on the absent answer.
• A fetch failure becomes a generic answer. Test: the law loader throws; the response is 200 with no answer and the log says no-laws. Mutation: the catch returns an empty list and proceeds. Red on the reason.

Every test in your file has a mutation, and the chapter’s rule from Chapter 3 applies to all thirteen. Delete the meant filter from retrieve and the stale-id test goes red. Make chooseLaws return the pool’s first eight instead of fallback and the seventh test goes red. Return the model’s answer on a scope: "other" reply and the Paris test goes red. Do each one, see red, put it back. A test you have never seen fail is a test you are taking on faith, and this file is the one place in the pipeline where faith is not required.