A Corpus You Can Trust

This chapter builds the offline half of the pipeline: the script that cuts a fetched section of a code into the subsections retrieval works with, the record that holds the index to the corpus, and the check that keeps a model-chosen passage from saying anything its page does not.

Every stage so far reads from the corpus, and the check in Chapter 3 assumes the corpus is right: a figure grounded on a quote is only as true as the quote. The deployed pipeline’s rule, from its first design document, is that the box answers only from what the repository committed and checked, and nothing else. This chapter is what “checked” means.

A state’s vehicle code is fetched from the legislature’s own site, or from the publisher it designates, one file per section, by a script that runs at one request a second with a named user agent and three retries. Beside the files is a record, SOURCES.json, with one entry per section: the URL it came from, the day it was fetched, the section’s citation in the state’s own style, and a hash of the text, which is a fixed-length fingerprint, in this pipeline SHA-256, that changes if a single byte of the text does. A test recomputes every file’s hash and compares it with the record, so a file that was edited by hand, or replaced by a later fetch nobody recorded, is red.

A code the script cannot fetch is reported, not worked around. Five states cannot be fetched: four designate LexisNexis as their code’s publisher, which serves a JavaScript-only reader behind a browser cookie check, and Oklahoma’s official text sits behind a robots.txt that forbids every scripted agent. Those states are recorded as not fetchable, with the reason, and their questions are answered from the manual and the agency’s pages. No mirror, however complete, is used: the pipeline’s claim is that every quote came from the official text, and a mirror cannot carry that claim.

The manuals came first and set the pattern. The deployed corpus’s metadata for Maryland records that the previous version of it was built from the open web, that 164 of its 221 laws carried no citation at all, and that 22 pointed at practice-test vendors. The current one is built from the manual’s PDF and nothing else, and every citation names the printed page.