Finding It by Its Words

This chapter builds the first stage of retrieval: forty lines that rank the laws by the words they share with the question, and return the best eight.

Chapter 2 sent all twenty-seven laws with every question, and with twenty-seven laws that is fine. The deployed pipeline publishes one file per state, and Maryland’s holds the manual’s 216 laws, the fifteen hand-written statutes, every subsection of 819 fetched sections of the Maryland Code, and the agency’s page rows. California’s file is 7.6 megabytes. Sending a state’s file with every question would cost dollars per answer and hand the model a book to find two sentences in. The rules say answer from the laws given; this stage decides which laws those are.

Before a law can be scored against a question, both have to be reduced to the words that carry meaning.

A stop word is a word that says nothing about which law is meant: a, the, how, can, i. They are dropped before anything is counted. The list is short and deliberate, and two words that would be stop words in a general-purpose search engine are kept: in a question about road law, park and far are the whole question.

Stemming is reducing a word to a form that matches its variants, so that hydrants finds hydrant. The deployed pipeline’s stemmer is one rule: a word longer than three letters that ends in a single s loses it. It is crude, and it is enough; bus keeps its s, buses becomes buse and matches nothing, and the Spanish question in the test file below matches nothing either, which is the point of Chapter 5.