The Check That Is Not the Model

This chapter builds the function that decides whether an answer is shown. It is sixty lines of code, it calls no model, and every chapter after this one exists to get the right text in front of the model so that this function passes more often. The function itself does not loosen.

Chapter 2 ended with a reply that cited MD-S01 and said 20 feet, and the observation that only a rule stood between it and the screen. The obvious repair is a second rule: check your figures against the laws given. The model would follow it, mostly, and you would be back where you started, with a rate.

The second obvious repair is a second call: send the answer and the laws to the model again and ask whether the answer is grounded. That is the self-assessment the introduction warned about. A model asked whether an answer is correct says yes at a rate that depends on the answer’s fluency more than on its correctness, and this answer was written by the same model to be fluent. You would have spent a second call to get a second opinion from the first opinion’s author.

What is wanted is a question that code can settle, and there are four of them. Each is a comparison between two strings you already have.

1. Did the model name any sources at all, and write anything? If not, the laws given did not cover the question, and the honest answer is to say so.
2. Is every id the model named one of the ids it was given? An id outside that set is a law nobody gave it, which means memory.
3. Does every page the answer names in prose belong to a law it cited? A page number the model wrote that no cited law is on is a page it made up.
4. Is every figure in the answer stated by a law it cited, or by the learner’s own question? A figure from nowhere is the failure the whole pipeline exists to prevent.

And one more, which is not about truth but about the register: is the answer short enough to be the two sentences the rules asked for, rather than a paragraph that drifted?

The function that asks them is a checker: code that decides whether a model’s output may be used, from the output and the inputs alone, without consulting the model. Grounding is the fourth question’s name: a figure is grounded when a source the answer cites states it, and an answer is grounded when every figure in it is.