Chapter 11 · section 2 of 7

The golden set

A golden set is a fixed list of inputs with the facts a right output must carry, checked against the source by a person before they went in. Each entry is a case: the question as a learner would type it, and an expect block that pins the few things that make an answer right or wrong.

The checks are partial on purpose. A case does not hold a reference answer to match; it holds a figure the answer must contain, a citation it must show, a word it must lead with, a scope the model must choose. An answer reworded is still right, and an answer with the wrong figure is wrong whatever else it says.

The deployed set’s committed fixture has 190 cases, 202 by its own documentation, and the way it was built is the method. It was not written in one sitting and not generated from the corpus. It grew one pull request at a time, alongside the behaviour each one added, so that every case was written by somebody who had just read the source it tests against. The cases come from three places:

The design document’s list. The first 33 were the questions the plan named before the box existed, with the app’s own facts as their expectations.
Ingesting a state. When a state’s code was fetched, a few cases were written for it in the same pull request: a fact a learner would ask about that the code states as a figure, the figure read off the section, the section in citesLaw. The expected answer comes from the statute, never from what the model said, so the model is graded against the source, and the citation check keeps it from being right by accident with the wrong law.
A failure. When a question was answered badly, the fix came with a case that would have caught it. notContains is how a case remembers a wrong answer: Florida’s hydrant case refuses 10 feet and 20 feet because an answer once said one of them. And when a right answer was marked wrong for saying “D.C.” where the case wanted “District of Columbia”, the fix was to the case, which now accepts either.

Listing: ask/eval-cases.ts

import type { Scope } from "./answer.js";

// The golden set: questions as a learner would type them, each with the few
// facts a right answer must carry. The checks are partial on purpose -- not
// a reference answer to match, but the figure that makes an answer right or
// wrong, read off the source by a person before the case went in.

export interface Case {
  question: string;
  expect: {
    scope: Scope;
    /** The first word, lower-cased: "yes" or "no". */
    lead?: string;
    /** What the shown answer must say. An entry that is a list is met by any one of it. */
    contains?: (string | string[])[];
    /** What it must not say: the wrong figure an answer once carried. */
    notContains?: string[];
    /** Text a shown citation must contain, e.g. "§ 21-1003(m)". */
    citesLaw?: string;
    /** The start of an id a shown citation must have, e.g. "MD-P-16-112". */
    citesId?: string;
    /** Nothing the model wrote may be shown. */
    unanswered?: boolean;
  };
}

export const CASES: Case[] = [
  { question: "how far from a fire hydrant can i park", expect: { scope: "road-law", contains: ["15"], citesLaw: "§ 21-1003(m)" } },
  { question: "can i leave my car right next to a fireplug", expect: { scope: "road-law", contains: ["15"], citesLaw: "§ 21-1003(m)" } },
  { question: "how far before a turn do i have to signal", expect: { scope: "road-law", contains: ["100"], citesLaw: "§ 21-604(d)" } },
  { question: "how close to a railroad crossing can i park", expect: { scope: "road-law", contains: ["50"], citesLaw: "§ 21-1003(t)" } },
  { question: "when do i have to turn my headlights on", expect: { scope: "road-law", contains: [["1,000", "1000"]] } },
  { question: "how close behind someone do i switch to low beams", expect: { scope: "road-law", contains: ["300"], notContains: ["500"] } },
  { question: "whats the fastest i can go on a residential street with no sign", expect: { scope: "road-law", contains: ["30"], citesLaw: "§ 21-801.1" } },
  { question: "what happens if i get pulled over and i don't have my license with me", expect: { scope: "road-law", contains: ["50"], citesId: "MD-P-16-112" } },
  { question: "if i fail my knowledge test how soon can i retake it", expect: { scope: "road-law", contains: [["7", "seven", "next business day"]], citesId: "MD-R-test-and-retake" } },
  { question: "can i wear headphones while driving", expect: { scope: "road-law", lead: "no", citesLaw: "§ 21-1120(c)" } },
  { question: "¿a qué distancia de un hidrante puedo estacionar?", expect: { scope: "road-law", contains: ["15"], citesLaw: "§ 21-1003(m)" } },
  { question: "what is the capital of France", expect: { scope: "other", unanswered: true } },
];

Twelve cases, each a fact from the corpus. Read the sixth: it refuses 500, because the fire-truck law says 500 feet and the low-beam law says 300, and a model given both could confuse them. Read the eleventh: it needs the index, and without Cloudflare credentials it will miss, which is the measurement telling you a stage is absent.