Case study · Howard's Lantern

A question box that cites the law.

Howard's Lantern is a driving-law exam-prep app. Its ask box answers a learner's question in two sentences from their own state's manual or vehicle code, with the section underneath. A wrong answer here teaches someone the wrong law, so every answer is checked against its sources before anyone sees it.

Ask it your own question → Inspect the pipeline

The problem

Learners ask in their own words, often in lower case and sometimes in Spanish, and never name the state. The answer has to come from that state's law, carry the right figure (a hydrant distance, a permit age, a fine) and name the section it came from. A fluent answer with the wrong number is worse than no answer.

The corpus

172,486 law rows across all 51 states, fetched verbatim, dated and hashed.
Each state's vehicle code and driver's manual, the courts' fine schedules, and the licensing agencies' own pages.

How an answer earns its sources

The model drafts the answer. Code, not the model, decides whether anyone sees it.

QuestionA learner asks in their own words, with their state chosen on screen.
Hybrid searchKeyword (BM25) and vector search over that state's laws, fused by rank.
RerankA rephraser and reranker pick the few sections that actually answer it.
DraftThe model writes two sentences and cites the sections it used by ID.
Check in codeEvery citation ID must be one it was given. Every figure must appear in the cited text.

Passes: shown to the learner with its sources underneath.

Fails: retried or rejected. The learner never sees an unchecked answer.

Measuring it: a 202-question golden set

The test suites never call the model, so they can't say whether the box answers well. A separate evaluation does: 202 questions, each with the few facts a right answer must carry, run through the real production pipeline against the live model.

{ "lang": "en", "state": "FL",
  "question": "How far from a fire hydrant can I park?",
  "expect": { "scope": "road-law", "contains": ["15"],
              "notContains": ["10 feet", "20 feet"], "cites": ["66"] } }

Expected values come from the statute, never from the model. Each case was written by someone who had just read the section it tests, and the citation check stops the model from being right by accident with the wrong law.
Every failure becomes a case. When a real question was answered badly, the fix shipped with a case that would have caught it. notContains is how a case remembers a wrong figure the model once gave.
Scored on what the learner would see. A right answer that the grounding check would have refused counts as a miss, because nobody would have read it.
Its table goes in the pull request. The evaluation runs before any change to the prompt, model, retrieval or corpus, and fails on any miss, so a reviewer reads the answers, not just a score.

Read the full evaluation method →

What this means for your project

The same pattern of hybrid retrieval, checks in code that reject unknown citations and unsupported figures, and a source-verified evaluation set is what the retrieval pipeline build delivers. The grounding and eval review measures an existing feature against it.

Talk about your project