Joel Stevick

Chapter 2 · section 3 of 5

The call

Listing: ask/answer.ts

import Anthropic from "@anthropic-ai/sdk";
import { fileURLToPath } from "node:url";
import { LAWS, STATE_NAME, type Law } from "./corpus.js";
import { ANSWER_MODEL, requireApiKey, textOf } from "./setup.js";

export const SCOPES = ["road-law", "other"] as const;
export type Scope = (typeof SCOPES)[number];

// The rules. Each is a sentence that was added after a measured failure, and
// each is held by name in a test, so a rewrite that lost one fails before
// anybody reads an answer.
export const RULES = {
  sourced:
    "Every statement comes from the laws given with the question. Nothing from memory about the law, this state or any other state. Put the ids of the laws you used in \"sources\". If no law given touches the question, leave \"answer\" and \"sources\" empty: the app says so itself.",
  figures:
    "Never write a number the laws given do not state. Write figures in digits.",
  register:
    "Write for a sixteen-year-old who asked a friend who knows: plain words, \"you\", each figure once, at most two sentences of under about 20 words each, and no law's name, section or page in the sentences; the app shows those underneath.",
  kinds:
    "A law marked as the court's fine schedule is what a driver actually pays for breaking the law listed above it. A law marked as the state's own page is the licensing agency's word on its own process: prefer it for process, and never for a rule of the road when the manual or the state's law given states the rule.",
  scope:
    "Scope. A rule of the road, a sign, a speed, a fine, an age, a document or a requirement of this state is \"road-law\". Anything else is \"other\", with \"answer\" and \"sources\" empty.",
} as const;

export function rulesText(): string {
  return [
    `You answer a learner driver's question about ${STATE_NAME} road law, from the laws given with the question and nothing else.`,
    "",
    ...Object.values(RULES).map((rule, index) => `${index + 1}. ${rule}`),
  ].join("\n");
}

export const ANSWER_SCHEMA = {
  type: "object",
  properties: {
    scope: { type: "string", enum: [...SCOPES] },
    answer: { type: "string" },
    sources: { type: "array", items: { type: "string" } },
  },
  required: ["scope", "answer", "sources"],
  additionalProperties: false,
} as const;

export type Reply =
  | { kind: "reply"; scope: Scope; answer: string; sources: string[] }
  // Why nothing came back. Written to the log, so a run of failed calls reads
  // differently from a run of refusals.
  | { kind: "failed"; why: "unreachable" | "unparsed" | "refused" };

/** One earlier turn as the learner saw it: what was shown, not what the model first wrote. */
export interface Turn {
  question: string;
  answer: string;
  sources: string[];
}

/** The reference alone: the page, or the section, without the quoted words after it. */
export function referenceOf(citation: string): string {
  const quoted = citation.indexOf(" — ");
  return quoted < 0 ? citation : citation.slice(0, quoted);
}

// The question as the model receives it: the learner's words, then the laws,
// each labelled by what it is. In the user turn, not the system prompt: the
// rules stay the same bytes on every request, and the laws change.
export function withLaws(question: string, laws: Law[]): string {
  if (laws.length === 0) {
    return `${question}\n\n(No law in the learner's state's manual or code was found for this question.)`;
  }
  const listed = laws.map((law) => {
    const reference = referenceOf(law.citation);
    switch (law.kind) {
      case "statute":
        return `[${law.id}] (the state's law, not the manual) ${law.law} (${reference})`;
      case "penalty":
        return `[${law.id}] (the court's fine schedule) ${law.law} (${reference})`;
      case "process":
        return `[${law.id}] (the state's own page) ${law.law} (${reference})`;
      default:
        return `[${law.id}] ${law.law} (manual: ${reference})`;
    }
  });
  return `${question}\n\nLaws from the learner's state retrieved for this question:\n${listed.join("\n")}`;
}

// The request as the SDK takes it, in a function of its own so that the
// measurement in Chapter 11 sends the very same request and cannot drift
// from it.
export function requestFor(
  question: string,
  laws: Law[],
  history: Turn[],
): Anthropic.MessageCreateParamsNonStreaming {
  return {
    model: ANSWER_MODEL,
    max_tokens: 1024,
    system: rulesText(),
    output_config: {
      // Two sentences from a page of law is not a reasoning task.
      effort: "low",
      format: { type: "json_schema", schema: ANSWER_SCHEMA },
    },
    messages: [
      // Replayed as logged: the assistant's turn is what the learner was
      // shown, so a refused turn is a refused turn in the thread too.
      ...history.flatMap((turn): Anthropic.MessageParam[] => [
        { role: "user", content: turn.question },
        {
          role: "assistant",
          content: JSON.stringify({ scope: "road-law", answer: turn.answer, sources: turn.sources }),
        },
      ]),
      { role: "user", content: withLaws(question, laws) },
    ],
  };
}

// What the model's message is read as: a reply in the fixed shape, or why not.
export function replyFrom(response: Anthropic.Message): Reply {
  if (response.stop_reason === "refusal") return { kind: "failed", why: "refused" };
  let parsed: { scope?: unknown; answer?: unknown; sources?: unknown };
  try {
    parsed = JSON.parse(textOf(response)) as typeof parsed;
  } catch {
    return { kind: "failed", why: "unparsed" };
  }
  if (
    typeof parsed.scope !== "string" ||
    !(SCOPES as readonly string[]).includes(parsed.scope) ||
    typeof parsed.answer !== "string" ||
    !Array.isArray(parsed.sources)
  ) {
    return { kind: "failed", why: "unparsed" };
  }
  return {
    kind: "reply",
    scope: parsed.scope as Scope,
    answer: parsed.answer,
    sources: parsed.sources.filter((id): id is string => typeof id === "string"),
  };
}

export type Ask = (question: string, laws: Law[], history: Turn[]) => Promise<Reply>;

// The asker the pipeline uses. A thrown error is a failure with a name, not
// an exception the caller has to know about.
export function askerFor(client: Pick<Anthropic, "messages">): Ask {
  return async (question, laws, history) => {
    try {
      return replyFrom(await client.messages.create(requestFor(question, laws, history)));
    } catch (error) {
      console.error("could not reach the model:", error instanceof Error ? error.message : error);
      return { kind: "failed", why: "unreachable" };
    }
  };
}

async function main(): Promise<void> {
  requireApiKey();
  const question = process.argv.slice(2).join(" ");
  if (question.length === 0) {
    console.error('usage: tsx ask/answer.ts "<question>"');
    process.exit(1);
  }
  const ask = askerFor(new Anthropic());
  // Every law in the corpus, for now. Chapter 4 is where choosing starts.
  console.log(JSON.stringify(await ask(question, LAWS, []), null, 2));
}

if (process.argv[1] === fileURLToPath(import.meta.url)) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : String(error));
    process.exit(1);
  });
}

Run it:

npx tsx ask/answer.ts "how far from a fire hydrant can i park"

You get a JSON reply with scope: "road-law", an answer that says 15 feet, and sources: ["MD-S01"]. Run it again with a question the corpus does not answer, say can I turn right on red, and the answer and sources come back empty, because the first rule says to leave them empty rather than reach for memory. Whether the model will always obey that rule is the subject of the next chapter. Whether it did this time, you can see.

The request, in the order its parts are sent: the rules in the system field, then the thread as the learner saw it, then the question with the laws listed under it. The rules are the same bytes on every request and go first; the laws change with every question and go last.

Four things in that file carry the design.

The laws are in the user turn, labelled. withLaws writes each law on its own line with its id in brackets, a label saying what kind of thing it is, and its reference in parentheses with the quoted words left out. The model is told which lines are the state’s law rather than the manual, which are the court’s schedule, which are the agency’s page, because the fourth rule tells it to treat them differently. The quoted words are left out because the model has the rule already and the quote was measured, in the deployed pipeline, at half of every law block the model read: 49 per cent of the characters for the drunk-driving laws, 48 per cent for Texas’s code chunks. The check in Chapter 3 grounds on the quote anyway, from the citation itself, so nothing the learner sees changes.

The thread is replayed as it was shown. history is a list of earlier turns, and each one’s answer is what the learner read, not what the model first wrote. When Chapter 3’s check refuses an answer, the turn is recorded with an empty answer, and that empty answer is what the model sees as its own earlier reply on the next question. A refused turn is a refused turn in the thread too, which keeps a follow-up from building on a sentence nobody saw.

A failure has a name. Reply is either a reply or one of three named failures, and askerFor converts a thrown error into the third. The pipeline in Chapter 10 writes the name to its log, and the names were chosen so that a day’s log reads as a diagnosis: a run of unreachable is a service problem, a run of refused is the model’s safeguards, and the difference matters when somebody is deciding what to fix.

The request is a function. requestFor returns the exact object the SDK sends, and nothing else in the pipeline builds one. Chapter 11 sends the same request another way, through an endpoint that takes many requests at once, at half the price, and it can do that without drifting from the deployed one only because the request is in one place.