Talking to a Model

This chapter builds one operation in TypeScript, field by field: you send some text to a server, and text comes back. Then it builds the same operation again with the reply in a shape your code can read without guessing. Every stage of the pipeline in the chapters after this one is one of those two operations, wrapped in structure.

A large language model, LLM for short, is a very large mathematical function that takes a sequence of text and produces a probability distribution over what text comes next. It was fitted to that job by being shown an enormous quantity of text and repeatedly adjusted until its guesses about what followed got good. The result is a fixed set of numbers, called weights, that do not change while you use them. Running those weights over an input to get an output is called inference, and it is the only thing that happens when you call the API: no training, no learning, no adjustment.

Two consequences fall straight out of that description, and this book is built on the first of them.

The model is generating, not retrieving. When it tells you that Maryland law keeps you 15 feet from a fire hydrant, it is not looking that up anywhere. It is producing the text that its fitted function scores as likely to follow your question. Most of the time the likely continuation and the true one are the same, because it was fitted on text where they usually were. Sometimes they are not, and nothing in the scoring separates a likely continuation from a true one. Chapter 2 is about giving the model the law so that it does not have to guess, and Chapter 3 is about checking that it did not guess anyway.

The second consequence is that the model has no memory. The weights do not change when it answers you. Whatever it appears to know about an earlier question, it knows because that question was sent along with this one.