In production

The deployed harness sends its model calls up as one batch: a request to the API’s batch endpoint carrying many message requests at once, billed at half the price and back in minutes, with results returned in any order and matched back by an id the caller chose. At 130 cases that is the difference between ten dollars and five, and it is why requestFor in Chapter 2 is a function: the batch is built from the very same request the worker sends, and the reply is read by the very same replyFrom, so a batch built from a copy of either could not drift from what the deployed box does.

The batching is one class, BatchedAsks, and its design has one measurement in it. Each case runs the pipeline’s stages before the model, the rephraser, the index, the reranker, the chooser, live, and parks its final request instead of sending it; when every case has parked, the parked requests go up together. The live stages are gated to three cases at a time:

because three concurrent callers once made the REST embedding endpoint answer null for everyone.

Three flags do what your --case does for a bigger set. --state XX runs one state’s cases while iterating, so a state’s pull request does not pay for fifty states’ calls; --store reads the laws from the production table rather than the committed corpus, so the run measures what is deployed; and --no-rerank measures the box as it was before the reranker, so a pull request can show both scores. The repository’s README reports the result: the eval showed the reranker in Chapter 7 beating any weighting of the fusion.