What it is not

Three things the deployed eval’s own documentation says it is not, each a thing people expect of it:

• Not deterministic, and not in npm test. A run can differ from the last one by a case or two; the table says which, and a reviewer decides whether that is the change or the model.
• Not a routing test. Whether the challenge fires on turn one, whether the ceiling holds, whether a turn is claimed before the model is asked: the worker suite proves those with a model that answers from a script, deterministically.
• Not complete. Forty-six states at a median of two cases each is a sample, not coverage. A question nobody has asked yet is not in the set until it is answered badly and somebody adds it.