Stories from the demo
Tuning the demo
How the pipeline behind the demo was tuned, one change at a time. Every number in these stories comes from an eval run committed next to the code it measured.
Story 1
“The passages don't list his job history.”
A 25-question eval on the live models took the demo from 21 of 25 to 23 or 24, for $1.96.
Story 2
Giving the answer the rest of the point
Widening each chosen passage with its neighbours answered 5 of 5 split-point questions instead of 4 for 60% more a question: past the bar I set, and switched on anyway.