Ingestion is ETL

What this episode covers

Extract the text of Markdown and PDF files, page by page. Split it into chunks at headings, then by size with an overlap. Load every chunk into SQLite with its vector and where it came from (the file, the heading or the page), with a hash per file so a second run re-indexes only what changed.

At the end: 287 chunks from 13 files in three and a half seconds, a second run that embeds nothing, and a search that finds the bench fee on page one of the PDF.