The LLM pipeline I forgot I shipped

This spring I built It's Half-Staff Because, a site that tracks U.S. flag half-staff status by scraping 52 government sources. This July, while rewriting the copy on my studio site, I found a sentence describing its "LLM parsing pipeline" and corrected it. The site has zero LLM, I said. I don't even have an OpenAI API key.

I was wrong on both counts, and the way I found out is the useful part.

The correction of the correction

After I removed the LLM references, the claim kept bothering me. The repo was right there, so I opened the pipeline and checked.

The scraper requires an OpenAI API key at startup and refuses to run without one. Every document a scraper collects goes through a parser that calls GPT-4o-mini with a fixed prompt, temperature zero, enforced JSON output, and a validation step. Parses with missing fields or a low confidence score are rejected. There is no fallback parser. Every record in the database has gone through the model.

And the site had current data on it that morning. Which meant the pipeline was running. Which meant the key existed. I opened my Railway dashboard, and there it was in the service's environment variables, where I had put it four months earlier: OPENAI_API_KEY.

I had shipped a production LLM pipeline, run it for four months, and forgotten it existed.

How you forget a thing like that

It never asked for attention.

Government proclamations are inconsistent prose. The model turns each one into structured fields: reason, dates, and issuer. Around that call I had added duplicate checks before storage, automatic expiry when a source stops publishing an order, and a link from every status back to the original proclamation. Any mistake can be checked against the source. GPT-4o-mini costs pennies a month at my volume, so the bill never drew my attention either.

The pipeline ran quietly for four months, which made it easy to forget. This is the only time I have been embarrassed by something working too well.

What I actually took from it

Read the system that actually runs. My memory said "no LLM." My README said otherwise, and I assumed the README was aspirational because I know how those files drift. Twenty minutes with the running code settled it.

The model call needs constraints around it. This pipeline assumes the model will occasionally be wrong. It enforces structure, rejects low-confidence output, catches duplicates, expires stale data, and links every claim to its source. Those checks are what let the model run unattended.

Keep a small operations note. A four-month-old API key in a dashboard I rarely open was doing real work, and I had no useful record of it in the place I would look. What runs, where it runs, and what it costs all belong in the project README. Mine had the first two and not the third — and the cost line is exactly the one that would have reminded me.

The map was correct the entire time I was insisting it couldn't be. That's the goal, I suppose. I'd just recommend remembering that you built it.