RSS Meets LLM: Challenges and Tradeoffs (Part 3)

Part 3 of a 4-part series on how EnggFeed, an RSS + LLM engineering blog aggregator, is built. Part 1 covered how it’s built, Part 2 covered the database schema’s evolution.

Part 2 closed on one design choice from the very first migration mattering more than expected: keeping blog.id as an internal UUID separate from the RSS guid. That choice didn’t come from foresight, it came from a bug. Here are the real problems that shaped the current codebase, and the tradeoffs that came out of fixing them.

Tags Empty for Signed-In Users

Issue: Blog cards rendered fine, but tags were always empty even for signed-in users.

Root cause: blog.id was originally the RSS guid itself, an externally-controlled string, used directly as the primary key. Guid format varies feed to feed. Dropbox uses full URLs, others use arbitrary strings or plain numeric IDs. That inconsistency meant blog.id and blog_tag.blog_id didn’t reliably match, so the join that was supposed to pull in tags returned nothing.

Fix: Split the two concerns apart. blog.id became a UUID generated by the app, a reliable primary key. blog.guid became its own Text column with a UNIQUE constraint, used only for ingest-time dedup. Every foreign key referencing a blog, blog_tag, blog_prerequisite, and at the time blog_chunk, points at the UUID, never the guid.

Lesson: An externally-controlled, format-inconsistent value should never be used as a primary key or foreign key anchor, even when it looks like a natural fit. See blog/models.py for how id and guid are separated today.

RSS Feeds Aren’t an Archive

Issue: Requesting Summary, Simplify, or Prerequisites for an older blog returned a 502.

Root cause: EnggFeed itself keeps every ingested blog forever, that part was never the problem. But EnggFeed never stores the raw article text, only the LLM’s output, so generating Summary or Simplify for the first time meant going back to fetch the article content again, on demand, by re-parsing the original source’s live RSS feed (Cloudflare’s, GitHub’s, whichever it came from) and matching on guid. Most source feeds only expose a rolling window of recent items, Cloudflare’s feed holds roughly 20 items, about two weeks back. Once a blog’s article aged out of that window on the source’s side, the on-demand fetch could never find it again, even though EnggFeed’s own database still had the blog listed. New blogs keep getting ingested and kept forever, but each source’s own feed only remembers its own recent past, so more and more already-ingested articles fell outside their source’s window every day, eventually covering most of the database.

Fix: Generate summary and simplify eagerly at ingest time, reusing the article content already fetched for the tag/prerequisite LLM call, while the article is still guaranteed to be sitting in the live feed. Content is still never stored raw, only the LLM’s output is persisted, keeping the original “no scraping, no content storage” constraint intact. The 7-day refresh cycle that used to re-fetch and regenerate summary/simplify was dropped entirely, since it was also what re-triggered the live RSS fetch and reintroduced this exact bug on every stale read. Prerequisites stayed generated on first click, since it has no content-aging risk.

Lesson: “Fetch on demand, never store” only holds up if the upstream source keeps the data around indefinitely. RSS feeds are a rolling window, not an archive. Caching hides staleness on repeat reads, but it does nothing for the first read of something, that fix has to change when the work happens, not how long the result is kept afterward.

GitHub Actions Said Failure, Ingest Said Success

Issue: The daily ingest workflow reported a failed run, a 504 from the curl step, even though the ingest pipeline had completed successfully server-side.

Root cause: POST /api/v1/ingest used to run the entire pipeline synchronously: fetch every RSS source, then for each new article make sequential LLM calls for tags/prerequisites, summary, and simplify, plus an embedding call per tag and prerequisite. Across the sources currently seeded (19, per eval/seed_blog_source.sql), this routinely took several minutes, well past the reverse proxy’s timeout in front of the app. The proxy killed the connection and returned a 504 to the GitHub Actions runner, but the FastAPI request handler kept running to completion on the server. The client-visible failure and the actual server-side outcome had quietly decoupled.

Fix: Made the endpoint asynchronous with FastAPI’s BackgroundTasks. trigger_ingest now only validates the ingest secret and schedules the job, returning immediately regardless of how long the pipeline actually takes. BackgroundTasks was chosen over a full queue like Celery or RQ deliberately: there’s exactly one async job type running on one instance, a broker and worker process would add operational weight for no real benefit at this scale.

Tradeoff accepted: GitHub Actions’ pass/fail status now only reflects whether the job was scheduled, not whether ingest actually succeeded. Background job failures are caught in ingest/controller.py’s _run_ingest_job and sent to Sentry instead of failing the HTTP response. Sentry is the real source of truth for ingest failures now, not the Actions run status, and that’s only as good as Sentry actually being configured correctly in production, worth double-checking rather than assuming.

Lesson: A synchronous endpoint doing meaningfully slow work behind a reverse proxy will eventually collide with that proxy’s timeout, and the failure it produces says nothing about whether the work itself succeeded. When a job’s duration is unbounded and nothing needs to wait on its result, decouple the response from the work instead of trying to outrun the timeout.

New Blogs Sorted by the Wrong Clock

Issue: A blog ingested today could still show up buried far down the feed, sometimes not visible on the first page at all.

Root cause: The feed listing originally sorted by published_at, the date printed on the article by its original source, not by when EnggFeed itself ingested it. Sources don’t always surface articles in strict date order. A source’s feed can show an article dated weeks earlier right after one dated yesterday. Sorting by that stamped date meant a blog that was genuinely brand new to EnggFeed, the first time any user could ever see it, could still rank behind older ingests that simply carried a later publish date, even though from a reader’s perspective both were “new” the moment they appeared.

Fix: Sort by created_at instead, the timestamp EnggFeed itself assigns the moment it ingests a blog. Whichever blog EnggFeed learned about most recently now shows up first, regardless of what date is printed on the original post.

Still not fully resolved: the sort order runs on created_at, but the date printed on the card, and on the Summary/Simplify detail pages, is still published_at. So a blog can legitimately sit near the top of the feed while showing an old-looking date, which reads as odd to anyone who notices, even though the ordering itself is correct.

Lesson: When the goal is “show what’s new,” new has to mean new to your system, not new according to a timestamp some other source controls. An externally-supplied date can quietly disagree with your own definition of recency.

Tier Reflects the Feed, Not the Article

Issue: An article’s content tier, and therefore whether it gets a Summary, Simplify, or prerequisites at all, isn’t a measure of the article’s real length. It’s a measure of how much text the source’s own RSS feed happened to expose for that item.

Root cause: EnggFeed prefers the <content:encoded> tag when a feed provides it, and falls back to the short <description> tag when it doesn’t. Word count, and the tier computed from it, comes from whichever of those two the feed exposes, never from the actual article at its source URL. Confirmed against two real feeds in Part 1: Cloudflare’s feed carries both, a 26-word <description> teaser alongside a 2,371-word <content:encoded> article, so EnggFeed picks up the full one. Discord’s feed has no <content:encoded> at all, only the short <description>, so Discord articles get tiered on a teaser’s word count no matter how long the real post actually is.

Status: Deliberately not fixed by scraping the article’s own URL as a fallback, that would break the same copyright constraint from Part 1 that ruled out storing article content in the first place. This is accepted as a known tradeoff: content tiers measure what the feed chooses to give, not the truth of the article.

Lesson: A derived signal is only as trustworthy as the thing it’s derived from. When the upstream source controls how much content you ever get to see, any decision built on top of that, tiering, gating, feature access, is really a decision about the source’s feed behavior, not about the article itself.

One Topic, Many Labels

Issue: The LLM never named the same underlying topic the same way twice. One article’s “Kafka,” another’s “kafka streaming,” a third’s “Apache Kafka,” three tags for one concept.

Root cause: Tag and prerequisite names come out of a freeform LLM call, not a fixed vocabulary. Without normalization, every slightly different phrasing became its own row.

Fix: Before inserting a candidate tag or prerequisite, normalize the string (lowercase, strip, collapse separators), embed it, and compare it against existing tags by cosine similarity. Above a threshold of 0.88, reuse the existing tag. Below it, insert a new one. The threshold itself wasn’t guessed, real candidate pairs were labeled “merge” or “separate” in Braintrust, and the threshold was picked from that labeled data, tightening the evaluation criteria until the score actually meant something, which moved agreement from 66% to 80%.

Lesson: When an LLM is the source of a label, treat “same concept, different words” as the default case, not the edge case, and validate any similarity threshold against real labeled examples rather than picking a number that feels reasonable.

TL;DR: Tradeoffs

No “it depends” answers, actual decisions made and why.

Pay-per-read vs. pay-at-ingest for LLM calls. Calling the LLM only when a user asked kept things cheap until someone actually read an article, but every such request could end up waiting on Claude. Now every article gets 2-3 LLM calls at ingest whether anyone ever reads it or not, but the read path never waits on an LLM again, and it’s net cheaper for anything read more than once.

pgvector vs. Qdrant for the vector store. pgvector is a Postgres extension, no extra service, no extra memory footprint, and it fits on a single small instance. Qdrant was ruled out for the opposite reason: a separate service with a higher memory cost, for a workload that’s only ever used for tag/prerequisite normalization at ingest, never for search.

FastAPI BackgroundTasks vs. a real queue. BackgroundTasks runs the job in the same process, no broker, no worker, no new failure modes to operate. The real cost is exactly what challenge #3 above accepted: the HTTP response can no longer tell you whether the job actually succeeded. A queue like Celery or RQ would fix that, at the cost of infrastructure that doesn’t pay for itself when there’s only one job type running once a day.

Sync vs. async, codebase-wide. The app stays synchronous almost everywhere, a plain SQLAlchemy Session, requests for HTTP, anthropic.Anthropic and openai.OpenAI’s sync clients, and plain def route handlers, which FastAPI already runs in a threadpool. That gets real concurrency across requests without needing every dependency in the chain to be async-safe. The genuine cost shows up in the ingest job itself: fetching all 19 sources happens one at a time, not concurrently, since the same DB session is reused across the whole job and isn’t safe to share across concurrent tasks, and the tag/prerequisite dedup check is a check-then-insert with no locking, running it concurrently risks two similar tags racing past the similarity check at once and both getting inserted.

One model for every call vs. model routing. Tag/prerequisite extraction and Simplify both stay on Sonnet, extraction has to correctly read an article to name the right tags and prerequisites, and Simplify has to restructure a technical article into an ELI5 explanation, both need real reasoning. Summary moved to Haiku, condensing an article into a short summary and key points doesn’t need the same reasoning depth, and Haiku is a fraction of the cost, $1/$5 per million input/output tokens versus Sonnet’s $3/$15. The cost of routing by call type is exactly that: one more thing to get right per call, route to the cheaper model where it’s safe and you save real money, route wrong and quality drops somewhere a cost dashboard won’t show you.

Coming up in Part 4: building a real Postman collection against the live API, and why it’s structured the way it is.