This is a follow-up to How I Built a RAG System on the SpaceX S-1 in One Weekend. If you haven’t read it, the short version: I built AskS1.com so you can ask questions about an IPO filing and get answers with page citations you can actually verify.
In June, I promised that when Anthropic filed, “AskS1 will be there.”
Anthropic filed its S-1 on October 26. By the end of this week, AskS1 had become a multi-filing product: a new generation model, a first real eval, a handful of reliability and mobile fixes, and a rewrite of the citation system that finally fixes the page numbers — the one thing I called out as the hardest part of the original build.
Here’s what changed, what broke, and what I’d still call unfinished.
TL;DR
AskS1 now covers two filings — Anthropic (default) and SpaceX — with a company switcher and one vector collection per filing.
Generation moved from Claude Haiku to Claude Sonnet 5.5.
Citations went from a fuzzy ±8-page window to the filing’s actual printed page numbers — down to the exact single page a fact appears on (including
F-financial statement pages). On my eval questions, every retrieved chunk now verifies against the PDF ground truth, versus 11% before.The Sources list now only shows what the answer actually cited.
A first, small, honest eval of the Anthropic pipeline.
A handful of reliability and mobile fixes that mattered more than they sound.
One disclosure first
AskS1 uses Anthropic’s Claude to answer questions about Anthropic’s own S-1. That’s a conflict worth stating out loud. The answers are grounded in retrieved text from the filing and cited to a page, so you can check every claim yourself — which is the whole point of the product — but you should know which model is reading which document. Nothing here is investment advice.
From “the SpaceX S-1 app” to “a filing app”
The v1 codebase was quietly a SpaceX app. The collection name, the “282 pages” in the system prompt, the page-clamp constant, the suggested questions, the page title — all hard-coded.
Adding a second company meant pulling every one of those into a single config object:
companies.ts
anthropic → collection: anthropic_s1, demo questions, suggestion chips, EDGAR link
spacex → collection: spacex_s1, demo questions, suggestion chips, EDGAR linkThe chat route now takes a company field. Retrieval hits that company’s Qdrant collection. The system prompt, demo cards, chat placeholder, header and ticker all read from the same config. Adding the third filing — whoever that is — is now a config entry plus an ingest run, not a refactor.
Two small decisions worth explaining:
One collection per filing, not one collection with a
companyfilter. It makes cross-contamination structurally impossible. In my eval I asked an Anthropic-selected chat a SpaceX/Starlink question. It didn’t leak Starlink numbers; it said the information wasn’t in the filing. That’s the behavior I want, and separate collections make it the default instead of something a filter has to get right.Anthropic is the default, SpaceX is one click (or
?company=spacex) away. Existing links to the SpaceX version keep working.
The ingestion pipeline moved into the repo
In v1 the ingestion lived in a Python prototype and was effectively a one-off. For v2 it’s a TypeScript script in the web repo:
npx tsx scripts/ingest.ts anthropic ../Anthropic_S1.pdf --skip-pages=3It uses the same chunking as before (400 words, 100-word overlap) and, importantly, the same embedder module the app uses at query time (all-MiniLM-L6-v2 via @xenova/transformers, 384 dimensions). Ingestion and retrieval importing the same function means the two can never drift apart. That’s a boring property that prevents a nasty class of bug.
The Anthropic filing comes from the real Form S-1 on SEC EDGAR, printed to PDF with headless Chrome. The final corpus is 794 chunks in anthropic_s1.
Ingest now also skips the front and back of the document:
Front matter — cover page, underwriter list, table of contents — all of which is dense with keywords and light on content. It’s the same problem as v1’s executive-summary issue, in a different costume.
Back matter — Part II, exhibit index, signatures. This is boilerplate that repeats data from earlier in the filing, so it only creates duplicate chunks competing for the same query.
The page number problem, again
If you read the first post, you know page citations nearly broke me. Chrome’s HTML→PDF reflow means the PDF page index doesn’t match the page numbers printed in the filing — I worked around that for SpaceX with position-based extraction, then hedged with a ±8 page window. For Anthropic I trusted the raw PDF index instead. I shouldn’t have.
I ran my first real eval, then checked every cited range against where the facts actually live in the filing:
Citation labels copied correctly from retrieval: 39 / 39 (0 fabricated)
Printed page actually inside the cited ±8 window: 4 / 35 (11%)Claude copied the ranges it was handed exactly, every time — the generator was faithful. The ranges themselves were wrong, drifting 40–80 pages from the printed numbers across the filing. The financial statements were the worst case: printed F-16–F-18, cited as Arabic p.284–300.
The fix reused the idea, not the code, from the SpaceX post: a printed page marker is a standalone 1–3 digit number (or F-N), horizontally centered, alone on its line. In the Anthropic PDF those markers sit mid-page — wherever Chrome happened to break the original EDGAR page — not in a predictable footer band, so I read text items by coordinate via pdf.js, filtered the sequence for monotonicity, and interpolated any page with no marker from its neighbors. That found 311 Arabic markers and 65 F-page markers, matching every ground-truth anchor I checked by hand.
My first version stopped at the chunk level — each chunk cited the whole printed span it covered, so a chunk straddling a page break cited both pages. Correct, but imprecise, since a 400-word chunk often spans two or three printed pages.
The fix: Chrome drops each marker exactly where the printed page ends, so every word between two markers belongs to a known page — the marker is a boundary in the text, not just a label for a chunk. I tag those boundaries directly into the stored chunk text:
...end of one printed page's text. [p.154] Start of the next page's text...The model cites the nearest tag before the fact it’s quoting, so answers now land on a single page, and the Sources list shows those exact pages instead of a span. The embedding is still computed from the clean, tag-free text, so this never touches retrieval quality.
Checked against the same eval questions, at both levels:
Before (raw PDF index): 4 / 35 chunks within the right window (11%)
Chunk-level printed spans: 45 / 45 correct
Word-level inline tags: 45 / 45 correctOne caveat: these are different samples over my own 12 eval questions, since retrieval shifts slightly on each re-ingest. A clean sweep on 45 chunks is a strong signal, not a universal guarantee — and exactness has a floor, since a sentence that itself crosses a page break still cites wherever it begins.
Sources should only list what the answer used
A smaller bug with outsized trust impact: the Sources section of an answer sometimes showed citations that never appeared in the answer text.
The cause was simple. Retrieval returns the top five chunks, and the UI listed all five. The model might only cite two of them. A reader sees p.88 in Sources, searches the answer for it, can’t find it, and reasonably wonders what else is off.
The fix is a small, boundary-safe filter in citations.ts: keep only retrieved chunks whose pages actually appear in the answer text. With exact-page citations, the matching works per page — a chunk counts as cited if the answer mentions any printed page it contains, and the Sources list shows those exact pages rather than chunk spans.
“Boundary-safe” matters. A naive substring check finds p.15 inside an answer that cites p.154. The match is escaped and requires that the citation isn’t immediately followed by another digit or a dash. If the model cites nothing recognizable, the UI falls back to showing all retrieved chunks rather than an empty list.
Haiku → Sonnet 5.5
v1 chose Claude Haiku on a benchmark: 4.7/5 at 2.8 seconds, with local 14B models roughly 10× slower for a 0.2-point quality gap.
For v2 I moved generation to Claude Sonnet 5.5 at medium effort, with no extended up-front thinking.
Anthropic's S-1 is a more nuanced document than a standard aerospace filing: a public benefit corporation, a Long-Term Benefit Trust with special governance rights, a Class F share structure, and risk factors about frontier AI itself. These are questions where getting the framing exactly right matters more than shaving a second off latency.
What it cost:
Haiku (v1)
Sonnet 5.5 (v2)
Typical answer latency
~2.8 s (v1 benchmark)
~6.2 s generation (v2 eval, mean)
Retrieval
local, fast
~240 ms (v2 eval, mean)
I also rewrote the system prompt: cite the printed prospectus page exactly as labeled, handle F- pages correctly, quote exact figures, and say so plainly when the context doesn’t contain the answer.
What does one answer cost?
People ask. Back-of-envelope, at Sonnet 5.5’s list prices ($2 per million input tokens, $10 per million output):
Input: 5 retrieved chunks (~3,600 tokens) + system prompt and question (~400) ≈ 4,000 tokens
Output: typical answer ≈ 500–900 tokens
≈ $0.008 input + $0.005–0.009 output → roughly 1.5¢ per answerA Sample Q&A card on the homepage is about a cent. Embeddings are free because they run locally in Node. Later questions in a thread resend prior turns, so the fifth question in a conversation is closer to 2¢.
A first real eval (small, and I’ll tell you how small)
Everything above was shipped on vibes and smoke tests until I asked myself the obvious question: did we evaluate it? We hadn’t. One retrieval check and a few demo cards isn’t an evaluation.
So I wrote a gold-fact question set from the Anthropic S-1 — 12 questions across four buckets: the three homepage demo questions, core questions (revenue, officers, customers, PBC structure, net revenue retention), hard questions (Class F voting mechanics), and negative controls (things that aren’t in the filing, like an IPO price or Starlink revenue). Each was run through live retrieval and live generation, then scored 0–5 on four axes: retrieval relevance, faithfulness, completeness, and citation quality.
Result: 12 of 12 passed, mean 4.69 / 5. Faithfulness stayed near-perfect (4.92), and citation quality was a clean 5.0 across all 12 cases — up from 4.83 when citations were spans, consistent with the exact-page fix above.
I re-ran this eval after the citation work, since the collection had been rebuilt twice. That turned out to matter: the mix of weak spots changed, not just the headline number.
Two previous misses got fixed. Net revenue retention, flagged earlier as the worst score (3.8) because retrieval never surfaced the 900% figure, now retrieves and states it (4.75). The Class F question, previously missing its headline 50.1% voting-power figure (4.0), now states it explicitly (4.5).
A new one showed up. The PBC-structure question dropped to 3.5 — retrieval now surfaces Public Benefit Fund financial mechanics instead of the actual legal-structure section the question asks about. The model handled it honestly, flagging that it didn’t have the Company Purpose text rather than inventing one, but the right content wasn’t in its retrieved chunks this time.
Every one of these misses, old and new, is a retrieval miss, not a generation miss — the model never fabricated a fact it didn’t have. The underlying cause is almost certainly re-ingestion itself: switching text extraction from pdf-parse to pdf.js for the page-tagging work shifted chunk boundaries slightly, which shifts which adjacent sentences end up sharing an embedding with which anchor. Fix one retrieval gap, open another, by chance, nearby. That’s a strong argument for re-running this eval after any re-ingest, not just the big ones — it’s now part of my checklist.
The remaining weak spots still point at the embedder, not the LLM. all-MiniLM-L6-v2 is small and fast, and it’s evidently inconsistent on which dense paragraph of a filing it surfaces for a given query. Swapping to a larger embedding model would likely move the needle more than any generation change. (I looked at MiniLM-L12: roughly +0.6 on STS-B, about four minutes to re-embed both collections. Not worth it by itself; worth trying alongside a hybrid keyword + vector approach.)
How much to trust this: 12 questions is a smoke test with a rubric, not a benchmark. The scoring was LLM-judged, not human-judged, and I ran it myself in the same session that built the system. Treat it as a regression suite and a map of weak spots — one that moves around a bit between ingests — rather than a single fixed scorecard. A real evaluation would use more questions, independent graders, and spot checks by someone who has read the filing.
Reliability: when Qdrant drops the connection
A quiet failure mode cost me an evening. Chat would intermittently return a 500. The logs showed ERR_HTTP2_INVALID_SESSION from Qdrant Cloud: the HTTP/2 session that Node had cached would go stale, and the next request through it would die.
The fix wasn’t clever: force HTTP/1.1 with bounded retries on the Qdrant client (via undici), in both the runtime path and the ingest script. Slightly less efficient on paper, and the intermittent chat 500s stopped. For a low-traffic app, boring connection handling beats clever connection reuse.
UX and mobile polish
None of this is glamorous, but it’s the difference between “a demo” and “a thing people use.”
Filing switch feedback. Switching from Anthropic to SpaceX re-renders the server-side page, which re-fetches demo answers. That takes a couple of seconds, during which the UI looked frozen — and users (me, first) assumed it was broken. Now the selection updates instantly and a loading overlay shows until the new filing is ready. It’s React’s
useTransitionplus a small overlay; the important part is the optimistic selected state.A friendlier failure mode for billing errors, and a hard session cap. If the Anthropic API call fails on a billing/credit error, the chat now shows plain-language copy instead of a raw API error, and the failed question doesn’t count against any limit. Separately, every browser session is capped at 8 questions total, independent of waitlist status — a blunt but effective guard against one visitor accidentally running up the bill.
Removed the Exchange / IPO-target stat blocks. They crowded narrow screens and, for a filing that hasn’t priced yet, mostly said “TBD.” On mobile, less is better.
Favicon and app icons. A proper
.ico, SVG icon, Apple touch icon and 192/512 PWA icons with a manifest, generated from a single script so they stay consistent.A quiet Blog link in the top bar and a “Read how it was built” link pointing at the first post.
What’s still unfinished
I’d rather list these than pretend:
Retrieval recall on numeric metrics. The NRR miss is the clearest example. Next experiments: a stronger embedder, hybrid keyword + vector retrieval, and a larger candidate pool before re-ranking.
A bigger, independently graded eval, with the SpaceX filing included so I can compare the two collections, and a re-run after every retrieval change.
Exact pages for SpaceX. The word-level tagging currently runs on the Anthropic collection; the SpaceX collection still cites chunk-level spans through the legacy path. It’s a re-ingest away.
Amendments. Anthropic will almost certainly amend its S-1 the way SpaceX did. The pipeline can re-ingest in minutes, but I’d like that to be a single command with a diff of what changed.
Stack, updated
Frontend: Next.js on Railway.
Vector storage: Qdrant Cloud — one collection per filing (
spacex_s1,anthropic_s1).Generation: Claude Sonnet 5.5, medium effort (was Haiku).
Embeddings:
all-MiniLM-L6-v2via@xenova/transformers, local in Node — unchanged, and shared between ingest and query.Ingestion: TypeScript script with position-based printed-page recovery and word-level page tagging (
pdf.jscoordinates).Networking:
undiciover HTTP/1.1 with retries for Qdrant.
What I took away from this
Per-document configuration is cheap if you do it early. The painful part of adding Anthropic wasn’t Anthropic. It was every hard-coded “SpaceX” and “282”.
Evaluate before you celebrate. The first eval found an 11% citation accuracy I’d have happily shipped.
Check the thing your model can’t check. Claude never fabricated a citation — it faithfully copied wrong page numbers I gave it. A faithful generator on top of a wrong index is still wrong, and you only find that by comparing citations to ground truth.
Reuse the lesson, not the code. The SpaceX fix (a footer band) didn’t transfer to Anthropic’s PDF. The idea behind it — use geometry, not text — did.
Try it at AskS1.com, switch between filings from the sidebar, and ask it something you’d normally dig through the filing for. If a citation is off, I want to know.
Questions, bugs, or filings you’d like to see next? Reply to this post.