Devops Monk

Filing45 publishes every stock trade disclosed by members of Congress, the fund managers who file 13Fs, and the executive branch — each row linked back to the filing it came from.

This post is not a feature tour. It is about the architecture: why the system is shaped the way it is, which constraints did the shaping, and where I got it wrong first.

The short version: almost every design decision in this system was forced by the documents, not chosen. Government disclosure data is not an API. It is four unrelated publishing systems, three document formats, one adversarial CDN, and a stack of scanned paper. The architecture is mostly a record of what each of those refused to do.

The shape of the problem

Four sources. Nothing in common.

There is no shared identifier, no shared schedule, no shared vocabulary, and no common failure mode. A design that treats them as four implementations of one interface will be wrong within a week — I know, because that was my first design.

flowchart TD
    subgraph Sources[Four independent government sources]
        HOUSE[House Clerk\nbulk ZIP index + PDFs]
        SENATE[Senate eFD\nHTML behind Akamai]
        EDGAR[SEC EDGAR\n13F quarterly datasets]
        OGE[OGE 278-T\nscanned paper PDFs]
    end

    subgraph Ingest[Ingest - independent cron jobs]
        P1[house: coordinate-aware\nPDF parsing]
        P2[senate: real Chrome\nvia Playwright]
        P3[edgar: CUSIP to ticker\nvia OpenFIGI]
        P4[oge: OCR repair\nwith confidence flag]
    end

    DB[(PostgreSQL 17\n19 tables - shared instance)]

    subgraph Serve[Next.js 15 standalone on PM2]
        PAGES[15 server-rendered routes]
        API[8 public JSON endpoints]
        ALERTS[Telegram + Web Push]
    end

    HOUSE --> P1 --> DB
    SENATE --> P2 --> DB
    EDGAR --> P3 --> DB
    OGE --> P4 --> DB
    DB --> PAGES
    DB --> API
    DB --> ALERTS

The system is about 9,200 lines of TypeScript. Roughly a quarter of it is ingest, and most of that quarter is handling documents that do not behave.

Decision 1: Four ingest jobs, not one pipeline

The instinct with multiple sources is a pipeline with pluggable adapters: one scheduler, one retry policy, one error path. I built that first. It was the wrong abstraction, and the reason is scheduling.

These sources publish on completely different rhythms:

17 12-23 * * 1-5   house     hourly on weekdays
35 13,21 * * 1-5   senate    twice daily
27 6 * * 1-5       oge       daily
41 5 * * 6         funds     weekly (13Fs are quarterly)
21 6 20 1,4,7,10   managers  quarterly, after the 13F deadline
51 5 1 * *         roster    monthly
5 5 * * 1-6        prices    daily
9 6 * * 1          sectors   weekly
3 * * * *          digest    hourly alert dispatch

A shared pipeline forces the House job — which has new filings most hours — to run at the pace of the 13F job, which has new data four times a year. Worse, it couples failures: when the Senate portal locks you out, you do not want the House ingest inside the same process.

So each source is its own bundled script on its own cron line, with its own log file. They share exactly two things: the database, and the amount-band parser. That is the whole contract.

The cost is real — nine cron entries and no central dashboard. The benefit is that Senate breaking is a Senate-shaped problem, visible in /var/log/filing45-senate.log and nowhere else. For a system whose sources fail independently and constantly, isolation beat abstraction.

The general rule I took from this: shared infrastructure is justified by shared behaviour, not by shared shape. Four things that all “fetch and store” do not belong together if they fetch on different days and fail for different reasons.

Decision 2: Rebuilding tables from PDF coordinates

House filings are e-filed PDFs. They have a real text layer, which sounds like good news until you extract it and discover that a PDF text layer is a bag of positioned strings with no notion of a table, a row, or a column.

Naive extraction gives you this:

NVDA  NVIDIA Corporation  P  12/29/2025  01/16/2026  $100,001 - $250,000  JT

…until a long asset name wraps, and then it gives you garbage, silently, in a way that looks like valid output.

The parser instead works from geometry. pdfjs-dist exposes every text item with its transform matrix, so:

1. Group items by y — that reconstructs a visual line.
2. Sort each line by x — that reconstructs column order.
3. Read columns positionally, using the header row’s x-offsets as boundaries.

This survives wrapping, because a wrapped continuation lands on its own y with an x inside the asset-name column, and can be appended to the right field rather than misread as a new row.

Three things had to be defended against, all of them found in production rather than in testing:

• NUL bytes. Some filings carry control characters inside the text layer. \u0000 is not whitespace to a regex, so \s+ splitting silently failed. Every extracted string is now scrubbed of [\u0000-\u001f] before anything touches it.
• Repeated page headers. Multi-page filings repeat the column header on every page, which parses cleanly as a trade with nonsense fields. Header text is matched and dropped.
• Header text bleeding into values. 286 rows had captured $1,001 - $15,000 Amount Cap. Gains > $200? as the amount — the band plus the next column’s header. The backfill after fixing this recovered bands for 285 of them, and moved one member’s disclosed total from $5.6M to $6.4M.

That last one is the interesting failure. It was not a crash. It was a parser producing confident, well-formed, wrong output for months. Which leads directly to the next decision.

Decision 3: The schema records what each filing can prove

The most consequential design choice in the system is not a technology. It is that uncertainty is a column, not a footnote.

Executive-branch filings are scanned paper, machine-read badly. “Purchase” comes through as lourchaso. Commas in amounts become spaces. Dates lose a slash. The repair heuristics work most of the time — and the rest of the time, a misread figure on a presidential disclosure is worse than a gap.

So official_trades carries a confident boolean, and the filing carries a count of what could not be read at all:

export const officialFilings = pgTable('official_filings', {
  // ...
  rowsParsed:     integer('rows_parsed').notNull().default(0),
  rowsUnreadable: integer('rows_unreadable').notNull().default(0),
});

The UI renders an unconfident amount with a marker rather than presenting it as fact. A reader can tell the difference between “we read $250,001–$500,000” and “we think we read $250,001–$500,000.”

The same principle shows up in three more places:

Paper filings are visible, not dropped. 55 of 531 House filings are handwritten scans with no text layer at all. Early on these were skipped, so eleven members showed “No filings on record” while actually having filed on paper — a false statement produced by silence. They are now stored with scanned: true and surfaced as existing-but-unparsed.

Executive filers are a separate table. officials and official_trades are deliberately not rows in members and trades:

Deliberately separate from members: these are not members of Congress, and mixing them into trades would corrupt every congressional figure on the site — the headline count, the leaderboard, the median delay.

Merging them would have saved two tables and several joins. It would also have quietly made every congressional statistic on the site wrong.

Reference data can never change a filed figure. Sector labels come from an outside source, and the schema comment states the boundary: used to colour composition, never to alter a number that came from a filing. Derived data and disclosed data are not the same kind of thing, and the schema says so.

The general rule: if your source can be wrong in a way you can detect, the detection belongs in the schema. Confidence that lives only in a parser’s local variable is confidence the reader never sees.

Decision 4: Idempotency by content hash

Ingest jobs re-run. They re-run on a schedule, after a crash, and whenever I re-parse a year to pick up a parser fix. Re-parsing must not duplicate trades — and more importantly, must not re-fire alerts to subscribers.

The Clerk gives disclosed rows no identifier. So identity is derived from content:

function rowHash(docId, t, index) {
  return createHash('sha1')
    .update([docId, index, t.ticker ?? '', t.assetName, t.transactionType,
             t.transactionDate, t.amountRange, t.account ?? ''].join('|'))
    .digest('hex');
}

A unique index on that hash makes re-ingest a no-op. Including index means two genuinely identical rows in one filing stay distinct; including docId means the same trade disclosed in an amended filing is correctly a different row.

Alert delivery is deduplicated separately, because “we stored this trade” and “we told this subscriber about it” are different facts:

export const deliveries = pgTable('deliveries', {
  subscriberId: integer('subscriber_id').notNull().references(() => subscribers.id),
  tradeId:      integer('trade_id').notNull().references(() => trades.id),
  sentAt:       timestamp('sent_at').notNull().defaultNow(),
}, t => ({
  uniqueIdx: uniqueIndex('deliveries_unique_idx').on(t.subscriberId, t.tradeId),
}));

Two hundred subscribers and a re-parsed year is exactly the scenario where a system without this sends two hundred duplicate notifications at 3am.

Dead endpoints are pruned on the same path. Telegram returns a distinct error when a user blocks the bot; Web Push returns 410 Gone when a subscription expires. Both mark the subscriber inactive rather than retrying forever.

Decision 5: Use a real browser where a real browser is required

efdsearch.senate.gov sits behind Akamai. It refuses plain HTTP clients, and it also refuses Playwright’s bundled headless Chromium — the fingerprint is detectable.

What works is Playwright driving actual installed Chrome:

const browser = await chromium.launch({
  channel: process.env.CHROME_CHANNEL ?? 'chrome',
});

The portal also gates search behind an agreement checkbox, and the click triggers a navigation. Clicking and then waiting is a race that loses intermittently; the click and the navigation wait have to be raced together:

const agree = page.locator('#agree_statement');
if (await agree.count()) {
  await Promise.allSettled([
    page.waitForURL('**/search/**', { timeout: 30_000 }),
    agree.click({ timeout: 15_000, noWaitAfter: true }),
  ]);
}

allSettled rather than all, because either half may legitimately lose the race; the check that matters is the one after it, which asserts the URL actually changed and the page is not an Akamai Access Denied.

This is the least elegant component in the system and the one I am least inclined to change. It is quarantined in its own script on its own schedule precisely because it is the most likely thing to break.

I also spent real time on OCR for the scanned House and OGE filings, and concluded it was a dead end: House paper filings are handwritten checkbox grids where Tesseract returns noise, and self-OCR of OGE scans is measurably worse than the embedded text layer. That is a negative result worth recording — the architecture now states the limit on the site instead of pretending to have solved it.

Decision 6: One Postgres, one box, boring deployment

Filing45 runs on a shared 2 vCPU VPS alongside four other applications. It uses the box’s existing PostgreSQL 17 and existing nginx rather than bringing its own.

Database: one shared instance, one database, 19 tables, 101 MB. The largest table is prices at 956,766 rows. This is nowhere near needing anything more interesting than Postgres, and treating it as if it were would be the actual architectural mistake.

Build: local, not on the server.

npm run build
npx esbuild src/ingest/run.ts --bundle --platform=node --format=cjs \
  --target=node20 --outfile=dist/ingest.cjs --external:pdfjs-dist

The comment in deploy.sh is the whole rationale: the box has 2 vCPU and runs several other apps, and a Next build would fight them for RAM. Next’s output: 'standalone' produces a self-contained bundle, and the ingest scripts are bundled to single .cjs files by esbuild so the server needs no node_modules for them at all.

Ship: rsync, then reload.

rsync -az --delete .next/standalone/ "$HOST:$REMOTE/app/"
rsync -az .next/static/          "$HOST:$REMOTE/app/.next/static/"
rsync -az public/                "$HOST:$REMOTE/app/public/"
rsync -az dist/*.cjs             "$HOST:$REMOTE/"
ssh "$HOST" "cd $REMOTE/app && pm2 restart filing45 --update-env"

Then a health check that fails the script if the site does not return 200.

That rsync -az public/ line was a bug fix. The deploy originally shipped only the standalone bundle and .next/static, so every image 404’d in production while working perfectly in development. A deploy script that ships a subset of what the dev server serves is a deploy script that will surprise you — and the surprise arrives in production, by definition.

Serve: nginx proxies to 127.0.0.1:3004, with immutable caching on hashed static assets and certbot-managed TLS. PM2 keeps the process up and restores it on reboot.

No containers, no orchestrator, no CI. For a single-process app with cron jobs on one box, each of those would add a failure mode without removing one.

Decision 7: The API shape follows from how it is used

Eight public JSON endpoints, added late, once the data was worth querying. Three choices are worth calling out.

Cursor pagination, not offset. New filings land at the top constantly. Offset pagination on a list that grows at the head silently skips and repeats rows as a client pages through. The cursor is the trade id, and id < cursor is stable regardless of what arrives mid-pagination.

Rate limiting in Postgres, not in memory. A fixed window per token in a table, not a counter in the process:

insert into api_usage (token_id, window_start, requests)
values (${token.id}, ${start.toISOString()}::timestamptz, 1)
on conflict (token_id, window_start) do update set requests = api_usage.requests + 1

An in-memory counter resets on every deploy and every PM2 restart — and this app restarts on every deploy. Postgres was already there; the table is trivial. (That ::timestamptz cast is not decorative: postgres.js refuses a raw Date in a templated insert.)

Session auth as a fallback to bearer tokens. The docs page has an in-browser sandbox, and pasting a token into a form to try an API is a bad first experience. So authenticate() falls back to the session cookie when there is no Authorization header:

const match = header.match(/^Bearer\s+(\S+)$/i);
if (!match) {
  const browsing = await sessionCaller(request);
  if (browsing) return browsing;
  return { status: 401, code: 'missing_token', /* ... */ };
}

This is only safe because of a property of the API itself: it sends Access-Control-Allow-Origin: *, and browsers refuse to pair a wildcard origin with credentialed cross-origin requests. The wildcard that makes the API publicly usable is the same thing that makes cookie auth safe. Worth understanding before copying — the two decisions are load-bearing on each other.

What the architecture still cannot do

Stating limits is part of the design, so:

• 13F holdings are capped at 40 positions per filing (TOP_HOLDINGS = 40). A 626-position portfolio arrives as its 40 largest. Fine for top holdings and quarter-over-quarter diffs; not a complete portfolio.
• 55 House filings are unparsed paper. They are visible as filings-on-record with no rows, which is honest, but it is still a gap.
• The Senate job is fragile by construction. It depends on a real Chrome binary and a CDN’s tolerance. It is isolated so that when it breaks, only it breaks.
• Disclosure lag is the floor, not a bug. The STOCK Act allows 45 days; the median in the data is 28. No architecture makes this data real-time, and a design that implied otherwise would be lying.

The through-line

Re-reading the decisions, they all reduce to the same principle applied in different places:

Design around what the data can actually prove, and make the gaps visible in the structure rather than in the prose.

A confident column. A separate table for executive filers. Paper filings stored rather than skipped. Reference data that can never alter a disclosed figure. Four ingest jobs that fail independently. A cursor that survives a list growing at the head.

None of those are clever. Each of them exists because the alternative produced output that looked right and was wrong — which, in a system whose entire value is that every row links back to a filing, is the only failure mode that actually matters.

The site is at filing45.devops-monk.com, and there is a public API at /api if you would rather query it than read it.