Bryan Hamilton / writing

bryanhamilton.info / writing / article

The Firehose Throttled My Server, So I Stopped Crawling It

I crawled a public API firehose one hundred pages deep looking for my own posts, and the API throttled my server for it. The fix was two requests: one profile page for the id list, one per-post endpoint for exact numbers.

  • November 15, 2026
  • 5 min read
  • php · api · webdev · ratelimit
  • More writing

For three weeks, the Coder Legion bar in my site's footer ran on a scraper that fetched the global post feed twenty items at a time and filtered it for my handle. It worked. Then my newest post went live, the bar stayed stale, and every attempt to fix it by crawling deeper made things worse - until the API itself started refusing my server. The fix wasn't crawling harder. It was two requests.

The bar that lied

The bar is a small strip at the bottom of bryanhamilton.info: articles, views, reactions, current level, a link to my latest post. It renders from a JSON endpoint on my own domain, /coderlegion.php, which pulls from Coder Legion's public API.

The API's list endpoint, /api/v1/posts?page=N, is a global firehose: the newest twenty posts across everyone, one page at a time. It ignores every author parameter I threw at it - author, user, handle, all of them return the same global feed. So the scraper fetched pages and filtered for my handle locally. That design had a flaw I chose not to see: I am one author among hundreds, and the firehose only remembers the present.

Pagination is not a strategy

When the bar went stale, I did the obvious thing. I deepened the crawl.

15 pages: 11 articles found, 676 views, 11 reactions. Then 40 pages: 12 articles, 762 views, 11 reactions. Then 100 pages - two thousand posts, fetched one at a time with a 12-second timeout each, an early-stop the moment a page's oldest post predated my join date: 11 articles, 676 views. Identical to the 15-page run.

That was the tell. Two crawls, twelve times the depth, byte-for-byte the same sums. The feed wraps or sheds old posts; either way, my oldest articles were never coming back no matter how deep I went. I almost went to 200 pages.

The throttle finds me

Meanwhile, the crawling itself started failing. After a morning of hundred-page sweeps, the firehose began timing out for my server's IP - 12-second waits, empty responses, a 120-second PHP time limit burning through pages that used to load in half a second.

A rate limit is easy to read as an obstacle. It's more useful to read it as a design review. The API was telling me my access pattern was wrong: one client, hammering a recency-ordered firehose, hunting for records that get older every hour. No amount of pagination fixes that. The design was the bug.

Two requests beat two thousand

The way out was already sitting there. My public profile page, /user/<handle>/posts?cat=articles, lists every article I've written - title, per-post reaction count, comment count - on a single page. It doesn't wrap. It doesn't shed. It's the source of truth I'd been approximating with a firehose.

And the API has a per-post endpoint: /api/v1/posts/<id> with an API key returns exact views, reactions, comments, creation date, and author level for one article. No pagination. Nothing to throttle. Fourteen posts, fourteen small requests.

The new scraper is one page fetch for the id list, then a per-post lookup for each:

$actx = stream_context_create(['http' => [
    'method' => 'GET', 'timeout' => 12,
    'header' => "X-API-Key: $key\r\n",
]]);
foreach ($items as &$it) {
    $raw = @file_get_contents(
        'https://coderlegion.com/api/v1/posts/' . $it['id'],
        false, $actx
    );
    if (!$raw) { continue; }
    $p = json_decode($raw, true)['data'] ?? null;
    if ($p) {
        $it['views']   = (int)($p['views'] ?? 0);
        $it['created'] = $p['created_unix'] ?? $it['created'];
    }
}

Merge, sum, cache for thirty minutes. The old scraper's worst day was a hundred sequential requests; the new one's worst day is fifteen.

The numbers, before and after

  • Firehose, 15 pages: 11 articles, 676 views, 11 reactions
  • Firehose, 40 pages: 12 articles, 762 views, 11 reactions
  • Firehose, 100 pages: 11 articles, 676 views - same as 15
  • Profile + per-post: 14 articles, 881 views, 21 reactions, 19 comments - exact

Views went up under the new scheme, because the per-post endpoint reaches articles the firehose had already fallen off the back of the truck. The bar is fully self-updating now: I publish post fifteen, it appears in the id list, the per-post calls pick it up, no code changes.

What I'd tell myself last week

  • Look for the object endpoint before the list endpoint. List APIs are for browsing; object APIs are for measuring. I used the wrong one for weeks because the right one required reading past the first endpoint I found.
  • A rate limit is a design review. When an API starts refusing you, it's not daring you to add backoff and retry. It's telling you the access pattern is wrong.
  • Compare whole runs, not increments. Two crawls at different depths returning identical sums was the single most informative datum of the day. I nearly missed it by planning a deeper crawl instead.
  • Small and often beats big and once. One page fetch plus N tiny object lookups on a 30-minute cache outperforms a heroic crawl that's stale before it finishes.

The bar's honest now. And my server IP is off the API's bad list, which - trust me on this one - is the better deal.

More from the workshop

All writingEvery article, on this domain. Open source11 production-extracted projects.