I run two sites. One of them, The DJ Calendar, holds the data: 99 ranked artists, tour dates, the works. The other, my portfolio, shows pieces of that data live - a top-10 marquee, a hero count-up chip.
For months this was boring. My portfolio's PHP fetched The DJ Calendar's JSON on a schedule, cached it, served it. That's the whole architecture. It worked so well I forgot it existed.
Then every fetch started returning a Cloudflare challenge page.
Confirming the block, all three ways
When a server-to-server call dies, my instinct is to blame my own code. So before touching anything, I shipped a read-only probe worker to the portfolio's host and tested every transport PHP offers:
file_get_contents()on a stream context: 403, HTML challenge page.curl_init()through php-curl: 403, HTML challenge page.exec('curl -s ...')shelling out to the binary: 403, HTML challenge page.
Three transports, one answer. Cloudflare had flagged the shared-hosting IP range, and from that box The DJ Calendar might as well not exist. This wasn't a rate limit I could back off from, a user-agent I could spoof, or a header I could tune. The block was on the address.
I want to be clear about what I did not do next, because it's the part I'm proudest of: I did not spend a night rotating proxies.
The pivot: stop competing with your visitors
Here's the thing about the block. It applied to exactly one client on the internet: my server.
My visitors' browsers were never blocked. Residential IPs sail through Cloudflare all day. And The DJ Calendar is my site - I don't need to scrape it, I can change it. One line:
header('Access-Control-Allow-Origin: *');...and a slim endpoint that emits just the top 10 (~2KB instead of the 1.3MB full dataset), and the architecture inverts. The browser - the visitor's browser - fetches the data cross-origin, directly from the source. My portfolio's server is out of the hot path entirely.
The old server-side proxy didn't die, though. It became the first attempt in a fallback chain:
fetchFeed('/slim-proxy.php', 'https://thedjcalendar.com/top10.php')Try same-origin first (it may still have a fresh cache). Empty payload at the edge? Fall through to the cross-origin endpoint. The contract changed from "the server fetches" to "the server tries, the browser finishes."
This whole pattern is now a tiny open-source repo, browser-direct-feed - one PHP endpoint, one ~15-line JS client, zero dependencies. If your server ever gets IP-blocked by a CDN, the fix is already built.
The off-by-one that no test caught
Of course, shipping the endpoint produced the fun bug.
The feed has a quirk: items[0] is the label line ("The DJ Calendar top 10 DJ's worldwide"), so the artist data is keyed 1-based - djs["1"] belongs to items[1]. The first version of my endpoint "cleaned" this up with PHP's array_values(), re-indexing to 0-based. Seemed tidy. Wasn't.
Every consumer indexes djs by its position in items. The re-index shifted every hover target by one artist. Number 1 showed Number 2's next show. Nobody errored; the marquee just quietly lied.
The lesson I keep re-learning: when a feed changes shape, the field names are the easy part. The key shape is the contract. PHP's json_encode turns [1..10]-keyed arrays into objects and 0-based arrays into lists, and consumers index accordingly. Replicate the original byte-for-byte before you improve it.
The error message that lied
One more, because it's the funniest one.
I built a demo page for the repo with a "try live fetch" button. Clicked it: "live fetch unavailable (offline, blocked, or endpoint gone)."
So the endpoint was gone? I curled it: 200, CORS headers open, all 10 DJs. My browser? Fine. My server? Fine. So I finally reproduced it in a headless browser and watched the network: the fetch succeeded. 200, Access-Control-Allow-Origin: *, valid JSON.
The failure was in my own demo code. I'd written the render function as a named function-expression IIFE - (function render(j){...})(synthData). That name only exists inside the expression. The button's success path called render(j), got a ReferenceError, and the catch-all handler translated that into "endpoint gone."
My catch block wasn't reporting an error. It was confessing to the wrong crime.
Two takeaways there. First, a catch-all that renders a human-friendly message is also a bug-eater - when something fails weirdly, make the handler show the actual error before you trust its story. Second, the fix for "is it the network?" is always the same: watch what the browser actually did, not what the UI claims happened.
The scoreboard
Total cost of un-blocking two sites: one header line, one slim endpoint, one fallback chain. Total servers involved in the hot path now: zero.
The block is still on, as far as I know. I stopped caring, because my data doesn't travel through my servers anymore. It travels through my visitors' - the one IP range on earth Cloudflare has no reason to touch.