tour-feed-pipeline Three feeds. One clean list.

The merge contract

Two sources report the same show. They'll disagree on almost everything except the fact of the show itself: the venue's spelling, the time zone, which details they bother to include. The rules, in order:

1. Every row is normalized first into one shape: a sortable date and display date guaranteed, venue, city and state trimmed and validated, ticket links split per provider, coordinates parsed, sold-out detected.
2. The first row seen wins the slot, keyed by date | city | venue.
3. Later duplicates fill gaps, never overwrite. Ticketmaster knows the price, Bandsintown knows the coordinates, so the merged row knows both.
4. The source badge upgrades to the richer feed when Ticketmaster joins a Bandsintown row, and both ticket links are kept so a fan can buy from either.
5. Rows with no city and no venue are dropped. Garbage in, nothing out.

// The loose key: sources disagree on venue spelling and time zones,
// but they rarely disagree on the date and city of a show.
function tourLooseMatchKey(array $row): string {
    if ($row['city'] === '' && $row['venue'] === '') return '';   // nothing to key on: dropped
    return $row['ymd'] . '|' . slug($row['city']) . '|' . slug($row['venue']);
}

Simplified sketch. The production functions are in src/ on GitHub.

War stories from real data

Every calendar site eventually learns these. This pipeline learned them the expensive way, and each one is in the demo above:

The venue that was the artist

Bandsintown sometimes sends a show whose "venue" is just the DJ's name echoed back. The pipeline checks the site's artist roster and blanks a venue that's really a person. Without that, artist pages grew phantom venues named after the artist.

The coordinate 999.0

Both feeds use 999 (and 0) to mean "unknown location". Those parse as perfectly valid numbers, and one of them puts a show in the Gulf of Guinea. Sentinels become "no coordinates", because wrong coordinates are worse than missing ones.

The date with no year

Curated data says "Mar 15" and means the next March 15, which is next year if you read it in September. The year is inferred and rolled forward, so a yearless date never comes out in the past.

The show that already happened

Past dates are pruned when the cache is written and when it's read, not just when the page renders, and the expiry step reports exactly what it removed.

Cache design

• Versioned filenames (<artist>.v5.json). The row shape has evolved five times. Bumping the version means old files are simply ignored and cleaned up lazily, instead of everything being invalidated mid-deploy.
• One bulk read per request. All cache files are loaded in a single pass, newest wins. On shared hosting, that turned dozens of per-artist file checks into one.
• Stale-while-revalidate. A fresh cache is served first; on a miss, a stale one can be served before paying for a network fetch. A slow day at Bandsintown never blanks an artist page.
• 12-hour freshness. Tour dates change, but not by the minute.

Design decisions

What changed from production

The repo is production code with the private parts removed, and it says so plainly. API keys, the Ticketmaster key holder and affiliate wrapper, the site's artist roster and the rendering layer stay private. In their place are placeholders and guarded fallbacks: with no Ticketmaster key, that source simply contributes zero rows. Everything else is the production code verbatim.

Try it locally

No network, no API keys, no database:

php demo/run.php          # the fixture run, end to end (what the demo above re-enacts)
php tests/run-tests.php   # the self-test that CI runs

Adding a source is one fetch function that returns rows in whatever shape the feed uses. Normalize them into the canonical row, then merge. No registry and no interface to implement.

From the repo's README. PHP 8.0+, with curl or the HTTP stream wrapper. The demo on this page is a JavaScript re-enactment of the same rules so it can run in your browser; the real pipeline is PHP.

Related projects