Bryan Hamilton / writing

bryanhamilton.info / writing / article

Making Your Site Legible to AI Crawlers: GEO on Two Live Domains

GEO - generative engine optimization - decides whether an AI assistant can even read your site. The audit that found AI bots blocked before Apache, the file layer that fixes it, and the checklist that keeps it true.

  • November 3, 2026
  • 7 min read
  • seo · geo · webdev · ai
  • More writing

Search engine optimization has a new sibling. If Google decides whether to rank you, that's SEO. If a language model decides whether to recommend, cite, or summarize you - that's GEO: generative engine optimization. Same discipline as ever (be legible to machines), new examiner.

I run The DJ Calendar and my portfolio at bryanhamilton.info. This is the working GEO setup on both live domains - including the audit that found my AI visibility was dead before Apache ever saw a request.

The audit that lied (and the toggle that didn't)

An AI-visibility audit scored one of my domains 24/100, "needs improvement - AI could not reach your site." Which was baffling, because my robots.txt explicitly welcomed the AI crawlers by name: GPTBot, ClaudeBot, OAI-SearchBot, the whole guest list.

Permissive robots.txt, unreachable site. That contradiction is the tell: something between the bot and Apache was answering for me. So I ran the cheapest diagnostic in the world - a user-agent matrix:

for ua in GPTBot ClaudeBot OAI-SearchBot Claude-Web PerplexityBot \
          Googlebot Bingbot "Mozilla/5.0"; do
  printf '%-16s %s\n' "$ua" "$(curl -s -o /dev/null -w '%{http_code}' \
    -A "$ua" https://example.com/)"
done

One signature mattered: a 25-byte, text/plain 403. My PHP never ran. Apache never logged it. That's a WAF-shaped hole - and it was Cloudflare's "Block AI Scrapy" toggle, doing exactly what it says on the tin.

The fix was one toggle. The diagnosis was the matrix. After flipping it, all 39 user-agents in my full sweep returned 200.

Re-run your audits, always

Here's the part that saves you from shipping a ghost: the audit still said 24/100 after the fix. Verdicts are snapshots. The tool had cached its crawl, the same way every cache in this stack does. Re-ran the scan - clean. An infra fix is not done when it's deployed; it's done when a fresh measurement says so.

The file layer: llms.txt and ai.txt

Two conventions matter here. llms.txt is a curated map of your site for language models - markdown, human-readable, with a short description of what the site is and links to the good stuff. ai.txt is a variant with the same job.

My setup on both domains:

  • llms.txt at the root: what the site is, the evergreen URLs, the writing hub.
  • ai.txt served as a byte-identical copy of llms.txt (verified with md5sum rather than trusting my own copy-paste). One wrinkle: the parser used by at least one auditor choked on typographic characters, so the file got a pure-ASCII rewrite. Em-dashes are not worth a failed ingest.
# bryanhamilton.info

> Full-stack engineer, 29 years in production. Zero-dependency web.

## Writing
- [From Hundreds of Rich Results Warnings to Zero](https://bryanhamilton.info/writing/...): JSON-LD at 1,000-page scale

IndexNow: tell the indexes now, not eventually

IndexNow is the ping protocol Bing and friends accept: you signal "these URLs changed" the moment they change, instead of waiting for a crawl schedule. Implementation is two files and one habit:

  • The key lives in a file named after itself at the domain root (served via an .htaccess rewrite), proving you control the domain.
  • Every publish or removal fires a batched ping. My endpoint returns HTTP 200 or 202 on success - I log both, they're equally valid.
curl -s -X POST https://api.indexnow.org/indexnow \
  -H 'Content-Type: application/json' -d @batch.json

The habit matters more than the plumbing: submit on publish, submit on removal. A dead URL in the index is as bad as a missing one.

The 429 red herring

One bot saw an HTTP 429 during testing. My limiter-phobia said "I'm being blocked!" - no. The site runs a per-IP rate limiter, and a burst of test requests looks like a burst of test requests. The 429 cleared in under a minute. Diagnose transients before you patch around them; a fix for a transient is a bug with extra steps.

Entity hygiene: canonicals and sameAs

LLMs build entity pictures the same way search engines do: who is this, what else do they appear as. The boring fundamentals carry the weight:

  • Every syndicated article's canonical points at the self-hosted original. (Gotcha for dev.to authors: the canonical_url front-matter overrides the API's canonical parameter. Ask me how I know.)
  • sameAs in JSON-LD mirrors every real profile, exactly once, no duplicates.
  • The agent-facing layer is explicit: a discovery endpoint describing the site's API and schemas, so a machine doesn't have to guess.

The checklist

  • UA matrix over robots.txt - verify reachability, not intention.
  • robots.txt welcomes the AI bots by name (or deliberately doesn't).
  • llms.txt + ai.txt, ASCII-clean, byte-verified.
  • IndexNow key live, ping on publish AND remove.
  • Canonicals point home on every syndication.
  • Re-run audits after infra changes - never trust a cached verdict.

None of this is glamorous. All of it is the difference between "the AI never heard of you" and "the AI quotes your production notes back to a hiring manager." The first audit said 24/100. The fresh one says the bots are through the door - because I checked what they actually see, not what my robots.txt wishes they saw.

More from the workshop

All writingEvery article, on this domain. Open source11 production-extracted projects.