User-agent: *
Allow: /

# Private, per-user or token pages — keep them out of the index.
Disallow: /account
Disallow: /track/
Disallow: /bus-track/
# Never index URLs that carry an API key.
Disallow: /*?apiKey=
# One-time key-verification links from email. Redeeming is a POST the crawler won't make,
# but there is nothing to index here and the URL is a credential in its own right.
Disallow: /get-key?token=
# Alert confirmation / unsubscribe links from email — same reasoning: the token IS the
# credential, and there is nothing here worth indexing.
Disallow: /alerts

# NOT LISTED HERE ON PURPOSE: /arrivals and /departures.
#
# Both are thin wall-display shells and both now carry <meta name="robots" content="noindex">.
# Adding a Disallow for them would BREAK that rather than reinforce it: a page the crawler is
# forbidden to fetch is a page whose noindex tag is never read, so a URL already in the index
# stays there — sometimes indefinitely, listed with no snippet. Disallow hides the page from the
# crawler; noindex removes it from the index. They are different tools and only one of them
# does what is wanted here.
#
# The entries above are Disallow rather than noindex because for those the URL itself is the
# problem — it carries a credential — so keeping the crawler away is the point.

# AI CRAWLERS ARE ALLOWED, DELIBERATELY, AND THERE ARE NO NAMED GROUPS FOR THEM BY DESIGN.
#
# GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, CCBot and the rest are covered
# by the wildcard above and get the same access as any other crawler. That is a decision taken on
# 2026-09-04, not an oversight: the site's problem is being found, the archive and guide pages
# exist to be read, an answer engine citing them is the same win as a search result, and the
# underlying rail and bus data is open data republished under OGL anyway.
#
# ADDING `User-agent: GPTBot` + `Allow: /` TO SAY SO WOULD BE A SECURITY BUG, NOT DOCUMENTATION.
# A named user-agent group REPLACES the wildcard group for that agent — a crawler reads the most
# specific group that matches it and only that group — so every Disallow above would stop applying
# to precisely the agents named. The credential-bearing paths (/account, /track/, /bus-track/,
# ?apiKey=, /get-key?token=, /alerts) would become crawlable for them alone, which is the opposite
# of the intent and would look like a tidy, explicit, well-meant file while doing it.
#
# So the policy lives in this comment. To CHANGE it — to block training crawlers while keeping the
# answer engines, say — each named group must repeat the full Disallow list above, or be written
# as `Disallow: /` for a full block where the omission does not matter.
#
# Nothing is blocked upstream either: checked 2026-09-04 against the live origin, ClaudeBot,
# GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, CCBot and Bingbot user-agents each got
# the full 82,457-byte day page, identical to a browser. Worth re-checking if AI referrals ever
# drop off, because Cloudflare has a one-click "block AI scrapers" control that would override
# everything in this file without changing it.

Sitemap: https://transitradar.co.uk/sitemap.xml