# Holland 1916 — robots.txt
#
# AI policy: block training/dataset crawlers; allow search engines and
# user-prompted / AI-search fetchers. The Content-Signal line declares the same
# intent in machine-readable form (https://contentsignals.org):
#   search=yes     — may index for search
#   ai-input=yes   — may use as input to answer a user's prompt (AI search / RAG)
#   ai-train=no    — may NOT use to train / fine-tune models
# Mirrors the hollandnameplate.com policy locked in HOS-5442, which supersedes the
# earlier HOS-5300 draft posture (that one listed ClaudeBot and Google-Extended as
# ALLOW; they are training crawlers/opt-out tokens and are blocked here — the
# allowed Anthropic agents are Claude-User and Claude-SearchBot).
#
# ⚠️ This file is only the ADVISORY half of the policy. Real enforcement is the
# Cloudflare WAF rule on the zone. The holland1916.com zone has never been
# audited for it — see the note in docs/cloudflare-ai-crawlers.md before cutover.

# Default: everyone allowed — browsers, search engines (Googlebot, Bingbot,
# Applebot, DuckDuckBot) and user-prompted AI fetchers.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /admin

# User-prompted AI + AI search engines — explicitly welcomed (answer engines,
# not training crawls) so people asking ChatGPT / Perplexity / Claude about
# Holland 1916 can retrieve the live page.
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-User
User-agent: Claude-SearchBot
Allow: /

# AI training / dataset crawlers — blocked (our content is not for model training).
# Google-Extended and Applebot-Extended are robots.txt opt-out TOKENS (Gemini /
# Apple Intelligence training), not separate crawlers — honored here; they do NOT
# affect Googlebot / Applebot search indexing.
User-agent: GPTBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Diffbot
User-agent: Omgilibot
User-agent: cohere-ai
User-agent: PanguBot
User-agent: Timpibot
Disallow: /

Sitemap: https://www.holland1916.com/sitemap.xml
Host: www.holland1916.com