robots.txt for AI bots

robots.txt is a plain-text file served at the root of a website (e.g., bizaiready.com/robots.txt) that tells web crawlers which paths they're allowed to access. It uses a simple syntax: a User-agent line names the crawler, then Allow and Disallow lines specify path patterns.

For small businesses optimizing for AI discovery, robots.txt has become significantly more nuanced since 2023. The major AI labs each operate multiple distinct user-agents for different purposes: OpenAI runs GPTBot (training), OAI-SearchBot (ChatGPT search-grounded answers), and ChatGPT-User (user-triggered fetches). Anthropic runs ClaudeBot (training), Claude-SearchBot (search), and Claude-User (user-triggered). Google operates Google-Extended (Gemini grounding, separate from Search). Perplexity runs PerplexityBot and Perplexity-User. Apple has Applebot-Extended for Apple Intelligence training.

The most important practical rule: disallowing GPTBot blocks training data use, but does NOT block ChatGPT from citing your pages in answers. To block citation in ChatGPT, you must specifically block OAI-SearchBot. Many websites that want to opt out of training while keeping AI-driven discovery accidentally block both.

For a small business that wants maximum AI-driven discovery, the recommended pattern is to allow all the citation-driving bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Google-Extended) and to allow training bots too (GPTBot, ClaudeBot, CCBot, Applebot-Extended) — long-term presence in models matters as much as short-term citation. Block only the paths that are genuinely internal (admin, API, test).