Robots.txt

Robots.txt is a plain-text file placed at the root of a domain, accessible at yourdomain.com/robots.txt. It uses a simple syntax to instruct crawlers, identified by user-agent, which URLs they are allowed or disallowed from crawling.

It is a directive, not an enforcement mechanism: well-behaved crawlers like Googlebot respect it, but it does not physically block access. Robots.txt controls crawling, not indexing, so blocking a page does not guarantee it stays out of search results if it is linked elsewhere.

Each block in a robots.txt file specifies a user-agent and a set of Allow or Disallow rules. For example, you can disallow crawling of an admin directory or a set of low-value URLs while allowing the rest of the site. You can also reference your XML sitemap in the file.

Misconfiguration is risky: an overly broad Disallow can accidentally block important pages or entire sections from being crawled. Always test changes carefully, since a single wrong line can remove pages from search.

Robots.txt now also governs AI crawlers. Bots like GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended can be allowed or disallowed here, which directly affects whether AI engines can read and cite your content.

Many sites accidentally blocked AI search crawlers while trying to block training bots. If you want to appear in AI answers, confirm your robots.txt allows the relevant search crawlers, since blocking them makes your content uncitable regardless of its quality.