Create a robots.txt file with full control over search engine and AI crawlers. Decide which bots can access your content.
Follow these tips to ensure your robots.txt file works correctly and supports your SEO strategy.
Your robots.txt file must be at the root of your domain (e.g., example.com/robots.txt). Files placed in subdirectories are ignored by crawlers.
Robots.txt is publicly accessible and only a suggestion, not a security measure. Never use it to hide sensitive content -- use authentication or server-side access controls instead.
Adding your sitemap URL helps search engines discover and index your pages faster. Use the full URL including the protocol (https://).
Use Google Search Console's robots.txt Tester to verify your rules work as expected before pushing to production. A single typo can block your entire site.
Crawl-delay tells bots to wait between requests. Google ignores this directive but Bing and others respect it. Only use it if your server struggles under crawler load.
AI crawlers like GPTBot and ClaudeBot are different from search crawlers. Blocking them prevents your content from being used in AI training, but may reduce your AI search visibility.
Common questions about robots.txt files and crawler management.
A robots.txt file is a plain text file placed at the root of your website that tells web crawlers and bots which pages they can and cannot access. It follows the Robots Exclusion Protocol, a standard used by all major search engines.
No, robots.txt only prevents crawling, not indexing. If other pages link to a URL that is blocked by robots.txt, search engines may still index it based on anchor text and other signals. Use the "noindex" meta tag to truly prevent indexing.
It depends on your goals. Blocking AI crawlers prevents your content from being used in AI model training, which some publishers prefer. However, allowing them can increase your visibility in AI-powered search results like ChatGPT browsing, Perplexity, and Google AI Overviews.
Disallow tells crawlers not to access specific paths. Allow explicitly permits access to paths that would otherwise be blocked by a broader Disallow rule. Allow takes precedence over Disallow when both match a URL, making it useful for exceptions.
User-agent: * is a wildcard that applies rules to all crawlers. You can also create specific rules for individual crawlers (e.g., User-agent: Googlebot) that override the wildcard rules for that particular bot.
Search engines typically cache your robots.txt file for up to 24 hours. Google may re-fetch it more frequently for popular sites. You can request a re-crawl in Google Search Console to speed up the process.
Crawl-delay tells bots how many seconds to wait between requests. Bing, Yandex, and some other crawlers respect it, but Google ignores it entirely. Use it only if your server is under heavy load from bots.
Yes, completely free. This robots.txt generator runs entirely in your browser -- no data is sent to any server. You can generate and download as many robots.txt files as you need.
GrandRanker helps you rank higher with AI-powered SEO content and expert strategies.