What Goes in a Robots.txt File
A robots.txt file uses a simple syntax built around user-agents and rules. Each block names a crawler with User-agent, then lists Disallow rules for paths to block and Allow rules for exceptions. A line like User-agent: * applies rules to all crawlers.
For a typical site, you allow most crawling and disallow only genuinely low-value or sensitive areas such as admin directories, internal search results, and checkout pages. It is good practice to reference your XML sitemap in the file with a Sitemap line, helping crawlers discover your important pages.
Creating Robots.txt on WordPress
WordPress generates a basic virtual robots.txt automatically, but to customize it you have two main options. SEO plugins like Yoast or Rank Math include a robots.txt editor in their tools section, letting you edit the file without touching the server. Alternatively, you can create a physical robots.txt file and upload it to your site's root directory via FTP or your host's file manager, which overrides the virtual one.
When editing on WordPress, take care not to disallow /wp-content/ broadly, since blocking resources like CSS and JavaScript can prevent Google from rendering your pages correctly. Allow the resources Google needs to render the site while blocking only genuinely private areas.
Avoiding Costly Robots.txt Mistakes
Robots.txt errors are among the most damaging in SEO because a single line can deindex large parts of a site. The classic mistake is a stray Disallow: / which blocks the entire site, sometimes left over from a staging environment. Always double-check before deploying.
Remember that robots.txt controls crawling, not indexing: a blocked page can still be indexed if linked elsewhere, and a page blocked in robots.txt will never show its noindex tag. Also confirm you are not accidentally blocking AI search crawlers if AI visibility matters to you. Testing in Search Console after any change is essential.