robots.txt Generator
Create a correct robots.txt file for your website in a click. This free robots.txt generator builds the file from simple options — including a safe WordPress preset — and lets you copy or download it, so search engines crawl what they should and skip what they shouldn't.
💡 Upload the file to the root of your site so it is reachable at yourbusiness.com/robots.txt. robots.txt only asks crawlers politely — it is not security, and it does not remove pages already indexed (use noindex or Search Console for that).
How the robots.txt generator works
Tick options, the file writes itself, download.
Choose access
Allow all for a live site, block all for staging.
Add rules & sitemap
Use the WordPress preset and add your sitemap URL.
Download & upload
Put robots.txt at your site's root — that's it.
What robots.txt does (and what it doesn't)
robots.txt is a small text file at the root of your website that tells search-engine crawlers which parts of the site they may or may not crawl. It is useful for keeping bots out of admin areas, internal search results, and duplicate or thin sections, and for pointing them to your sitemap so new pages are found faster.
It is easy to get wrong in ways that hurt: a single stray Disallow: / can hide your whole site from Google. That is why this generator uses safe defaults and a tested WordPress preset instead of asking you to hand-write directives.
robots.txt for WordPress — the safe setup
For most WordPress sites the right file is short: allow everything, block /wp-admin/ but allow admin-ajax.php (many plugins need it), optionally block internal search URLs, and list your sitemap. Don't block CSS or JS folders — Google needs them to render your pages. If you are unsure whether crawling issues are holding your site back, our WordPress support services and website maintenance plans include technical checks like this, and our local SEO services cover the wider on-site setup. New to WordPress? Start with how to install WordPress.
robots.txt generator — FAQs
Where do I put the robots.txt file?
At the root of your domain so it loads at yourbusiness.com/robots.txt. On WordPress, upload via FTP/file manager, or use an SEO plugin's robots editor.
Will robots.txt remove a page from Google?
No. It only stops crawling. To remove an indexed page, use a noindex tag or Search Console's removal tool.
Should I block AI crawlers?
That's your choice. Blocking GPTBot and similar stops them using your content for training, but it does not affect Google search. The option adds the common ones.
Does WordPress already have a robots.txt?
WordPress serves a basic virtual one if no file exists. A real file at the root overrides it and gives you full control.
Can you check my site's crawl setup?
Yes — technical checks are part of our maintenance plans and SEO services.
Want your whole site technically sound?
Crawling, speed, security, and updates — handled every month by people who do this daily.
Get Site Support →