Robots.txt in WordPress: What It Does and How to Edit It

Type your domain followed by /robots.txt into a browser and you will almost certainly see a few lines of text, even though you have never created such a file. Then you look in your hosting file manager and there is no robots.txt anywhere. Both are true, and that small puzzle is the best place to start understanding robots.txt in WordPress.

Once you know where that file comes from, the rest follows: how to edit robots.txt safely through an SEO plugin or as a real file, what a sensible version looks like for a business site, and the few lines that can do real damage. The advice here is deliberately cautious. For most sites the right amount of editing is very little.

What robots.txt is for

robots.txt is a plain text file at the root of a domain that gives instructions to crawlers: the automated programs search engines and other services use to fetch pages. It can say which parts of the site a crawler may or may not request, and it can point to your sitemap.

Two limits matter. First, the file is a request. Well-behaved crawlers such as Googlebot follow it, but nothing forces a badly behaved bot to. It is not a security tool and it does not hide anything; anyone can read it. Second, blocking crawling is not the same as keeping a page out of search results. A blocked page can still appear in Google, usually without a description, if other pages link to it. Keeping a page out of results is a job for noindex, which our post on noindexing a WordPress page explains. In fact, if you block a page in robots.txt, Google cannot fetch it to see the noindex instruction at all.

Crawling settings like this sit alongside the everyday SEO work described in our guide to the basics of WordPress SEO. For the rest of the site’s routine, see the WordPress owner’s guide.

The robots.txt WordPress makes for you

When no real robots.txt file exists in your site’s root folder, WordPress answers requests for /robots.txt itself, generating the text on the fly. On a typical site the default looks like this:

• User-agent: * — the rules below apply to all crawlers.
• Disallow: /wp-admin/ — do not crawl the admin area.
• Allow: /wp-admin/admin-ajax.php — except this one file, which some themes and plugins use on the public side of the site.
• Sitemap: followed by the address of WordPress’s built-in sitemap, added when the site is set to be visible to search engines and the built-in sitemap is active.

SEO plugins usually adjust this virtual file, for example replacing the sitemap line with their own sitemap address. This default is sensible. Many business sites never need anything more.

The “Discourage search engines from indexing this site” box in Settings → Reading works differently from robots.txt: in current versions it adds a noindex instruction to your pages rather than blocking crawlers in the file. If that box was left ticked after launch, see what the discourage search engines setting does.

How to edit robots.txt in WordPress

There are two routes. Use one, not both.

Through your SEO plugin

This is the easier option, because you never touch the server.

• Rank Math: in its General Settings there is a section for editing robots.txt. It edits the virtual file, so it only works when there is no real robots.txt file in the site’s root folder. If one exists, Rank Math tells you and the box cannot be used.
• Yoast SEO: its Tools area has a file editor, which can create and edit a real robots.txt file, as long as your server allows WordPress to write files. Some hosts block this, in which case the option is missing or does not save.

Whichever you use, save, then open yourdomain/robots.txt in a private browser window to confirm the change appears.

As a real file on the server

You can also create a plain text file called robots.txt and upload it to the site’s root folder, the same folder that holds wp-config.php on most installs. Use your hosting control panel’s file manager or an FTP program. Once a real file exists, it is served instead of WordPress’s virtual one, and plugin settings that edit the virtual file stop having any effect.

Our free robots.txt generator builds the file from simple options, including a WordPress preset, so you start from correct syntax rather than typing it by hand.

Before replacing an existing real file, download a copy. That is your undo button.

A sensible robots.txt for a business site

For a typical brochure site or small blog, the WordPress default plus your SEO plugin’s sitemap line is enough. Additions that are sometimes reasonable:

• Internal search results. Search result pages, which on WordPress use addresses with ?s=, can create an endless supply of thin pages. Some owners block them in robots.txt, but current versions of WordPress already mark search result pages noindex on their own, which usually does the job without a rule.
• Areas created by specific plugins that generate many useless addresses, such as some filtering or calendar views. Only block these if you understand what the addresses are and that no real page lives there.
• A crawler you do not want. You can address a single crawler by name with its own User-agent line. Some AI companies publish names for their crawlers, and some site owners choose to disallow them. This is a business decision, not an SEO fix, and it relies on that company respecting the file.

Google supports two pattern characters in rules: * matches any sequence of characters, and $ marks the end of an address. Other crawlers may not support them in the same way. Google also ignores the Crawl-delay line, so adding it will not slow Googlebot down.

Lines that can hurt your site

• Disallow: / under User-agent: *. This asks every crawler to stay away from the whole site. It is often left behind from development or copied from a staging site, and it is the most damaging single line you can have.
• Blocking /wp-content/ or /wp-includes/. Older advice suggested this. Those folders hold your theme’s CSS and JavaScript and your images. Google renders pages much as a browser does, and if it cannot fetch those files it cannot see the page properly. Leave them open.
• Blocking pages you want removed from Google. As explained above, this prevents Google from seeing a noindex instruction and can leave the page in results longer. Use noindex, and let Google crawl the page.
• Listing private areas you want kept secret. The file is public. Writing “Disallow: /secret-price-list/” tells everyone where to look. Protect private content with a password instead.

Checking the file after a change

1. Open yourdomain/robots.txt while logged out. Check it shows exactly what you expect, with each rule on its own line.
2. If the old version still shows, clear your caching plugin and any CDN cache. Some caches keep a copy of robots.txt too.
3. In Google Search Console, open Settings and look for the robots.txt report. It shows the version Google last fetched, when, and any problems it found. You can ask Google to fetch it again from there after a change you need picked up quickly.
4. Use the URL Inspection tool on one or two important pages to confirm they are not blocked by robots.txt.
5. Make sure the sitemap address in the file opens. If you are submitting it as well, see our post on how to submit your WordPress sitemap to Google.

Frequently asked questions

Why does my robots.txt show content when there is no file on the server?

WordPress generates a virtual robots.txt whenever no real file exists in the site’s root folder. The text is created each time someone requests the address. Your SEO plugin may add to it, usually a line pointing to its sitemap. Uploading a real file replaces this virtual version completely.

I edited robots.txt in my SEO plugin but nothing changed. What happened?

The most common reason is that a real robots.txt file already exists in the site’s root folder, so the plugin’s virtual version is never shown. Check your hosting file manager. Either edit that file directly, or download a copy, delete it, and manage the rules in the plugin instead.

Does a WooCommerce shop need a different robots.txt?

Not usually. Cart, checkout and account pages are typically handled with noindex by WooCommerce or your SEO plugin, which is a better tool for them. Be careful about blocking filter or sorting addresses unless you are sure no product or category page uses the same pattern.

Will Google pick up a robots.txt fix straight away?

Not always. Google keeps a cached copy of the file and refreshes it regularly on its own schedule rather than the moment you save. If you have fixed a serious mistake, such as a line that blocked the whole site, use the robots.txt report in Search Console to ask for a fresh fetch.

A Disallow: / line on a live site, or pages slipping out of search after someone edited robots.txt, should be traced, not guessed at, because the line that did the damage may come back with the next deployment or plugin update. Our WordPress bug fixing service handles faults like this from ₹1,499 per incident onwards.