Analytics says the site had a quiet week. The hosting panel says the opposite: processor use near the ceiling, a warning email about resource usage, and a couple of afternoons where pages took ages to load. Those two pictures do not fit together until you remember that analytics only counts people whose browsers run the tracking script. Most bots never do.
Spotting bot traffic slowing your website is harder than it should be, because WordPress itself rarely gives a hint, and sites with search, filters or a shop tend to suffer most. Below: which kinds of bots cause trouble, how to spot them in your access logs and host stats, and what to do about them in a sensible order, without blocking the search engines you actually want.
Not all bots are a problem
Plenty of automated traffic is welcome or harmless. It helps to sort what you find into groups before deciding what to do.
| Type | Examples | Usual approach |
|---|---|---|
| Search engine crawlers | Googlebot, Bingbot | Allow. Steer them away from useless URLs. |
| SEO and marketing tool crawlers | Crawlers from backlink and keyword tools | Allow, limit or block, depending on whether you use those tools. |
| AI and data crawlers | Crawlers collecting pages for AI models and datasets | A business decision. Many respect robots.txt. |
| Scrapers | Bots copying prices, content or product data | Rate limit or block. |
| Vulnerability scanners | Requests for old plugin files, config backups, admin tools | Block at a firewall. |
| Login and form attackers | Repeated requests to wp-login.php, xmlrpc.php, contact forms | Block and protect the login. |
The ones that hurt speed are rarely the big search engines. They are the bots that request large numbers of uncached pages quickly, or keep hammering endpoints that make WordPress work hard.
Why bots cost more than visitors
A person reads a page, scrolls, and maybe clicks to another. A bot can ask for hundreds of pages a minute. Worse, bots tend to wander into the URLs that are expensive to serve:
- Search results. Each
/?s=search is built fresh, with a database query, and is rarely cached. - Shop filters and sorting. Combinations of colour, size, price and sort order create a near-endless set of unique URLs, none of them cached.
- Add-to-cart links. Some themes print links with
?add-to-cart=in them. A crawler following those creates sessions and carts on every hit. - Calendars and archives. “Next month” links on event calendars can lead a crawler years into an empty future.
- Login and XML-RPC. Every login attempt runs PHP and checks the database, even when it fails.
Each of those requests needs a PHP process and processor time on your server, the same resources your customers need at the same moment.
Finding out if bot traffic is slowing your website: the access log
Your hosting panel almost certainly keeps a raw access log. In cPanel it is usually under Raw Access or in the Metrics section; Plesk shows logs per domain. Download a day that felt slow and open it in a text editor or spreadsheet.
Each line is one request. A typical line contains the visitor’s IP address, the time, the address requested, the response code and the user agent, which is the name the browser or bot gives itself:
203.0.113.5 [21/Oct/2026:11:02:14] "GET /?s=red+shoes&paged=48" 200 "ExampleBot/1.0"
(Real lines carry a few more fields. This one is shortened.)
What to look for:
- The busiest IP addresses. Sort or count by the first field. A single address with thousands of requests in a day is worth a closer look.
- The busiest user agents. Bots usually name themselves. Count which names appear most.
- The most requested paths. Long runs of search URLs, filter URLs,
wp-login.phporxmlrpc.phpare the classic signs. - The timing. Match bursts of bot requests to the times your host’s graphs showed high usage.
- Response codes. Masses of 404s for files you do not have often mean a vulnerability scanner working through a list.
If the panel offers a statistics tool such as AWStats, its robots and user agent sections give a quicker summary, though less detail than the raw log.
Check whether “Googlebot” is really Google
Anyone can call their bot Googlebot. Google documents a way to verify its crawlers: a reverse DNS lookup on the IP address should return a Google hostname, and a forward lookup on that name should return the same IP. If a heavy “Googlebot” fails that check, it is an impostor and can be blocked without any risk to your search presence.
What to do, in a sensible order
1. Close the expensive doors
Make the URLs bots love cheaper or less reachable. Use robots.txt to ask well-behaved crawlers to skip internal search and filter URLs. Remove ?add-to-cart= links from listings if your theme allows it, or mark them so crawlers do not follow them. Check whether calendar archives need to be crawlable at all.
robots.txt is a request, not a lock. Good crawlers obey it; scrapers and attackers ignore it. It still removes a large share of waste on many sites.
2. Protect the login and XML-RPC
Login floods are both a security issue and a speed issue. Limiting attempts, adding two-factor authentication and blocking access to xmlrpc.php where nothing uses it all reduce the load. The security side is covered in our guide to stopping WordPress brute-force attacks, and our post on whether to disable XML-RPC explains what might depend on it.
3. Rate limit the heavy hitters
Rate limiting slows or blocks any single source that makes too many requests in a short time. It can be applied at a CDN, by the host at server level, or less efficiently by a WordPress plugin. Higher up the chain is better, because a request stopped before it reaches WordPress costs your server nothing.
4. Block what you do not want
For specific bad bots, block by user agent or IP range. For tool crawlers you do not use, many respect a robots.txt rule addressed to their name. Keep a note of what you blocked and why, so a future problem is not mistaken for one of these rules.
5. Consider a web application firewall
A WAF, whether offered by your host, a CDN or a security service, filters known attack patterns and many bad bots before they reach the site. It is a broader security layer rather than a speed tool, but on sites that attract a lot of junk it removes a noticeable share of load. Our explainer on what a website firewall does covers the options.
Mistakes that make it worse
- Blocking all bots. Blanket rules can shut out search engines and the services that check your site for uptime or payments.
- Using crawl-delay for Google. Google ignores the crawl-delay line in robots.txt. If Googlebot genuinely strains a server, the server returning temporary errors makes Google slow down, but that should be a short measure, not a setting left on.
- Relying only on a security plugin. The plugin runs inside WordPress, so each blocked request has already used PHP. It helps, but blocking earlier helps more.
- Blocking by IP once and forgetting. Scrapers move addresses. Rules based on behaviour and rate last longer.
- Not checking the result. After changes, compare a day of logs and the host’s graphs with the day you started from.
Bad bots using server resources: what success looks like
You will know the work paid off when the access log is dominated by real browsers and a few named search crawlers, the host’s usage graphs flatten at the times that used to spike, and the slow afternoons stop. If usage is still high with bots under control, the cause is elsewhere, and our section guide on hosting and page speed covers the other server-side suspects. Bots are one item in the wider work of improving website speed, though on some sites they turn out to be the biggest single drain.
Keeping an eye on it afterwards
Bot traffic changes. New crawlers appear, scrapers rotate addresses, and a theme update can add fresh filter links overnight. A short monthly habit catches most of it: download one weekday’s access log, list the ten busiest IP addresses and user agents, and compare them with last month’s list. Anything new near the top deserves a look. Check the host’s usage graph at the same time. If both look normal, you are done for the month.
Frequently asked questions
Should I block AI crawlers to speed up my site?
Only if they actually show up as heavy in your logs, and only if you are comfortable with your pages not being collected by those services. Many AI crawlers state that they follow robots.txt, so a rule addressed to their names is the gentle option. Decide based on the log evidence and your own view of how your content is used.
Can blocking bots hurt my Google rankings?
Blocking genuine Googlebot from pages you want indexed can. Blocking impostors, scrapers and tool crawlers does not affect Google. That is why it is worth verifying any heavy crawler that calls itself Googlebot before blocking it, and why robots.txt rules for search and filter pages should be written carefully.
Why does a bot request pages that do not exist?
Those are usually vulnerability scanners. They work through lists of files known to belong to plugins with past security holes, backup files left by mistake or admin tools, hoping one exists on your site. Each request still costs a little server time, and a firewall rule can stop them before WordPress loads.
Will a CDN stop bot traffic on its own?
A CDN serving cached files absorbs some bot requests without bothering your server, and many CDNs add basic bot filtering. Requests for uncached pages, search, login and filters still pass through to your hosting unless you add rate limits or firewall rules, so the CDN helps but rarely solves the problem by itself.
Bot control sits halfway between speed and security, and the login side of it belongs firmly to security. Our website security maintenance starts with a full security audit, includes login and brute-force protection and security plugin setup, and offers ongoing monthly monitoring on maintenance plans.
