The web is now mostly machines
In June 2026, Cloudflare — which sits in front of a large share of the world's websites and can therefore measure this directly — reported that bots accounted for 57.5% of the HTML traffic on its network, against 42.5% from actual humans. That was the first time on record that machines made up the majority of web traffic. Read that again: more than half of the requests hitting websites are now automated.
Not all of that is new or sinister. Search engines have crawled the web for decades; that is how Google knows your site exists. Uptime monitors, feed readers, and security scanners all count as bots too. But a large and fast-growing slice of it is something genuinely new: AI crawlers. On Cloudflare's numbers, AI training and data-collection crawlers made up about 20% of verified bot traffic, with AI-powered search assistants adding several percent more. The names you would recognise are the ones doing the heavy lifting — GPTBot from OpenAI, ClaudeBot from Anthropic, Google's AI crawler, PerplexityBot, and a growing crowd of smaller ones.
The growth curve is the part that should get your attention. AI crawler traffic climbed several hundred percent over the course of 2024, and it has kept climbing since. GPTBot alone grew its crawling activity by roughly 300% year over year at one point. This is not a passing spike. It is a structural change in who — or what — is reading your website.
What AI crawlers actually cost you
When a crawler visits your site, it does the same thing a visitor's browser does: it requests pages, images, stylesheets, and scripts, and your server has to answer. Every one of those requests uses a little bit of processing power and a little bit of bandwidth. One visitor is nothing. Tens of thousands of automated requests a day, from dozens of different AI companies, each re-reading your entire site on its own schedule, is a different story.
On shared hosting — the cheap plans where hundreds of sites share one server — this shows up as sluggishness. Your host allocates each account a slice of the machine's resources, and when bots are hammering your site alongside your real customers, your real customers are the ones who feel the lag. On some budget hosts, sustained bot traffic is enough to trip a resource limit and briefly take your site offline. On metered or cloud hosting where you pay for bandwidth and compute, it shows up more directly: on your bill.
Here is the detail that reframes the whole conversation. Traditional search crawlers are a fair trade — Google reads your pages, and in return it sends you visitors. For every handful of pages Googlebot crawls, it sends roughly one human your way. AI training crawlers are nothing like that trade. Measurements in early 2026 put OpenAI's GPTBot at over a thousand pages crawled for every one visitor it referred back, and Anthropic's ClaudeBot was, at points, crawling many thousands of pages for each referral. In plain terms: they take a great deal of your content and, historically, sent almost nothing back. You pay the hosting cost; they get the content.
To be fair, that gap is narrowing as AI search matures and those tools start linking out to sources more often — and for some businesses, as I will get to, that trickle of referrals is turning into something worth having. But if you are running a small Canadian business site on a modest hosting plan, the immediate reality is that a chunk of what you are paying for is being consumed by machines that, for now, mostly take.
Three kinds of bot, and why the difference matters
You cannot make a sensible decision about AI crawlers until you separate them into their real categories, because they are not all the same and they do not all deserve the same treatment.
Training crawlers collect content to build and improve AI models. Cloudflare's data suggests just over half of AI crawler requests are for this purpose. This is the category with the worst trade for you: your content goes into a model, and you get no direct link, no visitor, and no credit. If you are a publisher or a business whose website is your product — a writer, a course creator, a stock-photo seller — this is the traffic you are most likely to want to limit.
AI search crawlers fetch your pages so an assistant can answer a live question and cite you. When someone asks ChatGPT or Perplexity "who does managed WordPress hosting on Vancouver Island," an AI search crawler may go and read relevant pages, then quote or link them in the answer. This is much closer to the old search bargain: you allow the crawl, and you get the chance to be the recommended answer. For a local service business, this is traffic you almost certainly want.
Unidentified scrapers are the bad actors: bots that ignore the rules, disguise themselves as ordinary browsers, and scrape aggressively for whatever purpose their operator has in mind. These do not announce themselves honestly and do not respect the polite signals below. Blocking them is a job for your host or a service like Cloudflare, not for a text file you edit yourself.
The takeaway is that "should I block AI bots" is the wrong question. The right question is "which of these do I want, and how do I keep the rest from costing me money."
Should you block them or welcome them?
This is genuinely a business decision, not a technical one, and the honest answer depends on what your website is for.
If your website exists to win you customers — a plumber, a dentist, a hosting company, a restaurant — then being visible inside AI assistants is quickly becoming as important as being visible in Google. People are asking ChatGPT and Google's AI answers for recommendations instead of scrolling through search results. If you block the AI search crawlers, you are opting out of that entirely. For most local and service businesses, that is the wrong move. You want to be found. There is real evidence the referrals matter, too: retailers saw AI-referred traffic multiply several times over through early 2026, and those visitors tended to convert better and stick around longer than traffic from other channels.
If your website is the product — your income comes from people reading your articles, buying your photos, or taking your paid content — the calculation flips. Training crawlers that ingest your work and return nothing are taking inventory off your shelves. Over two and a half million websites had chosen to disallow AI training crawlers by late 2025, and the majority of news publishers now block at least one. That is a rational response when your content is your livelihood.
Most small Canadian businesses sit in the first camp. The sensible posture for them is: welcome the AI search crawlers, be relaxed about or block the pure training crawlers if you like, and let your host deal with the dishonest scrapers. You do not have to pick a side in the larger fight between AI companies and publishers to make a good decision for your own site.
How to control AI crawlers on your site
You have more control than you might think, and it runs from a simple text file up to host-level tooling.
robots.txt is the oldest and simplest tool. It is a plain text file at the root of your site that tells well-behaved crawlers what they may and may not access. You can name specific AI crawlers and disallow them. For example, you can tell GPTBot and Google-Extended to stay out while leaving Googlebot free to index you for normal search. The important caveat: robots.txt is a request, not a wall. The honest crawlers obey it — and the major AI companies' crawlers do check it — but a dishonest scraper will drive straight past it. It is the right first step, and it is genuinely respected by the bots that matter, but it is not enforcement.
Your host or a CDN like Cloudflare is where real enforcement lives. In 2025 Cloudflare began blocking AI training crawlers by default for new sites and rolled out one-click controls that let any site owner allow, challenge, or block AI bots without touching code. They also introduced a "pay-per-crawl" system that lets site owners charge AI companies for access rather than simply blocking them. You do not need to be technical to use these — if your site is behind Cloudflare, most of it is a toggle in a dashboard. Many managed hosts now bundle similar bot management, which is one more reason managed hosting earns its price for a busy site.
An llms.txt file is the newer, friendlier counterpart to robots.txt. Rather than saying "keep out," it offers AI systems a clean, plain-language map of your most important content so they represent you accurately. It is an emerging convention rather than a settled standard, but it costs nothing to add and it signals that you want to be understood correctly. This very site publishes one.
The layered approach is: use robots.txt to state your preferences to the honest crawlers, use your host or Cloudflare to enforce against the rest, and consider an llms.txt if being represented well inside AI tools matters to you.
What I actually recommend
For a typical Canadian small business — the kind I host and build for every week — here is the plan I would follow.
First, check what is actually hitting your site before you change anything. Your hosting control panel almost certainly has traffic or "awstats"-style statistics that break visits down by user agent. If AI crawlers are a rounding error on your traffic, this is not a problem you need to spend an afternoon on. If they are a visible chunk of your bandwidth or your site has been feeling slow, it is worth acting.
Second, do not reflexively block everything. If customers finding you is the point of your website, staying open to AI search crawlers is an investment, not a cost. Blocking them to save a few pennies of bandwidth while your competitor becomes the answer ChatGPT gives is a poor trade.
Third, if bot traffic is genuinely straining your hosting, that is a signal your site may have outgrown a bargain shared plan. Moving to managed hosting with proper bot management, or putting a service like Cloudflare in front of your site, solves the performance problem and the scraper problem at the same time. That is the version of this I set up for clients: sensible defaults that keep the useful bots, block the abusive ones, and keep the real humans fast.
Fourth, publish an llms.txt if you care how AI tools describe your business — and increasingly you should, because a growing share of your future customers will first hear about you through an AI assistant rather than a search result.
The bottom line
The web quietly became majority-machine, and AI crawlers are the reason. That is not something to panic about, but it is something to be deliberate about. The costs are real — bandwidth, server load, and content taken without much given back — but so is the opportunity, because the same shift that sends bots to read your site is the shift that is turning AI assistants into a place where customers find businesses like yours.
The mistake is treating it as all threat or all opportunity. Sort the crawlers into training, search, and scrapers; decide which you want based on what your website is actually for; use robots.txt for preferences and your host for enforcement; and revisit it as AI search keeps growing. Handled that way, the bot surge becomes one more thing your hosting quietly manages for you, instead of a mystery draining your server.