Should You Block AI Crawlers From Scraping Your Website?

06 September 2026·5 min read

Quick answer: Blocking AI crawlers protects your content from being scraped and reused without credit, but it also removes you from the training and retrieval data that tools like ChatGPT and Perplexity use to recommend businesses. For most small businesses, the citation upside outweighs the protection concern — but a content-heavy publisher with a real IP risk may reasonably choose to block some or all of them. Here's the actual robots.txt syntax either way. ✨

Somewhere in the last two years, "should I let AI crawl my site" became a real question small business owners ask, and most get a confident-sounding answer from someone who hasn't actually thought through the trade-off. Blocking AI crawlers isn't free — it has a real cost most people don't mention.

This isn't a moral question, it's a business one: does being visible to AI search tools bring you more value than keeping your content out of their hands? For 💖 most small businesses, the answer is visibility, but not always — and you should make the call deliberately, not by copying a robots.txt file you found online.

What most businesses get wrong

The most common mistake is blocking every AI crawler reflexively because "AI stealing content" made headlines, without realising that many of the same crawlers are how tools like ChatGPT and Perplexity find and cite businesses when someone asks "who's a good [whatever] near me." Block the crawler, and you can't be recommended — you simply don't exist in that answer.

The opposite mistake is leaving robots.txt wide open without ever checking server logs, so a content-heavy site has no idea how much of its traffic is actually AI bots hammering the server versus real visitors, or whether specific crawlers are worth blocking for cost or IP reasons while others stay open for visibility.

Robots.txt rules for the major AI crawlers

Add these to your existing robots.txt file (usually at yourdomain.com/robots.txt). Each block controls one crawler — remove any block for a crawler you want to allow.

Block OpenAI's GPTBot (used for ChatGPT training):
User-agent: GPTBot Disallow: /

Block Google-Extended (Google's AI training crawler, separate from regular Googlebot — blocking this does NOT affect normal Google Search rankings):
User-agent: Google-Extended Disallow: /

Block PerplexityBot (Perplexity's crawler):
User-agent: PerplexityBot Disallow: /

Block CCBot (Common Crawl, used to train many AI models including some open-source ones):
User-agent: CCBot Disallow: /

To allow a crawler instead (so it can index you for citations), simply don't add a block for it, or explicitly allow it: User-agent: PerplexityBot Allow: /

Content-heavy blog worried about being scraped: A publisher producing long-form guides was seeing AI bots account for a noticeable share of server load, with no traffic or credit coming back. They blocked CCBot and GPTBot (used for model training with no real-time citation benefit) but left PerplexityBot open, since Perplexity actively links back to source pages in its answers.

Local service business weighing citation upside: A tradesperson's website had almost no proprietary content worth protecting — mostly service pages and FAQs — so the calculation was simple. They left every major AI crawler unblocked, reasoning that showing up when someone asks an AI assistant "who does this in my area" was worth far more than any scraping concern.

How to decide (and check what's happening now)

Start by checking your current robots.txt file at yourdomain.com/robots.txt to see what's already blocked — many website builders and SEO plugins add AI crawler blocks by default without telling you.

Weigh your content type. Original research, proprietary data, or long-form guides you've invested heavily in are the strongest candidates for blocking training crawlers like GPTBot and CCBot, since these are used to train models rather than to answer a specific real-time query. Crawlers used for live retrieval and citation, like PerplexityBot, are the ones worth keeping open if being recommended matters to you.

Check your server load if you're on shared or budget hosting — a high volume of AI bot traffic can genuinely slow your site for real visitors, which is a legitimate reason to block some crawlers regardless of the citation trade-off.

Remember robots.txt is a request, not a lock — well-behaved crawlers respect it, but it can't force compliance. It's the standard, appropriate tool for this, just not a guarantee.

💡 Google-Extended is not the same as Googlebot. Blocking Google-Extended stops your content from training Google's AI models but has no effect on your regular Google Search rankings — a distinction that trips a lot of business owners up.

Mistakes to avoid

Blocking every crawler out of general AI anxiety. This removes any chance of being cited by AI search tools your customers are increasingly using.
Confusing Google-Extended with Googlebot. Blocking the wrong one can accidentally hurt your actual search visibility.
Never checking your current robots.txt. Plugins and platforms sometimes add crawler blocks automatically — you might already be invisible to AI tools without knowing it.
Assuming robots.txt guarantees protection. It's a widely respected standard, not an enforceable barrier — treat it as best-effort, not bulletproof.
Setting and forgetting. New AI crawlers appear regularly; review your file every few months rather than assuming it still covers everything relevant.

Frequently asked questions

Does blocking AI crawlers hurt my regular Google SEO?

No — blocking GPTBot, PerplexityBot, CCBot or Google-Extended has no effect on standard Googlebot indexing or your organic search rankings. They're entirely separate crawlers with separate rules.

Will blocking crawlers actually stop my content being used?

It depends. Reputable AI companies generally respect robots.txt, but it can't stop every scraper, and content already used for training before you added the block isn't retroactively removed. It's a reasonable step, not a complete guarantee.

How do I know if AI crawlers are already visiting my site?

Check your server or hosting analytics for user-agent strings like GPTBot, PerplexityBot or CCBot in your access logs. Most hosting dashboards or a simple log-analysis tool can show this without extra software.

Should every business allow PerplexityBot even if they block the rest?

It's a reasonable default for most businesses because Perplexity tends to cite and link back to sources directly in its answers, but genuinely sensitive or original research may still warrant blocking it too — there's no single right answer for every site.

Keep reading 🤍

Written by

Kate, founder of Chronically Online

I help Gold Coast and Brisbane businesses grow with branding, websites and marketing that actually works.

Work with me ✦