Your robots.txt Can't Tell a Thief From a Customer
Here is the advice making the rounds right now: AI companies are scraping your site to train their models, so block their bots before they steal your work. Add the user agents to your robots.txt, paste in a few lines, feel protected. Done.
I understand the instinct. Nobody likes the idea of a trillion dollar company hoovering up years of your writing for free and giving you nothing back. That feels like theft, and in a lot of cases it basically is.
But here is the problem with the advice. A block list treats every bot the same way a bouncer treats a crowd with one rule: nobody gets in. And right now that rule is quietly throwing out the people you actually want inside.
A crawler and a retrieval bot are not the same animal
Most of the panic lumps all AI activity into one bucket. Bots. Scrapers. The machines. Block them.
The trouble is that "AI bot" describes at least two completely different jobs, and they have opposite consequences for your business.
The first is a training crawler. Its job is to go out, gather enormous amounts of text, and feed it into a model that gets trained months later. When that bot takes your content, you get nothing in the moment and very little later. Your words become a drop in an ocean of weights and parameters. Nobody can point to the drop. There is no link, no citation, no visit. That is the bot people are right to be nervous about.
The second is a retrieval bot. This one shows up when a real person asks an AI tool a live question and the tool goes out to fetch current information to answer it. That bot is not storing you for some future model. It is reading you right now, on behalf of someone who is right now trying to solve a problem that you might sell the solution to.
Read that again.
A training crawler takes from you. A retrieval bot can hand you a customer. Your block list cannot tell the difference, so it treats them both like criminals.
When you block the retrieval bot, you are not protecting your content. You are telling the AI answering your buyer's question that you have nothing to say. So it answers using somebody else who left the door open. Usually a competitor. Sometimes a competitor who is wrong.
The hammer problem
A block list is a hammer. It is good at exactly one thing: stopping access. And when access is a yes or no question, a hammer is fine.
The thing is, access stopped being a yes or no question. The real questions now are things like: who is accessing this, what are they going to do with it, do I get anything in return, and is this the moment a buyer is deciding whether I exist?
You cannot answer any of those with a hammer. You answer them with a policy.
And this is where I watch smart teams fall down. They treat the robots.txt file as a technical chore, something a developer edits in four minutes between two other tickets. A business decision worth real money gets made by whoever happened to have the file open. Nobody asked what the company actually wants from the AI ecosystem. Nobody decided. The default just got stricter because strict felt responsible.
A policy you never wrote is still a policy. It is just one you let happen by accident.
"Just block them all" is a position, so own it
Let me be fair to the other side, because there is a real argument here.
Some businesses genuinely should block broadly. If your entire value is proprietary content behind a paywall, and the AI tools are not sending you traffic or paying you for it, then yes, letting them read you for free is a bad trade. Blocking is a reasonable call.
But that is a call. A deliberate one, made by someone who understood the tradeoff and chose it on purpose. That is completely different from blocking by reflex because a blog post told you to.
Ask yourself the honest question. If you block the bot that fetches live answers, you are betting that being invisible in AI results costs you less than letting that AI read a page you already published for the whole public internet to see. Maybe that bet is right for you. Maybe it isn't. But you should be able to say which, and why.
Most of the people pasting in block lists right now could not tell you. They just wanted to feel like they did something.
Doing something is not the same as doing the right thing.
The part everyone skips: a block list only works if the bot respects it
Here is the uncomfortable bit nobody puts in the how to posts.
Your robots.txt is not a wall. It is a sign on the lawn that says please stay off the grass. Well behaved bots read the sign and obey it. The operators you are most worried about, the aggressive scrapers who genuinely do not care about your rights, are also the ones most likely to ignore the sign entirely.
So think about what the block list actually accomplishes. The polite bots, including the retrieval bots that might have cited you, read your sign and leave. The rude ones scrape you anyway. You have successfully locked out your guests and left the door open for the burglars.
If your goal is real protection from bad actors, robots.txt was never the tool. That is a server level and firewall level job, rate limiting, blocking by behavior, challenging suspicious traffic. Different conversation, different team, actual teeth.
The block list in your robots.txt is not security. It is etiquette. And right now a lot of people are using etiquette to turn away the one category of visitor who was going to be polite enough to send business back.
What a real bot policy actually asks
So what does deliberate look like? Not a copied snippet. A short set of decisions you can defend.
I walk clients through it roughly like this.
- What do we actually want from AI tools? Citations and referral traffic? To be the source an answer is built from? Or do we want nothing to do with them because our content is the product?
- Which bots serve that goal and which undermine it? A bot that fetches a page to answer a live buyer question is not the same as one quietly building a dataset. Treat them according to what they do, not the fact that both are automated.
- Where are we willing to be read, and where are we not? Your public blog and your pricing page exist to be found. Your gated research, your internal tools, your customer only areas do not. One policy does not fit both.
- How would we even know if we got this wrong? If you blocked a retrieval bot last quarter, would anyone notice the silence? Most teams have no way to tell, which is exactly why the decision gets made so carelessly.
Notice that only one of those is technical. The rest are strategy. That is the whole point. The config is the easy part. Knowing what you want is the work.
The default is drifting, and it is drifting toward quiet
Something is happening that I do not think most business owners have clocked yet. Every month there are more AI user agents to consider. Every month some tool launches a new crawler, renames an old one, or splits one bot into two with different purposes. The list is not static and it is not short.
So the person maintaining your block list is now in an arms race they did not sign up for. They either keep blocking everything new on sight, which means your policy gets quietly more restrictive over time with no decision behind it, or they ignore the new ones, which means your policy is really just a snapshot of whatever bots existed the day someone last touched the file.
Either way, drift wins. And drift almost always drifts toward silence, because blocking feels safe and allowing feels risky, even when the opposite is true.
Nobody ever got fired for blocking a bot. That is precisely why so many people block the wrong ones.
Here is what I keep telling people. The cost of blocking a retrieval bot does not show up on a dashboard with a red number next to it. There is no alert that says "a buyer asked an AI about your category today and you were not allowed to answer." The loss is invisible. It is the sale that went to the competitor whose page the AI could actually read.
Invisible losses are the most dangerous kind, because they never argue for themselves. They just accumulate.
Stop asking how to block. Start asking what you want to be seen by.
The framing of the whole conversation is backwards. The popular question is "how do I keep AI out?" That question produces block lists, and block lists produce the accidental, drifting, etiquette based non policy I just described.
The better question is "who do I want reading my content, and what do I want back from them?" That question produces a policy. It forces you to separate the bot that takes from the bot that brings. It forces you to decide, out loud, what role you want to play in how people now find answers.
Because that is what is really on the table. Not scraping. Not theft. Distribution. For a growing share of your buyers, the AI answer is the storefront window now. Blocking the bot that builds that window does not protect your store. It just takes you out of it.
A hammer is not a house. A block list is not a strategy.
And a sign that says keep off the grass does nothing except keep away the people who were polite enough to read it.