Blocking GPTBot Does Not Remove You From ChatGPT

GPTBot is not the crawler that fetches your page when somebody asks ChatGPT about your company. Block it and you have changed what future models are trained on, which is a real decision with real consequences, but you have not touched today's answers. A different agent does that job, it reads a different line in your robots.txt, and on most banking and insurance sites nobody has written that line.

We fetched the robots.txt of 37 banking, insurance and fintech domains across India, the United States and the United Kingdom on 23 September 2026 to see how the sector is handling this. Thirty-three returned a file we could parse. Twenty-two of them name no AI crawler at all.

Before going further it helps to see the four layers laid out together, because their timescales are what make a decision at one layer useless at another. The diagram below shows each one and how quickly it acts.

Four systems wearing one name

"AI crawler" gets used for four different things. A decision that is correct for one is often wrong for another, and the confusion is what produces the blocked-but-still-quoted outcome.

The training layer

Agents like GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended collect text that may inform future model weights.

The lag here is months to years. Blocking them shapes what a model released next year has absorbed. It does almost nothing to what an assistant says this afternoon, because the model answering today was trained on a corpus that closed some time ago.

The retrieval layer

Agents like OAI-SearchBot, PerplexityBot and Claude-SearchBot build and refresh the index the assistant actually searches at the moment somebody asks a question.

This is the layer that decides whether you can be cited today. Block it and you disappear from live answers, generally within weeks. Leave it open while blocking the training crawler and you remain fully quotable.

The live fetch layer

Agents like ChatGPT-User, Perplexity-User and Claude-User fire at question time, against a specific URL, usually because the assistant or the person named it.

The time budget here is seconds. This is where slow pages and content that only exists after JavaScript runs tend to fail, and it is the layer most likely to produce an answer about your product built from someone else's description of it.

The semantic layer

Not a crawler at all. Once your page is retrieved it gets split into passages, embedded and ranked. The assistant works with a passage, not your page.

No robots.txt directive reaches this layer. It is governed entirely by how your content is written and structured.

For an insurer this is where the most expensive failures happen. A policy wording is a cross-referential document: the coverage clause on one page is modified by an exclusion twelve pages later. Split that into passages and a retrieval system can surface the coverage without the exclusion attached. The answer a customer receives is then wrong in a direction that creates an expectation your policy does not honour.

Why the distinction matters more in regulated verticals

In most categories, getting the layers confused costs you some visibility. In banking and insurance it can cost you accuracy.

If your retrieval path is blocked but your competitors' are open, assistants answering questions about your products will assemble those answers from aggregators, forums and competitor comparison pages. You have not removed yourself from the conversation. You have removed your own version of the facts from it and left everyone else's in place.

That is the failure mode worth taking seriously: not absence, but being described by other people while your own pages sit unreadable.

What 33 financial services robots.txt files actually say

Most of the sector has not engaged

Twenty-two of 33 readable files name zero AI crawlers. Not allowed, not blocked, simply unaddressed.

Those firms do have a policy. It is the one they wrote for search engines years ago, now applied by default to a set of retrieval systems that did not exist when it was written. Among the silent: Policybazaar, Acko, Digit, Groww, Zerodha, Paytm, LIC India, Progressive, State Farm, Allstate, Stripe, SoFi, Bankrate, Policygenius, Monzo, Starling, Revolut, Admiral and Direct Line.

Several of those are firms whose whole proposition is that they are the technology-forward option in their market.

The ones who engaged got it right

We went in expecting to find the classic error: a file blocking GPTBot while leaving OAI-SearchBot open, so the firm gives up future training inclusion and keeps none of the protection it wanted.

We found one instance of that shape in the whole sample, and it looks deliberate rather than accidental. NerdWallet names Google-Extended and nothing else, which reads as a specific decision about Google's AI training rather than a blanket stance.

Every other firm that named an AI crawler named agents from both the training and retrieval categories. Bajaj Allianz names eighteen distinct agents. Investopedia names twenty. GEICO fifteen. Compare the Market twelve. HDFC ERGO eleven.

The split in this sector is not between firms getting it subtly wrong and firms getting it right. It is between firms who have thought about it at all and firms who have not.

Which agents get named

Across the files that engage, the most frequently named training agents were ClaudeBot (10 files), GPTBot (9), CCBot (7) and Google-Extended (7).

On the retrieval side, PerplexityBot appears in 10 files, with OAI-SearchBot and ChatGPT-User in 8 each. That PerplexityBot is named more often than OpenAI's search agent is a small tell about which crawler has made itself most visible in server logs.

India is ahead of the US and UK here

Indian insurers and fintechs engaged at a higher rate than American or British firms in this sample. Given the usual assumption that Indian BFSI trails on technical SEO, that is worth noting. Small sample, large brands, so read it as a signal rather than a census.

Four firms publish a policy no crawler can read

Four of the 37 domains returned an error or a bot challenge instead of their robots.txt: ICICI Prudential, Chime, Lemonade and Aviva.

Chime is the interesting one, because it behaved the same way when we ran a broader robots.txt study three weeks earlier. This is not a transient failure.

The consequence is direct. A crawler that cannot fetch your robots.txt cannot read your restrictions. Whatever access policy those firms believe they have published is not reaching the systems it was written for, and the practical result of an unreadable policy file is closer to no policy than to a strict one.

What the silence actually costs

A file that names no AI crawler is not neutral. It is a default, and the default is permissive for most agents.

That cuts both ways. A firm that wanted protection has none. A firm that wanted visibility has it by accident rather than design, which means nobody has checked whether the paths that matter are reachable, whether the quote flow is exposed, or whether a legacy Disallow written for Googlebot in 2019 is now hiding the product pages from retrieval agents.

Both firms are in the same position: their machine-reader policy was set by somebody who was solving a different problem years ago.

What to do instead of blocking GPTBot

Decide the two questions separately

They are genuinely different, and conflating them is what produces regret.

The first is whether you want your content informing future models. The second is whether you want to be quotable in answers today. A bank protecting proprietary research might reasonably answer no to the first and yes to the second. A broker trying to get recommended should answer yes to both.

Write the answer to each question into the file as its own group.

Remember that a named group replaces the catch-all

This catches out careful people. A crawler reads exactly one group: the most specific one naming it. Your `User-agent: *` rules are not merged in as a fallback.

Take a file that starts with sensible restrictions for everybody:

``` User-agent: * Disallow: /internal/ Disallow: /quote-engine/ Disallow: /admin/ ```

Then somebody adds a group for the AI crawler they read about:

``` User-agent: GPTBot Disallow: /pricing-models/ ```

Read quickly, GPTBot now looks more restricted than before. Read the way a crawler reads it, GPTBot is free to fetch `/internal/`, `/quote-engine/` and `/admin/`, because those lines live in a group it never opens. The author added a restriction and removed three.

In a regulated firm those paths are rarely trivial. Quote engines and internal tooling are exactly the directories a compliance team assumed were closed.

Know how the matching actually resolves

Within the group that applies, every matching rule is compared and the longest path pattern wins. When two rules match at equal length, Allow beats Disallow. A `*` matches any sequence and `$` anchors the end of a path.

This is the behaviour standardised as RFC 9309. It means a broad Disallow can be reopened by a longer Allow further down the file, which is usually how a supposedly blocked directory turns out to be reachable.

Watch for directives that parsers ignore

Some files in our sample carried Content-Signal lines, a newer syntax for declaring search, AI input and AI training preferences separately.

Standard robots.txt parsers ignore what they do not recognise. A directive like that sits alongside your rules expressing an intention rather than changing an outcome. It is worth adding if you want the intent on record, and it is not a substitute for naming agents in groups that parsers do act on.

Check what your file actually says per crawler

A single site-wide verdict is not an answer, because your file can say different things to different agents from the same lines. Our robots.txt tester evaluates a URL one crawler at a time and shows the line number of the rule that decided it, and the AI crawler checker covers which agents reach you at all.

Confirm the file is reachable

Fetch your own robots.txt from outside your network with a non-browser user agent. If your edge security returns a challenge, your policy is invisible, and four large financial brands in our sample are in exactly that position without apparently knowing it.

Fix the retrieval path before the policy

If the goal is being recommended rather than being left alone, robots.txt is the smaller half of the work. The page still has to be retrievable, parseable and quotable once the crawler arrives. That is the technical GEO and SEO side of it, and for regulated firms it interacts with everything covered in our financial services and insurance programmes.

The short version

Blocking GPTBot is a defensible decision about future training data. It is not a way to stop ChatGPT quoting you, and if that is what you wanted, the file needs to name the retrieval agents as well.

Before writing anything, check what your file currently says and whether a crawler can read it at all. In a sample of 37 financial brands, twenty-two had nothing written and four had written something nobody could fetch.