Does blocking GPTBot stop ChatGPT citing my site?

No. GPTBot collects training data for future models on a timescale of months to years. When ChatGPT searches the web to answer a question it uses OAI-SearchBot for the index and ChatGPT-User for live fetches, and each reads its own group in your robots.txt. Blocking GPTBot alone changes future training and leaves today's answers untouched.

What are the four layers of AI crawling?

Training crawlers collect text for future model weights. Retrieval crawlers build the index an assistant searches at answer time. Live fetch agents retrieve a specific URL at question time with a budget of seconds. The semantic layer is not a crawler at all: it splits retrieved pages into passages and ranks them, and no robots.txt directive reaches it.

Do AI crawlers execute JavaScript?

It varies by vendor and by which agent is involved, and it changes. Training, index and question-time agents are separate systems with separate infrastructure. For commercially important content the safe assumption is that it should not require execution to be readable.

Can AI crawlers read HTML tables?

Handling varies, and a cell's meaning depends on its headers, which does not always survive chunking into passages. Keep the table for readers and restate the two or three decisive figures in a sentence so there is a quotable version.

Why does my robots.txt work in a browser but fail for crawlers?

Bot protection at the CDN or WAF layer often challenges requests without browser characteristics, including requests for robots.txt itself. The file then becomes unreadable to the crawlers it was written for. Four of the 37 financial brands we tested on 23 September 2026 were in this state.

How many financial firms have an AI crawler policy?

In our sample of 33 readable files collected on 23 September 2026, eleven named at least one AI crawler and twenty-two named none. Indian firms engaged at a higher rate than American or British ones, with six of fifteen naming an agent against three of ten in the US and two of eight in the UK.

← Back to guides

Guide

How AI Crawlers Work - Training, Retrieval and Live Fetch

AI crawler names four systems, not one. What GPTBot, OAI-SearchBot and ChatGPT-User each do, what every layer can and cannot read, and what 37 banking and insurance robots.txt files actually say.

By Mehul JainLast updated September 23, 2026

"AI crawler" names four different systems, and a decision that is correct for one is routinely wrong for another. That confusion produces the most common complaint we hear from banks and insurers: we blocked the AI crawlers, and the assistants still quote us.

They do, because the agent you blocked collects training data for future models and the agent answering the question is a different one reading a different line in your robots.txt.

This guide separates the four layers, explains what each one can and cannot read, and covers what that means for a financial site where the important facts live in calculators, tables and policy documents. It draws on robots.txt from 37 banking, insurance and fintech domains across India, the US and the UK, and served HTML from fourteen product pages, both collected on 23 September 2026.

The four layers

Training

Agents including GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended collect text that may inform future model weights.

The lag is months to years. Blocking them shapes what a model released later has absorbed and does close to nothing to what an assistant says today, because the model answering now trained on a corpus that closed some time ago.

Retrieval index

Agents including OAI-SearchBot, PerplexityBot and Claude-SearchBot build and refresh the index an assistant searches at the moment a question arrives.

This layer decides whether you can be cited today. Block it and you leave live answers, generally within weeks.

Live fetch

Agents including ChatGPT-User, Perplexity-User and Claude-User fire at question time against a specific URL.

The budget is seconds. This is where slow pages, bot challenges and content that only exists after JavaScript runs tend to fail.

The semantic layer

Not a crawler. Once a page is retrieved it is split into passages, embedded and ranked, and the assistant works with a passage.

No robots.txt directive reaches this layer. It is governed entirely by how content is written and structured, and it is where the most expensive financial failures happen.

What the sector actually does

Most firms have written nothing

Of 33 readable robots.txt files in our sample, 22 named no AI crawler at all. Not allowed, not blocked. Unaddressed.

Those firms do have a policy: the one written for search engines years ago, now applied by default to retrieval systems that did not exist at the time.

The firms that engaged did it properly

We expected to find the classic error, a file blocking GPTBot while leaving OAI-SearchBot open. We found one instance across the whole sample, and it reads as deliberate.

Every other firm naming an AI crawler named agents from both the training and retrieval categories. Bajaj Allianz named eighteen distinct agents, Investopedia twenty, GEICO fifteen, Compare the Market twelve, HDFC ERGO eleven.

The divide in this sector is between firms that have considered the question and firms that have not.

India engaged at a higher rate

Market	Files naming an AI crawler	Sample
India	6	15
United States	3	10
United Kingdom	2	8

Against the usual assumption that Indian BFSI trails on technical SEO, that is worth noting. Small sample, large brands, so treat it as a signal.

Four firms publish a policy nobody can read

Four domains returned an error or bot challenge instead of their robots.txt. A crawler that cannot fetch the file cannot read the restrictions in it, so an unreadable policy behaves closer to no policy than to a strict one.

Two further sites returned a challenge instead of their product page to a standard desktop user agent, serving under 200 words where a full page should have been.

How a crawler reads your robots.txt

One group applies, and it replaces the rest

A crawler reads exactly one group: the most specific one naming it. The User-agent: * group is not merged in as a fallback.

Add a group naming GPTBot with a single restriction and you have not added a restriction. You have replaced your existing rules for that crawler and released it from everything else.

Longest match wins

Within the applicable group, every matching rule is compared and the longest path pattern wins. At equal length, Allow beats Disallow. A * matches any sequence and $ anchors the end of a path.

This is the behaviour standardised as RFC 9309, and it is usually how a directory somebody believed was blocked turns out to be reachable.

Unrecognised directives are ignored

Newer syntaxes for declaring search, AI input and training preferences separately are ignored by standard parsers. They record intent alongside your rules rather than changing outcomes.

What each layer can read

Server-rendered HTML

Reliable across every layer. If a fact is in the initial HTML response, it is available.

Client-rendered content

The least reliable channel, and the one financial sites depend on most. Premium calculators, eligibility checkers, quote flows and rate tables that populate after a call returns exist only after execution.

Whether a given agent renders is a per-vendor, per-layer question that changes. For commercially important content, assume it should not require execution.

In our sample, Indian insurance pages served substantial text before any JavaScript ran: 15,937 words on Policybazaar's term insurance page and 18,772 on HDFC ERGO's health page, against 2,213 for GEICO's auto page. The marketing copy arrives. The interactive machinery producing the numbers does not.

Tables

A table's meaning lives in the relationship between a cell and its headers, and that relationship does not always survive chunking. Indian pages in our sample carried between three and thirteen tables where the American pages carried none.

Keep the table for readers and restate the two or three decisive figures in prose.

Images

A rate card published as a PNG carries its numbers in pixels. Extraction varies by vendor, and alt text cannot realistically carry a rate table. Treat figures that exist only in an image as unquotable.

Symbols and regional number formats

An amount written as "50 lakh", as ₹50,00,000 and as "5000000" is one number to an Indian reader and three strings to a parser. Indian digit grouping puts separators where most training data does not.

BankBazaar's home loan page carried fifty lakh and crore references and no rupee character at all. Firms relying on retrieval systems to quote figures accurately in India have a normalisation problem their American and British competitors do not.

Documents and cross-references

Policy wordings are cross-referential by design: a coverage clause in one section qualified by an exclusion in another. Chunked, the coverage clause becomes a self-contained passage asserting cover, and the exclusion competes separately.

The result is an answer that quotes your document accurately and describes your product wrongly, which is harder to detect than an outright fabrication.

What to do

Decide the two questions separately

Whether you want your content informing future models, and whether you want to be quotable today, are different questions with different answers. Write each into the file as its own group.

Confirm the file is reachable

Fetch your own robots.txt from outside your network with a non-browser user agent. Four large financial brands in our sample are publishing a policy no crawler can read.

Read your own page as a crawler does

Fetch your most commercially important URL without a browser and read what comes back. If the figures a customer needs are absent, that is the surface area a non-rendering agent has.

Give important facts a stable home

Every number a customer might ask about should appear once in server-rendered prose, in a sentence naming it. A figure that exists only in a calculator is a figure you have declined to publish in quotable form.

Write conditions into the same passage as claims

Wherever a page states coverage, state the principal condition in the same paragraph rather than linking to it. A chunk containing the claim should contain the qualifier.

Publish exclusions as pages

Exclusions are the most-asked and least-published part of any policy. A plainly written exclusions page per product, each exclusion a self-contained sentence, is well suited to retrieval and genuinely useful.

Check the credential is not client-rendered

Registration numbers and disclosure text often sit in a footer component that hydrates with the application. If it renders client-side, the firm's most verifiable trust signal is invisible to any agent that does not execute.