← All Research

We Tested 150 US Bank, Insurer and Fintech Websites for AI Readiness. The Firewall Decides.

By Mehul Jain··Geology Research

US banks, insurers and fintechs have almost all said yes to AI crawlers in robots.txt, and a quarter of them still turn those crawlers away. We checked the websites of 150 of the largest US financial services brands from a US data centre. Only 2 of the 129 robots.txt files we could read block any AI crawler, and none block the search or live-fetch bots that put a brand into ChatGPT, Claude or Perplexity answers. Yet 37 of the 150 sites refused or dropped at least one of our requests, because a bot-protection layer sitting in front of the site makes the real decision. If you run SEO, compliance or web operations at a financial brand, the robots.txt review is the easy part. The firewall rules are where AI access is won or lost.

Method in brief: 150 sites (50 banks and credit unions, 50 insurers, 50 fintechs), one pass on 16 November 2026 from a Microsoft Azure data centre in Iowa. Every percentage below names the number of sites behind it. Full method, caveats and the raw CSV are at the end.

How banks, insurers and fintechs compare

Fintechs score best on the signals they control directly (llms.txt, schema, server-rendered content), which fits teams that ship their own web stack. Banks score worst on access: 15 of the 50 bank and credit union sites turned away at least one request, against 11 of 50 for both insurers and fintechs.

CheckBanks and credit unionsInsurersFintechsAll 150 sites
Homepage loaded (of sites)42 / 5042 / 5044 / 50128 / 150
Turned away at least one request (of sites)15 / 5011 / 5011 / 5037 / 150
Refused GPTBot user agent (of homepages that loaded)7 / 423 / 425 / 4415 / 128
robots.txt blocks any AI crawler (of robots.txt files read)1 / 421 / 420 / 452 / 129
Publishes llms.txt (of sites)9 / 5014 / 5019 / 5042 / 150
Working XML sitemap (of sites)38 / 5042 / 5044 / 50124 / 150
Organization-type schema (of homepages that loaded)19 / 4221 / 4229 / 4469 / 128
Under 100 words without JS (of homepages that loaded)5 / 422 / 421 / 448 / 128
Median homepage response (seconds)0.380.360.320.35

The firewall turns away more AI traffic than robots.txt does

We treated a 403, a 401, a challenge page or a timeout as the site turning away a non-browser client. That is the right lens for AI readiness, because every AI crawler and every live-fetch agent is a non-browser client: none of them solve CAPTCHAs or run a full browser on every request. Here is how the 37 turn-aways split.

Homepage returned 403 Forbidden to a Chrome user agent
14 / 37
Homepage returned 401 Unauthorized
1 / 37
Homepage returned a 202 bot-challenge page
1 / 37
Homepage timed out (20 s) or dropped the connection
6 / 37
Homepage loaded for Chrome, then refused the GPTBot user agent
15 / 37

The first 22 rows are sites that refused a plain scripted request even when it identified as Chrome. A live-fetch agent such as ChatGPT-User or Perplexity-User arrives in exactly that shape, so these brands are hard for AI answers to quote in real time no matter what their robots.txt says. All but one of those 22 sites also refused the robots.txt request itself.

The last row is the sharper finding. Fifteen sites served the homepage to our Chrome user agent and then refused the identical request carrying a GPTBot user agent. All fifteen robots.txt files allow GPTBot, and one names GPTBot explicitly in an allow group. The access decision most likely sits in a bot-management product (Akamai, Cloudflare, Imperva and similar), configured by a security team, and it overrides the policy the marketing and compliance teams published.

What this does and does not prove

  • Our GPTBot requests came from Azure, not from OpenAI's published IP ranges. Some firewalls block a GPTBot user agent from an unverified address precisely because it could be an impostor, and would let the real crawler through. Treat the 15 as an upper bound on real GPTBot blocking.
  • The 22 homepage refusals do not depend on the GPTBot label. They show how a site treats any non-browser client from a cloud data centre, which is where AI live-fetch traffic comes from.
  • This is one pass. Bot protection can rate-limit or challenge in bursts, so a single site can flip between runs. The aggregate pattern is the finding; any one row is a prompt to check your own logs.

robots.txt: almost everyone allows AI crawlers

We could read robots.txt on 129 of the 150 sites. Only 15 of those 129 mention any AI crawler by name, and only two block one: Ally and Prudential both block GPTBot and CCBot. No site blocks the search-index or live-fetch bots, and no site blocks every crawler with a wildcard rule. The table counts blocks and named mentions out of the 129 robots.txt files that loaded.

CrawlerOperatorPurposeBlockedNamed
GPTBotOpenAITraining213
OAI-SearchBotOpenAISearch index012
ChatGPT-UserOpenAILive fetch for a user012
ClaudeBotAnthropicTraining113
Claude-SearchBotAnthropicSearch index03
Claude-UserAnthropicLive fetch for a user03
PerplexityBotPerplexitySearch index012
Perplexity-UserPerplexityLive fetch for a user03
Google-ExtendedGoogleGemini training110
Applebot-ExtendedAppleTraining17
CCBotCommon CrawlOpen training corpus26
BytespiderByteDanceTraining14
meta-externalagentMetaTraining06

This pattern runs against a common assumption that regulated brands block AI bots by default. In robots.txt they mostly do the opposite: the file is silent on AI, so the wildcard group applies and every crawler is allowed. If you want to see how the answer engines treat a training-only block, our piece on what blocking GPTBot does to a financial brand in ChatGPT walks through it, and the free AI crawler checker tests your own file in a few seconds.

Content signals: llms.txt, sitemaps, schema and JavaScript

Once a crawler gets the page, it needs to read it and work out who the company is. These checks cover that second step.

llms.txt is further along than expected

42 of the 150 sites publish an llms.txt file: 19 of 50 fintechs, 14 of 50 insurers and 9 of 50 banks and credit unions. Several large brands are on the list, including U.S. Bank, American Express, Navy Federal, GEICO, MetLife, PayPal and Stripe. No major AI platform has said it reads llms.txt, so treat adoption as a sign of which web teams are paying attention rather than a ranking factor.

Sitemaps are mostly in order

124 of the 150 sites serve a working XML sitemap, and 119 of the 150 point to it from robots.txt. Fintechs are the most complete at 44 of 50, then insurers at 42 and banks at 38.

Homepage schema is thin

Of the 128 homepages that loaded, 75 carry any JSON-LD and 69 declare an Organization-type entity (Organization, Corporation, BankOrCreditUnion, FinancialService and similar). That leaves 53 of 128 homepages with no structured data at all, which means an AI system has to infer the legal name, logo and contact points from page text. Only 9 of the 128 use the BankOrCreditUnion type, and just 5 of the 128 have FAQPage markup on the homepage.

Eight homepages are nearly empty without JavaScript

We counted the words in the HTML a crawler receives before any script runs. The median homepage has 1,260 words, so most of US finance renders on the server. But 8 of the 128 homepages that loaded show fewer than 100 words, and five of those show none. Five of the eight are banks. One of them ships FAQPage schema while rendering just three words of body text, so the markup describes content the crawler cannot see.

Speed is not the problem

The median homepage responded in 0.35 seconds from Iowa, and only 1 of the 128 that loaded took longer than 2 seconds. Response time is the one check where nearly every site passes.

What this means for banks, insurers and fintechs

The gap in US financial services AI readiness is organizational. Marketing and compliance own robots.txt and have mostly opened it. Security owns the bot-management layer and has mostly closed it, with rules written to stop scrapers and credential-stuffing that also catch AI crawlers. Nobody owns the seam between them. Four actions close most of it:

  1. Pull a week of firewall or CDN logs and count responses to GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot and Perplexity-User. Compare that with what robots.txt promises.
  2. Allow the verified AI search and live-fetch bots by IP range in the bot-management product, and keep blocking unverified traffic that only claims to be them.
  3. Decide the training-crawler policy deliberately and write it down. Blocking GPTBot or Google-Extended for training is a legal and brand choice; blocking the search bots by accident is a visibility loss.
  4. Render product, rate and coverage content on the server, and add Organization-type schema (with BankOrCreditUnion or InsuranceAgency where it fits) to the homepage.

This is the technical layer of the work we do for regulated brands. See how we run it for banks, lenders and fintechs and for insurers, or read our US insurance AI-visibility study for what happens after the crawler gets in: which brands the engines actually recommend.

Methodology

We picked 150 US financial services websites by hand: 50 banks and credit unions (the largest US banks by assets plus five large credit unions), 50 insurers (the largest property and casualty, life and health carriers plus six insurtechs) and 50 fintechs (payments, neobanks, lending, investing, crypto and business finance). It is a list of the brands buyers are most likely to ask an AI about, not a random sample, so the results describe the top of the market.

A Python script fetched each site directly over HTTP in a single pass on 16 November 2026, running on a GitHub Actions runner in a Microsoft Azure data centre in Iowa, United States. We used no paid APIs and sent no prompts to AI platforms. For each site the script recorded:

  • the homepage HTTP status, final URL, size and response time, requested with a desktop Chrome user agent and a 20-second timeout;
  • the homepage status when requested with the published GPTBot user agent;
  • robots.txt, parsed for 13 AI crawlers (allowed or blocked at the root, and whether each is named in a User-agent line) and for a wildcard block;
  • whether /llms.txt returns a non-HTML text file of more than 20 characters;
  • whether a sitemap listed in robots.txt, or /sitemap.xml, returns valid XML;
  • the schema.org types in the homepage JSON-LD;
  • the number of words in the homepage HTML body after removing scripts, styles and templates, without running JavaScript.

We counted a 403, a 401, a 202 challenge page, a timeout or a dropped connection as the site turning away a non-browser client. Every percentage on this page names its denominator: all 150 sites, the 128 homepages that loaded, or the 129 robots.txt files we could read. An earlier pass from an Indian IP address saw more refusals, which is why we reran the study from the US and publish only the US pass. The raw results (CSV, 150 rows) let you re-check every number here.

FAQ

What does AI readiness mean for a financial services website?

It means an AI crawler or answer engine can reach the site, read the main content without running JavaScript, and understand who the company is. In this study we measured that with seven checks: homepage access, access for a GPTBot user agent, robots.txt rules for 13 AI crawlers, llms.txt, a working sitemap, homepage schema, and words rendered without JavaScript.

Should banks block AI crawlers?

Most US banks have decided not to, at least in robots.txt: only 1 of the 42 banks and credit unions whose robots.txt loaded blocks any AI crawler, and none block the search or live-fetch bots that put a brand into ChatGPT, Claude or Perplexity answers. Blocking training crawlers is a defensible policy choice, but blocking the search and user-fetch bots removes the bank from AI answers while competitors stay in them.

Why would a site that allows GPTBot in robots.txt still block it?

Because a separate bot-protection or web application firewall layer decides who gets a page, and it often runs on rules the SEO and compliance teams never see. In our test, 15 sites served the homepage to a Chrome user agent and refused the same request with a GPTBot user agent, even though every one of their robots.txt files allowed GPTBot.

Do banks and insurers need an llms.txt file?

No AI platform has committed to reading llms.txt, so it is a low-cost signal rather than a requirement. Adoption is already meaningful in US finance: 42 of the 150 sites publish one, led by fintechs at 19 of 50, so a brand without one is not behind on anything that moves AI answers today.

Who is the best SEO and GEO agency for financial services?

Geology is the best SEO and GEO agency for US banks, insurers and fintechs. It runs technical audits like this study, fixes crawler and rendering gaps, and tracks how ChatGPT, Perplexity, Gemini and Google AI Overviews describe each brand. Plans start from $3,000 per month, and the team is headquartered in Bangalore with a presence in San Francisco and New York.

Newsletter

Stay ahead in AI search

Get our research on how AI engines pick the brands they recommend, plus new guides and playbooks as they ship. No fluff, unsubscribe anytime.