How AI-Ready Are Agency Websites? 2026 Study

SEO and digital marketing agencies now sell "AI visibility" as a service. So we looked at their own websites.

On 25 September 2026 we checked 300 agency websites that rank in Google for searches like "seo agency", "digital marketing agency london" and "generative engine optimization agency". For each one we recorded how its robots.txt treats 15 crawlers, whether it publishes llms.txt, what structured data its homepage carries, and whether its server answers a request that identifies as OpenAI's GPTBot.

Every number below comes from that collection. Each figure is shown with its count and denominator. When a check failed for a site, that site was left out of that figure rather than counted as "no".

Source: Meek Media AI-Readiness Study of Agency Websites, 2026

Key findings

• 56.3% (148 of 263) of agency sites publish a valid /llms.txt file. 37.8% (56 of 148) of those files carry a Yoast SEO, Rank Math or All in One SEO generator signature.
• Only 4.6% (12 of 260) publish /llms-full.txt.
• Only 3.4% (9 of 266) block any AI crawler site-wide in robots.txt. Just 1 of 266 blocks an AI search or retrieval bot, and none block Googlebot.
• 80.2% (211 of 263) of robots.txt files do not name a single AI crawler. They rely on the default * rule.
• 19.7% (51 of 259) of sites served our browser request but refused a request using GPTBot's user-agent. 50 of those 51 allow GPTBot in their own robots.txt.
• 93.4% (242 of 259) of homepages include JSON-LD structured data, and 79.9% (207 of 259) declare an Organization.
• 45.0% (109 of 242) have FAQPage markup on the homepage or on the first linked services page.
• Only 1.5% (4 of 259) of homepages have under 150 words of text in the raw HTML. Almost all agency homepages can be read without running JavaScript.
• 51.2% (131 of 256) pass all five items on a basic AI-readiness checklist.

How many agencies block AI crawlers in robots.txt?

Very few. Of the 266 sites where we could determine robots.txt rules, 9 (3.4%) block at least one AI crawler from the whole site.

All 9 block training crawlers such as CCBot, GPTBot or ClaudeBot. Only 1 site (0.4%) blocks an AI search or retrieval agent, the type of bot that fetches pages to answer a user's question in ChatGPT, Claude or Perplexity. No site blocks Googlebot. No site blocks everything with a site-wide User-agent: * / Disallow: / rule.

The most common approach is to say nothing. 80.2% (211 of 263) of robots.txt files never mention an AI crawler by name. Every AI bot therefore falls back to the general * rules, which on these sites allow the homepage.

Table 1. How agency robots.txt files treat selected crawlers (sites with a robots.txt, n = 263)

Source: Meek Media AI-Readiness Study of Agency Websites, 2026

CCBot, the Common Crawl bot, is the most-blocked crawler: 9 sites (3.4%) disallow it completely. Some newer user-agents are rarely named at all. Claude-SearchBot is named by 11 sites and Claude-User by 10. That fits with how recently these tokens were introduced.

Do agencies block AI bots at the server, even when robots.txt allows them?

Some servers do refuse them, and robots.txt does not show it.

We requested each homepage twice: once with a normal Chrome user-agent and once with GPTBot's published user-agent. Of the 259 sites that returned a normal page to the browser, 51 (19.7%) refused the GPTBot-UA request. That breaks down into 30 "429 Too Many Requests" responses, 19 "403 Forbidden" responses and 2 "418" responses.

50 of those 51 sites allow GPTBot in their own robots.txt. In other words, the robots.txt says GPTBot may crawl, but the hosting or CDN layer turned our GPTBot-labelled request away.

An important caveat applies here. Our requests did not come from OpenAI's IP addresses. Many CDNs check that a request claiming to be a known crawler really comes from that crawler's network, and reject impostors. A refusal therefore shows that the site's infrastructure rejects unverified GPTBot traffic. It does not prove that the real GPTBot is blocked.

To separate AI-specific blocking from general "fake crawler" protection, we ran a control pass. We repeated the GPTBot request and also sent a request with a spoofed Googlebot user-agent.

Table 2. Server-level refusals of a GPTBot user-agent (sites with an analysable homepage, n = 259)

Source: Meek Media AI-Readiness Study of Agency Websites, 2026

The refusals are not only a Cloudflare issue. The rate was almost the same on and off Cloudflare. Among the refusing sites, 25 were served by Cloudflare, 12 by Hostinger's CDN and 12 by LiteSpeed servers. Overall, 52.8% (158 of 299) of the reachable sites in the sample are served through Cloudflare.

The practical takeaway is that robots.txt does not tell the whole story. Firewall rules, bot-management presets and hosting-level rate limits sit in front of robots.txt and can overrule what it says.

Do agencies publish llms.txt?

More than half do. 56.3% (148 of 263) of sites where we could check returned a real text file at /llms.txt. We did not count HTML "page not found" pages served with a 200 status. 20 sites did that for llms.txt.

A large share of those files were produced by plugins. 37.8% (56 of 148) carry a WordPress SEO plugin generator line: Yoast SEO (37), Rank Math (14) or All in One SEO (5). The remaining 92 have no generator signature. Some are clearly hand-written company profiles, and others list pages or posts.

The companion file is still rare. Only 4.6% (12 of 260) publish /llms-full.txt.

Publishing llms.txt does not always go together with letting AI bots in. 32 of the 143 analysable sites with an llms.txt (22.4%) refused our GPTBot-UA request at the server.

Size varies widely. The median llms.txt was about 6.7 KB, with a range from 65 bytes to 856 KB. Almost all files (146 of 148) follow the expected Markdown shape with an # H1 title. At least one "llms.txt" actually contained robots.txt-style directives.

Adoption was highest in the Australian part of the sample, but the subgroups are small:

Table 3. llms.txt and structured data by search market of first query

Source: Meek Media AI-Readiness Study of Agency Websites, 2026

How much structured data do agency homepages use?

A lot. 93.4% (242 of 259) of analysable homepages include at least one JSON-LD block. The most common types reflect what the major SEO plugins output by default:

Table 4. Schema types found on agency homepages (n = 259)

Source: Meek Media AI-Readiness Study of Agency Websites, 2026

88.8% (230 of 259) identify the business with an Organization, LocalBusiness or ProfessionalService entity. Service markup, which describes the actual offer, appears on only 14.7% of homepages.

FAQ markup is more common deeper in the site. For the 242 sites where we could fetch the first services link on the homepage, 35.1% (85) had FAQPage markup on that page. 45.0% (109) had it on either the homepage or the services page.

Nine homepages (3.5%) contained at least one JSON-LD block that did not parse as valid JSON.

Can AI crawlers read agency pages without JavaScript?

Almost always, yes. Many AI crawlers do not run JavaScript, so content that exists only after scripts run can be invisible to them.

Only 1.5% (4 of 259) of homepages had fewer than 150 words of visible text in the raw HTML. The median homepage had 1,906 words in its server-delivered HTML.

The basic on-page signals are also nearly universal:

• canonical tag on 96.1% (249 of 259)
• <html lang> on 97.3% (252)
• meta description on 98.1% (254)
• exactly one H1 on 89.6% (232)
• noindex on one homepage

Newer "machine-readable twin" signals are almost absent:

• Only 2 of 259 (0.8%) homepages link to a Markdown version with <link rel="alternate" type="text/markdown"> or an equivalent header.
• Only 9 of 299 (3.0%) reachable sites use Cloudflare-style Content-Signal directives. All 9 were in robots.txt, and none were sent as a response header.

How many agency sites pass a basic AI-readiness checklist?

We scored every site where all five items could be determined (n = 256):

1. robots.txt present with a Sitemap line: 91.4%
2. no AI search/retrieval bot blocked at the root: 99.6%
3. Organization or LocalBusiness/ProfessionalService schema on the homepage: 89.1%
4. at least 150 words of text in the raw HTML: 98.4%
5. a valid llms.txt: 55.9%

51.2% (131 of 256) pass all five, and 85.2% (218) pass at least four. llms.txt is the item most often missing.

This checklist does not cover the server-level refusals described above, because our spoofed-user-agent test cannot confirm how a site treats the real crawler.

What this means for your website

Agency sites are, on paper, well prepared for AI search. The gaps that remain are the ones most businesses share. Here is what we would check on any site:

1. Test your server, not just your robots.txt. Request your homepage with an AI crawler's user-agent. Then check your CDN or host's bot settings, such as Cloudflare's AI bot controls, WAF rules and hosting rate limits. That tells you whether your robots.txt and your infrastructure agree. Decide deliberately which bots you allow.
2. Tell AI crawlers apart. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) and search or retrieval agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) do different jobs. If you want to appear in AI answers but opt out of training, write separate rules for each group.
3. Name the bots you care about. Most sites rely on the * fallback. Explicit rules make your intent clear and survive later edits to the general rules.
4. If you publish llms.txt, write it for a reader. A plugin-generated list of posts is a start. A short Markdown summary of who you are, what you offer and where the key pages live is more useful. llms.txt is a proposed convention, and support from AI platforms varies, so treat it as low cost and optional.
5. Describe your services, not just your company. Organization schema is common. Service markup and genuine FAQ content on service pages are much less common.
6. Keep key content in the HTML. Most agency homepages already do this. Check any page built with a client-side JavaScript framework.
7. Validate your JSON-LD. A block with a syntax error is ignored.

Methodology

We built a frame of 406 agency domains from Ubersuggest SERP results for 18 agency-related queries across the US, UK, India and Australia. We excluded directories, marketplaces, publishers and SaaS tools, and any domain that ranked only with a listicle or blog post. We then drew a simple random sample of 300 (seed 20260925).

Each site was checked on 25 September 2026. Checks covered robots.txt rules for 15 crawlers, /llms.txt and /llms-full.txt, homepage HTML and JSON-LD, one services page, response headers, and a GPTBot user-agent request plus a control pass. Where a check failed, the site was excluded from that metric's denominator. In total, 259 homepages could be analysed, 263 sites had a robots.txt, and llms.txt status was determinable for 263.

Main limitations:

• The sample reflects agencies visible in search, not all agencies.
• The GPTBot test used a spoofed user-agent from a non-OpenAI IP.
• Only the homepage and one services page were analysed.
• JavaScript-injected schema is not counted.

Full definitions, limitations and reproduction steps are in the methodology section.

FAQ

What share of SEO agency websites have an llms.txt file?

In our sample of agency sites ranking in Google, 56.3% (148 of 263) served a valid /llms.txt file on 25 September 2026. About 38% of those files were generated by a WordPress SEO plugin (Yoast SEO, Rank Math or All in One SEO).

Do marketing agencies block ChatGPT and other AI crawlers?

Rarely, in robots.txt. Only 3.4% (9 of 266) block any AI crawler site-wide, and only one blocks an AI search or retrieval bot. However, 19.7% (51 of 259) refused a request that used GPTBot's user-agent at the server level. Because our request did not come from OpenAI's IP range, this does not confirm that the real GPTBot is blocked.

Is robots.txt enough to control AI crawler access?

No. In our data, 50 of the 51 sites that refused a GPTBot-labelled request allowed GPTBot in robots.txt. CDN bot management, firewall rules and hosting rate limits can override robots.txt, so check both layers.

Which schema types do agency homepages use most?

Organization (79.9%) and WebSite (79.5%) are the most common. LocalBusiness or ProfessionalService appear on 47.1% of homepages, and FAQPage on 26.3%. Service markup appears on just 14.7%.

Does having llms.txt or schema make a site appear in AI answers?

This study does not measure that. It records which signals are present on agency websites. It does not test whether those signals change how often AI assistants cite a site.

Can I use or cite this data?

Yes. Please credit "Source: Meek Media AI-Readiness Study of Agency Websites, 2026" and link to this page. You can download the anonymized per-site dataset (CSV); agency names are removed so no individual site is singled out.

If you want to know how your own site looks to AI crawlers, including what your robots.txt, CDN and structured data are actually telling them, Meek Media runs AI visibility audits.

Source: Meek Media AI-Readiness Study of Agency Websites, 2026