SEO & GEO

Structured Data, llms.txt and AI Crawlers: Technical SEO for AI

8 min read Updated: Czytaj po polsku
Structured Data, llms.txt and AI Crawlers: Technical SEO for AI

Technical SEO for AI rests on three layers: robots.txt decides which AI crawlers can read your site, schema.org structured data describes your business, products and articles in a machine-readable way, and llms.txt is an optional table of contents for language models. Only the first two have a real, documented effect. llms.txt is a 2024 proposal, not a standard — and Google states plainly that its search engine doesn’t use it.

Three layers: what does each one do?

Element Purpose Status Who uses it
robots.txt Access rules: which crawler may fetch which URLs Standardised as RFC 9309 (Robots Exclusion Protocol) Google, OpenAI, Anthropic, Perplexity — with exceptions for user-initiated fetches
schema.org structured data Describes entities: business, product, article, author Shared search-engine vocabulary, usually deployed as JSON-LD Google (rich results, understanding content), Bing and Copilot
llms.txt A Markdown list of a site’s key content for language models A 2024 proposal, not a standard Not used by Google Search; not treated as a signal in OpenAI, Anthropic or Perplexity crawler docs

robots.txt for AI crawlers: who’s who?

The key thing to understand is that the big AI companies split their crawlers by purpose. One collects training data, another builds a search index the assistant cites from, and a third fetches a page on a user’s request when someone pastes a link or asks a question. Based on official documentation from OpenAI, Anthropic, Perplexity and Google:

Crawler / token Company Purpose Effect of blocking in robots.txt
Googlebot Google Search, including AI Overviews and AI Mode You disappear from Google, AI answers included
Google-Extended Google A token (no separate crawler): Gemini training and grounding outside Search No effect on inclusion or ranking in Google Search
GPTBot OpenAI Model training Signals your content shouldn’t be used for training
OAI-SearchBot OpenAI ChatGPT search Excluded from ChatGPT search answers (navigational links aside)
ChatGPT-User OpenAI User-initiated fetches robots.txt rules may not apply
ClaudeBot Anthropic Model training Future content excluded from training data
Claude-SearchBot Anthropic Claude’s search index Reduced visibility in Claude’s search results
Claude-User Anthropic User-initiated fetches Claude won’t retrieve your page when a user asks about it
PerplexityBot Perplexity Perplexity search results (not training) Excluded from Perplexity results
Perplexity-User Perplexity User-initiated fetches Per Perplexity, generally ignores robots.txt

The practical upshot: blocking training (GPTBot, ClaudeBot, Google-Extended) is a decision about content rights, not SEO, because AI search visibility depends on different crawlers. A service business that wants ChatGPT, Claude or Perplexity to recommend it should, above all, let OAI-SearchBot, Claude-SearchBot and PerplexityBot in. We cover how to build visibility in those answers in our guide to generative engine optimisation (GEO).

Example: visible in AI search, opted out of training

robots.txt line What it does
User-agent: GPTBot Opens a rule group for OpenAI’s training crawler
Disallow: / Blocks it from the whole site
User-agent: ClaudeBot Group for Anthropic’s training crawler
Disallow: / As above
User-agent: Google-Extended Google’s token for Gemini training and grounding outside Search
Disallow: / As above — Google Search is unaffected
User-agent: * Everyone else, including Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot
Disallow: /basket/ An example section excluded for all
Sitemap: https://yourdomain.com/sitemap.xml Your sitemap location

The pitfalls we see most often

  • One group per crawler. Under Google’s robots.txt specification, only one group — the most specific match — applies to a given crawler; rules from User-agent: * are not merged in. Add a User-agent: OAI-SearchBot group containing only Allow: / and that bot stops honouring your Disallow: /basket/. Repeat important exclusions in every group.
  • Caching. Google generally caches robots.txt for up to 24 hours, and OpenAI says search changes take about 24 hours to apply. Don’t judge the effect after an hour.
  • Size and server errors. Google reads only the first 500 KiB. It treats a 4xx on robots.txt (except 429) as “no restrictions”, while a 5xx halts crawling of the site for the first 12 hours.
  • CDNs and firewalls. In July 2025 Cloudflare announced it was changing its default to block AI crawlers. If your site sits behind a CDN or WAF, check your logs to see whether OAI-SearchBot or PerplexityBot are getting 403s — robots.txt won’t tell you.
  • robots.txt is a request, not a lock. Well-behaved bots respect it, but it blocks nothing technically. Anthropic adds that IP-based blocking may be unreliable; real enforcement comes from WAF rules.

A newer addition is Cloudflare’s Content Signals Policy (September 2025): a robots.txt line such as Content-Signal: search=yes, ai-train=no separates consent for a search index (search), for feeding live AI answers (ai-input) and for training (ai-train). Cloudflare itself stresses these are expressions of preference, not technical countermeasures.

schema.org structured data: what’s worth it in 2026?

Structured data is a standardised description of a page’s content — that a string is a business address, a product price or an article’s author. Google recommends JSON-LD because it’s the easiest to implement and maintain, though Microdata and RDFa are equally valid.

Two statements matter in the AI context. Google’s guide to generative AI features says structured data isn’t required for AI Overviews and there’s no special schema for AI — keep using it for rich results. On Microsoft’s side, Bing’s Fabrice Canel confirmed at SMX Munich in March 2025 that schema markup helps Microsoft’s LLMs understand content (as reported by Search Engine Roundtable). The takeaway: schema isn’t a ticket into AI answers, it’s how machines read facts about your business without guessing.

Type Where Key properties Notes
Organization Homepage or About page name, url, logo, sameAs, contactPoint Ties your site to your profiles elsewhere
LocalBusiness (or a more specific type) Contact or location pages name and address (required), geo, openingHoursSpecification, telephone Don’t mark up reviews of yourself — Google reserves aggregateRating for sites reviewing other businesses
Product + Offer Product pages name, image, offers with price, currency and availability Must match the price on the page and in Merchant Center
Article / BlogPosting Blog posts headline, author, datePublished, dateModified, image Author as a Person linked to their profiles
BreadcrumbList Every page itemListElement Mirrors your site structure
FAQPage — — Google stopped showing FAQ rich results on 7 May 2026 (Google’s changelog)

Two rules from practice. First, build one coherent graph: an Organization with an @id that WebSite, articles (publisher) and location pages all reference — rather than several independent, contradictory blocks injected by different plugins. Second, everything in your markup must be visible on the page. A 5.0 rating in JSON-LD that appears nowhere in the content is a fast track to a manual action, not to an AI citation.

Validation: the Rich Results Test (what Google sees), the schema.org validator (syntax across the whole vocabulary) and URL Inspection in Search Console. On the websites and online shops we build, JSON-LD is generated from the same data as the page content — so prices and opening hours don’t drift apart at the first update.

llms.txt: what is it really?

llms.txt is a Markdown file served at /llms.txt, intended to help language models and AI agents find a site’s most important content quickly. Jeremy Howard proposed it on 3 September 2024 at llmstxt.org; the site itself calls it a proposal to standardise, open to community input. It isn’t an IETF or W3C standard.

The proposed format:

  1. an H1 with the project or business name — the only required element;
  2. a blockquote with a short summary;
  3. optional paragraphs or lists with more detail, without headings;
  4. H2 sections listing Markdown links, each optionally followed by a colon and notes;
  5. an “Optional” section of secondary links an agent can skip.

The proposal also recommends serving clean Markdown versions of pages at the same URL with .md appended.

What llms.txt does and doesn’t do

Popular claim Reality
“llms.txt is robots.txt for AI” No. It blocks and permits nothing — it’s a table of contents. Access is controlled by robots.txt.
“Google needs llms.txt for AI Overviews” No. On 15 June 2026 Google clarified that the file isn’t needed for Google Search and won’t affect visibility or rankings, positively or negatively.
“It’s an official standard” No. It’s a 2024 proposal maintained at llmstxt.org.
“Without llms.txt, ChatGPT can’t find me” According to OpenAI, appearing in ChatGPT search depends on OAI-SearchBot access, not on llms.txt.

So when should you add one? It makes sense for software and API documentation, knowledge bases and sites developers paste into AI tools. For a business website, it’s a fifteen-minute, low-priority job: harmless, but no substitute for indexable content, a correct robots.txt and structured data. Google explicitly says you’re free to keep one for other services.

Implementation checklist

  1. Check /robots.txt: returns 200, is under 500 KiB, points to your sitemap and doesn’t block CSS or JS.
  2. Decide on training (GPTBot, ClaudeBot, Google-Extended) separately from search (OAI-SearchBot, Claude-SearchBot, PerplexityBot).
  3. Repeat important Disallow rules in every bot-specific group.
  4. Review your CDN and WAF: AI bot blocking settings, firewall rules, and the response codes AI bots get in your server logs.
  5. Verify bots by IP — OpenAI and Anthropic publish their ranges; spoofing a user-agent string is trivial.
  6. Deploy JSON-LD: Organization and WebSite on the homepage, LocalBusiness on the contact page, Product in your shop, Article with an author on the blog, BreadcrumbList everywhere.
  7. Validate the markup and compare it against the visible content.
  8. Optionally add llms.txt listing your service pages, terms of engagement, FAQ and contact details.
  9. Monitor: AI bot visits in server logs, the Generative AI performance report in Search Console, and referrals from chatgpt.com or perplexity.ai in GA4.

To see how all this translates into visibility in Google’s own AI answers, read our article on AI Overviews and AI Mode.

Not sure what your robots.txt, CDN and schema markup are telling AI crawlers? Our SEO and GEO service covers configuration, server logs and structured data, and we’ll implement the fixes — get in touch.

Frequently asked questions

Is llms.txt an official standard?

No. It's a proposal Jeremy Howard published on 3 September 2024 at llmstxt.org, not an IETF or W3C standard. In 2026 Google clarified that the file isn't needed for Google Search and doesn't affect visibility or rankings.

Will blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects model training data, while ChatGPT search relies on a separate crawler, OAI-SearchBot. You can block GPTBot and still appear in ChatGPT search answers, provided OAI-SearchBot has access.

Will blocking Google-Extended remove me from AI Overviews?

No. Google-Extended covers Gemini training and grounding outside Search, and Google says it doesn't affect inclusion or ranking in Google Search. To opt out of AI Overviews and AI Mode, use the "Search generative AI" setting in Search Console.

Which structured data should a business website implement?

Start with Organization and WebSite on the homepage, LocalBusiness on the contact page if you serve customers locally, Article with an author on the blog and BreadcrumbList across pages. Shops add Product with an Offer. Everything in the markup must be visible on the page.

Do AI crawlers always respect robots.txt?

The major companies' training and search crawlers say they honour robots.txt, but user-initiated fetches are the exception: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. robots.txt isn't a technical safeguard; use WAF rules for hard blocking.

Sources

  1. Google Search Central — Optimizing your website for generative AI features on Google Search (2026)
  2. Google Search Central — Latest documentation updates (llms.txt clarification 15.06.2026, FAQ rich results 08.05.2026)
  3. Google Crawling Infrastructure — Google's common crawlers (Google-Extended)
  4. Google Crawling Infrastructure — How Google interprets the robots.txt specification
  5. IETF — RFC 9309: Robots Exclusion Protocol
  6. Google Search Central — Introduction to structured data markup
  7. Google Search Central — Local business (LocalBusiness) structured data
  8. OpenAI — Overview of OpenAI crawlers
  9. Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
  10. Perplexity — Perplexity crawlers
  11. llmstxt.org — The /llms.txt file (Jeremy Howard, 03.09.2024)
  12. Cloudflare — Giving users choice with Cloudflare's new Content Signals Policy (24.09.2025)
  13. Cloudflare — Content Independence Day: no AI crawl without compensation (01.07.2025)
  14. Search Engine Roundtable — Microsoft confirms schema helps its LLMs (Copilot) understand your content (20.03.2025)

Need help with this?

We grow rankings in Google and optimise for AI search engines. No “#1 in a week” promises — just Search Console data you can verify.