GEO: How to Get ChatGPT, Gemini and Perplexity to Recommend You
GEO in practice: how to get ChatGPT, Gemini and Perplexity to cite your business. AI crawlers, quotable content, brand mentions and a 90-day plan.
Read
Technical SEO for AI rests on three layers: robots.txt decides which AI crawlers can read your site, schema.org structured data describes your business, products and articles in a machine-readable way, and llms.txt is an optional table of contents for language models. Only the first two have a real, documented effect. llms.txt is a 2024 proposal, not a standard — and Google states plainly that its search engine doesn’t use it.
| Element | Purpose | Status | Who uses it |
|---|---|---|---|
| robots.txt | Access rules: which crawler may fetch which URLs | Standardised as RFC 9309 (Robots Exclusion Protocol) | Google, OpenAI, Anthropic, Perplexity — with exceptions for user-initiated fetches |
| schema.org structured data | Describes entities: business, product, article, author | Shared search-engine vocabulary, usually deployed as JSON-LD | Google (rich results, understanding content), Bing and Copilot |
| llms.txt | A Markdown list of a site’s key content for language models | A 2024 proposal, not a standard | Not used by Google Search; not treated as a signal in OpenAI, Anthropic or Perplexity crawler docs |
The key thing to understand is that the big AI companies split their crawlers by purpose. One collects training data, another builds a search index the assistant cites from, and a third fetches a page on a user’s request when someone pastes a link or asks a question. Based on official documentation from OpenAI, Anthropic, Perplexity and Google:
| Crawler / token | Company | Purpose | Effect of blocking in robots.txt |
|---|---|---|---|
Googlebot |
Search, including AI Overviews and AI Mode | You disappear from Google, AI answers included | |
Google-Extended |
A token (no separate crawler): Gemini training and grounding outside Search | No effect on inclusion or ranking in Google Search | |
GPTBot |
OpenAI | Model training | Signals your content shouldn’t be used for training |
OAI-SearchBot |
OpenAI | ChatGPT search | Excluded from ChatGPT search answers (navigational links aside) |
ChatGPT-User |
OpenAI | User-initiated fetches | robots.txt rules may not apply |
ClaudeBot |
Anthropic | Model training | Future content excluded from training data |
Claude-SearchBot |
Anthropic | Claude’s search index | Reduced visibility in Claude’s search results |
Claude-User |
Anthropic | User-initiated fetches | Claude won’t retrieve your page when a user asks about it |
PerplexityBot |
Perplexity | Perplexity search results (not training) | Excluded from Perplexity results |
Perplexity-User |
Perplexity | User-initiated fetches | Per Perplexity, generally ignores robots.txt |
The practical upshot: blocking training (GPTBot, ClaudeBot, Google-Extended) is a decision about content rights, not SEO, because AI search visibility depends on different crawlers. A service business that wants ChatGPT, Claude or Perplexity to recommend it should, above all, let OAI-SearchBot, Claude-SearchBot and PerplexityBot in. We cover how to build visibility in those answers in our guide to generative engine optimisation (GEO).
| robots.txt line | What it does |
|---|---|
User-agent: GPTBot |
Opens a rule group for OpenAI’s training crawler |
Disallow: / |
Blocks it from the whole site |
User-agent: ClaudeBot |
Group for Anthropic’s training crawler |
Disallow: / |
As above |
User-agent: Google-Extended |
Google’s token for Gemini training and grounding outside Search |
Disallow: / |
As above — Google Search is unaffected |
User-agent: * |
Everyone else, including Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot |
Disallow: /basket/ |
An example section excluded for all |
Sitemap: https://yourdomain.com/sitemap.xml |
Your sitemap location |
User-agent: * are not merged in. Add a User-agent: OAI-SearchBot group containing only Allow: / and that bot stops honouring your Disallow: /basket/. Repeat important exclusions in every group.A newer addition is Cloudflare’s Content Signals Policy (September 2025): a robots.txt line such as Content-Signal: search=yes, ai-train=no separates consent for a search index (search), for feeding live AI answers (ai-input) and for training (ai-train). Cloudflare itself stresses these are expressions of preference, not technical countermeasures.
Structured data is a standardised description of a page’s content — that a string is a business address, a product price or an article’s author. Google recommends JSON-LD because it’s the easiest to implement and maintain, though Microdata and RDFa are equally valid.
Two statements matter in the AI context. Google’s guide to generative AI features says structured data isn’t required for AI Overviews and there’s no special schema for AI — keep using it for rich results. On Microsoft’s side, Bing’s Fabrice Canel confirmed at SMX Munich in March 2025 that schema markup helps Microsoft’s LLMs understand content (as reported by Search Engine Roundtable). The takeaway: schema isn’t a ticket into AI answers, it’s how machines read facts about your business without guessing.
| Type | Where | Key properties | Notes |
|---|---|---|---|
Organization |
Homepage or About page | name, url, logo, sameAs, contactPoint |
Ties your site to your profiles elsewhere |
LocalBusiness (or a more specific type) |
Contact or location pages | name and address (required), geo, openingHoursSpecification, telephone |
Don’t mark up reviews of yourself — Google reserves aggregateRating for sites reviewing other businesses |
Product + Offer |
Product pages | name, image, offers with price, currency and availability |
Must match the price on the page and in Merchant Center |
Article / BlogPosting |
Blog posts | headline, author, datePublished, dateModified, image |
Author as a Person linked to their profiles |
BreadcrumbList |
Every page | itemListElement |
Mirrors your site structure |
FAQPage |
— | — | Google stopped showing FAQ rich results on 7 May 2026 (Google’s changelog) |
Two rules from practice. First, build one coherent graph: an Organization with an @id that WebSite, articles (publisher) and location pages all reference — rather than several independent, contradictory blocks injected by different plugins. Second, everything in your markup must be visible on the page. A 5.0 rating in JSON-LD that appears nowhere in the content is a fast track to a manual action, not to an AI citation.
Validation: the Rich Results Test (what Google sees), the schema.org validator (syntax across the whole vocabulary) and URL Inspection in Search Console. On the websites and online shops we build, JSON-LD is generated from the same data as the page content — so prices and opening hours don’t drift apart at the first update.
llms.txt is a Markdown file served at /llms.txt, intended to help language models and AI agents find a site’s most important content quickly. Jeremy Howard proposed it on 3 September 2024 at llmstxt.org; the site itself calls it a proposal to standardise, open to community input. It isn’t an IETF or W3C standard.
The proposed format:
The proposal also recommends serving clean Markdown versions of pages at the same URL with .md appended.
| Popular claim | Reality |
|---|---|
| “llms.txt is robots.txt for AI” | No. It blocks and permits nothing — it’s a table of contents. Access is controlled by robots.txt. |
| “Google needs llms.txt for AI Overviews” | No. On 15 June 2026 Google clarified that the file isn’t needed for Google Search and won’t affect visibility or rankings, positively or negatively. |
| “It’s an official standard” | No. It’s a 2024 proposal maintained at llmstxt.org. |
| “Without llms.txt, ChatGPT can’t find me” | According to OpenAI, appearing in ChatGPT search depends on OAI-SearchBot access, not on llms.txt. |
So when should you add one? It makes sense for software and API documentation, knowledge bases and sites developers paste into AI tools. For a business website, it’s a fifteen-minute, low-priority job: harmless, but no substitute for indexable content, a correct robots.txt and structured data. Google explicitly says you’re free to keep one for other services.
Disallow rules in every bot-specific group.To see how all this translates into visibility in Google’s own AI answers, read our article on AI Overviews and AI Mode.
Not sure what your robots.txt, CDN and schema markup are telling AI crawlers? Our SEO and GEO service covers configuration, server logs and structured data, and we’ll implement the fixes — get in touch.
No. It's a proposal Jeremy Howard published on 3 September 2024 at llmstxt.org, not an IETF or W3C standard. In 2026 Google clarified that the file isn't needed for Google Search and doesn't affect visibility or rankings.
No. GPTBot collects model training data, while ChatGPT search relies on a separate crawler, OAI-SearchBot. You can block GPTBot and still appear in ChatGPT search answers, provided OAI-SearchBot has access.
No. Google-Extended covers Gemini training and grounding outside Search, and Google says it doesn't affect inclusion or ranking in Google Search. To opt out of AI Overviews and AI Mode, use the "Search generative AI" setting in Search Console.
Start with Organization and WebSite on the homepage, LocalBusiness on the contact page if you serve customers locally, Article with an author on the blog and BreadcrumbList across pages. Shops add Product with an Offer. Everything in the markup must be visible on the page.
The major companies' training and search crawlers say they honour robots.txt, but user-initiated fetches are the exception: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. robots.txt isn't a technical safeguard; use WAF rules for hard blocking.
Insights

GEO in practice: how to get ChatGPT, Gemini and Perplexity to cite your business. AI crawlers, quotable content, brand mentions and a 90-day plan.
Read
How to get cited in Google AI Overviews and AI Mode: technical requirements, query fan-out, llms.txt myths and the new Search Console AI report, sourced.
Read
A step-by-step SEO audit checklist: indexing, redirects, content, Core Web Vitals, structured data and AI search visibility, with tools and priorities.
ReadWe grow rankings in Google and optimise for AI search engines. No “#1 in a week” promises — just Search Console data you can verify.
We use essential cookies and, with your consent, analytics (Google Analytics, Hotjar) and marketing cookies to understand how you use the site and improve it. Details in our privacy policy.