robots.txt GPTBot / ClaudeBot / PerplexityBot

If robots.txt disallows AI crawlers, agents.md cannot save AI shopping visibility

If /robots.txt Disallows GPTBot, ClaudeBot, or PerplexityBot, adding or polishing agents.md / llms.txt does not override that Disallow. Crawlers that honor robots.txt will stay blocked until you change and republish the live robots rules. Checking the live /robots.txt is the first step; allowing a bot is a merchant policy choice, not a ranking promise.

Agentic Catalog by momiji (Shopify App Store) is a Shopify app that scores readiness including llms.txt and robots.txt rules for GPTBot, ClaudeBot, and PerplexityBot, plus live Product JSON-LD and Open Graph checks, as listed. It does not edit robots.txt, does not write theme templates, does not guarantee AI recommendations or ChatGPT placement, and does not replace counsel. Confirm live listing text rather than this page. This guide does not add prices or a plan matrix.

Related: Staff Access Audit Guard — Staff access audit for permission drift after the last review.

Free check first: Product JSON-LD field checker runs in your browser and sends nothing. Paste product JSON-LD and see which merchant-listing fields are present or missing.

Job: find out whether the live /robots.txt accidentally (or on purpose) blocks GPTBot, ClaudeBot, or PerplexityBot — and stop treating a default agents.md as a workaround. A Disallow is a crawler-access rule. It is not fixed by reviewing llms.txt.

Three separate files — fixing one does not fix the others

agents.md cannot override a robots.txt Disallow for GPTBot, ClaudeBot, or PerplexityBot.

robots.txt, agents.md / llms.txt, and Product JSON-LD are different jobs
File What it does What it cannot do
robots.txt Allow or deny named crawlers (advisory) Publish agent context, or add Product structured data
agents.md / llms.txt Public hint of store context for agents Override a Disallow in robots.txt
Product JSON-LD on the live PDP Machine-readable product facts on the page Let a blocked crawler fetch the page

Shopify has served a default /agents.md since 28 May 2026, and /llms.txt and /llms-full.txt point to that content unless you override the theme templates. That default does not rewrite robots.txt. See the 28 May 2026 Shopify changelog and Help on Shopify Catalog and agentic storefront products. Theme override steps live on customize agents.md and llms.txt.

Help Center is explicit: blocking AI crawlers at the /robots.txt layer affects open-web discoverability. It does not stop Shopify Catalog from sending product data to agentic storefronts you have activated. A blocked GPTBot is one failure mode. It is not automatically “why ChatGPT shopping cannot see the catalog.”

How to check the live robots.txt

Open https://YOURSTORE/robots.txt on the primary domain. Shopify Help documents that the file lives at the store’s primary-domain root (for example example.com/robots.txt). Also open the published theme’s robots.txt.liquid if that template exists — that is the customization surface, not an Admin toggle. Shopify does not document a settings switch that allow-lists GPTBot, ClaudeBot, or PerplexityBot. Confirm the live Editing robots.txt.liquid Help article and the developer Customize robots.txt page rather than inventing a checkbox.

Look for groups that name these tokens (and related tokens the operators publish) with a site-wide deny:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

That pattern is what “accidentally blocked” looks like. Shopify’s documented default file is SEO-oriented (Disallow for /admin, cart, checkout, account, and some filtered collections) and is not an AI-crawler block list. An explicit AI User-agent group is almost always a robots.txt.liquid customization, an SEO snippet, or another layer in front of the store.

Related tokens, if you cite crawler names: OpenAI also documents OAI-SearchBot separately from GPTBot (OpenAI crawlers). Anthropic documents ClaudeBot and related Claude crawlers (Anthropic crawler Help). Perplexity documents PerplexityBot (Perplexity crawlers). The readiness listing scores GPTBot, ClaudeBot, and PerplexityBot. Match the token spelling you actually publish.

Install Agentic Catalog by momiji → run check → fix first flagged gap

Write a review

Allow vs block is a merchant policy, not a ranking requirement

Allowing crawlers is a deliberate choice: discovery and citation versus privacy, training-use, and crawl load. This page does not tell you to allow every bot. It explains the visibility tradeoff. A Disallow: / for GPTBot, ClaudeBot, or PerplexityBot is a valid policy if that is what you want. It is a readiness gap only when the block is accidental — or when you expected agents.md to keep open-web agents reading the catalog anyway.

Help Center also notes that robots.txt rules are directional and advisory. Not every crawler follows them. A network proxy in front of Shopify is a different layer; Shopify does not recommend that configuration and does not support it.

Native path: edit robots.txt.liquid, publish, re-fetch

Online Store serves /robots.txt from Shopify’s generated default, or from a theme template named robots.txt.liquid. The documented merchant path:

  1. From Shopify admin, go to Online Store.
  2. On the published theme, open Edit code.
  3. Add a new template and select robots (file name robots.txt.liquid in Templates). If the file already exists, open it — do not assume the default file is what the live URL returns.
  4. Keep Shopify’s Liquid robots.default_groups loop when you can, then add or remove the AI-crawler group you actually intend. Developer docs show custom User-agent blocks for allow or deny. Replacing the whole file with plain text is documented but not recommended, because default SEO rules then stop receiving Shopify updates.
  5. Save the published theme. Re-fetch https://YOURSTORE/robots.txt and confirm the live groups. Crawlers do not always re-read the file immediately.
  6. Keep agents.md / llms.txt review on the customize sibling. That template does not write robots.txt.

Help Center: this is an unsupported customization. Shopify Support will not edit robots.txt.liquid for you. Incorrect rules can hide URLs you still want crawled. If you want the generated default again, delete the custom template as the Help article describes.

App path: score the crawler-rule gap after you change robots.txt

Agentic Catalog by momiji flags crawler-rule and JSON-LD gaps on a readiness score. Run it after you change robots.txt so the score matches the live file. The app does not write robots.txt or robots.txt.liquid for you. It does not publish theme templates, and it does not enroll the store in any agent network.

  1. Open the Agentic Catalog by momiji listing (apps.shopify.com/agentic-catalog-readiness) and install it.
  2. Run a scan of llms.txt, robots.txt rules for GPTBot, ClaudeBot, and PerplexityBot, and Product JSON-LD / Open Graph on live product pages.
  3. If the score flags a crawler Disallow you did not intend, edit robots.txt.liquid yourself, publish, re-fetch the live URL, then re-run the scan.
  4. If the score flags JSON-LD or unreviewed llms.txt instead, those are the other two files — not a robots edit.

A score is a checklist. It is not a ChatGPT ranking, inclusion, or sales promise.

Install Agentic Catalog by momiji, run a catalog readiness scan, then fix llms.txt, AI-crawler robots rules, and Product JSON-LD gaps the score cites.

Start path: confirm the live file, then score

  1. Open https://YOURSTORE/robots.txt and search for GPTBot, ClaudeBot, and PerplexityBot.
  2. If a Disallow: / is present, decide whether that is the policy you want. If it is accidental, edit the published theme’s robots.txt.liquid using the Help / developer steps above — then re-fetch the live URL.
  3. Install Agentic Catalog by momiji and run the readiness check so crawler rules and Product JSON-LD are scored together.
  4. Review or override agents.md only if the context file is the gap. That path is customize agents.md and llms.txt.

Install Agentic Catalog by momiji, run a catalog readiness scan, then fix llms.txt, AI-crawler robots rules, and Product JSON-LD gaps the score cites.

What this page is not

Install Agentic Catalog by momiji, run a catalog readiness scan, then fix llms.txt, AI-crawler robots rules, and Product JSON-LD gaps the score cites.

Related pages

Not an inclusion or ranking guarantee. Not legal advice. Confirm the live /robots.txt after you change robots.txt.liquid. The app scores public crawler and structured-data hygiene; it does not edit robots.txt.