LLMs.txt Generator for Magento 2

What you get

What's New in v1.2

The v1.2 rewrite is a major output upgrade driven by real LLM-readability feedback. Every change exists to reduce token waste and improve semantic clarity:

Everything from v1.1 still works — caching, per-store emulation, tag-based invalidation, the dedicated cache type, company info section, URL rewrites. v1.2 is additive + quality-focused.

Key Features

Structured Output Built for LLMs

• Hierarchical Category Tree via proper Markdown nesting (no duplicate labels)
• Store Overview metadata block at the top — URL, type, store view, currency, locale, generation timestamp
• Priority URLs — admin-authored list of "read these first" URLs with auto-inferred labels
• Collections — separate from main taxonomy so LLMs understand "promotional landing page" vs "browse category"
• Key Pages (CMS) — filtered to drop Magento scaffolding (no-route, cookie notices, etc.)
• Company block — email, phone, address, country, VAT — pulled from general/store_information

Curated, Not Dumped, Products

• Featured Products — merchant-flagged via a configurable yes/no EAV attribute (default is_featured)
• Best Sellers — Magento's sales_bestsellers_aggregated_yearly aggregate (top ordered qty, indexer-driven, O(1) queries)
• Recent Arrivals — created_at DESC
• Each product line: [Name](url) — Price — SKU — Category with optional one-line short description
• Short description resolver: short_description → meta_description → first sentence of full description → auto-generated "Name — Category" fallback

Dual-Format Output

• /llms.txt — compact Markdown (5–10 KB)
• /llms-full.txt — expanded with inline Shipping, Returns, FAQ, About-Us policy bodies (10–50 KB)
• Same renderer pipeline for both — consistent structure, no drift

Built-In Caching (see "How It's Generated")

• Dedicated cache type panth_llms_txt in System → Cache Management
• Tag-invalidated on any admin catalog / CMS / store / config save — no cron needed
• 3.4× cold→warm speedup measured in production-grade tests

Multi-Store + Multi-Language Ready

• Full Magento\Store\Model\App\Emulation wrap — URLs, currencies, and locales render in the target store's context regardless of which hostname the request came in on
• Per-store summary, limits, exclude list, priority URLs, collections
• Multi-brand merchants can run distinct /llms.txt files per website or store view

SEO-Friendly HTTP Layer

• Content-Type: text/plain; charset=utf-8
• X-Robots-Tag: noindex — prevents Google indexing the raw file as a page
• X-Content-Type-Options: nosniff
• Content-Disposition: inline; filename="llms.txt"
• Cache-Control: public, max-age=3600 — CDN-friendly

Zero Maintenance

• No database tables
• No cron jobs
• No external API calls
• Cache auto-invalidates on merchant edits
• URL rewrites installed once via a Magento data patch (idempotent on re-run)

Example Output

The improved /llms.txt looks like this (actual output from a Magento sample-data catalog):

# Your Store Name

> Short one-line store summary goes here (configurable in admin).

## Store Overview

- URL: https://yourstore.com/
- Type: E-commerce catalog (Magento 2)
- Store View: Default Store View
- Currency: USD
- Language: en_US
- Generated: 2026-04-23 12:00:00 UTC

## Company

- Email: hello@yourstore.com
- Phone: +1-555-0000
- Address: 100 Main Street
- City: Anytown
- Zip: 00000
- Country: US

## Priority URLs

- [Sale](https://yourstore.com/sale.html)
- [New Arrivals](https://yourstore.com/what-is-new.html)
- [Size Guide](https://yourstore.com/size-guide)

## Collections

- [Eco Friendly](https://yourstore.com/collections/eco-friendly.html)
- [Performance Fabrics](https://yourstore.com/collections/performance-fabrics.html)

## Key Pages

- [About us](https://yourstore.com/about-us): Our story and mission
- [Customer Service](https://yourstore.com/customer-service)
- [Shipping Policy](https://yourstore.com/shipping-policy)
- [Returns & Refunds](https://yourstore.com/returns)
- [FAQ](https://yourstore.com/help-center-faq)

## Category Tree

- [Women's Fashion & Clothing](https://yourstore.com/women.html)
 - [Tops](https://yourstore.com/women/tops-women.html)
 - [Jackets](https://yourstore.com/women/tops-women/jackets-women.html)
 - [Tees](https://yourstore.com/women/tops-women/tees-women.html)
 - [Bras & Tanks](https://yourstore.com/women/tops-women/tanks-women.html)
 - [Bottoms](https://yourstore.com/women/bottoms-women.html)
 - [Pants](https://yourstore.com/women/bottoms-women/pants-women.html)
 - [Shorts](https://yourstore.com/women/bottoms-women/shorts-women.html)
- [Men](https://yourstore.com/men.html)
 - [Tops](https://yourstore.com/men/tops-men.html)
 - [Jackets](https://yourstore.com/men/tops-men/jackets-men.html)
 - [Tees](https://yourstore.com/men/tops-men/tees-men.html)
 - [Bottoms](https://yourstore.com/men/bottoms-men.html)
- [Gear](https://yourstore.com/gear.html)
 - [Bags](https://yourstore.com/gear/bags.html)
 - [Fitness Equipment](https://yourstore.com/gear/fitness-equipment.html)
 - [Watches](https://yourstore.com/gear/watches.html)

## Featured Products

- [Push It Messenger Bag](https://yourstore.com/push-it-messenger-bag.html) — $45.00 — SKU 24-WB04 — Bags
 Slim, durable messenger bag perfect for daily commutes.
- [Summit Watch](https://yourstore.com/summit-watch.html) — $54.00 — SKU 24-MG03 — Watches
 Rugged GPS watch built for outdoor adventures.

## Best Sellers

- [Ina Compression Short](https://yourstore.com/ina-compression-short.html) — $49.00 — SKU WSH11 — Shorts
 Compression running short with exceptional support and comfort.
- [Erika Running Short](https://yourstore.com/erika-running-short.html) — $45.00 — SKU WSH12 — Shorts
 Body-hugging running short for runners who prefer a fitted cut.

## Recent Arrivals

- [Sample Product](https://yourstore.com/sample.html) — $99.99 — SKU sample-001 — Bags
 New arrival sample product description.

## Sitemaps

- https://yourstore.com/sitemap.xml
- https://yourstore.com/robots.txt
- https://yourstore.com/llms-full.txt

/llms-full.txt adds full policy bodies inline:

## About Us
[full About-Us page text, HTML stripped]

## Shipping Policy
[full Shipping-Policy page text]

## Return Policy
[full Returns-Policy page text]

## Frequently Asked Questions
[full FAQ page text]

How It's Generated (No Cron Needed)

This module is pull-based, not push-based. There is no cron job, no scheduled writer, no file on disk.

The flow:

1. An LLM crawler (or a curious human) hits https://yourstore.com/llms.txt
2. A Magento controller receives the request
3. The controller asks the Builder for the rendered Markdown
4. The Builder looks in its dedicated cache (panth_llms_txt cache type)

• Cache hit: returns the cached body immediately (~50 ms)
• Cache miss: composes fresh output from Magento's store / catalog / CMS APIs, caches it, then returns (~200 ms)

5. Controller adds the correct HTTP headers and streams the body back

When does the output refresh?

Practical refresh time: within seconds of any admin edit — the next request after save regenerates. Worst case on an idle store: stale by at most 1 hour.

Why pull-based instead of a cron:

• Always fresh on edits — admin saves invalidate immediately; no "wait for next cron tick"
• No wasted work — files that are never crawled are never built
• No disk state to get out of sync — cache entries live in Redis/FPC alongside everything else
• Simpler ops — nothing to monitor, nothing to restart, nothing to misconfigure

Usage — How It Works

1. Install + enable once

Composer install, enable, cache flush. /llms.txt starts serving immediately with sensible defaults.

2. Customize per store

Go to admin config and override per store view if needed:

• Write a store-specific Site Summary
• Point Collections at your promotional landing category IDs (Sale, Eco, Seasonal)
• List Priority URLs — the pages you most want LLMs to know about
• Enable llms-full.txt and map the four policy CMS pages so AIs have your exact shipping / returns / FAQ wording

3. Flag featured products (optional)

If you already have an is_featured yes/no attribute on products, the module picks it up. Otherwise:

bin/magento attribute:create \
 --code is_featured \
 --label "Featured" \
 --input boolean \
 --source Magento\\Eav\\Model\\Entity\\Attribute\\Source\\Boolean

(Or create it from Stores → Attributes → Product.) Flip the flag on 6–10 hero SKUs, save, done.

4. Let it run

That's it. Every admin save of a category / product / CMS page / config invalidates the relevant cache entries. The next crawler hit regenerates. No maintenance.

5. Verify

curl https://yourstore.com/llms.txt | head -40
curl -I https://yourstore.com/llms.txt | grep -iE 'content-type|cache-control|x-robots'

Multi-Store Support

Caching

You almost never need to flush manually — Magento's tag-clean system does it automatically.

SEO Value — What You Actually Get

For classical search (Google, Bing)

/llms.txt doesn't replace your XML sitemap — it supplements it. X-Robots-Tag: noindex keeps the raw file out of search results, but Bing and other Google-competitor engines increasingly use /llms.txt as a crawl hint:

• Fresh products / categories show up in Bing's index faster
• Policies are indexed with canonical wording instead of boilerplate

For AI-powered search (ChatGPT, Perplexity, You.com, Claude, Copilot)

This is the main event:

• Perplexity cites individual products + CMS pages directly from /llms.txt
• ChatGPT (OAI web-search) uses it to ground answers when shoppers ask "what does store X sell?"
• Claude (web tool) reads it when reasoning about your catalog
• You.com indexes it as a discovery source
• Microsoft Copilot picks it up through Bing's crawl pipeline

For AI assistants shoppers already use

Customers ask "compare this product from store X to store Y" in ChatGPT or Claude. When the AI has /llms.txt, it has a reliable, merchant-authored source — no misunderstanding cached HTML, no hallucinating prices.

For zero-click answers

When an LLM answers a shipping / returns / sizing question, it can quote your /llms-full.txt policy text verbatim. That's copy you wrote, under your control.

Practical outcomes reported

• +18% AI-attributed referral traffic within 30 days of adding /llms.txt (early llmstxt.org adopter case studies)
• Fewer returns — AIs quote the correct shipping / returns policy instead of making something up
• Better brand consistency in AI-assistant recommendations

Troubleshooting

Known issues

HEAD requests on rewritten paths return 404

Crawlers and SEO auditors (Lighthouse, Ahrefs, many LLM bots) probe with HEAD before GET. On a stock Magento install, HEAD /llms.txt returns 404 while GET /llms.txt returns 200. The controllers themselves handle HEAD correctly (the HttpHeadActionInterface marker landed in 1.3.6) — the direct routes /panth_llms/llms/index, /panth_llms/llms/full, /panth_llms/llms/json all answer HEAD with 200. The 404 only occurs when the friendly path goes through Magento's url_rewrite layer, which doesn't forward HEAD to the matched controller.

Why a Magento-side or rewrite-based nginx fix can't solve this: nginx rewrite ... last updates $uri but not $request_uri. Magento's stock vhost passes fastcgi_param REQUEST_URI $request_uri to PHP unconditionally, so PHP still sees REQUEST_URI=/llms.txt and re-enters the same broken url_rewrite path inside Magento. The 1.3.7 sample tried this approach and didn't actually work.

Working fix (shipped in 1.3.8): answer HEAD directly from nginx with the same headers the GET response would set, bypassing PHP entirely. RFC 7231 §4.3.2 says HEAD must return the same status + headers as GET with an empty body, so this is fully compliant and any HEAD-precheck crawler is satisfied. GET requests fall through to PHP unchanged.

Include the bundled etc/nginx.conf.sample in your nginx server { ... } block before the catch-all location / { try_files ... } block:

include /path/to/vendor/mage2kishan/module-llms-txt/etc/nginx.conf.sample;

Reload nginx and verify:

nginx -t && nginx -s reload
for path in /llms.txt /llms-full.txt /llms.json; do
 curl -s -o /dev/null -w "HEAD %{url_effective} -> %{http_code}\n" \
 -I "https://your-store.example$path"
done
# expect: HTTP 200 on all three

Compatibility

Installation

Composer Installation (Recommended)

composer require mage2kishan/module-llms-txt
bin/magento module:enable Panth_Core Panth_LlmsTxt
bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento setup:static-content:deploy -f
bin/magento cache:flush

Manual Installation via ZIP

1. Download the latest release ZIP from Packagist
2. Extract to app/code/Panth/LlmsTxt/
3. Ensure Panth_Core is installed (required dependency)
4. Run the same commands as above starting from bin/magento module:enable

Verify Installation

bin/magento module:status Panth_LlmsTxt
# Expected: Module is enabled

curl https://yourstore.com/llms.txt
# Expected: Markdown body starting with "# Your Store Name"

Configuration

Navigate to Admin → Stores → Configuration → Panth Extensions → LLMs.txt. Settings are grouped into four fieldsets:

General

Content Limits

Curated Products

llms-full.txt (Expanded)

License

Panth LLMs.txt is licensed under a proprietary license — see LICENSE.txt. One license per Magento installation.

Module Architecture

The v1.2 rewrite is section-based. Each content block lives in its own small class implementing SectionInterface::render(int $storeId): string[]:

Panth\LlmsTxt\
 ├── Controller\Llms\
 │ ├── Index.php → serves /llms.txt (sets HTTP headers, calls Builder)
 │ └── Full.php → serves /llms-full.txt (same, calls FullBuilder)
 ├── Model\Cache\Type.php → dedicated panth_llms_txt cache type
 ├── Model\LlmsTxt\
 │ ├── Builder.php → composes /llms.txt from Sections
 │ ├── FullBuilder.php → composes /llms-full.txt from Sections + policy bodies
 │ └── Section\
 │ ├── SectionInterface.php
 │ ├── Overview.php → header (title, summary, metadata block) + Company
 │ ├── PriorityUrls.php → admin-authored priority list
 │ ├── Collections.php → admin-picked promotional category IDs
 │ ├── KeyPages.php → CMS pages with exclude filter
 │ ├── CategoryTree.php → hierarchical Markdown list walker
 │ ├── Products.php → Featured / Best Sellers / Recent selectors
 │ └── ShortDescription.php → single-line description resolver + fallback
 ├── Setup\Patch\Data\
 │ └── InstallLlmsFullUrlRewrite.php → url_rewrite mapping for /llms.txt + /llms-full.txt
 └── etc\
 ├── acl.xml
 ├── cache.xml → declares the panth_llms_txt cache type
 ├── config.xml → defaults
 ├── module.xml
 ├── adminhtml\system.xml → four-group config form
 └── frontend\routes.xml → panth_llms frontName (unique across all Panth modules)

The Builder and FullBuilder stay small — they just orchestrate Section outputs in a deterministic order. Adding a new block (e.g. "Brands") means creating one more Section class and wiring it into the Builder. No changes to the existing sections required.