What is Magento IndexNow ?
Magento IndexNow wires the IndexNow protocol (launched 2021 by Microsoft Bing + Yandex) into Magento so the storefront pushes URL changes directly to participating search engines instead of waiting on organic crawl. Via the mage2kishan/module-indexnow extension, Magento auto-pings api.indexnow.org on cms_page_save, product_save, and category_save events — cutting Bing time-to-index from 4 – 14 days to hours. Honoured by Bing, Yandex, Seznam, Yep, and Naver. Not Google.
Five steps from key generation to live Bing pings
IndexNow is not a black-box service — it’s a documented HTTP protocol with one verification file, one POST endpoint, and a small payload. Here is the Magento wiring, end to end.
-
01
Generate the IndexNow API key
IndexNow keys are 32 – 128 character alphanumeric strings used to prove site ownership when pinging the endpoint. The
mage2kishan/module-indexnowmodule generates one automatically on first install and stores it in admin config underpanth_indexnow/indexnow/api_key. Hand-rolled implementations typically use a UUID-style generator. The key is shared across all pings from the site — you can rotate it any time from admin, but you must also re-publish the verification file (see step 02) or every subsequent ping returns422 invalid key. -
02
Publish the key verification file at site root
IndexNow requires a plain-text verification file served at
https://yoursite.com/<key>.txtwithContent-Type: text/plainand a body containing exactly the same key. This proves the pinging entity controls the domain. The Magento module exposes the file via a frontend controller route — typicallypanth_indexnow/key/<key>— which means no static file uploads are needed and key rotation is fully self-service. Verify withcurl https://yoursite.com/<key>.txtafter every rotation. -
03
Configure trigger events and partner endpoints in admin
In Magento admin go to
Stores → Configuration → Panth → IndexNow. Enable or disable per event type —cms_page_save,catalog_product_save,catalog_category_save— pick which partner endpoints to ping (api.indexnow.org,www.bing.com/indexnow,yandex.com/indexnow, or all three), and toggle auto-ping. Most stores enable all three event types and post toapi.indexnow.orgonly — that endpoint fans out to every partner search engine automatically. -
04
URL change event fires — queue then POST the batch
When a CMS page, product, or category is saved, the module’s observer (on
cms_page_save_afterand siblings) builds the canonical URL of the changed entity and queues it inpanth_indexnow_queue. A cron job (panth_indexnow_dispatch, default every 5 minutes) batches the queue and POSTs tohttps://api.indexnow.org/indexnowwith JSON{host, key, keyLocation, urlList: [...]}— up to 10,000 URLs per request. Cron batching is critical: hand-rolled single-URL pings on every save trigger IndexNow’s rate limit on bulk imports. -
05
Bing and Yandex fetch, validate, and index
The receiving endpoint returns
200(accepted, will crawl),202(received but key validation pending — rare), or422(invalid key, host mismatch, or malformed payload). On200, Bing typically schedules a crawl of the listed URLs within minutes for established sites (hours for newer domains), vs days to weeks for purely organic discovery. The Magento module logs every response code tovar/log/panth_indexnow.log— tail this if pings stop reaching Bing Webmaster Tools’ Submitted URLs dashboard.
Four scenarios where IndexNow is the obvious answer
IndexNow is free and the setup is one afternoon. These four scenarios are where it pays for itself in the first week.
-
High-velocity content sites — blog, news, glossary
If you ship multiple new URLs a day — daily blog posts, news items, glossary entries (the Tier-5 wave on this site is a textbook example) — time-to-index is a direct lever on traffic. Organic Bing/Yandex discovery for a brand-new URL averages 4 – 14 days; IndexNow brings that to hours for established sites. On a 200-post-per-year blog that compounds into thousands of indexed-URL-days you would otherwise be waiting on. Enable the
cms_page_savetrigger plus a CLI batch job for archive backfill. -
E-commerce price/stock changes you want crawled fast
When a SKU comes back in stock, a price drops below a competitor, or a sale-only product goes live for 48 hours, Bing’s organic re-crawl interval is too slow — the URL might not re-crawl until after the sale ends. Pinging IndexNow on
catalog_product_save_afterpushes the change to Bing within minutes so the new price/stock signal hits Bing Shopping and the SERP before the sale window closes. Especially useful for Black Friday / flash-sale stores. -
Site migrations and bulk URL changes
After a domain change, URL-structure refactor, or category-tree restructure, you typically have hundreds or thousands of new canonical URLs that need re-crawling. Sitemap-and-wait works for Google but Bing is dramatically slower to pick up sitemap deltas. Run
bin/magento panth:seo:indexnow:ping --all-urls(or the equivalent module CLI) to push the entire new URL set in a single batch — Bing typically reflects the new URL structure inside 24 – 48 hours instead of 2 – 4 weeks. -
Brand-mention indexing for new product / service pages
When a new product page ships or a new service landing page goes live tied to a PR push, paid-media campaign, or social burst, the URL needs to be indexable the moment the campaign starts — not 5 days later when organic discovery catches up. Ping IndexNow as part of the launch checklist (alongside GSC URL inspection for Google). Bing is increasingly important for AI-powered search (Bing Chat / Copilot pull from Bing’s index), so fast Bing indexing also feeds the AI-citation funnel.
Three IndexNow mistakes that get your key rate-limited
Every IndexNow audit I’ve been called in to fix came from one of these three mistakes. Check yours before going live.
-
Pinging URLs that 404 or redirect
IndexNow’s spec says submitted URLs must return
200 OKwithin hours of the ping — pinging a 404, 410, or 301-chained URL gets your key flagged and eventually rate-limited or down-ranked. The Magento module checks each queued URL is live before posting; hand-rolled scripts usually skip the pre-flight check. If you’re bulk-pinging an archive, run a quickcurl -Isweep first and drop any URL not returning200. -
Hammering the endpoint on bulk imports
IndexNow rate-limits unverified keys at roughly
10,000URLs per day and rejects bursts — if you trigger a 50,000-product reimport and the observer fires one ping per save, the first ~10k pings succeed and the rest get429d or silently dropped. Always use the batch endpoint withurlList: [...](up to 10,000 URLs per single request) and queue them with cron-deferred batching, not per-save. The module does this by default; custom observers often do not. -
Rotating the API key but forgetting the verification file
If you regenerate the IndexNow key in admin but the old
<oldkey>.txtis the only file Bing can fetch, every new ping returns422 key mismatchwithin an hour. The module auto-emits the new key file via its controller route, so admin rotation Just Works — but if you’ve hard-coded the verification file as a staticpub/upload (common in tutorials), you must re-upload it after every rotation. Verify withcurl https://yoursite.com/<newkey>.txtimmediately after rotating.
Magento IndexNow — frequently asked questions
-
Does Google support IndexNow?
No — not in production as of 2026. Google announced in November 2021 that they would test the IndexNow protocol, then announced again in 2024 that testing was still ongoing, but as of this writing there is no Google IndexNow endpoint and Google does not honour pings to api.indexnow.org for crawl-priority. For Google, use the standard channels: an up-to-date XML sitemap submitted in Google Search Console, the URL Inspection tool for one-off urgent submissions, and (where available) the Indexing API for job-posting and live-stream URL types. IndexNow remains the right tool for Bing, Yandex, Seznam, Yep, and Naver — just not Google. -
Is IndexNow free? Is there a quota?
Yes, completely free — no cost, no paid tier, no API-key fees. The only quota is the rate limit, which sits at roughly 10,000 URLs per day per unverified key for the public endpoint. Verified sites (those with a Bing Webmaster Tools account linked to the same domain as the IndexNow key) get a higher effective ceiling. Bursts are rate-limited regardless of verification status — if you push 20,000 URLs in a single minute you will see 429 responses. That is why cron-deferred batching is the recommended pattern: queue the pings, batch them up to 10,000 per request, and POST one batch per cron tick. -
How fast does Bing actually crawl post-ping?
For established sites with healthy crawl budget, the typical IndexNow-to-crawl latency is hours, sometimes minutes for top-priority URLs (recent content from a domain Bing trusts). New domains and low-authority sites can wait 12 – 48 hours even with IndexNow, because the protocol guarantees crawl scheduling, not crawl-rate priority — Bing still applies its normal crawl-budget heuristics. The headline win is the comparison against organic discovery: without IndexNow, a brand-new URL waits a median of 4 – 14 days for Bing to find it via sitemap re-fetch or link-following. With IndexNow, the median drops to under 24 hours. -
What happens if I send a URL Bing already crawled recently?
No harm done — Bing returns 200 (accepted) and schedules the URL for re-evaluation. Whether it actually re-crawls depends on its own freshness heuristics: a CMS page that hasn’t changed since the last crawl might be skipped, while a product page with a price change in the meta will be re-fetched. There is no penalty for over-pinging a URL within reason — the rate limit is a per-key, per-day ceiling on the count of submitted URLs, not on duplicates. The mage2kishan module deduplicates within a 24-hour window by default to avoid wasting quota on no-op re-pings, but you can override that in admin. -
Can I share one IndexNow key across multiple sites?
Yes — the protocol explicitly supports this. The key is a credential proving you control <em>at least one</em> host. The verification file must be served at every host you ping for, and each ping sets the <code>host</code> parameter explicitly so Bing knows which site the URLs belong to. This is convenient for agencies and multi-store Magento setups where one key serves five or six storefronts — rotate the key in one admin, re-publish the verification file via the controller route on every storefront, and you’re done. The mage2kishan module supports multi-store setups out of the box and writes the same key + per-store host into each ping. -
Does the mage2kishan/module-indexnow module handle key rotation cleanly?
Yes. Rotating the key from admin (<code>Stores → Configuration → Panth → IndexNow → Regenerate Key</code>) does three things atomically: it updates the stored key in <code>panth_indexnow/indexnow/api_key</code>, it invalidates the old verification file served via the controller route (now 404), and it begins serving the new verification file at <code>/<newkey>.txt</code>. Subsequent pings include the new key in the JSON body and the new <code>keyLocation</code> URL, so Bing re-validates within minutes. Old pings that were already queued at rotation time are re-keyed before being POSTed to avoid 422 mismatches. Verify with <code>curl https://yoursite.com/<newkey>.txt</code> immediately after rotation and check <code>var/log/panth_indexnow.log</code> for the first post-rotation ping response code — you want a clean <code>200</code>.
Want IndexNow installed on your Magento store?
Send your storefront URL — I will install mage2kishan/module-indexnow, generate and verify the key, wire the cron-batched queue, and confirm the first ping shows up in Bing Webmaster Tools. Fixed-price, typically shipped inside 48 business hours.