What happened
Google significantly expanded its help documentation for merchants whose product pages are blocked from StoreBot or Googlebot, as Search Engine Roundtable reported on July 17. The updated doc clarifies which crawler user-agents you need to allow, walks through checking for robots.txt blocks, and covers the sneakier case: IP-based blocks that stop Google's crawlers even when robots.txt is clean.
One number in the doc does a lot of work: once you fix a block, pages take roughly 12 to 48 hours to reprocess. If your Merchant Center listings or local-inventory ads depend on Google reaching your product pages — and they all do — this doc is now the reference for diagnosing why they can't.
Why this matters
Crawl blocks are a silent failure mode. Your store looks perfect in a browser, orders keep coming from other channels, and meanwhile Shopping listings disapprove in batches because StoreBot can't fetch the landing pages. The usual culprits aren't in robots.txt at all — they're bot-management rules at the CDN or firewall layer, a security plugin that got aggressive, or staging-era blocks that shipped to production. That's exactly why Google spelling out the IP-block case matters: it's the block most merchants never think to check.
The 12–48 hour reprocess window is the practical gold here. The classic failure pattern goes like this: a merchant fixes the block correctly, checks an hour later, sees no recovery, concludes the fix was wrong, and starts changing other things — creating a second problem on top of the solved one. Knowing the window exists is the difference between waiting calmly and thrashing.
Be clear-eyed about what this isn't: nothing changed in how Google crawls. It's a documentation update. The value is having one canonical checklist instead of piecing the answer together from forum threads mid-outage.
What to do about it
Test crawler access now, not during an outage
Pull up your robots.txt and confirm nothing disallows Google's shopping and search crawlers from product URLs. Then check the layer robots.txt can't show you: your CDN or firewall's bot-management rules. Cloudflare and similar WAFs can block by IP or bot score, and those blocks are invisible until listings start dying.
Bookmark the doc in your incident runbook
When Merchant Center disapprovals point at page availability or crawl problems, this doc is now step one. Put the link where whoever handles your feed will actually find it at 9pm on a Friday, next to your Merchant Center and hosting logins.
After any fix, wait the full window before changing anything else
Log the exact time you removed the block, then give it the 12-48 hours Google states before judging the result. Stacking additional changes inside that window makes it impossible to know what worked — and usually breaks something that wasn't broken.
Re-check access after every CDN or security change
New WAF rules, a security plugin update, a hosting migration — these are the moments crawl blocks get introduced, not random Tuesdays. Add a crawler-access check to the sign-off list for any change touching your CDN, firewall, or robots.txt, and you'll catch the block the day it ships instead of the week your listings disapprove.