What happened
A classifieds operator, Javier Lorente Murillo, put a sharp question to Google on LinkedIn on July 27. His platform holds about 5,000 stable indexed URLs but mints roughly 10,000 new listings every month, and each listing lives 24 to 72 hours before it expires. He wanted to use the unavailable_after directive — the meta tag that tells Google to stop serving a URL after a set date — and push the date forward whenever a listing gets renewed.
Gary Illyes answered. Pushing the date forward is probably fine, he said, but Google only registers the new date when it re-fetches the page in full — "we rarely see HEAD requests so we'd still download the content." No re-crawl, no update. Google keeps whatever date it read last time.
Illyes also drew the boundaries tight. The directive controls serving eligibility and nothing else: it doesn't change your HTTP status codes, doesn't reduce crawl cost, and doesn't touch 404 handling. The operator's real pain — more than half of Googlebot's requests on his site return 404s, and his error template ranks among his ten most-viewed pages — is a problem the tag was never built to solve.
Why this matters
Run the math on a 24–72 hour listing against Google's re-crawl cadence and the tag falls apart. Unless Googlebot happens to return inside that window, the expiry date sitting in Google's index is fiction for the entire life of the offer. Google might drop a live listing on a stale date, or keep a dead one eligible long after it's gone.
Meta directives feel precise because you set them to the second. This one is only precise at crawl time — and you don't control crawl time. That gap should stop you before you wire expiry tags into flash sales, event pages, or any catalog where inventory turns fast.
The sturdier patterns are boring on purpose. Gone for good? Serve a real 404 or 410 and let the URL die honestly. Recurring promotion? Keep one permanent URL and rotate the offer on it. The page never expires, only the deal does, and the URL's crawl history and links keep compounding instead of resetting with every new listing.
What to do about it
Check your 404 share in Crawl Stats
Open Search Console → Settings → Crawl stats and look at the response-code breakdown. The operator in this story found over half of Googlebot's requests hitting 404s. If your fast-turning catalog shows anything close to that, your churn architecture — not a meta tag — is the project.
Give recurring sales a permanent URL
Stop minting a new URL for every flash sale or seasonal event. One stable /flash-sale/ or /black-friday/ page that swaps offers keeps its rankings, links, and crawl history year over year. On BigCommerce that means updating the page content and leaving the path alone.
Serve 410 for products that won't return
A discontinued product with no successor should return a 410 or 404, with internal links to it removed. Don't paper over it with an expiry tag — Illyes just confirmed the tag doesn't govern 404 handling or crawl behavior at all.
Verify re-crawl timing before trusting the tag
If you already use unavailable_after, check your server logs for Googlebot's last full fetch of those URLs. If the gap between fetches runs longer than your offer lifespan, the dates Google holds are stale by definition.