What happened
A question reached Google's John Mueller about a Shopify store whose internal search pages kept landing in the index even though /search was clearly disallowed in robots.txt. Mueller's answer: precedence. Robots.txt is specific-beats-general. If your file contains a user-agent: Googlebot section, Googlebot reads only that section. Everything under user-agent: * — including the disallow that should have stopped the problem — gets ignored entirely.
The failure mode was ugly: search-box spam pages getting crawled and indexed because the one rule meant to stop it lived in the block Google never reads. Nothing was misspelled. The syntax was fine. The file simply had two sections, and Google honors exactly one of them.
Mueller's fix options are straightforward. List the user-agents together above one shared rule set. Or repeat every rule inside the Googlebot section so nothing depends on the general block. Or delete the bot-specific section if you don't actually need it. And for internal search pages specifically, he pointed to noindex — robots.txt was never the right tool for keeping already-discovered URLs out of the index.
Why this matters
Almost nobody writes a Googlebot section on purpose. They inherit one — from an old developer, an SEO plugin, an app that appended its own rules, or a platform default that predates the current team. The file grows by accretion, each addition looks harmless, and the day someone adds a bot-specific block, every general rule silently stops applying to the one crawler that matters most.
The nastier part is that this failure has no error state. Search Console won't flag it. The file validates. A human reading top to bottom sees the disallow and assumes coverage. You find out when the symptoms surface — indexed search results, crawled faceted URLs, parameter junk in site: queries — and by then you're diagnosing from the wrong end.
There's a second lesson buried here: disallow doesn't deindex. Blocking a URL in robots.txt stops crawling, not indexing — Google can index a blocked URL from links alone. For internal search results, faceted pages, and anything else you want out of the index, noindex is the mechanism, and the page has to stay crawlable for Google to see the tag. Stores mix these two up constantly, and this case is both mistakes living in one file.
What to do about it
Open your robots.txt right now
Load yourdomain.com/robots.txt and scan for any user-agent line naming a specific bot — Googlebot, Googlebot-Image, Bingbot, anything. If a Googlebot section exists, every rule you care about must appear inside it, or the file needs restructuring so shared rules sit under a combined user-agent list.
Verify with Search Console, not your eyes
Don't trust a visual read. Use the robots.txt report in Search Console, then run your problem URLs — an internal search URL, a faceted URL — through URL Inspection and confirm what Googlebot is actually allowed to fetch. The tool resolves precedence for you, which is exactly the thing human eyes get wrong.
Move index control to noindex where it belongs
For internal search results, put a noindex robots meta tag on the search template and drop the robots.txt disallow so Google can crawl in and see it. On BigCommerce that's a one-time edit to the search template in your theme — do it once and this entire class of spam stops mattering.
Re-check after every app install
Apps and plugins that edit robots.txt are the usual source of surprise bot-specific sections. Add a robots.txt diff to your post-install checklist, and keep a dated copy of the known-good file so any change stands out immediately.