Robots.txt is blocking Google: test it before the next deploy
A single Disallow can hide the page, its assets, or the sitemap. Check the live file, the exact user-agent group, and what the page needs to render.
Robots.txt is a public instruction, not a security layer
The file tells crawlers which paths they may request. It does not hide content: anyone can read it, and a blocked page can still be linked or indexed from elsewhere. Use noindex for pages that should not appear in search, and authentication for private data.
A staging rule copied into production is the most common accident. Read the file served on the live host, not the one in the repository.
curl -sS https://example.com/robots.txtCheck the group that applies
Rules are matched by user-agent group. A Disallow under one bot does not necessarily apply to Googlebot, and a wildcard group can override what you expected. Test the exact path against the live file.
- Disallow: / blocks the entire site.
- A trailing wildcard such as /*? can block parameter URLs.
- Blocking /_next/ or static assets can prevent Google from rendering the page.
- Blocking the sitemap path can stop sitemap discovery.
- A 404 robots.txt is treated as allow; a 5xx can pause crawling.
How to verify without guessing
Fetch the URL with a plain client and confirm the response. Then check the page in Search Console’s URL Inspection: the rendered HTML shows whether Google could load the page and the resources it needs.
If a page is blocked, remove the rule that matches it, deploy, and fetch robots.txt again. Then inspect the page and request indexing once the path is open.
| Symptom | Likely rule | Fix |
|---|---|---|
| Whole site missing | Disallow: / | Remove the blanket rule in production |
| Pages load without styles | Assets blocked | Allow CSS and JS paths |
| Sitemap not discovered | Sitemap path disallowed | Allow the XML path |
| Only parameter pages missing | Wildcard query rule | Narrow the rule to the parameters you intend to block |
After you unblock
Unblocking does not push the page into the index. Confirm the page is crawlable, not noindexed, and linked from somewhere useful, then request indexing for the most important URLs.
Keep robots.txt small and reviewed. Every rule should have a reason and a path it applies to; delete staging leftovers before a launch.
Frequently asked questions
Can I block a page from search with robots.txt? You can block crawling, but the URL may still be indexed from links. Use noindex if you want it out of results.
Does Google ignore robots.txt if it cannot fetch it? A 5xx response can pause crawling, which is worse than a 404. Keep the file reachable.
I removed the rule but Google still does not index the page. Crawl access was only one blocker; check noindex, canonical, content and internal links next.
See whether the page and its assets are reachable
Run a free audit to catch blocked fetches and noindex directives before they cost you a launch.
Run a free audit