When a crawler cannot fetch your site: Cloudflare, WAFs, and false 403s
How to tell a real application error from a bot challenge, rate limit, DNS issue, or firewall rule—and what to change without turning security off.
Look at the response, not just the status
A 403 with a Cloudflare or WAF signature, a browser challenge, or a rate-limit header is a different problem from your app returning an authorization error. Record the final URL, response headers, content type, and a short body preview when diagnosing it.
Keep the security boundary narrow
Allow ordinary GET requests for public HTML, robots.txt, sitemap.xml, and assets. Keep admin, mutation, and API routes protected. If a provider offers verified bot management, prefer an explicit rule for legitimate search crawlers over disabling the WAF globally.
- Check DNS and TLS before changing firewall rules.
- Review recent WAF events and rate limits for the exact path.
- Test from a clean user agent and a normal browser.
- Re-test robots.txt and sitemap.xml after every CDN change.
Ready for a pre-launch audit?
Run the public analyzer and get a prioritised report for the URL you are about to share.
Run the SEO audit