robots.txt and sitemap debugging: the trailing-slash 404 trap

The three URLs a crawler can disagree about

A route may appear in a sitemap without a trailing slash, redirect to a slash version, and have a canonical tag pointing somewhere else. Humans see one page; crawlers see three URL signals. Choose one normalized form and use it in links, canonicals, sitemap loc entries, and redirects.

curl -I https://example.com/pricing
curl -I https://example.com/pricing/
curl -s https://example.com/sitemap.xml

Inspect robots.txt as a response

A dashboard saying robots is configured does not prove that the file is reachable. Request it directly and verify a 200 response, text/plain content, and the sitemap URL. A CDN challenge, HTML error page, or a blanket Disallow can prevent discovery even when the file exists in your repository.

• 200 response, not a soft 404 or a challenge page.
• No Disallow: / unless you intentionally want a private site.
• Sitemap URL uses the final HTTPS origin.
• Rules for staging hosts do not leak into production.

Fix the sitemap at the source

Every loc should be absolute, public, and reachable. Remove login, dashboard, API, and parameter-only URLs. If a route intentionally redirects, place the final URL in the sitemap instead. Validate XML syntax and sample several URLs with HEAD or GET requests before submitting the file.