Discoverability · 30 points
robots.txt
One check, stated three ways: what we measure, what would make it wrong, and how to confirm it yourself.
What this measures
Whether robots.txt is served and readable, whether it declares a Sitemap, and whether a declared sitemap can actually be read.
Where this check can be wrong
No robots.txt is a legitimate configuration, not a fault — it simply means “everything allowed”. A file that returns 404 through a CDN rule looks missing from here even when your origin has one.
Verify it without taking our word for it
curl -s https://your.site/robots.txt -i and look at the status line, not the file in your repository.