Discoverability · 30 points

Indexable

One check, stated three ways: what we measure, what would make it wrong, and how to confirm it yourself.

What this measures

The meta robots tag and the X-Robots-Tag response header, for noindex, none and nofollow.

Where this check can be wrong

A staging hostname can carry noindex for a while and we will read the production URL's own headers, so a shared IP or a CDN rule may be reporting on the wrong environment.

Verify it without taking our word for it

curl -sI https://your.site | grep -i x-robots-tag, plus view-source and search for name="robots".

Related