AI visibility · 15 points
Answer-engine access
One check, stated three ways: what we measure, what would make it wrong, and how to confirm it yourself.
What this measures
robots.txt parsed into User-agent groups, then tested for five retrieval agents — OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-WebBot, anthropic-ai — against the audited page's path. Longest matching rule wins, Allow beats Disallow on a tie, an empty Disallow blocks nothing, and an agent with no group of its own falls back to the * group.
Where this check can be wrong
Robots.txt is a request, not a wall: a crawler can ignore it, and a CDN or WAF rule can block the same agent regardless of what the file says. We read the file you control, so we cannot see the layer above it.
Verify it without taking our word for it
curl -s https://your.site/robots.txt, find the group naming each agent, and confirm with curl -A "OAI-SearchBot" https://your.site -I that nothing else in the path returns 403.