HTTP bot accessibility

Why it matters

HTTP probes with agent user-agent strings show how the sampled page responds from the scanner. They do not establish access from provider IPs or successful retrieval of the expected answer.

How to fix it

Return HTTP 200 to legitimate bots and do not block them in the WAF or CDN.

1. Request the home page with a bot user agent and record the HTTP status.

2. When the status is 401 or 403, add a WAF or CDN rule that allows known AI user agents.

   map $http_user_agent $is_ai_bot {
     default       0;
     "~*GPTBot"    1;
     "~*ClaudeBot" 1;
   }

3. Keep rate limits, but do not answer verified AI crawlers with a challenge page.

Verify

Run a new scan for this checkpoint: