About Fetchproof's scanner

Every request carries this user agent:

Robots.txt tokens it obeys: FetchproofBot (current) and AgentReadyBot (former name).

• robots.txt first. If it asks our scanner not to fetch the homepage, we stop there.
• The homepage, once, as plain HTML. The scanner never runs JavaScript, never logs in and never submits forms.
• /llms.txt, if the site publishes one.

• Each check makes a handful of requests, with short timeouts and at most two retries. Rate limits (HTTP 429) and Retry-After are respected.
• Pages larger than 5 MB are not read.
• Requests come from cloud servers and identify themselves with the user agent shown above.

• Our scanner obeys rules written for FetchproofBot, and still obeys rules written for its former name, AgentReadyBot. If a site names either, the stricter of those rules applies; otherwise the site-wide (*) rules apply.
• Rules take effect on the next check. We do not cache robots.txt.

To block it, add:

A site owner who wants Fetchproof to check a site that blocks unrecognised bots can verify ownership with a DNS TXT record at _agentready-verify.<your domain>. Verification only lets the owner check their own site; reports still say what robots.txt asks.

Questions or concerns: support@vaqeno.com.