Meet the RobotsGuard Bot
A lightweight, respectful crawler that checks your pages for indexing and crawlability signals, nothing more.
What the RobotsGuard Bot does
Our bot visits only the specific URLs you've configured for monitoring. Here's exactly what it checks:
Reads robots.txt
Fetches your robots.txt to verify which bots are allowed or blocked. Checks for syntax errors and conflicting rules.
Checks meta robots tags
Inspects the HTML <head> of your monitored pages for noindex, nofollow, and other directives.
Reads X-Robots-Tag headers
Checks HTTP response headers for server-level indexing directives that can override page-level settings.
Validates canonical tags
Verifies that canonical URLs are correct, self-referencing where expected, and not pointing to wrong domains.
What our bot does NOT do
Bot identity & behavior
Our bot identifies itself transparently so you can always recognize it in your logs.
RobotsGuard-Pro/2.0 (+https://robotsguard.com/bot)Published at robotsguard.com/bot/ips.jsonRequests to a domain are spaced seconds to tens of seconds apart, never sent in bursts.RobotsGuard-ProRespectful pacing
Our bot only visits URLs a customer explicitly added for monitoring. Requests to the same domain are deliberately spaced out, a few seconds to tens of seconds apart, depending on the platform we detect, so we never put load on your server.
No disguises
We never send requests under any other User-Agent, and we never pretend to be a web browser. Every single request carries the string above.
Why our bot might get blocked
If RobotsGuard can't reach your pages, here are the most common causes and how to fix them.
Cloudflare Bot Protection
Most commonCloudflare's Bot Fight Mode or Super Bot Fight Mode may classify our bot as automated traffic and block it with a challenge or 403 error.
How to fix:
Web Application Firewalls (WAF)
CommonWAFs like Sucuri, Wordfence, AWS WAF, or Akamai may block automated requests that don't look like regular browser traffic.
How to fix:
Rate Limiting
OccasionalIf you have aggressive rate limiting configured, our monitoring requests might get throttled or blocked, especially on the first scan.
How to fix:
robots.txt blocking
Ironic but commonYour robots.txt might have a blanket Disallow rule that also blocks our bot. We respect robots.txt. If we're blocked, we can't monitor.
How to fix:
Server-level IP blocking
Less commonSome hosting providers or .htaccess rules block requests from data center IPs or specific geographic regions.
How to fix:
Geographic restrictions / Geo-blocking
RareIf your site restricts access by country, our monitoring servers may be blocked if they're outside your allowed regions.
How to fix:
Whitelisting guide
Step-by-step instructions for the most common platforms.
How to stop us monitoring a URL
We don't crawl the web looking for pages. The only URLs we ever check are ones a customer deliberately added to their own RobotsGuard account.
We check those URLs the way a major search engine would, so the customer can see what search engines actually see. Fetching the page is the point of the check, including pages meant to be disallowed, since that's the only way to tell a customer whether their disallow rule is actually working. robots.txt rules on your site are evaluated and reported back to the customer, but they don't stop us from checking a URL they've already configured.
You added the URL
Remove it from your RobotsGuard account. That stops us checking it immediately.
Someone else is monitoring your site
If you'd rather no monitoring tool touched your site at all, block the User-Agent above at your firewall or WAF. We honour that block rather than retry around it.
Still getting blocked?
Questions about our bot's behaviour on your site? Email us at [email protected] and we'll check your configuration and find the exact cause.