AI crawler monitoring identifies each request, verifies it where the vendor provides a method, maps it to the vendor's documented purpose, and compares it with your access policy. Do not collapse training crawlers, search indexers, and user-triggered fetchers into one "AI bot" bucket. Names and IP ranges change, so vendor documentation—not a copied list—is the source of truth.
This guide is the practical playbook for monitoring AI crawler traffic. How to identify each bot, how to verify it is the bot it claims to be, how to read its behavior, what to allow versus block in robots.txt, and how to alert when an AI crawler is the actual cause of your next 5xx spike.
This is the inbound side of AI. For the outbound side - whether your brand appears in AI answers - see AI Search Visibility Monitoring.
Why AI Crawler Monitoring Is Suddenly A Topic
Three things changed at once:
- AI search products started citing live web content. ChatGPT Search, Perplexity, Google AI Overviews, Bing Copilot — all fetch pages in real time to ground answers. If their crawler cannot reach you, you do not get cited.
- AI labs started crawling to train models. Separate user agents, sometimes overlapping IP ranges, often more aggressive than search crawlers, and not always declared.
- Volume scaled. Real-time AI search traffic plus training crawls plus agent-style "browse the web" fetches now adds up to a meaningful share of bot requests, and shows up as latency spikes, 429s, and 5xx storms on under-provisioned sites.
If your site logs do not separate AI crawlers from human traffic, you are flying blind on three problems at once: SEO/AEO visibility, infrastructure load, and content licensing.
For broader robots.txt and sitemap regressions that block crawlers entirely, see Sitemap & robots.txt Monitoring.
The AI Crawler Roster (As Of 2026)
User agents change. Always verify against the vendor's docs before alerting on a specific string. Categories below reflect the declared intent in each vendor's documentation - not a guarantee.
OpenAI
| User agent | Purpose | Documented? |
|---|---|---|
GPTBot |
Crawls content that may be used to train foundation models | Yes |
OAI-SearchBot |
Links and surfaces websites in ChatGPT Search | Yes |
ChatGPT-User |
On-demand user-initiated fetches | Yes |
OpenAI publishes separate IP-range JSON for these agents. Treat them separately: OAI-SearchBot and GPTBot have independent robots.txt settings, while OpenAI says robots.txt may not apply to user-initiated ChatGPT-User requests. See OpenAI's crawler documentation.
Anthropic
Anthropic's official crawler policy is the source of truth for these roles:
| User agent | Purpose | Documented? |
|---|---|---|
ClaudeBot |
Model training | Yes |
Claude-User |
User-directed retrieval | Yes |
Claude-SearchBot |
Search indexing and result quality | Yes |
Perplexity
| User agent | Purpose | Documented? |
|---|---|---|
PerplexityBot |
Indexing for Perplexity Search | Yes |
Perplexity-User |
On-demand user query fetches | Yes |
Perplexity documents both crawlers and current IP JSON endpoints. Monitor them separately and record observed policy mismatches, but do not present historical allegations as the crawler's documented behavior.
| User agent | Purpose |
|---|---|
Googlebot |
Classic search index |
Google-Extended |
Robots.txt product token for Gemini training and grounding in some non-Search systems; not a request user agent |
GoogleOther |
Misc product crawls |
Google-Extended is a robots.txt product token, not a request user agent. Google says it controls training and grounding in some non-Search systems; it does not control inclusion in AI Overviews or AI Mode, which use Googlebot and standard Search preview controls.
Microsoft
| User agent | Purpose |
|---|---|
Bingbot |
Bing search index |
Bing-Copilot (varies) |
Live Copilot grounding fetches |
Meta
| User agent | Purpose |
|---|---|
Meta-ExternalAgent |
Meta AI training |
Meta-ExternalFetcher |
On-demand fetches |
FacebookExternalHit |
Link previews (not AI but commonly mis-classified) |
Apple
| User agent | Purpose |
|---|---|
Applebot |
Spotlight / Siri search |
Applebot-Extended |
Opt-out flag for Apple AI training |
Amazon
| User agent | Purpose |
|---|---|
Amazonbot |
Alexa answers and Amazon AI training |
Others worth watching
CCBot— Common Crawl, the upstream dataset many models train on.Bytespider— ByteDance / TikTok. Aggressive, not always well-behaved on robots.txt.Diffbot— knowledge-graph extractor used downstream by AI products.cohere-ai,Cohere-AI— Cohere training.YouBot— You.com search.Mistral-Crawl— Mistral training (sporadic).
Maintain this list in code, not in your head. Bots change names. New ones appear monthly.
Verify Before You Trust The User Agent
User-agent strings are trivially spoofable. A scraper claiming to be GPTBot may be a competitor scraping your prices. Verify with one of:
1. Vendor-documented DNS verification
Use reverse-then-forward DNS only for vendors that explicitly document valid hostname suffixes and this verification procedure. For example, Google documents it for Googlebot:
# 1. Reverse DNS the source IP
host 66.249.66.1
# -> a documented Google crawler hostname
# 2. Forward DNS the result back to an IP
host the-ptr-record-you-got.googlebot.com
# 3. Compare to the original IP — must match
For a vendor that documents this method, a failed forward-confirmed reverse-DNS check means the request is not verified as that vendor's crawler.
2. Published IP ranges
Some vendors publish IP ranges or DNS-verification instructions. Pull only documented endpoints, cache the last valid set, validate schema and freshness, and keep user-agent-only requests labelled unverified. Anthropic advises that IP blocking is not a durable opt-out because its bots need to read robots.txt.
3. HTTPS verifier endpoints
Some providers offer signed verifier endpoints. Useful for higher-trust signals when blocking is a destructive action.
Anything that does not pass at least one verification path is "claimed bot, unverified" - log it, do not trust it, and watch for misbehavior.
For the broader pattern of distinguishing real traffic from automation, see DDoS Detection & Traffic Spike Monitoring.
What To Log Per Request
Add these fields to your access logs (or analytics pipeline) for every request:
ua_string- full user agentbot_family-gptbot,claudebot,perplexitybot,googlebot,unknown, etc.bot_purpose-train,search,user-fetch,index,unknownbot_identity_status-verified,unverified, ormismatch, with the vendor-specific evidence methodsrc_ippath,status,bytes,response_time_msrefereraccept_languagerobots_decision- what robots.txt would have said for this UA + path
The bot_verified flag is the single most important field. Aggregate everything else against it.
robots.txt: Opt-In vs Opt-Out
robots.txt is the contract layer. AI bots that respect it will honor your rules; the ones that ignore it are exactly the ones you want to detect and alert on.
A pragmatic default for most sites:
# Allow live citations and search indexing
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Disallow model training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: cohere-ai
Disallow: /
Sitemap: https://example.com/sitemap.xml
This template:
- Allows the documented automatic search crawler and robots-aware fetchers shown.
ChatGPT-Useris omitted because OpenAI says it is user-initiated, does not govern Search inclusion, and robots.txt may not apply. - Blocks the training/product tokens explicitly listed;
PerplexityBotremains allowed because Perplexity documents it as a search-results crawler, not a foundation-model training crawler. - Leaves classic search (Googlebot, Bingbot) untouched.
Adjust to your strategy. The point is: be deliberate, version the file, and monitor for unintended regressions.
For monitoring robots.txt itself - regressions, missing files, unintended Disallow: / after a deploy - see Sitemap & robots.txt Monitoring.
Monitoring Patterns
Volume by bot family
Daily and hourly counts per bot_family, segmented by bot_purpose. New spikes worth alerting on:
- A bot family appearing for the first time.
- An order-of-magnitude jump in request count.
- A new path becoming the top destination.
Verified vs unverified
If requests claiming to be GPTBot fall outside OpenAI's current published IP data, classify them as identity mismatches. Alert thresholds should follow your observed traffic and abuse policy.
Status code distribution per bot
| Bot | 2xx | 3xx | 4xx | 5xx |
|---|---|---|---|---|
| GPTBot | measured | measured | measured | measured |
| ClaudeBot | measured | measured | measured | measured |
| PerplexityBot | measured | measured | measured | measured |
A 5xx increase means your delivery path failed those requests; correlate origin, CDN, WAF, dependency, and load telemetry before assigning cause.
Latency per bot
p50, p95, p99 response time per bot_family. AI crawlers tend to hammer specific paths; latency rising on the path they are crawling tells you whether your origin is keeping up.
Observed robots.txt policy mismatches
For every verified request whose applicable rule would have disallowed that path, log a robots_policy_mismatch event with the exact rule and parser version. Aggregate by bot family and investigate sustained mismatches; robots.txt is a cooperative crawling protocol, not access control.
Cost / bandwidth attribution
Track bytes served per bot family so legal, finance, and platform teams can quantify crawler-specific egress.
Sample Alert Rules
| Severity | Trigger | Window | Action |
|---|---|---|---|
| Critical | Bot traffic exhausts a user-facing error budget | SLO window | Page on-call, snapshot top paths |
| Critical | New claimed bot causes material load or abuse | Confirmed window | Apply abuse runbook |
| High | Identity-mismatch rate moves materially above baseline | Rolling baseline | Notify SRE channel |
| High | Sustained robots-policy mismatch from verified bot | Confirmed window | Notify SEO + legal |
| Medium | Bytes/day per bot > N (capacity budget) | 24 h | Notify platform |
| Info | New bot family observed | 1 h | Slack digest |
Tune thresholds to your traffic. For the alerting policy that keeps these from becoming pager fatigue, see Alert Fatigue.
Rate Limiting Without Breaking Citations
If a crawler overwhelms a path, apply a documented capacity policy. A 429 communicates rate limiting more accurately than manufacturing a 5xx, but crawler retry behavior and citation impact are vendor-specific and not guaranteed.
Pattern:
- Per-bot family token bucket (separate from anonymous-user limits).
- Higher limit for search and user-fetch bots you deliberately allow (
OAI-SearchBot,Perplexity-User,Claude-User,Claude-SearchBot). - Separate policy for training bots (
GPTBot,ClaudeBot) and search indexing (PerplexityBot). - Return the status that accurately describes the condition; use
429withRetry-Afterfor deliberate rate limiting and503for genuine temporary unavailability.
For the broader rate-limit pattern - thresholds, headers, retry semantics - see API Rate Limit Monitoring.
Edge Cases You Will Hit
User-driven fetches
User-triggered fetchers have different purposes from bulk indexing or training crawlers. Configure their rules independently, monitor observed requests, and compare behavior with each vendor's current documentation.
Spoofed Googlebot for SEO cloaking
Scrapers pretend to be Googlebot to bypass paywalls or get cleaner HTML. Reverse-DNS verification catches this. Combine with SEO Cloaking Detection.
Bot traffic during JavaScript-heavy renders
If your site is a SPA, the bot may render JS. That changes performance and cost. See JavaScript SEO & Googlebot Rendering.
User-agent: * interactions
Disallow: / under User-agent: * does not automatically apply to AI bots if they have their own block - many bots only check their own section. Be explicit per bot.
CDN / WAF caching
If Cloudflare or your WAF serves cached responses to bots, your origin logs will under-count. Capture bot identity at the edge (Cloudflare Workers, Fastly VCL, etc.), not just at the origin.
Security headers and bot identity
Aggressive bot blocking can leak HTML structure or break previews. Validate your headers and content policies. See HTTP Security Headers Monitoring.
A Reference Dashboard
The bare minimum AI crawler dashboard:
- Top: total bot requests vs human requests today, week-over-week.
- By bot family: stacked area chart of requests/hour, last 14 days.
- Compliance: robots.txt violations per bot, last 24 h.
- Verification: verified vs unverified rate per bot, last 7 d.
- Health: 5xx and p95 latency per bot, last 24 h.
- Top paths per bot: the URLs each AI crawler is hammering.
- New bots: any user agent seen for the first time in the last 7 days.
Tie this to alerts. A dashboard without alerts is a museum exhibit.
AI Crawler Monitoring Checklist
- Per-request
bot_family,bot_purpose,bot_verifiedfields logged - Vendor-documented identity verification implemented for priority bot families
-
robots.txtreflects a clear policy for training, search indexing, and user-triggered fetches -
robots.txtfile is monitored for regressions on deploy - Volume, 5xx rate, latency, and bytes alerts per bot family
-
robots_policy_mismatchevents tracked and reviewed - Rate limits configured by documented crawler purpose and capacity budget
- Dashboard reviewed weekly; new bot families investigated
- Edge / CDN identity capture (not just origin) so cache hits are counted
- Legal / SEO stakeholders looped into training-crawler policy decisions
How Webalert Helps
Webalert focuses on the external side of crawler monitoring - making sure crawlers can actually reach your site and that what they fetch is what you intend:
- External HTTP monitoring - Detect when AI-traffic spikes cause 5xx for real users, regardless of internal dashboards.
- Content validation - Catch the day a deploy serves your homepage as a login page to GPTBot. See Response Body Validation Monitoring.
- robots.txt monitoring - Alert when
robots.txtchanges unexpectedly or returns 5xx. See Sitemap & robots.txt Monitoring. - Multi-region checks - Some AI bots crawl from a specific region; verify you respond healthily from there.
- TLS and DNS checks - Bots will not crawl a site with broken certs or DNS issues; catch these before crawl budget is wasted.
- Status page - Communicate downtime to bots and humans alike, via a real status page that you control.
Example Webalert check tuned for crawler health:
- URL:
https://example.com/robots.txt - Method:
GET - Expected status:
200 - Must contain:
User-agent: GPTBot,Sitemap: - Must not contain:
Disallow: /directly underUser-agent: *unless intentional - Response time: under 800ms
- Region: US + EU
- Alert: immediate on any change to body
A second check on your sitemap, a third on a representative page that AI bots should be able to reach, and you have closed the loop.
Summary
AI crawler monitoring separates declared purpose, identity confidence, access policy, volume, errors, latency, and bandwidth. Keep search indexing, user fetch, and model-training controls distinct.
Crawler logs can show that a documented bot requested a page; they cannot prove that a product cited the page or that a particular request entered model training. Join crawler telemetry with separately measured answer citations and infrastructure health.