For SEO professionals and webmasters, managing AI crawlers is paramount. While allowing AI bots to access your content is crucial for visibility in emerging AI discovery engines, uncontrolled crawling can quickly exhaust server resources, leading to performance issues and unexpected hosting costs. Official documentation for AI user-agent strings is often incomplete or outdated, making effective management a challenge.
To address this, we've compiled a verified list of AI crawlers based on actual server logs, complete with user-agent strings and, where available, official IP lists for validation. This resource will be regularly updated to reflect new crawlers and changes to existing ones.
The Complete Verified AI Crawler List (December 2025)
| Name | Purpose | Crawl Rate of SEJ (pages/hour) | Verified IP List | Robots.txt disallow | Complete User Agent |
|---|---|---|---|---|---|
| GPTBot | AI training data collection for GPT models (ChatGPT, GPT-4o) | 100 | Official IP List | User-agent: GPTBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot) |
| ChatGPT-User | AI agent for real-time web browsing when users interact with ChatGPT | 2400 | Official IP List | User-agent: ChatGPT-User Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
| OAI-SearchBot | AI search indexing for ChatGPT search features (not for training) | 150 | Official IP List | User-agent: OAI-SearchBot Allow: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ClaudeBot | AI training data collection for Claude models | 500 | Official IP List | User-agent: ClaudeBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]) |
| Claude-User | AI agent for real-time web access when Claude users browse | <10 | Not available | User-agent: Claude-User Disallow: /sample-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected]) |
| Claude-SearchBot | AI search indexing for Claude search capabilities | <10 | Not available | User-agent: Claude-SearchBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com) |
| Google-CloudVertexBot | AI agent for Vertex AI Agent Builder (site owners’ request only) | <10 | Official IP List | User-agent: Google-CloudVertexBot Allow: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.7390.122 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) |
| Google-Extended | Token controlling AI training usage of Googlebot-crawled content. | User-agent: Google-Extended Allow: / Disallow: /private-folder |
|||
| Gemini-Deep-Research | AI research agent for Google Gemini’s Deep Research feature | <10 | Official IP List | User-agent: Gemini-Deep-Research Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 |
| Gemini’s chat when a user asks to open a webpage | <10 | ||||
| Bingbot | Powers Bing Search and Bing Chat (Copilot) AI answers | 1300 | Official IP List | User-agent: BingBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Applebot-Extended | Doesn’t crawl but controls how Apple uses Applebot data. | <10 | Official IP List | User-agent: Applebot-Extended Allow: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| PerplexityBot | AI search indexing for Perplexity’s answer engine | 150 | Official IP List | User-agent: PerplexityBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-User | AI agent for real-time browsing when Perplexity users request information | <10 | Official IP List | User-agent: Perplexity-User Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
| Meta-ExternalAgent | AI training data collection for Meta’s LLMs (Llama, etc.) | 1100 | Not available | User-agent: meta-externalagent Allow: / Disallow: /private-folder |
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Meta-WebIndexer | Used to improve Meta AI search. | <10 | Not available | User-agent: Meta-WebIndexer Allow: / Disallow: /private-folder |
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Bytespider | AI training data for ByteDance’s LLMs for products like TikTok | <10 | Not available | User-agent: Bytespider Allow: / Similar News Sauron Names Sonos Exec CEO, Delays High-End Security LaunchThis article details Sauron, a high-end home security startup founded by Kevin Hartz and Jack Abraham, its new CEO Maxime Bouvat-Merlin from Sonos, and its delayed launch of AI-driven, military-grade security systems for affluent customers amidst rising crime concerns.  Thomas Lee Young's Unlikely Path to AI Industrial SafetyThis article explores the journey of Thomas Lee Young, CEO of Interface, an AI startup focused on preventing industrial accidents. It details how his unconventional background and experiences have become a significant advantage in the competitive world of industrial technology and venture capital.  Perplexity AI: Unpacking How AI Search & AEO WorkThis article delves into an interview with Jesse Dwyer of Perplexity AI, exploring the fundamental differences between traditional and AI-driven search, the concept of sub-document processing, and the evolving landscape of Answer Engine Optimization (AEO).  IBM Acquires Confluent for $11B to Boost AI, DataIBM's $11 billion acquisition of Confluent aims to significantly strengthen its data and AI capabilities, addressing the increasing need for real-time data management in cloud and AI environments. AI Reshapes B2B Sales: The 2021 Playbook Is DeadJason Lemkin, CEO and Founder of SaaStr, shares critical insights from SaaStr AI London on the profound shifts in B2B sales and go-to-market strategies driven by artificial intelligence. He argues that while core sales "plays" still work, the 2021 playbook is obsolete, demanding new approaches to value, competition, and product expertise. |
