For SEO professionals and webmasters, managing AI crawlers is paramount. While allowing AI bots to access your content is crucial for visibility in emerging AI discovery engines, uncontrolled crawling can quickly exhaust server resources, leading to performance issues and unexpected hosting costs. Official documentation for AI user-agent strings is often incomplete or outdated, making effective management a challenge.
To address this, we've compiled a verified list of AI crawlers based on actual server logs, complete with user-agent strings and, where available, official IP lists for validation. This resource will be regularly updated to reflect new crawlers and changes to existing ones.
The Complete Verified AI Crawler List (December 2025)
| Name | Purpose | Crawl Rate of SEJ (pages/hour) | Verified IP List | Robots.txt disallow | Complete User Agent |
|---|---|---|---|---|---|
| GPTBot | AI training data collection for GPT models (ChatGPT, GPT-4o) | 100 | Official IP List | User-agent: GPTBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot) |
| ChatGPT-User | AI agent for real-time web browsing when users interact with ChatGPT | 2400 | Official IP List | User-agent: ChatGPT-User Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
| OAI-SearchBot | AI search indexing for ChatGPT search features (not for training) | 150 | Official IP List | User-agent: OAI-SearchBot Allow: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ClaudeBot | AI training data collection for Claude models | 500 | Official IP List | User-agent: ClaudeBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]) |
| Claude-User | AI agent for real-time web access when Claude users browse | <10 | Not available | User-agent: Claude-User Disallow: /sample-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected]) |
| Claude-SearchBot | AI search indexing for Claude search capabilities | <10 | Not available | User-agent: Claude-SearchBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com) |
| Google-CloudVertexBot | AI agent for Vertex AI Agent Builder (site owners’ request only) | <10 | Official IP List | User-agent: Google-CloudVertexBot Allow: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.7390.122 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) |
| Google-Extended | Token controlling AI training usage of Googlebot-crawled content. | User-agent: Google-Extended Allow: / Disallow: /private-folder |
|||
| Gemini-Deep-Research | AI research agent for Google Gemini’s Deep Research feature | <10 | Official IP List | User-agent: Gemini-Deep-Research Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 |
| Gemini’s chat when a user asks to open a webpage | <10 | ||||
| Bingbot | Powers Bing Search and Bing Chat (Copilot) AI answers | 1300 | Official IP List | User-agent: BingBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Applebot-Extended | Doesn’t crawl but controls how Apple uses Applebot data. | <10 | Official IP List | User-agent: Applebot-Extended Allow: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| PerplexityBot | AI search indexing for Perplexity’s answer engine | 150 | Official IP List | User-agent: PerplexityBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-User | AI agent for real-time browsing when Perplexity users request information | <10 | Official IP List | User-agent: Perplexity-User Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
| Meta-ExternalAgent | AI training data collection for Meta’s LLMs (Llama, etc.) | 1100 | Not available | User-agent: meta-externalagent Allow: / Disallow: /private-folder |
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Meta-WebIndexer | Used to improve Meta AI search. | <10 | Not available | User-agent: Meta-WebIndexer Allow: / Disallow: /private-folder |
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Bytespider | AI training data for ByteDance’s LLMs for products like TikTok | <10 | Not available | User-agent: Bytespider Allow: / Similar NewsSaaStr's AI SDRs: $1M Revenue in 90 DaysSaaStr, a leading SaaS community, details its six-month journey deploying five specialized AI Sales Development Representatives (SDRs). This article uncovers the actual performance metrics, unexpected insights, and the substantial human commitment necessary to achieve over $1 million in revenue in just 90 days, fundamentally reshaping their sales development strategy.  Tesla FSD Texting Feature Sparks Safety, Legal DebateTesla CEO Elon Musk has stirred controversy by stating that the latest Full Self-Driving (Supervised) software allows drivers to text, a practice illegal in most U.S. states. This revelation comes amidst intensifying scrutiny from federal regulators and a legal battle with California authorities over the safety and marketing of Tesla's driver assistance systems.  Thomas Lee Young's Unlikely Path to AI Industrial SafetyThis article explores the journey of Thomas Lee Young, CEO of Interface, an AI startup focused on preventing industrial accidents. It details how his unconventional background and experiences have become a significant advantage in the competitive world of industrial technology and venture capital.  Leona Health Secures $14M from a16z for AI Doctor Co-pilotLeona Health, a startup founded by former Uber Eats and Rappi executive Caroline Merin, has raised $14 million in seed funding to tackle the overwhelming volume of patient communications doctors in Latin America receive via WhatsApp. Its AI co-pilot aims to streamline healthcare interactions and save physicians valuable time.  Nvidia Unveils Advanced AI Weather Forecasting ModelsNvidia has launched a new suite of AI-powered weather forecasting models, including Earth-2 Medium Range, Nowcasting, and Global Data Assimilation. These tools aim to significantly enhance forecast accuracy and speed, making advanced meteorological insights more accessible worldwide. |
