For SEO professionals and webmasters, managing AI crawlers is paramount. While allowing AI bots to access your content is crucial for visibility in emerging AI discovery engines, uncontrolled crawling can quickly exhaust server resources, leading to performance issues and unexpected hosting costs. Official documentation for AI user-agent strings is often incomplete or outdated, making effective management a challenge.
To address this, we've compiled a verified list of AI crawlers based on actual server logs, complete with user-agent strings and, where available, official IP lists for validation. This resource will be regularly updated to reflect new crawlers and changes to existing ones.
The Complete Verified AI Crawler List (December 2025)
| Name | Purpose | Crawl Rate of SEJ (pages/hour) | Verified IP List | Robots.txt disallow | Complete User Agent |
|---|---|---|---|---|---|
| GPTBot | AI training data collection for GPT models (ChatGPT, GPT-4o) | 100 | Official IP List | User-agent: GPTBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot) |
| ChatGPT-User | AI agent for real-time web browsing when users interact with ChatGPT | 2400 | Official IP List | User-agent: ChatGPT-User Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot |
| OAI-SearchBot | AI search indexing for ChatGPT search features (not for training) | 150 | Official IP List | User-agent: OAI-SearchBot Allow: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ClaudeBot | AI training data collection for Claude models | 500 | Official IP List | User-agent: ClaudeBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected]) |
| Claude-User | AI agent for real-time web access when Claude users browse | <10 | Not available | User-agent: Claude-User Disallow: /sample-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected]) |
| Claude-SearchBot | AI search indexing for Claude search capabilities | <10 | Not available | User-agent: Claude-SearchBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com) |
| Google-CloudVertexBot | AI agent for Vertex AI Agent Builder (site owners’ request only) | <10 | Official IP List | User-agent: Google-CloudVertexBot Allow: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.7390.122 Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) |
| Google-Extended | Token controlling AI training usage of Googlebot-crawled content. | User-agent: Google-Extended Allow: / Disallow: /private-folder |
|||
| Gemini-Deep-Research | AI research agent for Google Gemini’s Deep Research feature | <10 | Official IP List | User-agent: Gemini-Deep-Research Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 |
| Gemini’s chat when a user asks to open a webpage | <10 | ||||
| Bingbot | Powers Bing Search and Bing Chat (Copilot) AI answers | 1300 | Official IP List | User-agent: BingBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Applebot-Extended | Doesn’t crawl but controls how Apple uses Applebot data. | <10 | Official IP List | User-agent: Applebot-Extended Allow: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| PerplexityBot | AI search indexing for Perplexity’s answer engine | 150 | Official IP List | User-agent: PerplexityBot Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-User | AI agent for real-time browsing when Perplexity users request information | <10 | Official IP List | User-agent: Perplexity-User Allow: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
| Meta-ExternalAgent | AI training data collection for Meta’s LLMs (Llama, etc.) | 1100 | Not available | User-agent: meta-externalagent Allow: / Disallow: /private-folder |
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Meta-WebIndexer | Used to improve Meta AI search. | <10 | Not available | User-agent: Meta-WebIndexer Allow: / Disallow: /private-folder |
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) |
| Bytespider | AI training data for ByteDance’s LLMs for products like TikTok | <10 | Not available | User-agent: Bytespider Allow: / Similar News Guardio Nabs $80M to Combat AI-Generated Cyber ThreatsThis article details cybersecurity startup Guardio's recent $80 million funding round led by ION Crossover Partners. The investment will fuel Guardio's expansion into detecting and combating malicious code and phishing sites generated by AI tools, alongside enhancing its consumer-focused security offerings.  Hinge CEO Launches AI Dating App OvertoneFormer Hinge CEO Justin McLeod has announced his departure to launch Overtone, an innovative AI-powered dating application. Backed by Match Group, Overtone seeks to revolutionize online dating by leveraging artificial intelligence and voice tools for more meaningful connections, addressing a market increasingly disillusioned with traditional dating apps.  50+ US AI Startups Secure $100M+ Funding in 2025The U.S. AI industry continues its explosive growth, with more than 50 startups securing funding rounds of $100 million or greater in 2025. This article details the companies, their innovations, and the significant investments driving the sector forward.  TechCrunch Warns Businesses of Rising Impersonation ScamsTechCrunch has issued a warning about an increase in sophisticated impersonation scams. Fraudsters are posing as TechCrunch reporters and event leads, targeting companies to illicitly obtain sensitive business information. The article details common scam tactics, potential motivations, and crucial verification steps businesses can take to protect themselves and uphold journalistic integrity.  Pacific Fusion finds cheaper path to fusion energyThis article details how Pacific Fusion's recent experiments at Sandia National Laboratory's Z Machine revealed a cheaper method to initiate fusion reactions, potentially accelerating the path to commercial fusion energy. |
