Indexes your content for AI-powered search results.
Recommended action: Allow access to maintain AI search visibility.
Category
AI Search Crawler
Primary use case
AI search indexing
robots.txt
Respected
What is xAI-SearchBot?
xAI-SearchBot is the live-search crawler operated by xAI, the AI company behind Grok. When a Grok user asks a question that needs current information, xAI-SearchBot fetches relevant web pages on-demand so Grok can ground its answer in fresh, citable content. The full user-agent is Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai). It is the xAI counterpart to OAI-SearchBot (ChatGPT search), Claude-SearchBot (Anthropic search), and PerplexityBot (Perplexity index). The bot obeys robots.txt and identifies itself clearly in the User-Agent header. Allowing it means your content is eligible to appear as a cited source inside Grok's answers; blocking it removes that eligibility but does not affect any separate xAI training data collection.
What xAI-SearchBot means for your site
xAI-SearchBot is how your content becomes citable inside Grok — xAI's AI assistant integrated into X (for paid subscribers) and available as a standalone product at grok.com. Each visit corresponds to a real human asking Grok a question that your page is a good match for; Grok then fetches the page on-demand and its answer may quote, summarize, or link to your content with a visible citation. Two things make xAI-SearchBot strategically distinct from OAI-SearchBot, Claude-SearchBot, and PerplexityBot. First, its primary audience is X's paid user base — a US-skewed, news- and politics-engaged cohort that other AI products do not reach the same way. Second, when Grok cites your content inside an answer that appears on X, that answer can be screenshotted, quoted, or shared by users, creating a secondary distribution loop through X's social graph that no other AI search crawler taps into. Pages with clear direct answers, strong branding, and timely angles tend to win citations more often.
What should you do?
- Allow xAI-SearchBot in robots.txt — blocking removes your content from Grok citations entirely
- Lead pages with the literal question and answer so Grok can lift a clean, quotable passage
- Add a recognisable brand element (name and visual) near the top so users notice you when cited inside Grok's UI
- Prioritise news-pegged and 'current state of X' content — Grok's real-time positioning over-indexes on freshness
- Track citation frequency per page in BotSights and double down on what Grok keeps returning to
- Watch for spoofed traffic: xAI does not publish an IP allowlist, so UA-only filters are brittle — use defense-in-depth for high-value endpoints
See xAI-SearchBot on your own site
BotSights tracks every xAI-SearchBot visit in real time, including which pages it crawls, how often, and from where. See plans →
How to identify xAI-SearchBot
xAI-SearchBot announces itself with the User-Agent string "Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)". The "+https://x.ai" self-identifier follows xAI's convention, similar to how Googlebot signs with "+http://www.google.com/bot.html". Verification is harder than for OpenAI or Anthropic crawlers: xAI does not publish an IP allowlist (no equivalent of openai.com/searchbot.json or Anthropic's bots.json), and there is no documented reverse-DNS pattern. In practice the UA string is the primary signal, which means UA-only matching is more spoofable for xAI than for its peers. For high-value endpoints, layer behavioural checks and rate limits on top. BotSights cross-references each xAI-SearchBot hit against known cloud infrastructure ranges and flags suspicious sources automatically.
xAI-SearchBotxai-searchbot How to block xAI-SearchBot
Three robots.txt options below. Pick the one that matches your goal. Each snippet lists every known xAI-SearchBot user-agent pattern so the rules apply regardless of which one the bot announces.
Edit robots.txt with care
A single misplaced line can de-index your entire site. Common mistake: pasting User-agent: * followed by Disallow: / blocks every bot, not just xAI-SearchBot, including Googlebot. Always paste the snippet between existing rules (not over them), keep the User-agent line scoped to xAI-SearchBot's patterns, and verify with Google's robots.txt tester before deploying. If you are not sure, ask a developer first.
Option 1: Block all access
Tells xAI-SearchBot not to crawl any URL on your site. Use this when you want the bot completely off your content.
User-agent: xAI-SearchBot User-agent: xai-searchbot Disallow: /
Option 2: Block specific paths only
Keep public content crawlable but exclude sensitive or non-public sections. Add one Disallow: line per path. Replace the example paths with your own.
User-agent: xAI-SearchBot User-agent: xai-searchbot Disallow: /admin/ Disallow: /private/ Disallow: /checkout/
Option 3: Slow down with a crawl delay
Crawl-delay is a voluntary directive that asks the bot to wait the given number of seconds between requests. Useful when xAI-SearchBot is hammering your origin and slowing the site down for real visitors, but you do not want to block it outright. The value is in seconds, so 10 means at most one request every ten seconds. Not all bots honour this directive (Googlebot ignores it; Bingbot, Yandex, and many AI crawlers do respect it).
User-agent: xAI-SearchBot User-agent: xai-searchbot Crawl-delay: 10
Frequently Asked Questions
What is the exact User-Agent for xAI-SearchBot?
Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai). The substring "xAI-SearchBot" is what you match in robots.txt rules and server-side filters.
Should I block xAI-SearchBot?
Almost certainly no. Blocking removes your content from Grok's answers entirely, which kills both the citation to X's paid user base and any chance of being quoted in a Grok answer that goes viral on X. Allow by default; block only if you have a concrete reason (staging environment, paywall enforcement, observed abusive crawl rate).
Does xAI publish IP ranges I can verify against?
No. Unlike OpenAI (openai.com/searchbot.json) and Anthropic (claude.com/crawling/bots.json), xAI has not published a verifiable IP allowlist or a formal bot policy page. The User-Agent string is the primary identifier today. This makes UA-only filtering more spoofable for xAI than for its peers — BotSights compensates by flagging UA-versus-network mismatches automatically.
Is xAI-SearchBot the same as the xAI training bot?
No, and importantly: xAI has not publicly named a separate training crawler. xAI-SearchBot is documented as a live-fetch bot for Grok answers. If xAI runs a separate training pipeline, it most likely uses a different and currently unannounced UA. Blocking xAI-SearchBot therefore only stops Grok citations; it does not necessarily stop any future xAI training data collection.
How is xAI-SearchBot different from OAI-SearchBot, Claude-SearchBot, and PerplexityBot?
Mechanically they all do the same job — fetch live web pages so an AI assistant can ground its answer. The strategic differences are audience and distribution. xAI-SearchBot exclusively serves Grok inside X and at grok.com; OAI-SearchBot serves ChatGPT Search (mass-market); Claude-SearchBot serves Claude (researcher-leaning); PerplexityBot powers Perplexity (answer engine). Only Grok citations propagate through X's social graph as a secondary distribution channel.
How often does xAI-SearchBot visit my site?
On-demand, not on a schedule. A hit happens when a Grok user asks something Grok decides your page can answer. Volume scales with how often your content is the best live match — not with your site size. Pages that win frequent citations get repeat visits; less-relevant pages may get few or none.
Does xAI-SearchBot respect robots.txt?
xAI states the bot honors robots.txt. Compliance has not been independently verified at scale, so the safe practice is to monitor in BotSights after adding a Disallow rule. If you see continued xAI-SearchBot hits after a robots.txt block, that is a strong signal to escalate to server-side blocking (Cloudflare WAF, NGINX rules).
How can I see which of my pages Grok actually cites?
BotSights tracks every xAI-SearchBot hit per page, plus follow-up click-through from real users who land on your site via a Grok citation. This gives you page-level AI visibility that you cannot derive from server logs alone — server logs show the fetch, but not whether Grok surfaced the page in a user-facing answer.
Track how AI search engines discover your content
Monitor which pages AI search crawlers index, how often they visit, and whether your content is visible in AI-powered search.
- AI search crawler activity by page
- Compare AI crawl coverage with traditional search
- Spot indexing gaps before they cost visibility
Signups reopen soon — we're upgrading our infrastructure.