meta-webindexer

AI Search Crawler

Operated by Meta

Allow

Indexes your content for AI-powered search results.

Recommended action: Allow access to maintain AI search visibility.

What is meta-webindexer?

meta-webindexer browses the internet to improve search results for Meta AI users, analyzing online content to make Meta AI's responses more relevant with proper citations.

What meta-webindexer means for your site

meta-webindexer proactively indexes your site for an AI-powered search engine, operated by Meta. Unlike AI assistants that fetch on-demand, this bot discovers and catalogs your content so it can surface in AI search results. Being indexed means your content is eligible to appear when users search through AI platforms. Not being indexed means you are invisible in this channel.

What should you do?

Allow meta-webindexer to crawl your site
Ensure your most important pages are accessible and well-structured
Monitor which pages are being indexed through crawl activity
Review robots.txt to ensure you are not accidentally blocking meta-webindexer

How to identify meta-webindexer

meta-webindexer uses the user-agent "meta-webindexer" and respects robots.txt. It crawls systematically, similar to traditional search engine bots.

meta-webindexer

Frequently Asked Questions

Should I block meta-webindexer?

Blocking meta-webindexer removes your content from Meta's AI search index. If you want visibility in AI-powered search, allow it.

Is meta-webindexer different from Googlebot?

Yes. Googlebot indexes for traditional Google Search. meta-webindexer indexes for AI-powered search, which presents results differently, often as conversational answers with citations.

Does meta-webindexer affect my site speed?

Like any crawler, meta-webindexer consumes some server resources. If crawl rate is excessive, you can set a crawl-delay in robots.txt.

Official Meta documentation