Search Engine Crawlers
The Search Engine dashboard shows how search engines discover and crawl your website.
What We Track
Search engine crawlers (also called spiders or bots) visit your pages to index them for search results. We track all major search engines:
- Googlebot — Google Search
- Bingbot — Microsoft Bing
- DuckDuckBot — DuckDuckGo
- Baiduspider — Baidu (China)
- Yandex — Yandex (Russia)
- And many more
Dashboard Sections
Hero Metrics
- Total Crawls — total search engine visits, with trend
- Search Engines — number of unique search engines detected
- Bots — total bot visits (all types)
- Humans — total human visits, with traffic percentage
Crawl Trends
Line chart showing crawl activity per search engine over time.
Search Engine Breakdown
Ranked list of all search engines by visit count. Click any bot name to see its detail page.
Top Crawled Pages
Which pages search engines visit most. Click any URL to see the page detail view.
Bots vs Humans
Pie chart showing the split between bot and human traffic.
Why This Matters
Monitoring search engine crawl patterns helps you understand:
- Crawl budget — are search engines spending time on your important pages?
- Index coverage — are all your pages being discovered?
- Crawl frequency — how often do search engines return?
- New crawlers — are new search engines finding your site?