Collects analytics data without affecting your rankings.
Recommended action: Allow access. Block only if crawl rate is excessive.
Category
SEO Tool
Primary use case
SEO analytics and backlink data
Trust level
Generally safe
Trust Levels
- Trusted
- Generally safe
- Review recommended
- Caution advised
Trust levels are an indication based on category, operator, and robots.txt compliance. Always review bot activity for your specific situation.
Learn how we assess trustrobots.txt
Respected
SemrushBot Traffic (Last 90 Days)
What is SemrushBot?
SemrushBot crawls websites to collect data for Semrush's SEO analytics platform, including backlinks, keyword rankings, and competitive analysis.
What SemrushBot means for your site
SemrushBot crawls your site to collect data for Semrush. This data powers backlink analysis, keyword tracking, and competitive intelligence tools used by SEO professionals. It does not affect your search rankings or site performance in any meaningful way. Its presence simply means your site is being analyzed as part of the broader web.
What should you do?
- Allow SemrushBot unless crawl rate is excessive
- Set a crawl-delay in robots.txt if server load is a concern
- No SEO action needed, this bot is informational only
See SemrushBot on your own site
BotSights tracks every SemrushBot visit in real time, including which pages it crawls, how often, and from where.
How to identify SemrushBot
SemrushBot uses the user-agent "SemrushBot" and respects robots.txt. It is one of the most active non-search crawlers on the web.
SemrushBotsemrushbotHow to block SemrushBot
Three robots.txt options below. Pick the one that matches your goal. Each snippet lists every known SemrushBot user-agent pattern so the rules apply regardless of which one the bot announces.
Edit robots.txt with care
A single misplaced line can de-index your entire site. Common mistake: pasting User-agent: * followed by Disallow: / blocks every bot, not just SemrushBot, including Googlebot. Always paste the snippet between existing rules (not over them), keep the User-agent line scoped to SemrushBot's patterns, and verify with Google's robots.txt tester before deploying. If you are not sure, ask a developer first.
Option 1: Block all access
Tells SemrushBot not to crawl any URL on your site. Use this when you want the bot completely off your content.
User-agent: SemrushBot
User-agent: semrushbot
Disallow: /Option 2: Block specific paths only
Keep public content crawlable but exclude sensitive or non-public sections. Add one Disallow: line per path. Replace the example paths with your own.
User-agent: SemrushBot
User-agent: semrushbot
Disallow: /admin/
Disallow: /private/
Disallow: /checkout/Option 3: Slow down with a crawl delay
Crawl-delay is a voluntary directive that asks the bot to wait the given number of seconds between requests. Useful when SemrushBot is hammering your origin and slowing the site down for real visitors, but you do not want to block it outright. The value is in seconds, so 10 means at most one request every ten seconds. Not all bots honour this directive (Googlebot ignores it; Bingbot, Yandex, and many AI crawlers do respect it).
User-agent: SemrushBot
User-agent: semrushbot
Crawl-delay: 10Frequently Asked Questions
What is the User-Agent for SemrushBot?
SemrushBot identifies itself with the User-Agent string "SemrushBot" (alternate forms: semrushbot). Use this in robots.txt or firewall rules.
Does SemrushBot affect my Google rankings?
No. SemrushBot collects data for Semrush, separate from any search engine's ranking algorithm. Allowing or blocking it has zero SEO impact on Google, Bing, or other search engines.
Should I block SemrushBot?
Usually not. Blocking only prevents your site from appearing in Semrush backlink and SEO database, which means competitors who use the tool cannot analyze your site. Some site owners prefer this for privacy. Most have no reason to block.
How do I block SemrushBot if I want to?
Add this to your robots.txt: User-agent: SemrushBot / Disallow: /. SemrushBot respects robots.txt.
Why is SemrushBot crawling my site so frequently?
SEO tools tend to re-crawl popular sites often to keep their backlink and ranking data fresh. If the load is excessive, set a Crawl-delay directive in robots.txt: "User-agent: SemrushBot / Crawl-delay: 10" — most respect this.
Is data SemrushBot collects shared with my competitors?
Semrush's customers can typically see basic crawl data about any indexed site (backlinks, top pages, keyword visibility). If you do not want this exposed, blocking is the only way to opt out.
Track SEO crawlers visiting your site
See which SEO and link intelligence bots hit your pages, how often they appear, and whether the crawl load is worth allowing.
- See which pages SEO crawlers hit most
- Monitor crawl load from non-search bots
- Decide which crawler access is worth allowing
Free plan available. No credit card required. Setup in 2 minutes.