Cloudflare Worker Setup

Track bot and AI crawler visits on any website using Cloudflare. Works with all Cloudflare plans (Free, Pro, Business, Enterprise).

Time to set up: ~5 minutes

Prerequisites

• A Cloudflare account with your domain added and proxied (orange cloud enabled)
• A BotSights account with a project created for your domain
• Your API Key (found in BotSights → Account → Projects → API Key)

Important: Check your Cloudflare bot settings before setup.

Cloudflare may block AI bots by default, which prevents BotSights from tracking them.

• Go to Security → Bots → Block AI Bots and turn it OFF. If enabled, Cloudflare blocks AI crawlers at the network level before they reach the Worker, so BotSights can never detect them.
• Go to Security → Bots → Robots.txt Management and turn it OFF if you want to manage your robots.txt yourself.

These settings are enabled by default on some Cloudflare plans. If you want to control AI bot access, use robots.txt instead of Cloudflare's blocking. BotSights will still detect and log the visits.

Step 1: Create the Worker

1. Log in to your Cloudflare Dashboard
2. Select your domain (e.g. yourdomain.com)
3. In the left sidebar, go to Workers & Pages (under Compute / Workers)
4. Click Create application
5. Select Start with Hello World!
6. Name the worker (e.g. botsights) and click Deploy

Step 2: Edit the Worker Code

1. After deploying, click Edit Code in the top right
2. Select all existing code in the worker.js file and delete it
3. Paste the following script:

const API_KEY = 'YOUR_API_KEY_HERE';
const API_URL = 'https://www.botsights.com/api/log';
 
export default {
  async fetch(request, env, ctx) {
    const startTime = Date.now();
    const response = await fetch(request);
    ctx.waitUntil(sendAnalytics(request, response, startTime).catch(() => {}));
    return response;
  },
};
 
async function sendAnalytics(request, response, startTime) {
  const url = new URL(request.url);
  const ua = request.headers.get('user-agent') || '';
 
  const ext = url.pathname.split('.').pop();
  const skip = [
    'css','js','png','jpg','jpeg','gif','svg','webp','avif',
    'ico','woff','woff2','ttf','eot','map','mp4','webm','mp3',
  ];
  if (ext && skip.includes(ext.toLowerCase())) return;
 
  // Skip AJAX/form/API requests from humans (not pageviews).
  // Bots may legitimately POST (e.g. Googlebot JS rendering) — keep those.
  const isBot = /bot|crawler|spider|scraper|slurp|crawling/i.test(ua);
  if (!isBot && request.method !== 'GET' && request.method !== 'HEAD') return;
 
  const verifiedBotCategory =
    (request.cf && request.cf.verifiedBotCategory) ? String(request.cf.verifiedBotCategory) : null;
 
  await fetch(API_URL, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      api_key: API_KEY,
      ua,
      url: url.pathname + url.search,
      ip: request.headers.get('cf-connecting-ip') || '',
      referer: request.headers.get('referer') || '',
      status_code: response.status,
      duration_ms: Date.now() - startTime,
      content_type: response.headers.get('content-type') || '',
      http_method: request.method,
      source: 'cloudflare_worker',
      verified_bot_category: verifiedBotCategory,
    }),
  });
}

4. Replace YOUR_API_KEY_HERE with your actual API key from BotSights
5. Click Deploy

Step 3: Add a Route to Your Domain

1. Navigate back to your Cloudflare dashboard and select your domain
2. In the left sidebar, click Workers Routes
3. Click Add route in the HTTP Routes section
4. Under Route, enter your domain followed by /*:

If your site uses www:

www.yourdomain.com/*

If your site does not use www:

yourdomain.com/*

If you're not sure, add both routes to cover all traffic:

yourdomain.com/*
www.yourdomain.com/*

5. Select the botsights worker you just created
6. Under Failure mode, select Fail open (proceed) — this ensures your website keeps working even if the Worker has an error
7. Click Save

Important: The route must match exactly how visitors access your site. If your site is on www.yourdomain.com but you set the route to yourdomain.com/*, the Worker won't run. Check your browser's address bar to see which version your site uses.

Step 4: Verify It Works

1. Visit your website in a browser
2. Open your BotSights dashboard → Realtime page
3. Your visit will appear as a "Human" visit

To test bot detection, run this in your terminal:

curl -H "User-Agent: ChatGPT-User" https://yourdomain.com/

This simulates a ChatGPT bot visit.

Troubleshooting:

How It Works

The Worker sits between Cloudflare and your origin server. Every request passes through it transparently. After serving the response, analytics data is sent asynchronously — your visitors experience zero added latency.

Static assets (images, CSS, JS, fonts) are automatically skipped.

FAQ

Will this slow down my website? No. Analytics are sent after the response is delivered using waitUntil(). Zero latency added.

Does this work with Cloudflare's free plan? Yes. The free plan includes 100,000 Worker requests per day. Set failure mode to "Fail open" so your site works if the limit is reached.

What about cached pages? The Worker runs on every request, including cached pages.

Can I use this alongside the WordPress plugin? We recommend one or the other to avoid double-counting. The Worker is preferred for Cloudflare sites.