Cloudflare Integration
Set up Cloudflare Workers to track AI bot visits and improve tracking accuracy
Cloudflare Workers provide a powerful way to track AI bot visits at the edge, giving you real-time insights into how AI models interact with your website.
Why Use Cloudflare Workers?
Cloudflare Workers offer several advantages:
- Edge Tracking: Track AI bots at Cloudflare's edge before they reach your origin
- Real-time Detection: Identify AI bots instantly using user agent and referer analysis
- No Impact on Performance: Tracking happens asynchronously without blocking requests
- Comprehensive Coverage: Track all AI bot visits, not just those that reach your server
Prerequisites
Before setting up Cloudflare Workers, you need:
- A Cloudflare account
- Your website added to Cloudflare
- Access to Cloudflare Workers & Pages
- A Ranketta account with API access
Setting Up Cloudflare Workers
Step 1: Get Your Site ID
- Navigate to Traffic in your Ranketta dashboard
- Click Set up Cloudflare tracking
- Your unique Site ID will be generated automatically
- Copy the Site ID (you'll need it for the Worker code)
Step 2: Create a Cloudflare Worker
- Log in to your Cloudflare dashboard
- Go to Workers & Pages
- Click Create Application
- Select Start with Hello World!
- Click Deploy to create the worker
Step 3: Edit the Worker Code
- In your Worker, click Edit Code
- Delete the default code
- Paste the Worker code provided by Ranketta
- The code includes:
- Your Site ID
- API endpoint for tracking
- AI bot detection patterns
- Request forwarding logic
Step 4: Deploy the Worker
- Click Deploy to save your changes
- The Worker will be active immediately
- Test by visiting your website with an AI bot
Step 5: Add a Route
- In your Worker settings, go to Domains & Routes
- Click Add Route
- Enter your domain (e.g.,
example.com/*) - Select your Worker from the dropdown
- Click Add Route
How It Works
AI Bot Detection
The Worker detects AI bots using multiple methods:
User Agent Detection
The Worker checks user agent strings for known AI bots:
- OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
- Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
- Google: Googlebot, Gemini, Google-CloudVertexBot
- Microsoft: bingbot, BingPreview, msnbot
- Perplexity: PerplexityBot, Perplexity-User
- Apple: Applebot
- Others: DuckDuckBot, YouBot, Phind, and more
Referer Detection
The Worker also checks referer headers to identify traffic from AI chat interfaces:
- ChatGPT (chatgpt.com, ai.com, openai.com)
- Claude (claude.ai, anthropic.com)
- Perplexity (perplexity.ai)
- Gemini (gemini.google.com, bard.google.com)
- Copilot (copilot.microsoft.com)
- And other AI interfaces
Tracking Process
- Request Interception: Worker intercepts all requests to your domain
- Bot Detection: Analyzes user agent and referer headers
- Data Collection: Gathers request metadata (URL, IP, country, etc.)
- Async Tracking: Sends tracking data to Ranketta API (non-blocking)
- Request Forwarding: Forwards original request to your origin server
Privacy & Security
- IP Anonymization: Last octet of IP addresses is removed for privacy
- API Key Authentication: Secure API key prevents unauthorized tracking
- Rate Limiting: Built-in rate limiting prevents abuse
- Error Handling: Failures don't affect your website's functionality
Viewing Tracked Data
Traffic Dashboard
After setup, visit Traffic in Ranketta to see:
- Real-time AI bot visits
- Breakdown by AI provider
- Geographic distribution
- Page-level analytics
Data Availability
- Data appears within minutes of setup
- Historical data accumulates over time
- Filter by date range, provider, or country
Troubleshooting
Worker Not Tracking
If the Worker isn't tracking:
- Verify the route is configured correctly
- Check that your Site ID is correct in the Worker code
- Ensure the Worker is deployed and active
- Test with a known AI bot user agent
Missing Data
If data isn't appearing:
- Wait a few minutes for initial sync
- Verify AI bots are actually visiting your site
- Check Worker logs in Cloudflare dashboard
- Ensure API endpoint is accessible
Performance Issues
If you notice performance issues:
- Verify the Worker code uses async tracking
- Check Cloudflare Worker limits
- Review rate limiting settings
- Monitor Worker execution time
Advanced Configuration
Custom Detection Rules
You can modify the Worker code to:
- Add custom user agent patterns
- Adjust detection sensitivity
- Add custom tracking logic
- Filter specific paths or domains
Multiple Routes
You can add multiple routes:
- Track different subdomains separately
- Apply different tracking rules per route
- Aggregate data across all routes
Environment Variables
Use Cloudflare environment variables for:
- API endpoints
- Site IDs
- Custom configuration
- Feature flags
Best Practices
Regular Monitoring
- Check tracking data weekly
- Monitor Worker execution metrics
- Review error logs periodically
- Verify data accuracy
Performance Optimization
- Keep Worker code lightweight
- Use async operations for tracking
- Implement proper error handling
- Monitor execution time
Security
- Keep API keys secure
- Use environment variables for sensitive data
- Regularly update Worker code
- Monitor for suspicious activity
Comparison with Other Methods
vs. Google Analytics
- Cloudflare: Tracks at edge, no JavaScript required
- Google Analytics: Requires JavaScript, may miss some bots
- Best: Use both for comprehensive coverage
vs. Server-Side Tracking
- Cloudflare: Lower latency, no server load
- Server-Side: More control, requires server access
- Best: Cloudflare for edge tracking, server-side for detailed logs
Next Steps
After setting up Cloudflare Workers:
- Connect Google Analytics for additional insights
- Set up Looker Studio to visualize your data
- Review your prompts to optimize visibility
