Bot tracking

Crawlers and AI agents do not run JavaScript, so the browser script never sees them. To count them, your server sends one small request to twelvebit for each request from a known bot.

The three kinds#

Kind	What it means	Bots
`ai_answer` (AI answers)	An assistant fetches the page because a person asked about it now. These visits are the closest thing to a reader from ChatGPT, Claude or Perplexity.	ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, PerplexityBot, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, YouBot
`indexing` (Search indexing)	A search engine crawls the page for its index.	Googlebot, Bingbot, DuckDuckBot, Applebot, YandexBot, Baiduspider, Yahoo Slurp, Bravebot, Kagibot, SeznamBot, PetalBot, Qwantbot
`training` (AI training)	A crawler collects the page as training data for a model.	GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, FacebookBot, Amazonbot, cohere-ai, cohere-training-data-crawler, Diffbot, AI2Bot, Omgilibot, ImagesiftBot, Timpibot, PanguBot

Unknown user agents are ignored, so you can send every request and twelvebit keeps only the bots on this list. The list is maintained on the server, so it stays current without changes to your code.

The endpoint#

curl -sS -X POST https://twelvebit.app/api/v1/bots -H "Authorization: Bearer $TWELVEBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"user_agent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)", "path": "/pricing"}'

Field	Type	Notes
`user_agent`	string	Required. The request's `User-Agent` header.
`path`	string	The requested path. A full URL is reduced to its path.

The response is 202 with { "tracked": true, "bot": "GPTBot", "kind": "training" }, or { "tracked": false } for a user agent that is not on the list. The limit for this endpoint is 1,200 requests per minute per key. Read the counts with GET /api/v1/bots?period=30d.

Use a site key (tbk_…) from the environment. Never send the key to the browser.

Next.js (proxy.ts)#

Next.js 16 calls middleware "proxy". Put this file next to your app folder (in src/ if you use it). event.waitUntil sends the request after the response, so it adds no delay. If you already have a proxy.ts (or middleware.ts in Next.js 15 and earlier), add the if block to your function.

// proxy.ts
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

// Known crawlers and AI agents. Other requests are not sent.
const BOTS = /chatgpt-user|oai-searchbot|claude-user|claude-searchbot|perplexity-user|perplexitybot|duckassistbot|mistralai-user|meta-externalfetcher|youbot|gptbot|claudebot|anthropic-ai|ccbot|google-extended|applebot-extended|bytespider|meta-externalagent|facebookbot|amazonbot|cohere-ai|cohere-training-data-crawler|diffbot|ai2bot|omgilibot|imagesiftbot|timpibot|pangubot|googlebot|bingbot|duckduckbot|applebot|yandexbot|baiduspider|slurp|bravebot|kagibot|seznambot|petalbot|qwantbot/i;

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const ua = request.headers.get("user-agent") ?? "";
  const key = process.env.TWELVEBIT_API_KEY;
  if (key && BOTS.test(ua)) {
    event.waitUntil(
      fetch("https://twelvebit.app/api/v1/bots", {
        method: "POST",
        headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
        body: JSON.stringify({ user_agent: ua, path: request.nextUrl.pathname }),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

export const config = {
  // Skip build files and images. robots.txt, sitemap.xml and llms.txt still match.
  matcher: ["/((?!_next/static|_next/image|favicon.ico|.*\\.(?:png|jpe?g|gif|webp|avif|svg|ico|css|js|woff2?)$).*)"],
};

Express#

// Node.js 18+ has fetch built in.
const BOTS = /chatgpt-user|oai-searchbot|claude-user|claude-searchbot|perplexity-user|perplexitybot|duckassistbot|mistralai-user|meta-externalfetcher|youbot|gptbot|claudebot|anthropic-ai|ccbot|google-extended|applebot-extended|bytespider|meta-externalagent|facebookbot|amazonbot|cohere-ai|cohere-training-data-crawler|diffbot|ai2bot|omgilibot|imagesiftbot|timpibot|pangubot|googlebot|bingbot|duckduckbot|applebot|yandexbot|baiduspider|slurp|bravebot|kagibot|seznambot|petalbot|qwantbot/i;

app.use((req, res, next) => {
  const ua = req.get("user-agent") || "";
  if (process.env.TWELVEBIT_API_KEY && BOTS.test(ua)) {
    // Fire and forget: do not await, do not block the response.
    fetch("https://twelvebit.app/api/v1/bots", {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.TWELVEBIT_API_KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify({ user_agent: ua, path: req.path }),
    }).catch(() => {});
  }
  next();
});

Other servers#

Any backend works: for a request whose User-Agent matches the pattern above (case-insensitive), send POST /api/v1/bots with user_agent and path, in the background, with a short timeout.