# Bot tracking

Crawlers and AI agents do not run JavaScript, so the browser script never sees them. To count them, your server sends one small request to twelvebit for each request from a known bot.

## The three kinds

| Kind | What it means | Bots |
|---|---|---|
| `ai_answer` (AI answers) | An assistant fetches the page because a person asked about it now. These visits are the closest thing to a reader from ChatGPT, Claude or Perplexity. | ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot, Perplexity-User, PerplexityBot, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher, YouBot |
| `indexing` (Search indexing) | A search engine crawls the page for its index. | Googlebot, Bingbot, DuckDuckBot, Applebot, YandexBot, Baiduspider, Yahoo Slurp, Bravebot, Kagibot, SeznamBot, PetalBot, Qwantbot |
| `training` (AI training) | A crawler collects the page as training data for a model. | GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, FacebookBot, Amazonbot, cohere-ai, cohere-training-data-crawler, Diffbot, AI2Bot, Omgilibot, ImagesiftBot, Timpibot, PanguBot |

Unknown user agents are ignored, so you can send every request and twelvebit keeps only the bots on this list. The list is maintained on the server, so it stays current without changes to your code.

## The endpoint

~~~sh
curl -sS -X POST https://twelvebit.app/api/v1/bots -H "Authorization: Bearer $TWELVEBIT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"user_agent": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)", "path": "/pricing"}'
~~~

| Field | Type | Notes |
|---|---|---|
| `user_agent` | string | Required. The request's `User-Agent` header. |
| `path` | string | The requested path. A full URL is reduced to its path. |

The response is `202` with `{ "tracked": true, "bot": "GPTBot", "kind": "training" }`, or `{ "tracked": false }` for a user agent that is not on the list. The limit for this endpoint is 1,200 requests per minute per key. Read the counts with `GET /api/v1/bots?period=30d`.

Use a site key (`tbk_…`) from the environment. Never send the key to the browser.

## Next.js (proxy.ts)

Next.js 16 calls middleware "proxy". Put this file next to your `app` folder (in `src/` if you use it). `event.waitUntil` sends the request after the response, so it adds no delay. If you already have a `proxy.ts` (or `middleware.ts` in Next.js 15 and earlier), add the `if` block to your function.

~~~ts
// proxy.ts
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

// Known crawlers and AI agents. Other requests are not sent.
const BOTS = /chatgpt-user|oai-searchbot|claude-user|claude-searchbot|perplexity-user|perplexitybot|duckassistbot|mistralai-user|meta-externalfetcher|youbot|gptbot|claudebot|anthropic-ai|ccbot|google-extended|applebot-extended|bytespider|meta-externalagent|facebookbot|amazonbot|cohere-ai|cohere-training-data-crawler|diffbot|ai2bot|omgilibot|imagesiftbot|timpibot|pangubot|googlebot|bingbot|duckduckbot|applebot|yandexbot|baiduspider|slurp|bravebot|kagibot|seznambot|petalbot|qwantbot/i;

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const ua = request.headers.get("user-agent") ?? "";
  const key = process.env.TWELVEBIT_API_KEY;
  if (key && BOTS.test(ua)) {
    event.waitUntil(
      fetch("https://twelvebit.app/api/v1/bots", {
        method: "POST",
        headers: { Authorization: `Bearer ${key}`, "Content-Type": "application/json" },
        body: JSON.stringify({ user_agent: ua, path: request.nextUrl.pathname }),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

export const config = {
  // Skip build files and images. robots.txt, sitemap.xml and llms.txt still match.
  matcher: ["/((?!_next/static|_next/image|favicon.ico|.*\\.(?:png|jpe?g|gif|webp|avif|svg|ico|css|js|woff2?)$).*)"],
};
~~~

## Express

~~~js
// Node.js 18+ has fetch built in.
const BOTS = /chatgpt-user|oai-searchbot|claude-user|claude-searchbot|perplexity-user|perplexitybot|duckassistbot|mistralai-user|meta-externalfetcher|youbot|gptbot|claudebot|anthropic-ai|ccbot|google-extended|applebot-extended|bytespider|meta-externalagent|facebookbot|amazonbot|cohere-ai|cohere-training-data-crawler|diffbot|ai2bot|omgilibot|imagesiftbot|timpibot|pangubot|googlebot|bingbot|duckduckbot|applebot|yandexbot|baiduspider|slurp|bravebot|kagibot|seznambot|petalbot|qwantbot/i;

app.use((req, res, next) => {
  const ua = req.get("user-agent") || "";
  if (process.env.TWELVEBIT_API_KEY && BOTS.test(ua)) {
    // Fire and forget: do not await, do not block the response.
    fetch("https://twelvebit.app/api/v1/bots", {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.TWELVEBIT_API_KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify({ user_agent: ua, path: req.path }),
    }).catch(() => {});
  }
  next();
});
~~~

## Other servers

Any backend works: for a request whose `User-Agent` matches the pattern above (case-insensitive), send `POST /api/v1/bots` with `user_agent` and `path`, in the background, with a short timeout.

---

Source: https://twelvebit.app/docs/bots · All docs: https://twelvebit.app/llms.txt