Agent Detection

Product · Developer tool by RudderStack

Detect AI agents, crawlers, and scrapers in your event stream.

Agent Detection is an intelligent bot classification tool for RudderStack that labels event traffic by type and intent in real time. It helps data teams distinguish between human users, helpful AI agents, and harmful scrapers before data reaches downstream tools.

Languages
English

Category: Data Governance

Classification Categories
5 (Human, AI Agent, AI Crawler, Search Crawler, Scraper)
Intent Score Range
0.0 to 1.0
Confidence Score Range
0.0 to 1.0
Deployment Mode
Event Stream Transformation
Supported Models
Jev (hosted), Laya (self-hosted), Kev (self-hosted)
Integration Count
200+ downstream destinations
  • Classifies traffic into 5 categories: human, AI agent, crawler, search, and scraper
  • Provides a 0.0 to 1.0 probability score for bad intent and kind confidence
  • Runs in-flight as an Event Stream Transformation before data hits the warehouse
  • Supports self-hosted model endpoints for enhanced data privacy

Overview

Agent Detection is a specialized transformation product for RudderStack Event Stream that identifies and labels automated traffic in real time. It classifies every incoming event into five distinct categories: human, AI agent, AI crawler, search crawler, and scraper. By enriching events with a "bad intent" probability score and a "kind confidence" metric, it allows businesses to distinguish between helpful AI assistants acting for users and harmful scrapers that skew analytics. The product runs in-flight, ensuring data is cleaned and labeled before it reaches the data warehouse or any of over 200 downstream integrations. Developers can manage the detection as plain transformation code, adjusting thresholds or swapping between hosted models like Jev and self-hosted models like Laya to meet privacy requirements.

Detection and Labeling

  • Traffic Classification: Classifies every event into one of five categories: human, AI agent, AI crawler, search crawler, or scraper.
  • Acts-for-User Flagging: Flags events where a real person is behind an automated action, such as a shopping assistant browsing on their behalf.
  • Kind Confidence Metric: Provides a probability score from 0 to 1 indicating the model's certainty about the assigned traffic category.

Implementation and Control

  • In-Flight Processing: Operates as a Transformation template within the RudderStack Event Stream to classify data before it lands in downstream tools.
  • Schema Compatibility: Adds a classification object to the event context without altering existing schemas, ensuring downstream destinations continue to function.
  • Automated Actioning: Enables developers to route, filter, or drop events based on the detected category and intent score directly in the transformation code.

Downstream Use Cases

  • Marketing Optimization Optimization: Keeps scrapers and automation out of conversion events and retargeting audiences to optimize ad spend.
  • Product Analytics Integrity: Splits funnels and experiments by category so bot traffic stops skewing product usage results.

Use cases

  • Marketing teams keep scrapers and automation out of conversion events to ensure cleaner ad signals.
  • Product analysts split funnels by category so bot traffic stops skewing real usage metrics.
  • Data engineers manage bot detection as code, adjusting thresholds and model endpoints within transformations.

Specifications

Integrations
RudderStack Event Stream, Jev, Laya, Kev, and 200+ downstream destinations
Deployment
Cloud or self-hosted model endpoints (Laya, Kev) via Event Stream Transformation
Classification Categories
5 (Human, AI Agent, AI Crawler, Search Crawler, Scraper)
Intent Score Range
0.0 to 1.0
Confidence Score Range
0.0 to 1.0
Deployment Mode
Event Stream Transformation
Supported Models
Jev (hosted), Laya (self-hosted), Kev (self-hosted)
Integration Count
200+ downstream destinations

How it works

  1. Collect: Events arrive from web, mobile, and server SDKs through the RudderStack Event Stream.
  2. Classify: The transformation sends request signals to a System 1 model (hosted or self-hosted) for a fast verdict.
  3. Enrich: Classification fields like isBot, category, and badIntent are written to the event context.
  4. Act: The labeled data is routed, filtered, or passed through to the warehouse and over 200 destinations.

FAQ

Why not just block all bots?

AI agents now browse and buy for real people, and search crawlers affect discovery; classification allows businesses to keep desired traffic while blocking harmful bots.

What is a System 1 model?

It is a small, fast classifier built for quick, pattern-based decisions, suitable for running on every event in a data pipeline.

Can I run inference in my own infrastructure?

Yes, the transformation can be pointed at a self-hosted endpoint so request data never leaves the customer's environment.

Does it break existing tracking?

It only adds a classification object to the event context, meaning existing destinations, tracking plans, and warehouse models continue to work.

Read the docs

Added by AntSpace from public sources (rudderstack.com).

Related

More products by RudderStack