Raindrop

How to Evaluate AI Agent Behavior in Production

For AI crawlers and low-bandwidth access.

TL;DR

  • To evaluate an agent in production, pick the behaviors that matter and measure how often each happens on live traffic.
  • An LLM judge reads every trace it scores, so its cost grows with traffic, trace length, and the number of behaviors. Most teams sample to keep the bill down.
  • Raindrop Signals build a classifier for each behavior you describe. Code settles most runs; a model reads only the ambiguous ones. A saved Signal runs on every new event. No sampling.
  • When you compare platforms, ask about coverage, cost at your volume, trends, release comparisons, and access to the underlying runs.

What does it mean to evaluate agent behavior in production?

It means turning a question into a rate. "Is the agent good?" can't be measured. "How often does the agent say a refund is done when no refund call succeeded?" can: 2.1% of conversations this week, up from 0.8% last week.

A pre-release eval scores a dataset someone wrote. A production eval scores what real users asked, including requests no dataset anticipated. You need both.

Why does an LLM judge get expensive?

A judge is a prompt that reads a trace and answers yes or no. It needs no training data and handles nuance. Braintrust's online scoring and Langfuse's LLM-as-a-judge evaluators both work this way.

The judge reads the whole trace every time:

judge cost = runs × tokens per trace × behaviors × price per token

Take an illustrative month: 500,000 runs, 40,000 tokens per trace, 5 behaviors, and $1 per million input tokens. That is 100 billion tokens, or about $100,000, before output tokens and retries.

So teams sample. Langfuse's guide says to "lower the sampling rate if needed," and Braintrust rules take a sampling_rate. A 5% sample cuts the bill to about $5,000 and cuts what you see by the same amount:

  • Rare failures vanish. A failure in 0.2% of runs leaves about 50 examples a month. Split those by customer or release and most slices are empty.
  • Small accounts go dark. A failure that hits one customer may never reach the sample.
  • Trends get noisy. Small daily counts hide real regressions.

How do Raindrop Signals work?

A Signal labels every event that shows a behavior, so you get a rate over time and the conversations behind each match.

  1. Describe the behavior in one sentence, such as "the agent says a refund is complete when no refund call succeeded."
  2. Review the draft. Raindrop reads your traces, writes policy rules, builds a classifier, and shows real matches. Label a few Match or Not Match.
  3. Save it. The Signal labels every new event and backfills the last 90 days, sampled, so you see a rate on day one.

You can also create Signals from Claude Code, Cursor, Codex, or Devin through the Raindrop MCP server. When a Signal gets a match wrong, select the events, click Refine Signal, and say what to change.

A Signal can check facts, judge meaning, or do both:

TypeWhat decides itExample
DeterministicCode that reads tool calls, inputs, results, and orderThe same tool failed three times with identical inputs
SemanticA model that reads the conversationThe user is frustrated
BothCode gathers evidence; a model judges itEvery refund call failed, and the agent said the refund is complete

The third row explains the cost difference. Signals 2.0, built on rd-signal-2, reasons once, when it builds the classifier. For the refund Signal, code checks each trace for a successful refund call. Most runs have one or never mention refunds, and code settles them without a model. Only runs with a failed call and a refund claim go to the model, and it reads only those calls and the reply.

judge cost  = traffic × full-trace reasoning
rd-signal-2 = build once + code on every trace + ambiguous cases × compact context

On Raindrop's launch evaluation in August 2026, rd-signal-2 reached 71% precision and 87% recall at 1/1,600 of the per-trace cost of GPT-5.6 Sol xhigh, which reached 91% recall at the same precision. Each day, frontier models check a sample of live classifications; when they find drift, Raindrop retunes the classifier. Signals are included in Raindrop, with no inference bill and no model key to bring.

Seven presets need no drafting: User Frustration, Task Failure, Forgetting, Refusals, Jailbreaking, NSFW, and User Praise. Tool errors, SDK feedback such as thumbs up and down, and Self Diagnostics reports also arrive as Signals.

What do Signals not do?

Signals measure behaviors you chose. Other parts of Raindrop cover the rest.

Question it answers
TracesWhat happened in this run?
SignalsHow often does this behavior happen?
Issue DetectionWhat recurring problem started that nobody defined?
StumblesWhich single conversation went wrong?
SimulationsWhat will this change do before I merge it?

When Issue Detection finds a failure worth tracking, turn it into a Signal and watch whether it returns.

How do you compare behavior across releases?

Experiments compare two groups of events by date, model, prompt version, feature flag, or property. Pick the Signals that define success, and Raindrop shows each group's rate, the difference, a 95% confidence interval, and a verdict such as likely worse. Experiments need Pro.

Change Reports start from a merged pull request and track the Signals and Stumbles in the traffic that followed. Signal Alerts post to Slack when a rate crosses a threshold.

Say you move the refund agent to a faster model behind a flag. A week later, an Experiment shows the false-refund rate is higher with the flag on, and the interval excludes zero. You open the matches and read the conversations where the new model said "refund complete" after a timeout. A judge on a 5% sample would give you a twentieth of those matches.

What should you ask a vendor?

Ask each vendor the same questions, with your own traffic numbers.

AskLLM judge on sampled trafficRaindrop Signals
Coverage: which runs get labeled?The share you can afford to judgeEvery new event
Cost: what does one more behavior cost at your volume?Another full-trace call per judged runIncluded; custom Signals use plan slots
Trends: can you split a rate and still trust it?Noisy for rare behaviorsA rate over time with every match
Releases: can you compare variants with a confidence interval?Possible, with fewer matchesExperiments and Change Reports
Runs: can you open the conversations behind a number?Usually, for judged tracesEvery match, with Refine Signal
Accuracy: what has the vendor published?Depends on judge and prompt71% precision, 87% recall

When is an LLM judge still the right call?

Use a judge when volume is low, when the question is subjective and changes often, or when you score a fixed dataset before release. At a few thousand runs a month, judging every run may cost less than building anything else. Signals pay off when the agent is busy, traces are long, and you track the same behaviors every day.

FAQs

Do Signals sample production traffic? No. A saved Signal runs on every new event. Only the one-time 90-day backfill is sampled.

Is a Signal just a prompt sent to an LLM? No. rd-signal-2 builds a classifier per behavior. Code settles the clear cases; a model trained for that behavior handles the rest.

Which plans include Signals? Every plan. Hobby (free) includes 1 custom Signal, Pro includes 10, and Enterprise sets custom limits. Presets, tool errors, SDK feedback, and Self Diagnostics never count toward the limit. See current plans.

How to test this on your own traffic

Pick one behavior you have seen in a support ticket. Write it as a Signal and as a judge on the platform you're comparing, and run both for a week. Compare the matches each found, the judge's cost at the sampling rate you would actually use, and whether you could split the rate by release and open the runs. Get started for free.