rd-signal-2: Frontier Classification at Production Scale

Today we're launching Signals 2.0, powered by rd-signal-2. It builds binary classifiers that detect specific agent behaviors in production traces.

rd-signal-2 approaches GPT-5.6 Sol xhigh accuracy at 1/1,600 of the cost. It costs 1/260 as much as GPT-5.6 Luna xhigh.

Signals 2.0 is available to all Raindrop customers today at no additional cost.

We're also releasing Signal Builder for training and hosting custom classifiers with Zero Data Retention, including in environments with strict data requirements such as healthcare.

Defining what to detect

In "Why Language Models Hallucinate", OpenAI framed hallucination as a binary classification problem: deciding whether a statement is true or false.

A yes-or-no answer can still require a difficult judgment. For agent behavior, a team first has to agree on what counts as a failure and work through the edge cases. Then it needs a classifier that matches that definition.

Raindrop helps teams define, train, and run these classifiers.

Classifying agent behavior

In June 2025, we launched the first version of Signals: a pipeline for training small classification models.

Those classifiers evaluated a single input-and-output pair. That worked for chatbot behaviors where one turn contained all the evidence.

Agent failures can span multiple turns, tool calls, and subagents, sometimes across hundreds of thousands of tokens. Finding a rare failure in that history requires code to collect the relevant evidence and a model to interpret it.

Running a frontier model over every trace is expensive and slow at production scale. Smaller models cost less, but struggle with complex behaviors and long context.

For each behavior, rd-signal-2 studies production traces, writes code to collect the relevant context, and uses it to train a model for that task.

How rd-signal-2 builds Signals

In this trace, an agent calls the same tool three times. Every attempt times out, but the agent tells the user it succeeded:

Detecting this failure requires comparing the tool results with the final response. rd-signal-2 generates code to do that:

export function run(event) {
1  const attempts = findToolCalls(event.trace, "update_record");
 
2  const repeatedWithoutChange =
    attempts.length >= 3 &&
    haveIdenticalInputs(attempts) &&
    attempts.every(call => call.failed);
 
3  if (!repeatedWithoutChange) {
    return { matched: false };
  }
 
4  return classify(
    formatForReview(attempts, event.finalResponse),
    "The assistant claims the operation succeeded",
    { spanIds: attempts.flatMap(call => call.spanIds) }
  );
}

The code checks for at least three calls to the same tool, all with identical inputs and all failed. If those conditions aren't met, the Signal returns a non-match without calling a model.

Otherwise, it passes the failed attempts and the final response to a task-specific classification head and our in-house reasoning model. Together, they judge whether the assistant claimed success.

This example fits within one turn. Signals can also collect context from sessions that span days or weeks.

Reasoning at build time

rd-signal-2 does most of its reasoning while building a classifier. It tries several approaches and checks the results against the customer's definition before deploying one.

A full-trace LLM judge processes the trace each time it runs:

Prompt classifier cost = traffic × full-trace reasoning

With rd-signal-2, each trace runs through the generated code. Only cases that need a model are sent for classification, with the relevant context already extracted:

rd-signal-2 cost = build once + deterministic execution
                    + ambiguous candidates × compact context

This keeps Signals inexpensive enough to include in Raindrop and run across billions of traces per month, without a separate inference bill.

Working through edge cases

The hardest part is often defining what the classifier should detect. For the repeated tool-call example, a team needs to decide how to handle cases where:

• the arguments changed slightly but the strategy didn't;
• the first three attempts failed but the fourth succeeded;
• the agent admitted that the operation failed;
• a subagent completed the task.

Those decisions change which traces match.

We wrote four variations of the same one-sentence behavior and ran each over the same 2,000 production traces. Match rates ranged from 0.9% to 4.6%. Of the traces matched by at least one variation, 67% were rejected by another. We're continuing to research how to make classifiers match the user's intent more closely.

Keeping Signals accurate

Changes to models and harnesses affect the traces a Signal sees. Customers also find gaps in their definitions as they review more data. A Signal can become less accurate even when its code hasn't changed.

Each day, we use frontier models to evaluate a random sample of production Signals. When we find drift or a regression, we rerun prompt optimization and retune the classifier.

Customers can also inspect and edit the code and prompt behind each Signal.

Running Signals at scale

Running millions of evaluations a day requires reliable model serving, queueing, and retries.

We treat generated Signal code as untrusted. Each customer's Signals run in an isolated environment with no credentials or outbound internet access. The runtime can access only that organization's data.

We fetch trace context once and cache it near the evaluation workers so multiple Signals can reuse it. Each Signal runs its code and sends only the cases that need a model for classification.

At production scale, the median trace is classified in 100 ms. This infrastructure currently evaluates over 20 billion traces per month.

Using Signals across Raindrop

Raindrop Issue Detection uses the same architecture to find and track new failure patterns. Customers can turn any detected issue into a Signal, refine its definition, and use it in Experiments to A/B test changes.

You can create Signals yourself, ask Triage Agent to build them, or use your coding agent through MCP. Every Signal runs through the same evaluation and monitoring infrastructure.

Creating a Signal with a coding agent

Signals 2.0 API

Developers can call the Signals API to detect behaviors in their own systems.

For teams with stricter data requirements, ZDR Signals supports training and running classifiers without retaining production data.

Try Signals 2.0

Create a Signal in Raindrop, or schedule a call with our team to build one together.