Describe what matters. Raindrop builds a classifier, learns your definition from real examples, and watches for that behavior across production. From subtle failures to the moments your agent gets it right.
SignalsAgent responded to web request without browser tool
NegativeTracking
Agent responded to web request without browser tool
The agent answered a request to check the web without using its browser tool.
2.8%of events match
Last 7 days
MonTueWedThuFriSatSun
Policy
Match when the user explicitly asks for a web lookup and the agent gives a substantive answer without calling an available browser tool for that request.
Matched events84 of 3,000
One behavior. A measurable rate. The conversations behind every match.Inside rd-signal-2
A practical guide
From a sentence to a production Signal.
Start with a behavior you have actually seen. The best specification includes both an example that should match and one that should not.
1
Describe the behavior.
Open Signals → New Signal. Be specific about what should count, and what should not. Raindrop drafts a policy and classifier from your traces.
2
Review real examples.
Label matches and non-matches. Explain edge cases in chat, then rerun the draft until it reflects your intent.
3
Read the rules. Save.
Review the policy and classifier code. Choose a name and sentiment, then create the Signal.
4
Track and refine.
New events are scored automatically, with a sampled 90-day backfill. Correct mistaken matches to refine the live Signal.
Signals connect what your agent did with what it meant. Code follows the sequence of tool calls, inputs, and results. Semantic reasoning reads the conversation against that evidence to decide whether the behavior matches.
Unsupported refund confirmationExample Signal
TrajectoryWhat happened
User request“Please refund order ORD-2048.”
1
create_refundFirst attempt
{ order_id: "ORD-2048" }PAYMENT_GATEWAY_TIMEOUT
2
create_refundRetry
{ order_id: "ORD-2048" }PAYMENT_GATEWAY_TIMEOUT
Two attempts. No successful refund result in the trace.
Semantic reasoningWhat it means
Assistant response
“Your refund is complete. The money will reach your card in 5–10 days.”
The assistant tells the user their requested refund is complete.
Read the reply in the context of the request and the tool results.
Signal matches
The trace contains no successful refund result, but the response presents the refund as complete. Together, they match this Signal.
Follow the trace. Interpret the response.
This Signal checks whether the agent claims a refund is complete without a successful tool result to support it. Follow each step to see how the trajectory and conversation come together.
"The final reply claims the requested refund is complete.",
{spanIds:refundCalls.flatMap(call=>call.spanIds)}
);
}
Production economics
Enough judgment to be useful. Cheap enough to keep on.
In our published evaluation, rd-signal-2 reached 71% precision and 87% recall at 1/1,600 of the per-trace cost of GPT-5.6 Sol xhigh.
The tradeoff is visible: Sol reached 91% recall at the same precision. These are results from our evaluation, not a guarantee for every behavior or dataset.
A new model, a different harness, or an edge case can change what your traces look like. A classifier needs attention even when its definition has stayed the same.
Raindrop samples production classifications daily with frontier models. When it finds drift or regressions, it reruns prompt optimization and retunes. You can also inspect and adjust the code and prompt yourself.
Evaluate. Refine. Repeat.
Production examples feed the next iteration. Your corrections make the intended boundary more explicit.
Isolated by design.
Generated Signals run without credentials or internet access. Each organization’s runtime can access only its own data.
Share context, save work.
Trace context is fetched once and cached near evaluation workers so multiple Signals can reuse it.
MCP & automation
Your coding agent can build Signals too.
Ask Claude Code, Codex, Cursor, or Devin to turn a production finding into a Signal. Raindrop’s MCP exposes the authoring workflow, examples, and refinement tools inside the environment where you already work.
Raindrop MCPExample workflow
“Create a Signal for agents answering web lookup requests without using their browser tool. Show me the examples before saving it.”
1
start_signal_session
Draft a classifier and scan real traffic.
2
label_signal_batch · refine_signal_session
Review examples in an MCP app or in chat. Tighten the definition.
3
close_signal_session
Confirm the Signal and start tracking the behavior.
A recurring behavior you want to measure: an agent repeating failed tool calls, forgetting something the user told it, inventing a successful result, or recovering from an error. Positive outcomes matter just as much as failures.
Is a Signal just a prompt sent to an LLM?
rd-signal-2 builds a classifier for the behavior. Code gathers the relevant evidence and handles deterministic checks. A task-specific classification head and semantic reasoning model handle the judgment that remains.
Can a Signal read more than a single message?
Yes. Signals can gather context across tool calls, multiple turns, and longer sessions. This lets them detect a relationship between events, such as failed tool calls followed by an inaccurate claim of success.
How do Signals relate to issue detection?
A Signal tracks a defined behavior over time. Issue detection investigates emerging changes in production. Turn a discovered failure into a Signal to track whether it returns.
Can I create and refine Signals through MCP?
Yes. Coding agents can draft, review, create, and refine Signals through Raindrop MCP. Creating and refining requires signal write access. Read the MCP guide.
Can I use the classifier outside Raindrop?
Signals 2.0 also introduces direct API access and Signal Builder for custom classifiers, including Zero Data Retention options. Talk with the team about your use case.
