What are Signals
Signals are ground truth indicators for agent performance. They can be positive (eg. Thumbs Up), negative (eg. Agent Forgetting), or neutral (eg. Creating Personal Website).
Raindrop’s Default Signals
Raindrop has a set of default signals you can enable:- Forgetting: agent forgetting important information about the user
- Task Failure: agent failing to complete its task due to an issue or error
- User Frustration: user expressing frustration at the agent
- NSFW: user or AI trying to discuss explicit content, violence, or self-harm
- Jailbreaking: user attempting to jailbreak the AI by prompt injection or other means
- Laziness: agent returning outputs that were blank, empty, or low-effort
- Wins: user praising, thanking, or complimenting the assistant
Custom Signals
You can define any custom signal and see incident rates over millions of events. If you can describe it, Raindrop can track it.Classifier Signals
You can create Classifier Signals using Deep Search.1
Prompt Deep Search
Getting specific helps — eg. “users complaining about the website’s aesthetics” will provide better results than “ugly website”
2
Mark Examples
Review a couple examples by marking them as correct or incorrect so Deep Search can learn
3
Update Search
Hit the “Update Search” button and Deep Search automatically generates rules based on your refinements
4
Start Tracking
To track these signals, we’ve built a pipeline that trains small, custom models that adapt to the unique shape of each product and search. It takes about an hour to train the model and then it will review every single event you send to Raindrop. It will also backfill the previous 3 days.
5
(Optional) Additional Refinements
If you need to further refine the signal, you can hit the “x” on any event from its signal page to mark it as incorrect and give the model feedback.
Keyword / Regex Signals
You can define specific keywords or regex patterns to match in your events. This works great for:- Syntax / formatting patterns
- Specific phrases (eg. “you’re absolutely right”)
- Common behaviors (eg. the assistant apologizing)
Instrumented Signals
You can manually track any signal directly from your app using the SDK: Common manual signals include Thumbs Up / Down, Regenerations, and Shares.
Tool Errors
If you send traces to Raindrop, the application will automatically pull tool errors out into signals that you can track and set up alerts for.