Skip to content
Raindrop
Chat with us

Follow what happens after you ship.

Get updates on changes to your agent’s behavior every time you ship with Change Reports.

Connect to GitHubChat with us

Raindrop is trusted by

VercelSpeakClayFramerAngelListTolan

Raindrop watches
every PR.

Connect your GitHub repository. Raindrop reads the prompts, models and tools that make up each agent in your repo, and works out which production traffic belongs to each one.

When a pull request merges to your default branch, Raindrop works out which agents it changed and how:

  • Tools added or removed
  • Model or settings changed
  • Instructions substantially changed
Merged pull requestsAgent changes found
Each merged pull request, matched to the agents it changed.

Sometimes nothing going wrong
is something going wrong.

RaindropAPP9:35 AM

Change Report: Added MCP guidance to the system prompt

sprinkle-ai/assistant · #291 · Merged by @Jackson · 1 hour ago

The updated system prompt has been observed in traces, and the advice has been offered to multiple users.

Some important notes:

  • MCP Guidance: The agent explained how to connect MCP to 23 users.
  • 7 users connected MCP after the change went live.
View change report →Snooze
RaindropAPP9:35 AM

Change Report: Lowered compaction threshold to 148k tokens

sprinkle-ai/assistant · #284 · Merged by @Jackson · 1 hour ago

The updated compaction settings are now live (first observed 48 minutes ago). Since then, 238 users have had conversations that triggered compaction.

Some important notes:

  • Slightly elevated forgetting after compaction: Users reminded the agent of earlier context: “i said that already” and “i put the plan above”.
  • Cost has barely budged: $1,048 vs. ~$1,069 previously (2% lower).
View change report →Snooze

An early check,
then the full report.

Reports go to your project’s Slack channel, or to the PR author’s DMs. Each one compares against an equal-length window before the change. If the agent saw no usage, nothing is sent.

Early check

Soon after the merge. If anything in the first stretch of traffic plausibly relates to the change (the new tool, the changed rule, the new model), you hear about it right away. If nothing does, no message.

Full report

After the change has been live for a while: a plain-language verdict, how many events and users the agent handled, and whether each changed tool was actually called and by how many users.

Highlights and lowlights

Negative signals on events that used the change, related stumbles, and quotes from real conversations, so you can see the evidence behind each claim.

Questions

What counts as a change?
Anything in your agent's harness: tools added or removed, a model or settings change, instructions substantially changed, cache settings. Raindrop reads each pull request merged to your default branch and works out which agents it changed and how.
When do reports arrive?
An early check soon after the merge, if anything in the first stretch of traffic plausibly relates to the change. The full report once the change has been live for a while. Nothing is sent if the agent saw no usage.
What if the change barely gets used?
That is often the point. A web search tool that never gets called, or a caching change that doesn't move cost, never raises an issue. The report tells you whether each changed tool was actually called, and by how many users.
How is this different from Simulations?
Simulations predict what a change will do before it ships. Production is the source of truth, and many changes can only be monitored there. Change Reports cover that side, in the hours after the merge.
Where do reports go?
Your project's Slack channel, or the PR author's DMs (falling back to the channel when the author has no matching Slack account). Every report also lives in Briefs, with the conversations behind each claim.
What do I need to set up?
Connect GitHub from Briefs → Change Reports and turn on change reports per agent under Settings → Notifications. Log the commit SHA with your events (recent SDKs do this automatically) so usage is scoped by commit rather than by time.

Build with the evidence

Find the next failure.
Verify the next fix.

Start building Book a demo

Start free. No credit card required.