Skip to main content
Raindrop carries the evidence; your coding agent investigates, reproduces, and patches; your repository owns the check, review, merge, and deploy.

​
What you will build

Detect, Fix, Eval, Verify: the four stages of the self-healing loop with Raindrop surfaces labeled underneath
In one pass you will name a production failure, replay its trace against your branch, write a check that fails before the fix and passes after, and confirm with an experiment that production behavior changed. Run the loop manually once before automating it. Manual review shows where evidence or reproduction becomes unreliable.

​
Prerequisites

  • An instrumented application. See Integrations or run the raindrop-setup skill.
  • Issue Detection on Pro, or a signal alert on any plan.
  • Raindrop MCP connected to your coding agent.
  • Workshop:
  • A Query API key in the shell that runs Workshop:
    Create the read-only key in Settings. Never commit it or your ingest key.
  • A coding agent that supports MCP: Claude Code, Codex, Cursor, Devin, or OpenCode.

​
Step 1. Start from a detected failure

Open the issue or signal alert in Slack. An issue reports a broader distribution change, such as a failure spreading across users or a regression after a deploy. A signal alert fires when a defined behavior crosses its threshold.
Slack issue notification with the prompt for AI agents block and copy affordance
Copy the prompt block into your coding agent. It already carries the evidence the agent needs: the affected tools and models, the event and user counts, the last-seen time, the root-cause hypothesis, and the issue ID. If Devin is connected to your organization, click Fix with Devin on the issue page to open a session with the same context preloaded.

​
Step 2. Read the evidence yourself (optional)

Open the issue and check its severity, trend, affected tools and models, event and user counts, and matching events.
Issue detail page showing severity, event and user counts, affected tools and models, activity chart, and agent analysis
Two questions decide whether to continue:
  • Does the issue describe one failure or several? Issues can group a shared symptom with different causes.
  • Is a code change worth making? Ignore inaccurate or low-value issues.

​
Step 3. Give the agent a focused evidence request

Use the prompt block from your Slack alert, or ask the agent to investigate in this order:
The raindrop-investigate skill packages the same flow:

​
Step 4. Import the production trace into Workshop

Start Workshop:

​
Import the run and find the cause

Ask your coding agent:
import_cloud_trace needs a verified event ID and RAINDROP_QUERY_API_KEY. Workshop opens the run as a span tree: every model call, tool call, and sub-agent in the order they ran, with the duration of each beside it. This is the same execution you saw counted in the issue, now on your machine. Read the tree before opening payloads, since repeated tool calls, empty sub-agent results, and long spans usually point at the cause.
The imported run in Workshop's span tree, showing model calls, tool calls, and a sub-agent in the order they ran

​
Mark what you found

Record the finding on the span with annotate. The note stays attached to the run, so your coding agent reads it as context on every later pass.
Annotating a span in Workshop

​
Step 5. Replay the failure against local code

Ask your coding agent to scaffold replay:
The skill creates .raindrop/agents.yaml and a local HTTP endpoint with GET /health and POST /replay. Register it:
Replay from Workshop or your coding agent:
Replay needs a registered local endpoint. It leaves the imported production trace untouched and adds a second run beside it: the production trace stays as your reference, and the replay shows what your current code does with the same input. Expect the replay to fail here. That reproduction is the point of the step. If it diverges from production in any other way, fix the scaffold before trusting the result.

​
Step 6. Write a check that fails before the fix

Ask your coding agent:
Keep the check in your own test suite. Your repository chooses the file, framework, and command.
  • Assert the tool-call sequence and forbidden calls.
  • Assert tool arguments.
  • Assert state after the run.
  • Assert structural output such as a citation or schema.
Your test command and CI own the check. Keep the suite small. A few checks that each encode a real production failure beat a large collected set, because production monitoring covers the long tail. See Thoughts on evals.

​
Step 7. Ship the fix through the normal review process

Open a pull request with the fix and check together. Review, merge, and deploy remain your repository’s process.

​
Step 8. Verify the change in production

After deploy:
  • Watch the signal or issue for a lower failure rate.
  • Run an experiment comparing cohorts before and after the change.
  • Break the comparison down by the relevant signals.
Experiments compare traffic you have already logged.
Experiment comparison showing negative signals dropping between Baseline and Experiment cohorts
If the signal does not move, inspect events logged after deploy and return to Step 3.

​
Automate only after the check earns trust

Start with a reviewed manual run:
  1. Manual trigger, agent investigates, human reviews the diff.
  2. Alert triggers the agent; the pull request waits for review.
  3. Agent opens pull requests unattended only where an existing check proves the fix.
Automation depends on check coverage. An agent that cannot show a failing-then-passing check should not merge code unattended.

​
Pitfalls

Replay diverges from production. Fix missing tool behavior or dropped context before trusting the replay. The check never failed. Run it before the fix and confirm the failure. The issue groups several causes. Split the investigation and trace each cause separately. The final response looks correct. Assert on the trajectory, not only the final message.