Build a self-healing loop

Raindrop carries the evidence; your coding agent investigates, reproduces, and patches; your repository owns the check, review, merge, and deploy.

What you will build

In one pass you will name a production failure, inspect its trace, write a check that fails before the fix and passes after, and confirm with an experiment that production behavior changed. Run the loop manually once before automating it. Manual review shows where evidence or reproduction becomes unreliable.

Prerequisites

An instrumented application. See Integrations or run the raindrop-setup skill.
Issue Detection on Pro, or a signal alert on any plan.
Raindrop MCP connected to your coding agent.

Workshop:

curl -fsSL https://raindrop.sh/install | bash

A Query API key in the shell that runs Workshop:
export RAINDROP_QUERY_API_KEY=...
Create the read-only key in Settings. Never commit it or your ingest key.
A coding agent that supports MCP: Claude Code, Codex, Cursor, Devin, or OpenCode.

Step 1. Start from a detected failure

Open the issue or signal alert in Slack. An issue reports a broader distribution change, such as a failure spreading across users or a regression after a deploy. A signal alert fires when a defined behavior crosses its threshold.

Slack issue notification with the prompt for AI agents block and copy
affordance

Copy the prompt block into your coding agent. It already carries the evidence the agent needs: the affected tools and models, the event and user counts, the last-seen time, the root-cause hypothesis, and the issue ID. If Devin is connected to your organization, click Fix with Devin on the issue page to open a session with the same context preloaded.

Step 2. Read the evidence yourself (optional)

Open the issue and check its severity, trend, affected tools and models, event and user counts, and matching events.

Issue detail page showing severity, event and user counts, affected tools and models, activity
chart, and agent analysis

Two questions decide whether to continue:

Does the issue describe one failure or several? Issues can group a shared symptom with different causes.
Is a code change worth making? Ignore inaccurate or low-value issues.

Step 3. Give the agent a focused evidence request

Use the prompt block from your Slack alert, or ask the agent to investigate in this order:

Use the Raindrop MCP server to investigate issue <issue-id>.

1. get_issue for the report and root-cause hypothesis
2. get_issue_events to list the events behind it
3. get_event on two or three representative events
4. get_trace on the worst event for the full execution tree

Tell me the root cause and responsible code path.
Do not change code yet.

The raindrop-investigate skill packages the same flow:

npx skills add raindrop-ai/skills --skill raindrop-investigate

Step 4. Import the production trace into Workshop

Start Workshop:

raindrop workshop

Import the run and find the cause

Ask your coding agent:

Use the Workshop MCP tools to pull the failing run
and find what went wrong.

1. import_cloud_trace with event_id <event-id>
2. get_run_outline for the shape of the run
3. query_traces to find repeated or failing spans
4. get_span_payload on the span that went wrong
5. search_run to find a string across the payloads

Tell me where this run diverges from expected behavior.

import_cloud_trace needs a verified event ID and RAINDROP_QUERY_API_KEY. Workshop opens the run as a span tree: every model call, tool call, and sub-agent in the order they ran, with the duration of each beside it. This is the same execution you saw counted in the issue, now on your machine. Read the tree before opening payloads, since repeated tool calls, empty sub-agent results, and long spans usually point at the cause.

The imported run in Workshop's span tree, showing model calls, tool calls, and a sub-agent in
the order they ran

Mark what you found

Record the finding on the span with annotate. The note stays attached to the run, so your coding agent reads it as context on every later pass.

Step 5. Write a check that fails before the fix

Ask your coding agent:

Using the imported trace as evidence, write a test in
this repository that fails on this behavior.
Follow the existing test conventions.

Run it, show me the failure, then fix the code and
show me the passing result.

Keep the check in your own test suite. Your repository chooses the file, framework, and command.

Assert the tool-call sequence and forbidden calls.
Assert tool arguments.
Assert state after the run.
Assert structural output such as a citation or schema.

Your test command and CI own the check. Keep the suite small. A few checks that each encode a real production failure beat a large collected set, because production monitoring covers the long tail.

Step 6. Ship the fix through the normal review process

Open a pull request with the fix and check together. Review, merge, and deploy remain your repository’s process.

Step 7. Verify the change in production

After deploy:

Watch the signal or issue for a lower failure rate.
Run an experiment comparing cohorts before and after the change.
Break the comparison down by the relevant signals.

Experiments compare traffic you have already logged.

Experiment comparison showing negative signals dropping between Baseline and Experiment
cohorts

If the signal does not move, inspect events logged after deploy and return to Step 3.

Automate only after the check earns trust

Start with a reviewed manual run:

Manual trigger, agent investigates, human reviews the diff.
Alert triggers the agent; the pull request waits for review.
Agent opens pull requests unattended only where an existing check proves the fix.

Automation depends on check coverage. An agent that cannot show a failing-then-passing check should not merge code unattended.

Pitfalls

The check never failed. Run it before the fix and confirm the failure. The issue groups several causes. Split the investigation and trace each cause separately. The final response looks correct. Assert on the trajectory, not only the final message.

Issues and Signals for detection
Workshop for inspecting traces locally
MCP overview for production evidence
Experiments for production verification

Getting Started

SDK

Integrations

Platform

Organization

Cookbooks

Workshop

Query API

MCP

Security

Support

Build a self-healing loop

What you will build

Prerequisites

Step 1. Start from a detected failure

Step 2. Read the evidence yourself (optional)

Step 3. Give the agent a focused evidence request

Step 4. Import the production trace into Workshop

Import the run and find the cause

Mark what you found

Step 5. Write a check that fails before the fix

Step 6. Ship the fix through the normal review process

Step 7. Verify the change in production

Automate only after the check earns trust

Pitfalls

​What you will build

​Prerequisites

​Step 1. Start from a detected failure

​Step 2. Read the evidence yourself (optional)

​Step 3. Give the agent a focused evidence request

​Step 4. Import the production trace into Workshop

​Import the run and find the cause

​Mark what you found

​Step 5. Write a check that fails before the fix

​Step 6. Ship the fix through the normal review process

​Step 7. Verify the change in production

​Automate only after the check earns trust

​Pitfalls

​Related

What you will build

Prerequisites

Step 1. Start from a detected failure

Step 2. Read the evidence yourself (optional)

Step 3. Give the agent a focused evidence request

Step 4. Import the production trace into Workshop

Import the run and find the cause

Mark what you found

Step 5. Write a check that fails before the fix

Step 6. Ship the fix through the normal review process

Step 7. Verify the change in production

Automate only after the check earns trust

Pitfalls

Related