Simulations: Now in Public Beta

We launched Simulations in Early Access in September. Since then, customers have run more than 33,000 simulations across more than 1,700 pull requests. Teams at companies like Caddy and Leap CRM are using Simulations to test changes to their agents before they ship.

Today, Simulations is available in Public Beta.

A change to your agent's prompt, model, tools, or code might fix one issue while introducing unexpected behavior somewhere else. Evals help you test for known issues, but they depend on predefined test cases. You can't anticipate every scenario your agent will encounter in production.

With Simulations, you can test changes against real production scenarios before you merge, so you can see how your agent's behavior changes across different workflows.

How it works

Even with production traces, testing changes to agent behavior is difficult. A trace captures one execution path. If your updated agent calls a different tool or takes a new path, replaying that trace won't capture the new behavior.

Raindrop uses historical production traces to recreate your agent's tools, data, and external services in a simulated environment. Your agent's code runs in an isolated sandbox without access to production services or credentials, using its real prompts, models, and agent loop. Raindrop simulates interactions with external tools and services, keeping the environment consistent even when your agent takes a different path.

Simulations on every PR

By default, Simulations runs when you open a pull request that changes your agent's prompts, tools, or model settings. You can also comment @raindrop simulate to run or rerun a simulation.

Raindrop replays recent production conversations against your updated agent and compares the results with what actually happened in production. Results appear directly in GitHub, showing differences in responses, tool calls, execution paths, and metrics such as tool failures and execution duration.

You can verify whether your change fixes the intended issue, investigate unexpected changes across other workflows, and iterate before merging.

The harness playground

Test changes to your agent directly in Raindrop's Prompt Playground, without opening a GitHub PR. Traditional prompt playgrounds were built for chatbots, not agents that use tools, skills, and complex workflows. Raindrop runs your full agent harness in a simulated environment, so you can replay real production scenarios, see how changes affect your agent's behavior, and compare results side by side with what actually happened in production.

Early customers

Caddy: Getting agent behavior right before shipping

“Simulations shows me whether a prompt change actually fixes the problem I’m trying to solve. I can test it against realistic conversations, see what worked and what didn’t, and keep iterating before I ship.”

Chris Catanese

Founding Engineer, Caddy

Caddy is a personal AI assistant that helps people manage everyday tasks through text messages, from drafting emails to tracking flight prices.

Caddy's engineering team noticed that its agent sometimes misinterpreted sarcasm in users' messages. They needed to update its prompt to recognize jokes without mistakenly treating serious messages as sarcasm.

Chris, Caddy's founding engineer, used Simulations to test each version of the updated prompt against real production conversations. Early iterations caught some jokes but missed others. After three or four rounds, the agent was consistently recognizing sarcasm without misinterpreting ordinary messages.

Leap CRM: From production issues to verified fixes

“Raindrop helps us go from finding an issue in production to verifying the fix before it ships. Our coding agents generate PRs, and Simulations lets us replay the original user issue in a full world to see whether the change actually improves the agent’s behavior. It’s especially valuable now that agents are writing most of our PRs.”

Mel George

Leap CRM, Leap Legal

Leap Legal provides legal practice management software to law firms worldwide. Its Leap CRM team uses Raindrop to find issues in production and test fixes before merging.

Recently, Raindrop identified an issue where Leap CRM's AI agent couldn't clearly explain how CRM statuses and stages worked. The team created a PR, but Simulations showed the response was still unclear. They refined the fix, ran another simulation, and shipped the improvement.

Available now

To get started, connect your GitHub repository.