Raindrop

How Opine finds and fixes silent agent failures

"It’s like having a full-time AI engineer working for you, watching everything all the time."
Charlie Duong
Charlie Duong
Co-Founder, Opine

Enterprise selling is complex. Deals span months and require coordination across multiple stakeholders on both the buyer and vendor side. Context is spread across tools like Salesforce, Slack, and Granola, making it hard to get the full picture of any deal.

Opine brings that context together. It turns transcripts, email threads, CRM records, Slack channels, and other data sources into a single, legible context layer, so revenue teams have the full picture of every deal in one place. On top of that layer, Opine runs agents for everything from instant deal briefings and live-call coaching to keeping the CRM up-to-date.

As their customers adopted these agents for more workflows, the Opine team needed to monitor and understand how they behaved in production.

The team used LangSmith for tracing and evaluations, and sent its own agents to crawl those traces for problems. That caught known issues, but only after the fact, at a high token cost, and only for scenarios someone had thought to define.

"We wanted a system watching for us, flagging the things we don’t even know to look for because we haven’t seen those behaviors yet."

Charlie Duong
Charlie Duong
Co-Founder, Opine

That’s when Opine turned to Raindrop.

Catching the unknown unknowns

Opine started testing Raindrop with a group of customers who rely heavily on its agents, and quickly found issues the team hadn’t caught. Raindrop flagged early signs of trouble as Stumbles and escalated recurring patterns into Issues, getting sharper as more traces came in.

One discovery was a subtle problem with deal search. When users asked Opine for “open deals” or “active opportunities,” its agent sometimes returned closed deals too. The answers looked plausible, and no customer had reported the problem.

Raindrop’s Filter Omission Signal caught it. Before the fix, the agent dropped the requested filter in 96–100% of searches mentioning “open” or “active.” After the fix, that rate fell to 0% in subsequent production measurements.

Another alert showed an agent reporting success while merging unrelated competitors. Companies with similar names were being treated as the same competitor, even though the agent reported that the matching had succeeded. After the team fixed the issue, replaying the same 140-name input showed zero incorrect merges, down from nine.

Other issues were just as difficult to spot. Raindrop caught an agent incorrectly claiming that an available editing tool couldn’t be used, leaving customer automations blocked. After the team investigated, blocked runs dropped by 80%.

"Raindrop’s alerts are really high-value."

Charlie Duong
Charlie Duong
Co-Founder, Opine

Fixing new features in early testing

Raindrop also helped Opine catch problems in newer features before they could become widespread.

One catch came from live-call coaching, a new Opine feature that helps reps handle objections mid-call. When a prospect mentioned problems with other vendors’ scaling and pricing, the assistant mistakenly treated those comments as objections to Opine and suggested rebuttals the rep didn’t need.

Raindrop flagged the behavior, and the team used the alert to pinpoint the issue and ship a fix while the feature was still in early testing with enterprise customers.

"It was a brand-new product we were testing with enterprise customers, so getting that feedback and fixing it quickly was really valuable."

Charlie Duong
Charlie Duong
Co-Founder, Opine

Raindrop surfaced performance issues that traditional error monitoring hadn’t explained, too. A single timeout in the search query-rewrite step accounted for 89% of errored LLM spans and was responsible for every error in two workflows.

The team traced the problem to resource contention between search and other AI workloads. After adjusting timeout settings and moving AI work onto dedicated infrastructure, query-rewrite timeouts fell from more than 300 per day to near zero. The 90th-percentile latency for successful rewrites improved from 5.3 seconds to 2.2 seconds.

For Opine, that meant fewer failed workflows and faster search, without having to wait for customers to encounter and report the problems themselves.

Like a full-time AI engineer on staff

As a small team, Opine leans on AI wherever it can, and its Raindrop setup shows it. Raindrop alerts land in a dedicated Slack channel. The team built an agent that watches the channel, checks each issue through the Raindrop MCP, and turns real problems into a pull request or a task for their coding agent. As new patterns emerge, the team defines custom Signals so Raindrop tracks exactly what matters to Opine.

The value goes beyond catching individual errors. Raindrop gives Opine a way to discover unexpected agent behavior, investigate its root cause, and measure whether a fix actually worked in production.

As Opine’s agents take on more work for its customers, Charlie expects Raindrop’s role to grow with them.

"It’s like having a full-time AI engineer working for you, watching everything all the time."

Charlie Duong
Charlie Duong
Co-Founder, Opine