Raindrop

How Templi delivers reliable agents for financial advisors

"It’s a layer above tracing. That’s what I see Raindrop as doing: automating error analysis."
Daniel Cooke
Daniel Cooke
Co-Founder & CTO, Templi

In the UK, financial advisors spend hours preparing suitability reports, documenting client recommendations, and meeting regulatory requirements. Templi builds agents that automate that work, so advisors can spend more time with clients.

For Templi’s customers, FCA-regulated advice firms, reliability comes first. As more firms adopted its agents, the team wanted a consistent way to monitor performance and keep improving across customers and workflows.

Raindrop helps Templi spot issues early, often before a customer notices, and turn them into lasting fixes and regression tests.

From weekly manual trace reviews to Slack alerts

Before Raindrop, Templi relied on tools like Sentry and LangSmith to trace agent behavior in production, but identifying issues still required manual review.

Every week, its engineering team met for an hour to review agent conversations and investigate potential problems. Understanding issues within subagent interactions meant digging into individual chats.

"We had no way to see half these problems without manually going in and looking at agent chats."

Daniel Cooke
Daniel Cooke
Co-Founder & CTO, Templi

Today, Raindrop flags issues directly in Slack, guiding engineers to the conversations that need attention.

"I don’t want to dig through traces to find problems. Just tell me what the problems are."

Daniel Cooke
Daniel Cooke
Co-Founder & CTO, Templi

Turning production insights into lasting improvements

Templi has built Raindrop Signals into its evaluation and development workflows. When Raindrop flags an issue, the team investigates the trace and identifies the kind of improvement needed.

Refining agent instructions. When an agent’s instructions don’t cover a particular case, the team adds the trace to its evaluation suite, updates the prompt, and verifies that the case passes. The test then runs nightly to catch regressions.

Improving consistency across customers. When the instructions are correct but the agent follows them inconsistently, Templi gathers datasets of 100–200 PII stripped production traces, evaluates performance, and iterates until the agent handles a broader range of cases.

Signals surfaces the examples needed for these datasets, so the team can assemble them without reviewing conversations by hand.

From detection to fix in 30 minutes

Much of this workflow now runs autonomously. Templi connects Raindrop Signals to its issue tracker and to coding agents, which investigate problems, prepare fixes, and run tests.

In October alone, 41 tickets originating from Raindrop Signals were fixed autonomously, with little to no direct engineering involvement.

That speed has surprised customers. In some cases, Templi has detected a problem, contacted the affected customer, and delivered a fix within approximately 30 minutes.

Measurable improvements in agent quality

Between its initial monitoring period in September and the following month, Templi reported 74% fewer issues across four key production signals. This included an 83% reduction in unretried tool call errors and a 76% reduction in refusals.

These results reflect a repeatable process for improving agent quality: identifying issues in production, making targeted changes, and testing those changes across customer workflows.

For Dan, Raindrop connects production monitoring with a clear path to improvement.

"It’s a layer above tracing. That’s what I see Raindrop as doing: automating error analysis."

Daniel Cooke
Daniel Cooke
Co-Founder & CTO, Templi

The team has also retired its weekly, hour-long manual trace-review meeting. Engineers can spend that time improving agent performance, with production examples guiding what to improve and what to test next.