How Tolan's AI Engineers monitor their 'Alien' AI companions with Raindrop

"Raindrop has been invaluable as we've been growing quickly. It's critical for us to keep issue incidence below an acceptable threshold and become aware of any spikes. It's like if we see an iOS crash report in Sentry, but for our AI capabilities."
Evan Goldschmidt
CTO

Tolan is one of the most popular AI companion apps on the app store, pairing users with alien "best friends" from the planet Portola. Each Tolan needs to authentically embody the lore of Portola, remembering personal details and supporting users through life's ups and downs.

As Tolan's user base grew, their AI engineering team faced a critical challenge: it was difficult to quickly identify and quantify user issues, making it hard to measure if their fixes actually worked.

"Scaling an AI consumer product generates a huge volume of corner cases that you can't discover reliably by talking directly with users. we quickly realized we needed a smarter solution to monitor issues at scale"
Quinten Farmer
CEO

To gain clear visibility into these issues, Tolan turned to Raindrop. With Raindrop they were able to discover new issues, monitor for spikes, and assess solutions.

Impact

27.8% Reduction in Memory Issues

Raindrop helped identify incidents where Tolans forgot crucial user details or failed to send reminders. The team rapidly tested and validated different retrieval and storage solutions, reducing memory issues by 27.8%.

Nearly Eliminated Lore Inconsistencies

Tolan's creative team developed a set of lore benchmarks, such as the Tolan correctly identifying its home planet. Based on this benchmark and Raindrop's monitoring, the engineering team made prompt changes that reduced instances of benchmark failure below 0.1%.

"Without Raindrop we couldn't tell if we were actually fixing the problem. Were we coming up with good interventions and driving that incidence rate down to an acceptably low level? Now we can isolate test cases and see if our improvements are having a material impact on the end-user experience."
Evan Goldschmidt
CTO

Tolan's AI engineering team has continued to rely on Raindrop as they've been growing rapidly, to monitor every single issue to see how that rate goes down with their solutions and understand if there are anomalous spikes.