How New Computer Monitors Dot's Memory and Personality with Raindrop
"Raindrop accelerated our development cycles by surfacing issues early and letting us see how well our solutions worked. It's a must have for any AI product in production."Sam WhitmoreCEO/CTO
Dot, created by New Computer, is one of the first-ever personal AI assistants, designed to fill roles ranging from accountability buddy to thought partner to friend. To achieve this goal, the team pioneered early memory systems and agent workflows to ensure Dot remembered user details and handled tasks reliably.
To maintain this level of reliability, New Computer's AI engineering team needed effective monitoring to quickly identify issues, deploy targeted fixes, and measure the success of their solutions.
Maintaining user privacy is one of New Computer's top priorities. They were able to use Raindrop to monitor user issues in bulk while keeping individual user data anonymized to get a broad overview of what they should be solving.
"Raindrop has been an invaluable partner in discovering hidden bugs and understanding which features are resonating with users. We have a high commitment to user privacy, so we wanted to do this type of analysis in bulk only, which they support."
Sam WhitmoreCEO/CTO
Immediate Issue Notifications
Raindrop's daily alerts highlight both user successes and issues. These notifications helped the New Computer team quickly pinpoint critical memory issues such as Dot missing reminders and tone issues that affected Dot's persona, such as pestering or speaking in platitudes (ChatGPT-esque tone), which would annoy users.
The team used these notifications to get a pulse on issue rates, understand any spikes, and see where they needed to dive in deeper.
"The daily notification feature has been particularly valuable, letting the whole team see what's going on at a glance, and letting us know where we need to deep dive."
Sam WhitmoreCEO/CTO
Precise Monitoring and Rapid Iteration
Raindrop helped New Computer monitor critical issues, like Dot lying about its capabilities which was a large source of user frustration. Connecting Raindrop to their eval platform, the team rapidly iterated on their prompt and pipeline, eventually implementing a new capabilities system with a default set of memories about what Dot can do.
Raindrop also highlighted user frustration caused by excessive proactive messages, enabling the team to swiftly reduce message frequency and improve the overall user experience.
"Did our fix actually work after deploying it? Are we still seeing the same problems pop up over time? Anyone who works with AI knows that this type of monitoring is super hard to pull off in production."
Sam WhitmoreCEO/CTO
Continuous Monitoring
Raindrop has become an essential part of New Computer's workflow, continuously monitoring interactions to ensure rapid identification and resolution of any issues.