How Unstuck AI monitors their learning agents with Raindrop
"Raindrop has become table stakes for building AI products. We need to understand how our agents perform in production to ensure they're working well for our millions of users."Sid BendreCTO
Unstuck AI, an academic copilot developed by Oleve, is one of the most popular AI tools in EdTech with over 2 million users, and part of Oleve's edtech products, a portfolio that's generating $6mm+ in ARR with a team of just 5 people.
Students rely on Unstuck's learning agent to generate study notes from lecture recordings, summaries of videos, create flashcards and even quizzes. As the team rapidly expanded tool use capabilities, they needed an effective way to monitor changes and quickly identify issues.
"Building an agent-native platform, Raindrop is a core part of our feedback loop. We need to monitor agent improvements as soon as we ship them to see if they're working for users. This is core to the UX of the product, so it's critical that we have a very strong pulse on it."
Sid BendreCTO
Highlights
Caught Tool Use Issues
Quickly identified and resolved critical issues with their summarization tool
Use Notifications to Prioritize
Real-time notifications flagged urgent issues like incomplete lecture notes or missing math explanations
Issue Notifications
Raindrop sends Unstuck daily alerts highlighting user wins and surfacing emerging issues. When users started receiving incomplete summaries and lecture notes, these notifications allowed the team to quickly prioritize fixes.
"Raindrop has been super high leverage for us. Every day when I'm on my way to the office, I check the notification to see what issues users have been facing and that impacts what I prioritize."
Sid BendreCTO
Monitoring Agentic Tool-Use
As soon as Unstuck's AI engineering team pushed their fixes for the summarization issues, they were immediately able to jump into the Raindrop dashboard and see if the issue incidence rate started decreasing.
"We had just pushed out a major change for the summarization issue, and then we were able to go into Raindrop, filter User Frustration by the summarization tool, and immediately get feedback on whether our changes were impactful"
Sid BendreCTO
Feature Prioritization
Raindrop's notifications don't just flag issues, they highlight wins too. This lets the Unstuck team quickly see which features users love and identify unexpected ways people engage with the product.
"Raindrop has been invaluable in helping us assess demand for a feature over another. It directly led to us launching flashcards and quizzes!"
Sid BendreCTO
Beyond Unstuck
Unstuck is part of Oleve's suite of AI-driven consumer products. Raindrop has become so essential to Oleve's AI engineering workflow that the team now integrates it from day one in every new product they build, just as they did with Unstuck.