How Spiral solved a critical launch bug with Raindrop
"The slack messages I get every day are brilliant. I can’t live without them now. Being able to see the top 5 issues from the last 24 hours, go into the product and actually understand the chat and trace is just super helpful."Danny AzizGM of Spiral at Every
Spiral used Raindrop to rapidly resolve critical tool errors during their V3 launch.
Spiral is an AI writing partner with taste. Founders, marketers, and content creators love using Spiral to write both short-form social media posts and long-form blogs. Spiral recently launched their V3, where they take a unique, conversational approach to creating content. Instead of starting with writing, Spiral interviews the user to clarify all of the concrete details. Then, if the user is not sure of what angle they want to take with their writing, Spiral offers 3 different approaches so the user can explore a world of options.
Spiral integrated Raindrop ahead of their V3 launch so they could understand if their agent was working effectively. This let the team quickly identify areas needing attention with Raindrop’s alerts and maintain a real-time pulse on agent performance with signals.
Critical Launch Bug
The week of launch, Raindrop alerted the Spiral team to a critical bug, corroborating user reports and surfacing how frequently the issue was occurring. Their agent was making a tool call to create a document, but users weren’t able to see the document due to a rendering problem. Despite the model believing its task was successful, users saw nothing, leading to repeated user frustration and the model stuck in a loop calling the tool. As Danny Aziz noted, “other logs that I have wouldn't have shown this because from the model's perspective, it was totally fine.”
Spiral was able to solve this critical bug the day it was reported, eliminating an issue that was affecting around 1/3 of users during launch week.
Continuous Improvement
In addition to issue alerts, Spiral uses Raindrop to track Signals to monitor their agent’s performance. This includes negative signals like the model not adhering to a user’s desired writing style, the model falling pre to “AI-isms” (tell-tale signs that a section of writing was written by an AI), and falling back to English despite the user wanting to write in another language. As the team is pushing prompt changes in the weeks after their launch, they’re able to see if their changes actually reduce the rates of these negative signals.
Impact
Solved Launch Issue affecting 1/3 of users
Spiral identified and fixed a critical rendering bug affecting 1/3 of their users on the same day it was reported, preventing widespread frustration during their V3 launch.
Measurable Quality Improvements
By tracking signals for 'AI-isms' and style adherence, the team can quantitatively measure if their prompt engineering updates are actually improving the agent's performance.
"If we make a prompt change and all of a sudden Spiral is using all these AI-isms instead of the user’s desired tone you start to lose trust with users. Tracking signals with Raindrop lets us discover these spikes so we can address them quickly."
Danny AzizGM of Spiral at Every
Raindrop’s proactive monitoring has become part of Spiral’s daily routine to continuously improve their agent and rapidly fix any emergent issues.