A practical guide to choosing decision models

Before models decide, we decide on models
Using a general-purpose LLM like Claude to make every decision in your codebase feels like bringing a bazooka to a knife fight. You’re spending all that intelligence on decisions too complicated for deterministic code, but nowhere near complicated enough to justify frontier intelligence (and its price tag). So it’s unsurprising that a new class of models, decision models (like Typesafe’s Jev), has caught on.
Their appeal is quite simple: they make judgments quickly and cheaply. But with so many models to choose from now, deciding which one to use, or whether to use one at all, has become a challenge in its own right.
Uncertainty around decision models
How do the models stack up against each other in judgment accuracy?
Are some decision models better than others at judging certain kinds of questions?
Do they share LLMs’ sensitivity to wording and framing? In other words, could repeating information or adding unrelated events to a question’s context turn a correct judgment into a mistake?
And most importantly, how can knowing these things help me choose the right model for my codebase?
I found the answer depends on what you’re asking a model to judge. Different models were better at answering different kinds of questions. Overall, however, most changes to context had little effect on accuracy.
Building the benchmark
To see whether different decision models do better on different kinds of questions, I put together a collection of benchmark questions testing 15 distinct cognitive capabilities. The goal was to get more specific than broad categories used by other benchmarks, like “knowledge and reasoning” or “language understanding”, where answering a single question may require drawing on several of these capabilities.
The inspiration came from a field called comparative cognition that studies cognitive differences between species. AI can, in many ways, feel like an alien mind. So the idea of studying it as we might an unfamiliar species, by testing its cognitive capabilities separately to understand where its strengths lie, seemed fitting.
| Capability | What it asks of a model | Question sources | Sample question |
|---|---|---|---|
| 1. Retrieval | Find information explicitly stated in the context. | Read a sampleKing Louie is a fictional character introduced in Walt Disney's 1967 animated musical film, The Jungle Book. Unlike the majority of the adapted characters in the film, Louie was not featured in Rudyard Kipling's original works. King Louie was portrayed as an orangutan who was the leader of the other jungle primates, and who attempted to gain knowledge of fire from Mowgli, in order to become more human. is king louie in the jungle book book Options: • Yes • No Answer: No | |
| 2. State tracking and updating | Keep track of what changes as a sequence of events unfolds. | View sources (4) | Read a sampleAlice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Rodrigo, Claire is dancing with Jamie, Dave is dancing with Izzi, Eve is dancing with Sam, Fred is dancing with Karl, and Gertrude is dancing with Helga. Throughout the song, the dancers often trade partners. First, Alice and Claire switch partners. Then, Bob and Fred switch partners. Then, Claire and Dave switch partners. Then, Alice and Fred switch partners. Then, Claire and Alice switch partners. Then, Bob and Eve switch partners. Finally, Eve and Gertrude switch partners. At the end of the dance, Claire is dancing with Answer: Rodrigo |
| 3. Concepts and categorization | Identify which category something belongs to, such as a topic, emotion, or intent. | View sources (13) | Read a sampleLSU Tumbles, Auburn Rises in AP Poll (AP) AP - LSU fell out of the top 10 Sunday while Auburn moved in, the result of a one-point game decided by a second-chance extra point. Which news topic best describes this article? Options: World / Sports / Business / Science/Technology Answer: Sports |
| 4. Relations | Work out how people or things are related, such as family connections or relative order. | View sources (2) | Read a sample[Asia] is a proud sibling of the amazing [Russell]. [Russell] has mixed feelings towards his mother, [Alma]. What is Alma’s relationship to Asia? Answer: mother |
| 5. Analogy | Recognize the same pattern of relationships in different situations. | View sources (1) | Read a sampleThe interviewer upset the job applicant, causing the job applicant to rebuke the interviewer. The onlookers cheered the football team, but the football team ignored the onlookers. This is an example of Options: A. literal similarity. B. an analogy. C. a cross mapping. D. surface similarity. E. a false analogy. F. only objects similarity. G. no similarity. Answer: no similarity. |
The questions, themselves, came from various public benchmarks and were screened independently by OpenAI’s Sol 6.1 and Anthropic’s Sonnet 5.5, both at high reasoning effort. I included a question in a capability area only if both models could agree that it tested that capability, resulting in 1,005 questions across 15 capabilities, with 67 questions each.
Those 1,005 questions were then given to 11 decision models to answer, alongside OpenAI’s Astra and Anthropic’s Opus 5.5, to see whether decision models could match or exceed the accuracy of general-purpose LLMs. Finally, I tested the 11 decision models on roughly 24 altered versions of each question (24,563 in total), changing contextual details that shouldn’t affect the correct answer to find out whether those changes would nevertheless sway their judgments.
| Manipulation | What changes | Variations |
|---|---|---|
| Text length | Change the amount of surrounding text while retaining the decisive information. | 512 / 2,048 / 8,192-token targets |
| Extra events | Add irrelevant events without changing the events needed to answer the question. | 2, 6 or 12 unrelated events added |
| Evidence position | Move answer-relevant information within the context while preserving what it says. | Beginning / middle / end |
| Evidence order | Reorder how evidence is presented while preserving what happened and when. | Original / reversed / shuffled presentation |
| Repetition | Repeat the same evidence without introducing new facts or events. | The same evidence presented twice or three times |
GLiDE consistently came out ahead in accuracy
Overall, Fastino Lab’s GLiDE v1.2 decisively led the pack on accuracy, scoring an impressive 94.9% (954/1,005), compared with Astra’s 99.4% (999/1,005) and Opus 5.5’s 99.4% (997/1,003), followed by Perplexity’s Decider 1.1 at 91.4% (919/1,005) and Typesafe’s Jev at 88.8% (892/1,005). I went in expecting Jev to come out as the decisive leader in overall judgment accuracy, so this came as a bit of a surprise.
View exact scores and coverage
| Model | Correct / answered | Accuracy | Answered / selected |
|---|---|---|---|
| 999 / 1005 | 99.4% | 1005 / 1005 | |
| 997 / 1003 | 99.4% | 1003 / 1005 | |
| 954 / 1005 | 94.9% | 1005 / 1005 | |
| 919 / 1005 | 91.4% | 1005 / 1005 | |
| 892 / 1005 | 88.8% | 1005 / 1005 | |
| 846 / 1004 | 84.3% | 1004 / 1005 | |
| 835 / 1005 | 83.1% | 1005 / 1005 | |
| 777 / 997 | 77.9% | 997 / 1005 | |
| 768 / 1005 | 76.4% | 1005 / 1005 | |
| 678 / 1005 | 67.5% | 1005 / 1005 | |
| 429 / 1005 | 42.7% | 1005 / 1005 | |
| 416 / 1005 | 41.4% | 1005 / 1005 | |
| 408 / 1005 | 40.6% | 1005 / 1005 |
Overall accuracy scores, however, have a way of hiding weaknesses on particular kinds of questions. So the value of the capability areas is that they gave me a way to look beneath the totals to see where each model’s strengths lay, including whether lower-scoring models could outperform GLiDE on particular kinds of questions.
Even with the results broken down, though, GLiDE v1.2 still led or tied for best accuracy in 11 of the 15 capability areas. The exceptions were Numerical cognition, where Jev led (65/67), Physical cognition and Social cognition, where Decider v1.1 led (64/67 each), and Uncertainty monitoring, where Microsoft Decision-1 led (65/67).
| Capability | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 100% | 100% | 96% | 100%* | 100% | 97% | 99% | 100% | 70% | 82% | 81% | |
| 97% | 81% | 90% | 66% | 67% | 57% | 55% | 45% | 36% | 31% | 33% | |
| 97% | 96% | 97% | 94% | 96% | 94% | 97% | 88% | 66% | 72% | 72% | |
| 88% | 82% | 82% | 67% | 63% | 53%* | 55% | 39% | 16% | 19% | 16% | |
| 87% | 79% | 64% | 52% | 87% | 69% | 51% | 40% | 25% | 7% | 19% | |
| 97% | 97% | 81% | 73% | 69% | 79% | 69% | 72% | 33% | 19% | 21% | |
| 94% | 96% | 94% | 90% | 88% | 92%* | 87% | 78% | 31% | 34% | 27% | |
| 99% | 97% | 78% | 93% | 69% | 60% | 66% | 69% | 33% | 33% | 36% | |
| 88% | 85% | 97% | 76% | 69% | 70%* | 54% | 58% | 36% | 28% | 19% | |
| 99% | 91% | 93% | 87% | 93% | 79% | 99% | 60% | 52% | 55% | 51% | |
| 91% | 84% | 96% | 97% | 93% | 76% | 91% | 75% | 57% | 37% | 31% | |
| 100% | 97% | 99% | 99% | 100% | 97% | 93% | 90% | 33% | 61% | 57% | |
| 91% | 96% | 88% | 84% | 88% | 87% | 78% | 70% | 61% | 51% | 51% | |
| 99% | 99% | 96% | 96% | 90% | 85% | 84% | 69% | 57% | 54% | 57% | |
| 99% | 94% | 84% | 93% | 78% | 73% | 72% | 61% | 34% | 36% | 39% | |
When misses are costly, consider Jev
Valuable as it is, accuracy is only one factor to consider when choosing a model. Just as important, especially for us here at Raindrop, is a model’s error profile: how it balances false alarms against missed detections. Our systems frequently judge whether suspicious agent behavior should be flagged for human review, so getting that balance wrong means either sending people after imaginary problems or letting real ones slip through.
With that in mind, looking just at the 214-question binary subset of the original 1,005 questions revealed a reason to consider Jev despite GLiDE’s accuracy lead: Jev only missed three cases to GLiDE’s six, though it raised five false alarms to GLiDE’s one. Choosing between them therefore means weighing the cost of unnecessary flags against the cost of problems going undetected.
Precision and recall
View exact precision, recall and error counts
| Model | False alarms | Missed detections | Recall | Precision |
|---|---|---|---|---|
| 2 | 1 | 98.9% | 97.8% | |
| 1 | 0 | 100.0% | 98.9% | |
| 1 | 6 | 93.4% | 98.8% | |
| 5 | 9 | 90.1% | 94.3% | |
| 5 | 3 | 96.7% | 94.6% | |
| 8 | 9 | 90.1% | 91.1% | |
| 4 | 9 | 90.1% | 95.3% | |
| 17 | 24 | 73.6% | 79.8% | |
| 3 | 12 | 86.8% | 96.3% | |
| 27 | 25 | 72.5% | 71.0% | |
| 57 | 25 | 72.5% | 53.7% | |
| 43 | 45 | 50.5% | 51.7% | |
| 54 | 43 | 52.7% | 47.1% |
For speed and cost, consider Jev or self-host GLiDE
Speed and cost add another trade-off. Among the three decision models I tested through APIs, Jev looked particularly appealing: its median request time was 219 ms, with model charges of roughly $0.028 per 1,000 requests. Microsoft Decision-1 was slightly cheaper at $0.015 for the same number of requests, but slower at 330 ms and less accurate (84.3% versus Jev’s 88.8%). The other eight ran on the same rented A100 80GB cloud GPU, and among them, GLiDE v1.2 still proved to be the better choice even when speed was taken into account: its median prediction time was 137 ms, compared with Decider v1.1’s 145 ms.
Accuracy and request time
Accuracy and API cost
View exact costs and response times
| Model | Accuracy | Medianrequest time | p95request time | Costper 1,000 requests |
|---|---|---|---|---|
| 99.4% | 3,217 ms | 5,801 ms | $7.319 | |
| 99.4% | 1,753 ms | 3,906 ms | $4.633 | |
| 88.8% | 219 ms | 360 ms | $0.028 | |
| 84.3% | 330 ms | 506 ms | $0.015 | |
| 77.9% | 375 ms | 650 ms | $0.045 |
Accuracy held up across context changes
Does changing a question’s context change a model’s decision? For the most part, no. Most changes to context, including paraphrasing and adding irrelevant details, had little net effect on accuracy. The exception was adding short notes expressing someone else’s proposed answer alongside the original context, such as “A reader thinks: [misleading detail]”. In an agentic application, the equivalent might be an agent claiming it completed a task or another model claiming that agent got the task right.
| Model | Text length | Extra events | Evidence position | Evidence order | Repetition | Invented names | Irrelevant details | Similar distractors | Misleading details | Unrelated examples | Paraphrasing | Option order |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Misleading claims reduced accuracy in 10 of the 11 models, with GLiNER2.5 Decide and Strands suffering the largest declines: 24.4 and 20.8 percentage points, respectively. Much to my surprise, Microsoft’s Decision-1 was hurt more by correct claims: its accuracy fell by almost 10 points, while misleading claims left it roughly unchanged. Other models, such as GLiNER2.5 Decide, meanwhile benefitted from the correct hints, with its accuracy jumping from 42.7% to 78.7%.
Making the model decision
So where does this leave us? We’ve got answers now, but how exactly should we be using them to choose the right models for our codebases?
I’ve found the results are most useful when treated as a checklist of sorts for narrowing down the model choices. Essentially, by working through the following considerations in order, the model that makes the most sense for your codebase usually becomes apparent:
- Hosting: whether the model must run on your own infrastructure,
- Question type: what kind of decisions your code needs to make,
- Error costs: how costly false alarms and missed detections are to your use case,
- Cost: how much you can spend on API requests, and
- Speed: how quickly you need an answer.
For instance: if hosted APIs are acceptable and you need quick answers, choose Jev. If, however, the model must run on your own infrastructure and be effective at retrieving facts from a record or applying explicit rules, GLiDE then is probably the better choice. On the other hand, if you need it to make decisions that rely on physical understanding, Decider might be your best bet. Unless of course, error costs carry a high penalty. In that case, GLiDE may still be the better choice. I could keep going, but rather than walk you through every possible combination, I’ll leave you with a way to explore your own requirements using the model finder below:
The questions, recorded model answers, and code are available in the companion repository on Github.
