Raindrop

A practical guide to choosing decision models

A practical guide to choosing decision models

Before models decide, we decide on models

Using a general-purpose LLM like Claude to make every decision in your codebase feels like bringing a bazooka to a knife fight. You’re spending all that intelligence on decisions too complicated for deterministic code, but nowhere near complicated enough to justify frontier intelligence (and its price tag). So it’s unsurprising that a new class of models, decision models (like Typesafe’s Jev), has caught on.

Their appeal is quite simple: they make judgments quickly and cheaply. But with so many models to choose from now, deciding which one to use, or whether to use one at all, has become a challenge in its own right.

Uncertainty around decision models

How do the models stack up against each other in judgment accuracy?

Are some decision models better than others at judging certain kinds of questions?

Do they share LLMs’ sensitivity to wording and framing? In other words, could repeating information or adding unrelated events to a question’s context turn a correct judgment into a mistake?

And most importantly, how can knowing these things help me choose the right model for my codebase?

I found the answer depends on what you’re asking a model to judge. Different models were better at answering different kinds of questions. Overall, however, most changes to context had little effect on accuracy.

Building the benchmark

To see whether different decision models do better on different kinds of questions, I put together a collection of benchmark questions testing 15 distinct cognitive capabilities. The goal was to get more specific than broad categories used by other benchmarks, like “knowledge and reasoning” or “language understanding”, where answering a single question may require drawing on several of these capabilities.

The inspiration came from a field called comparative cognition that studies cognitive differences between species. AI can, in many ways, feel like an alien mind. So the idea of studying it as we might an unfamiliar species, by testing its cognitive capabilities separately to understand where its strengths lie, seemed fitting.

CapabilityWhat it asks of a modelQuestion sourcesSample question
1. RetrievalFind information explicitly stated in the context.
View sources (8)
Read a sample
King Louie is a fictional character introduced in Walt Disney's 1967 animated musical film, The Jungle Book. Unlike the majority of the adapted characters in the film, Louie was not featured in Rudyard Kipling's original works. King Louie was portrayed as an orangutan who was the leader of the other jungle primates, and who attempted to gain knowledge of fire from Mowgli, in order to become more human. is king louie in the jungle book book Options: • Yes • No

Answer: No

2. State tracking and updatingKeep track of what changes as a sequence of events unfolds.
View sources (4)
Read a sample
Alice, Bob, Claire, Dave, Eve, Fred, and Gertrude are dancers at a square dance. At the start of a song, they each have a partner: Alice is dancing with Lola, Bob is dancing with Rodrigo, Claire is dancing with Jamie, Dave is dancing with Izzi, Eve is dancing with Sam, Fred is dancing with Karl, and Gertrude is dancing with Helga. Throughout the song, the dancers often trade partners. First, Alice and Claire switch partners. Then, Bob and Fred switch partners. Then, Claire and Dave switch partners. Then, Alice and Fred switch partners. Then, Claire and Alice switch partners. Then, Bob and Eve switch partners. Finally, Eve and Gertrude switch partners. At the end of the dance, Claire is dancing with

Answer: Rodrigo

3. Concepts and categorizationIdentify which category something belongs to, such as a topic, emotion, or intent.
View sources (13)
Read a sample
LSU Tumbles, Auburn Rises in AP Poll (AP) AP - LSU fell out of the top 10 Sunday while Auburn moved in, the result of a one-point game decided by a second-chance extra point. Which news topic best describes this article? Options: World / Sports / Business / Science/Technology

Answer: Sports

4. RelationsWork out how people or things are related, such as family connections or relative order.
View sources (2)
Read a sample
[Asia] is a proud sibling of the amazing [Russell]. [Russell] has mixed feelings towards his mother, [Alma]. What is Alma’s relationship to Asia?

Answer: mother

5. AnalogyRecognize the same pattern of relationships in different situations.
View sources (1)
Read a sample
The interviewer upset the job applicant, causing the job applicant to rebuke the interviewer. The onlookers cheered the football team, but the football team ignored the onlookers. This is an example of Options: A. literal similarity. B. an analogy. C. a cross mapping. D. surface similarity. E. a false analogy. F. only objects similarity. G. no similarity.

Answer: no similarity.

Table 1. The 15 capabilities, their benchmark sources, and sample questions. Each capability has 67 independent questions.

The questions, themselves, came from various public benchmarks and were screened independently by OpenAI’s Sol 6.1 and Anthropic’s Sonnet 5.5, both at high reasoning effort. I included a question in a capability area only if both models could agree that it tested that capability, resulting in 1,005 questions across 15 capabilities, with 67 questions each.

Those 1,005 questions were then given to 11 decision models to answer, alongside OpenAI’s Astra and Anthropic’s Opus 5.5, to see whether decision models could match or exceed the accuracy of general-purpose LLMs. Finally, I tested the 11 decision models on roughly 24 altered versions of each question (24,563 in total), changing contextual details that shouldn’t affect the correct answer to find out whether those changes would nevertheless sway their judgments.

ManipulationWhat changesVariations
Text lengthChange the amount of surrounding text while retaining the decisive information.

512 / 2,048 / 8,192-token targets

Extra eventsAdd irrelevant events without changing the events needed to answer the question.

2, 6 or 12 unrelated events added

Evidence positionMove answer-relevant information within the context while preserving what it says.

Beginning / middle / end

Evidence orderReorder how evidence is presented while preserving what happened and when.

Original / reversed / shuffled presentation

RepetitionRepeat the same evidence without introducing new facts or events.

The same evidence presented twice or three times

Table 2. Context manipulations designed to preserve the correct answer.

GLiDE consistently came out ahead in accuracy

Overall, Fastino Lab’s GLiDE v1.2 decisively led the pack on accuracy, scoring an impressive 94.9% (954/1,005), compared with Astra’s 99.4% (999/1,005) and Opus 5.5’s 99.4% (997/1,003), followed by Perplexity’s Decider 1.1 at 91.4% (919/1,005) and Typesafe’s Jev at 88.8% (892/1,005). I went in expecting Jev to come out as the decisive leader in overall judgment accuracy, so this came as a bit of a surprise.

Decision modelsLLM baselines
AstraOpenAI
Opus 5.5Anthropic
GLiDE v1.2Fastino Labs
Decider v1.1Perplexity
JevTypeSafe
Decision-1Microsoft
ClefCloudflare
GPT-6 Luna DecisionsOpenAI
Clef FlashCloudflare
Strands DeciderStrands Agents
GLiNER2.5 DecideFastino Labs
LayaConvai Innovations
CalDec v1K. Grozdanovski
View exact scores and coverage
ModelCorrect / answeredAccuracyAnswered / selected
AstraOpenAI · LLM baseline999 / 100599.4%1005 / 1005
Opus 5.5Anthropic · LLM baseline997 / 100399.4%1003 / 1005
GLiDE v1.2Fastino Labs954 / 100594.9%1005 / 1005
Decider v1.1Perplexity919 / 100591.4%1005 / 1005
JevTypeSafe892 / 100588.8%1005 / 1005
Decision-1Microsoft846 / 100484.3%1004 / 1005
ClefCloudflare835 / 100583.1%1005 / 1005
GPT-6 Luna DecisionsOpenAI777 / 99777.9%997 / 1005
Clef FlashCloudflare768 / 100576.4%1005 / 1005
Strands DeciderStrands Agents678 / 100567.5%1005 / 1005
GLiNER2.5 DecideFastino Labs429 / 100542.7%1005 / 1005
LayaConvai Innovations416 / 100541.4%1005 / 1005
CalDec v1K. Grozdanovski408 / 100540.6%1005 / 1005
Figure 1. Overall accuracy across 1,005 questions, calculated over answered questions. Outlined bars show the two general-purpose LLM baselines.

Overall accuracy scores, however, have a way of hiding weaknesses on particular kinds of questions. So the value of the capability areas is that they gave me a way to look beneath the totals to see where each model’s strengths lay, including whether lower-scoring models could outperform GLiDE on particular kinds of questions.

Even with the results broken down, though, GLiDE v1.2 still led or tied for best accuracy in 11 of the 15 capability areas. The exceptions were Numerical cognition, where Jev led (65/67), Physical cognition and Social cognition, where Decider v1.1 led (64/67 each), and Uncertainty monitoring, where Microsoft Decision-1 led (65/67).

Capability
GLiDE v1.2
Decider v1.1
Jev
Decision-1
Clef
GPT-6 Luna Decisions
Clef Flash
Strands Decider
GLiNER2.5 Decide
Laya
CalDec v1
100%100%96%100%*100%97%99%100%70%82%81%
97%81%90%66%67%57%55%45%36%31%33%
97%96%97%94%96%94%97%88%66%72%72%
88%82%82%67%63%53%*55%39%16%19%16%
87%79%64%52%87%69%51%40%25%7%19%
97%97%81%73%69%79%69%72%33%19%21%
94%96%94%90%88%92%*87%78%31%34%27%
99%97%78%93%69%60%66%69%33%33%36%
88%85%97%76%69%70%*54%58%36%28%19%
99%91%93%87%93%79%99%60%52%55%51%
91%84%96%97%93%76%91%75%57%37%31%
100%97%99%99%100%97%93%90%33%61%57%
91%96%88%84%88%87%78%70%61%51%51%
99%99%96%96%90%85%84%69%57%54%57%
99%94%84%93%78%73%72%61%34%36%39%
* = one or more unanswered questions0%100%
Figure 2. Accuracy across the 15 capabilities, with 67 questions each. Cells show accuracy, rounded to the nearest percent. Outlines mark the highest decision-model accuracy in each row, including ties.

When misses are costly, consider Jev

Valuable as it is, accuracy is only one factor to consider when choosing a model. Just as important, especially for us here at Raindrop, is a model’s error profile: how it balances false alarms against missed detections. Our systems frequently judge whether suspicious agent behavior should be flagged for human review, so getting that balance wrong means either sending people after imaginary problems or letting real ones slip through.

With that in mind, looking just at the 214-question binary subset of the original 1,005 questions revealed a reason to consider Jev despite GLiDE’s accuracy lead: Jev only missed three cases to GLiDE’s six, though it raised five false alarms to GLiDE’s one. Choosing between them therefore means weighing the cost of unnecessary flags against the cost of problems going undetected.

Precision and recall

PrecisionRecall
AstraLLM baseline97.8%98.9%
Opus 5.5LLM baseline98.9%100.0%
GLiDE v1.298.8%93.4%
Decider v1.194.3%90.1%
Jev94.6%96.7%
Microsoft Decision-191.1%90.1%
Clef95.3%90.1%
Luna Decisions79.8%73.6%
Clef Flash96.3%86.8%
Strands Decider71.0%72.5%
GLiNER2.5 Decide53.7%72.5%
Laya51.7%50.5%
CalDec v147.1%52.7%
View exact precision, recall and error counts
ModelFalse alarmsMissed detectionsRecallPrecision
AstraOpenAI · LLM baseline2198.9%97.8%
Opus 5.5Anthropic · LLM baseline10100.0%98.9%
GLiDE v1.2Fastino Labs1693.4%98.8%
Decider v1.1Perplexity5990.1%94.3%
JevTypeSafe5396.7%94.6%
Decision-1Microsoft8990.1%91.1%
ClefCloudflare4990.1%95.3%
GPT-6 Luna DecisionsOpenAI172473.6%79.8%
Clef FlashCloudflare31286.8%96.3%
Strands DeciderStrands Agents272572.5%71.0%
GLiNER2.5 DecideFastino Labs572572.5%53.7%
LayaConvai Innovations434550.5%51.7%
CalDec v1K. Grozdanovski544352.7%47.1%
Figure 3. Precision and recall on 214 binary questions.

For speed and cost, consider Jev or self-host GLiDE

Speed and cost add another trade-off. Among the three decision models I tested through APIs, Jev looked particularly appealing: its median request time was 219 ms, with model charges of roughly $0.028 per 1,000 requests. Microsoft Decision-1 was slightly cheaper at $0.015 for the same number of requests, but slower at 330 ms and less accurate (84.3% versus Jev’s 88.8%). The other eight ran on the same rented A100 80GB cloud GPU, and among them, GLiDE v1.2 still proved to be the better choice even when speed was taken into account: its median prediction time was 137 ms, compared with Decider v1.1’s 145 ms.

Accuracy and request time

AccuracyMedian request time · ms · log scaleAstra: 99.4% accuracy; median request time 3,217 ms; model cost $7.319 per 1,000 requests.AstraOpus 5.5: 99.4% accuracy; median request time 1,753 ms; model cost $4.633 per 1,000 requests.Opus 5.5Jev: 88.8% accuracy; median request time 219 ms; model cost $0.028 per 1,000 requests.JevMicrosoft Decision-1: 84.3% accuracy; median request time 330 ms; model cost $0.015 per 1,000 requests.Microsoft Decision-1Luna Decisions: 77.9% accuracy; median request time 375 ms; model cost $0.045 per 1,000 requests.Luna Decisions

Accuracy and API cost

AccuracyModel cost per 1,000 requests · USD · log scaleAstra: 99.4% accuracy; median request time 3,217 ms; model cost $7.319 per 1,000 requests.AstraOpus 5.5: 99.4% accuracy; median request time 1,753 ms; model cost $4.633 per 1,000 requests.Opus 5.5Jev: 88.8% accuracy; median request time 219 ms; model cost $0.028 per 1,000 requests.JevMicrosoft Decision-1: 84.3% accuracy; median request time 330 ms; model cost $0.015 per 1,000 requests.Microsoft Decision-1Luna Decisions: 77.9% accuracy; median request time 375 ms; model cost $0.045 per 1,000 requests.Luna Decisions

Timing includes the full request. API times run from preparing a request to receiving an answer, including network travel and routing through Vercel AI Gateway, the service I used to access these models.

Cost reflects actual usage. Costs show model inference charges per 1,000 requests, including priced refusals.

View exact costs and response times
ModelAccuracyMedianrequest timep95request timeCostper 1,000 requests
AstraOpenAI · LLM baseline99.4%3,217 ms5,801 ms$7.319
Opus 5.5Anthropic · LLM baseline99.4%1,753 ms3,906 ms$4.633
JevTypeSafe88.8%219 ms360 ms$0.028
Decision-1Microsoft84.3%330 ms506 ms$0.015
GPT-6 Luna DecisionsOpenAI77.9%375 ms650 ms$0.045
Figure 4. Accuracy, costs and response times for the original 1,005 questions.

Accuracy held up across context changes

Does changing a question’s context change a model’s decision? For the most part, no. Most changes to context, including paraphrasing and adding irrelevant details, had little net effect on accuracy. The exception was adding short notes expressing someone else’s proposed answer alongside the original context, such as “A reader thinks: [misleading detail]”. In an agentic application, the equivalent might be an agent claiming it completed a task or another model claiming that agent got the task right.

ModelText lengthExtra eventsEvidence positionEvidence orderRepetitionInvented namesIrrelevant detailsSimilar distractorsMis­leading detailsUnrelated examplesPara­phrasingOption order
GLiDE v1.2Fastino Labs
Decider v1.1Perplexity
JevTypeSafe
Decision-1Microsoft
ClefCloudflare
GPT-6 Luna DecisionsOpenAI
Clef FlashCloudflare
Strands DeciderStrands Agents
GLiNER2.5 DecideFastino Labs
LayaConvai Innovations
CalDec v1K. Grozdanovski
Sorted = the provider’s input processing fixes choice order; order sensitivity isn’t tested.
Accuracy change · percentage points−700+70
Figure 5. Change in accuracy between matched versions of the same questions, in percentage points. Select a cell to see the versions compared, answer changes and counts; negative values indicate a decline.

Misleading claims reduced accuracy in 10 of the 11 models, with GLiNER2.5 Decide and Strands suffering the largest declines: 24.4 and 20.8 percentage points, respectively. Much to my surprise, Microsoft’s Decision-1 was hurt more by correct claims: its accuracy fell by almost 10 points, while misleading claims left it roughly unchanged. Other models, such as GLiNER2.5 Decide, meanwhile benefitted from the correct hints, with its accuracy jumping from 42.7% to 78.7%.

Making the model decision

So where does this leave us? We’ve got answers now, but how exactly should we be using them to choose the right models for our codebases?

I’ve found the results are most useful when treated as a checklist of sorts for narrowing down the model choices. Essentially, by working through the following considerations in order, the model that makes the most sense for your codebase usually becomes apparent:

  1. Hosting: whether the model must run on your own infrastructure,
  2. Question type: what kind of decisions your code needs to make,
  3. Error costs: how costly false alarms and missed detections are to your use case,
  4. Cost: how much you can spend on API requests, and
  5. Speed: how quickly you need an answer.

For instance: if hosted APIs are acceptable and you need quick answers, choose Jev. If, however, the model must run on your own infrastructure and be effective at retrieving facts from a record or applying explicit rules, GLiDE then is probably the better choice. On the other hand, if you need it to make decisions that rely on physical understanding, Decider might be your best bet. Unless of course, error costs carry a high penalty. In that case, GLiDE may still be the better choice. I could keep going, but rather than walk you through every possible combination, I’ll leave you with a way to explore your own requirements using the model finder below:

1. Must the model run on your own infrastructure?
2. Which kinds of questions will it judge?
3. Which mistake is more costly?
4. What is your API cost limit ($/1M input tokens)?
5. How quickly should it respond (ms)?

Your match

GLiDE

GLiDE
API ↗

Highest accuracy for your selected questions.


The questions, recorded model answers, and code are available in the companion repository on Github.