We were using GPT-6 Sol through the Responses API to make small decisions for our support bot. It would read a message and pick a support topic, or work out which action someone was asking for.
These calls only needed a choice from a list. When we looked at the prices for JEV and the Decisions API, the question was fairly obvious: could we do the same job for a lot less?
A model for small decisions
Decision models handle narrow tasks: pick a category, check a condition, or give something a score. They return structured answers and probabilities that your code can use directly. Responses is more general-purpose; for our classifier, we asked it to generate JSON containing the answer and a confidence score.
We compared JEV, TypeSafe's System One model, with OpenAI's Decisions API. We ran this experiment during the Decisions API's public beta, using GPT-6 Luna.
The price difference got our attention
These are the public list prices per million tokens as of October 20, 2026. The input comparison uses uncached tokens and OpenAI's standard short-context pricing.
| API and model | Input per million tokens | Output per million tokens |
|---|---|---|
| Responses with GPT-6 Sol | $2.00 | $10.00 |
| Responses with GPT-6 Luna | $0.10 | $0.50 |
| JEV 1.13 | $0.042 | No charge |
| Decisions with GPT-6 Luna | $0.10 | No charge |
On input price, JEV is 97.9% cheaper than SOL, the model we were using, and 58% cheaper than Luna Responses. Decisions is 95% cheaper than SOL and costs the same as Luna Responses for input. Both JEV and Decisions have no output-token charge.
Sources: OpenAI model pricing, Decisions pricing, and JEV pricing.
That was enough to get our attention. We needed to see what happened to the answers.
What our bot needs to decide
I've written about how we use OpsBot for engineering support before. For this experiment, we needed to classify a user's message and detect the action they were asking for. Think "I can't log in" → an access topic, or "assign this to me" → reassign the ticket.
We ran evals
We used Luna for both Responses and Decisions, alongside JEV 1.13, so the two OpenAI endpoints used the same model. SOL was our production cost baseline; the results below compare against Luna Responses.
We ran 80 topic examples and 100 action examples through each API, keeping the task inputs and choices fixed. All 540 calls returned parseable answers. These were classification tests; we didn't execute ticket actions. Reference labels were AI-reviewed, and the sample deliberately included historical failures and unknown cases.
Latency was the big win
Both decision APIs brought p95 latency below a second for both tasks.
| API | Action classification p95 | Topic classification p95 |
|---|---|---|
| Responses with Luna | 3.313 s | 5.483 s |
| Decisions with Luna | 0.478 s | 0.652 s |
| JEV 1.13 | 0.685 s | 0.921 s |
Decisions was about 7 times faster for actions and 8 times faster for topics at p95. JEV was about 5 to 6 times faster.
Before you quote this as a "trust me bro" benchmark 😁: latency varies with context size, load, and model provider. Your numbers may differ, but our evals showed both decision APIs were substantially faster than Responses.
Did they pick the right answer
First, we looked only at whether the selected answer matched an acceptable reference label. We ignored confidence for this part.
| API | Correct action matches | Correct topic matches |
|---|---|---|
| Responses with Luna | 98/100 | 67/80 |
| Decisions with Luna | 96/100 | 60/80 |
| JEV 1.13 | 95/100 | 64/80 |
For actions, Decisions and JEV were close to Responses. Topics were harder for all three, and Responses still matched the most labels.
The speed improvement was clear, but we still had one more rule to apply: confidence. For Responses, that meant literally asking the LLM, "How confident are you?" Yes, we were using its answer to decide whether to accept its answer 😁. That rule had worked pretty well for us in production so far. Once we applied the same cutoff to all three APIs, the results looked different.
The answer was correct but we rejected it
Our evaluation required an answer to match the expected label and have at least 80% confidence before counting it as correct and accepted.
Across both tasks:
| API | Correct matches without a cutoff | Correct and accepted at ≥80% confidence |
|---|---|---|
| Responses with Luna | 165/180 (91.7%) | 150/180 (83.3%) |
| Decisions with Luna | 156/180 (86.7%) | 120/180 (66.7%) |
| JEV 1.13 | 159/180 (88.3%) | 131/180 (72.8%) |
At the 80% confidence cutoff, the correct-and-accepted rate dropped to 66.7% for Decisions and 72.8% for JEV.
We tried threshold tuning first
We went back to the saved results and tried lower cutoffs. We looked for thresholds that accepted more correct answers without increasing the wrong answers passing compared with each API's own 80% baseline.
| Task | API | Cutoff change | Correct and accepted before → after | Wrong accepted before → after |
|---|---|---|---|---|
| Actions | Decisions | 80% → 60% | 71/100 → 89/100 | 0 → 0 |
| Actions | JEV | 80% → 68% | 89/100 → 93/100 | 2 → 2 |
| Topics | Decisions | 80% → 75% | 49/80 → 53/80 | 6 → 6 |
| Topics | JEV | 80% → 78% | 42/80 → 44/80 | 3 → 3 |
At the tuned cutoffs, the correct-and-accepted rate rose to 78.9% for Decisions and 76.1% for JEV. We chose these cutoffs on the same examples, so they still need validation on unseen data.
Confidence calibration
In simple terms: if the confidence score is 80%, does that mean the answer is correct about 80% of the time? That's what we'd want if we're treating the score as a probability of correctness.
OpenAI's guidance recommends using labeled examples from your own application to set thresholds. Our tuning was a first step in understanding which scores we could use. We adjusted the cutoff; calibrating the scores would mean learning a mapping from those scores to observed correctness.
To do that, we'd compare scores with actual results, learn the mapping on one set, and check it on unseen examples. Scikit-learn explains how to check calibration and how to calibrate a classifier.
I'm still new to this part of ML and learning as I go. We've tried threshold tuning so far; we haven't fitted or deployed a calibration mapping.
Key takeaways
-
For classification tasks like ours, decision-only models can be much cheaper and faster than generative models.
-
These APIs may need tuning and customisation to fit an existing system. Our confidence cutoff was a good example.