An LLM cascade with basal in front of Claude Opus got within 0.1 points of Opus alone, at 31% of the cost. Jev, TypeSafe's decision model available as an API, got within 0.2 points at 25%. On their own, none of the cheaper models met our requirements; as filters in front of Opus, all of them did.
Do not send every small decision to the most expensive model, and do not trust a cheap model on its own either. A lot of business intelligence at scale is small, bounded decisions made thousands of times. That suits decision models, so when basal, the Polish open-weights decision model from Remigiusz Kinas, got a lot of attention, I tested it on 1 such task from our work at EmpoweredHouse. Here is the whole test: how we built the data and the answer key, the rules we fixed in advance, and where every model broke.
What I tested: claim judging
Claim judging decides whether 1 fact bears on 1 business question. A claim is 1 fact taken from notes, docs or threads. A business question is an open question about the company, such as "is the cloud migration paying off?". For every claim against every open question, the judge model reads both and picks 1 of 4 answers:
- supports: evidence the answer is yes;
- counters: evidence the answer is no;
- neutral: context that pushes neither way;
- irrelevant: does not bear on the question.
At the time of the test, Opus judged every pair, and it was the most expensive step. The question was whether a cheaper model could replace it, on its own or as a filter that drops irrelevant pairs before Opus sees them.
A test set built to be hard
On our real claims, more than 90% of pairs were plainly irrelevant. A test on them mostly shows that a model can reject an off-topic claim. So we built a set that tests the hard decisions instead.
- A fictional company: Kestrel, a software house of about 120 people, with a consistent history from 2023 to 2026.
- 5 business questions in English, written in different styles: do referral hires stay longer, do fixed-price projects finish on budget, is the cloud migration paying off, is on-call sustainable, are seniors grown inside.
- 993 claims, each written for 1 question. About 1 in 4 also bears on another question.
- About 30% traps: the same words and the same people, but a different question.
- 4,965 pairs: every claim against every question, 19.3% of them relevant.
How we built the answer key
2 Claude Opus agents labelled every pair independently: the agent that wrote the claim, and a second one that did not see the first label. They agreed on 98.9% of pairs. A third agent settled the 56 disputes.
All 3 annotators are the same model under the same rules. Their agreement shows that the labels are consistent. It does not show that a person would agree with every one, so read the results as "how close does this model judge to Opus".
The main score is Cohen's kappa (κ) against this answer key: agreement corrected for chance, where 1.0 is perfect. Accuracy, the share of the 4,965 decisions that match the key, is easier to read, so I give both. Accuracy alone would flatter every model, because 81% of the pairs are irrelevant; that is why κ is the main score.
The rules we fixed before the run
We fixed the requirements before the run and did not change them after seeing the results:
| Requirement | Threshold |
|---|---|
| Relevant pairs kept (relevant recall) | at least 95% |
| Irrelevant pairs wrongly included (false positives) | at most 3% |
| Stance agreement | at least 95% |
| Wrong "counters" calls (false counters), the most harmful error | at most 5% |
For filters we set 1 more target: pass on at least 99% of the relevant pairs, because a pair the filter drops is lost and Opus never sees it.
Jev and basal return probabilities, so their filter threshold can be tuned. We tuned it on 4 questions and tested it on the fifth, which it had not seen, for each question in turn. Haiku returns only an answer, so it has no threshold.
Results: alone, only Opus passes
4 models went into the ring:
- Claude Opus, high effort, the reference;
- basal-omni-1.0-ultra, the 31B open-weights version of basal, built on Gemma 4, Apache 2.0 licence, on 1 rented H100;
- Claude Haiku, Anthropic's smaller model;
- Jev, a decision model from TypeSafe, used through its API.
| Model alone | Accuracy | Mistakes per 1,000 | κ | Relevant kept | Stance agreement | Wrong "counters" | Meets the rules |
|---|---|---|---|---|---|---|---|
| Claude Opus | 98.6% | 14 | 0.959 | 98.5% | 97.0% | 0.0% | yes |
| basal-omni-1.0-ultra | 96.9% | 31 | 0.905 | 92.5% | 92.2% | 0.9% | no: recall, stance |
| Claude Haiku | 96.6% | 34 | 0.902 | 97.4% | 93.3% | 1.5% | no: stance |
| Jev | 95.4% | 46 | 0.858 | 89.6% | 88.0% | 1.5% | no: recall, stance |
All 3 cheaper models miss the stance requirement. Haiku finds almost every relevant claim but gets the stance wrong more often than the rules allow; basal and Jev also miss relevant claims. Most of their mistakes, 75 to 88%, involve "neutral": a plan, a rule or a context fact read as evidence, evidence read as context, or context dropped as irrelevant.
They also fail in opposite directions. Haiku stretches: it keeps 97.4% of the relevant claims but lets through the most noise, 2.0% false positives. Basal and Jev are cautious: almost no noise, 0.3% and 0.7%, but they miss relevant claims, about 7.5% and 10%.
Results: as a filter in front of Opus, all 3 come within 0.007 κ of Opus
In the cascade, the cheap model reads every pair and drops what is clearly irrelevant. Opus judges the rest, about 1 in 4 pairs. The cost counts both parts.
| Setup | Accuracy | κ | Sent to Opus | Relevant kept by the filter | Cost, Opus alone = 100% |
|---|---|---|---|---|---|
| Opus alone | 98.6% | 0.959 | 100% | not applicable | 100% |
| Basal, then Opus | 98.5% | 0.957 | 24.9% | 99.0% | 31% |
| Jev, then Opus | 98.4% | 0.953 | 23.9% | 98.3% | 25% |
| Haiku, then Opus | not stated | 0.952 | 20.4% | 97.4% | 24% |
According to the test report, all 3 cascades meet the 4 requirements and lose at most 0.007 κ against Opus alone. Only basal also met the filter target: on questions it was not tuned on, it kept 99.0% of the relevant claims. Jev came close at 98.3%. Basal did much better than I expected.
Where every model breaks
2 kinds of pairs pulled every score down.
| Pairs | Count | Opus κ | Haiku κ | Jev κ | Basal κ |
|---|---|---|---|---|---|
| Ordinary | 4,627 | 0.97 | 0.92 | 0.90 | 0.94 |
| Traps | 338 | 0.88 | 0.74 | 0.58 | 0.68 |
| Disputed by the annotators | 56 | 0.44 | 0.42 | 0.24 | 0.15 |
On traps, false positives were 19% for Haiku, 10% for Opus, 8% for Jev and 2% for basal.
The other weak spot was questions with 2 parts. "Is the cloud migration paying off" means cost and stability; "is on-call sustainable" means response times and load. On-call scored lowest for every model, and for Jev and basal the cloud question came second lowest. Jev kept only 78% of the relevant claims on the cloud question.
Why an LLM cascade works on this task
An LLM cascade sends every case to a cheap model first and passes only some cases to an expensive one. FrugalGPT (Chen, Zaharia and Zou, 2023) is a research example: an LLM cascade that learns which combinations of models to use for different queries, to reduce cost and improve accuracy.
In our test the filter has 1 job: drop irrelevant pairs. 75 to 80% of pairs never reached Opus, and the filters still passed 97.4 to 99.0% of the relevant ones.
Cost: what a quarter of the bill buys
| System | Cost per 1,000 pairs | How it was measured |
|---|---|---|
| Jev | $0.017 | API bill, 1 claim × 5 questions per request |
| Claude Haiku | $0.056 | API bill, 25 claims per request |
| basal-omni-1.0-ultra | $0.099 | a 10-minute run on 1 rented H100 PCIe at $2.89 an hour |
| Claude Opus | $1.68 | API bill, 25 claims per request |
| Basal, then Opus | $0.52 | filter on every pair, Opus on what passed |
| Jev, then Opus | $0.42 | as above |
The whole run cost $10.52, including $1.31 for starting and testing the GPU machine for basal.
By my arithmetic, at 25 to 31% of the cost, the same budget covers 3 to 4 times as many decisions.
Which filter: Jev or basal
On this test, the 2 filters trade places:
- Jev is the cheapest filter. About 6 times cheaper than basal per pair, about 0.2 seconds per filter request, and it runs as an API with no infrastructure of our own.
- Basal kept the most relevant claims as a filter, and it has open weights: we ran it on our own rented GPU.
The test report recommends Jev as the filter: the same quality as Opus alone (κ 0.953 against 0.959) at $0.42 instead of $1.68 per 1,000 pairs. My own take: basal's quality out of the box and its independence from a model provider may be what points us its way.
The test covers only the step that judges pairs. At the time of the test, Opus still judged every pair.
How to run this test on your own task
Here is the method as steps you can reuse:
- Build a hard test set. Add traps that share words with the right answer. A set of easy rejections only proves that a model can reject.
- Label twice, settle the disputes. 2 independent annotators and a third for disagreements. Say plainly who the annotators are.
- Fix the pass mark first. Recall, false positives and the most harmful error, written down before any model runs.
- Tune thresholds on data the model has not seen. Leave 1 question out, tune on the rest, test on the one left out.
- Measure the filter by what it keeps. A pair the filter drops is lost for good, so measure filter recall, not only the filter's own accuracy.
- Price both parts. The filter on every case plus the large model on what passes.
What comes next: tuning the cheaper judges
Next we tune the cheaper judges for this task and run the same 4,965 decisions again. My take: with tuning, one of them could take this job from Opus on its own.
Remigiusz Kinas, congratulations on basal.
Empowered Intelligence is software by Empowered House that reads the tools a company already uses, such as Notion, Airtable and Attio, and shows leaders how the company runs, with sources behind its answers. Claim judging is 1 of the steps behind those answers.
FAQ
What is basal?
Basal is a Polish decision model from Remigiusz Kinas. The version in this test, basal-omni-1.0-ultra, has 31B parameters, is built on Gemma 4 and has open weights under the Apache 2.0 licence.
How accurate was basal compared with Claude Opus?
On 4,965 claim-judging decisions, basal alone reached 96.9% and Claude Opus 98.6%. As a filter in front of Opus, basal reached 98.5%, at 31% of the cost of Opus alone.
What is an LLM cascade?
An LLM cascade sends each case to a cheaper model first and passes only some cases to a more expensive one. You pay for the large model only on the cases the cheaper one passes on.
Is Jev or basal the better filter?
On this test, Jev was the cheapest filter, at about a sixth of basal's cost per pair, and the test report recommends it. As a filter, on questions it was not tuned on, basal kept more relevant claims, 99.0% against 98.3%, and it has open weights.
How do you test an LLM as a judge?
Build a labelled test set with traps, fix the pass mark before the run, tune thresholds on data the model has not seen, and report both quality and cost per 1,000 decisions.
Does Empowered Intelligence use basal in production?
Not at the time of the test: Claude Opus judged every pair. The test covers only that 1 step.
If you run a cascade on a similar task, where does your filter fail most: on neutral context too?
