Most of What Your Agent Calls an LLM For Is a Yes or a No

Four decisions and one sentenceHover to explore

Every so often a product launch is worth reading for its premise rather than its benchmarks. This one has a premise I think is right and benchmarks nobody outside the company has checked, which is an unusual combination and worth separating carefully.

This article works from LangChain's "Building a harness with Jev" by Sydney Runkle and Hunter Lovell, published 17 September 2026; TypeSafe AI's own announcement; a practical API walkthrough; and MindStudio's sceptical read of the claims. Every performance figure below is TypeSafe's own. Section 7 says what that means. The argument from section 2 onward is mine.

1. What was actually shipped

TypeSafe AI released Jev, which it calls a System One model — a class it defines as models built to make fast, structured decisions that software consumes directly, rather than to generate text. LangChain published a harness built on it the same week.

The shape of the thing: you hand it a state and a set of typed questions, and it answers all of them in one pass with probabilities attached. Three question types:

• Choice — pick one of up to 255 options, with a probability per option
• Score — place the input on an ordered scale of 2 to 10 levels
• Noul — a yes-or-no, returned as a single number between 0 and 1

from typesafe_sdk import Choice, Noul, TypeSafeClient

client = TypeSafeClient()
response = client.system_one(
    state={"ticket": "I was charged twice"},
    questions={
        "department": Choice(
            instructions="Which team handles this",
            criteria={"billing": "Payment issues", "technical": "Bugs"},
        ),
        "urgent": Noul(instructions="Message conveys urgency"),
    },
)

Two architectural differences do the work. The model samples all outputs in one query in parallel rather than autoregressively, which is where the latency claim comes from. And it is trained with what TypeSafe calls RLCD — Reinforcement Learning for Calibrated Decisions — which optimises not for an answer a human rater approves of, but for an answer paired with a probability that reflects how often that answer is right.

The published figures: 70–500ms end to end, $0.042 per million input tokens with output free, and on the company's own workflow evaluations 193.6× faster and 444.6× cheaper than the frontier models it compared against. Context is capped at 64k tokens total, 32k for the state.

2. The premise, which is the part that survives

Strip away the vendor and look at what an agent turn actually consists of.

Then, once, at the end: write the answer. That one needs a language model, and nothing here replaces it.

Every one of those decisions currently goes through a model built to produce sentences, priced per token, generating autoregressively, and then gets parsed back down to a single word. You are paying generation prices and generation latency for work whose entire output fits in a byte.

That claim costs nothing to evaluate and does not depend on any of TypeSafe's numbers being right. It is the same move as separating OLTP from analytics, or a cache from a database: two access patterns wearing one component because nobody had yet noticed they were two things.

3. The claim with the longest shelf life is calibration, not speed

Speed and cost advantages get competed away. Within two quarters there will be four of these, and the price will be whatever the second-cheapest one charges. Calibration is the part I would actually build against.

Here is the difference in practice. An LLM asked "is this urgent?" returns urgent. A calibrated classifier returns 0.71.

A label gives you a branch. A probability gives you a threshold, and a threshold is a dial:

1. You can set different bars for different consequences.

   Block at 0.9, flag for review at 0.6, log below that. One question, three behaviours, no extra model calls. With a bare label you get one behaviour and no way to trade precision against recall.

2. You can tune the bar against the capacity you actually have.

   This is the control I argued for in the piece on Anthropic's oversight numbers: decide how many items a human will review each week, then move the threshold until that is what arrives. You cannot do that arithmetic on labels.

3. Confidence becomes a routing signal rather than a vibe.

   Low confidence is precisely the case that should escalate to a slower, more capable model. Most routing today guesses complexity from the input; a calibrated score measures uncertainty about the output, which is the thing you actually wanted to know.

The important word is calibrated. Any model will emit a number if you ask it for one, and the softmax probability on a token is not a claim about correctness. Whether Jev's calibration holds is exactly the sort of thing that needs third-party measurement — and has not had any.

4. What "zero type errors" does and does not promise

TypeSafe claims Jev has a guaranteed 0% structured-output error rate, framed as a mathematical property of the architecture rather than a benchmark result. Taken literally, and I have no reason to doubt it, that means the model cannot return something outside the schema you declared.

That said, eliminating malformed output is a real operational win, and if you have ever shipped an agent you know why: the retry-on-parse-failure path is where a surprising amount of latency, cost and strange behaviour lives. Removing an entire class of failure is worth something even when it is not the class that worries you most.

5. Measure your own System One share this week

Before evaluating any product here, find out whether the premise describes your system. This is an hour of work and the answer is durable regardless of which vendor you end up using.

Instrument every model call your agent makes and bucket it:

Then compute two numbers: the share of calls in the first two buckets, and the share of spend. In most agent stacks I have seen, the first number is high and the second is lower but nowhere near proportionate — decisions are usually short prompts, so they are cheap individually and numerous enough to matter in aggregate, and they dominate latency because they sit on the critical path of every loop.

The threshold I would use:

If decisions and extraction are under a third of your calls, this is an optimisation and you should ignore it until something else is fixed. If they are over two thirds — which is the common case for anything with a tool loop — then your architecture already has two workloads in it and you are one vendor decision away from being able to split them.

6. What to do about it without joining a waitlist

Jev is early access, so the practical question is what you can do now. Three options, in increasing order of effort.

That last one is the real prerequisite. You cannot safely move a decision to a cheaper model without a way to tell whether it got worse, and building that set is work you owe yourself whether or not a System One model ever enters the picture.

7. What I would not claim

Every performance number here is self-reported. No independent party has benchmarked Jev. The model is behind a waitlist, the evaluations are not on public benchmarks, and the comparison numbers come from the company selling the product. That does not make them false; it makes them unverified, and they should carry exactly the discount you would apply to any vendor's own figures.

TypeSafe discloses its own methodology limits, and they are real. The workflow evaluations were built by its model capabilities team, which the company acknowledges could bias them. The comparison baseline is the average of two frontier models. The demos used simplified queries with human-readable keys. Credit for publishing all of that; it does not stop it mattering.

"193.6× faster, 444.6× cheaper" is a ceiling, not an expectation. TypeSafe says as much — it expects these to be at the higher end of real-world gains. Quoting the headline figure as what you will get would be misreading the company's own caveat.

I have not used it. Everything above is read from published material. I am recommending the architectural question, not the product.

Calibration claims are the ones I would most want checked. It is the part of this with real engineering value and the part hardest to verify from outside, and a calibration curve that holds on the vendor's workflows may not hold on yours. If you pilot this, measure calibration on your own data before you wire a threshold to anything that matters.

The honest summary

The launch will be argued about on its numbers, and that argument cannot be settled yet by anyone outside the company.

The part that does not depend on the numbers is the decomposition. An agent loop is mostly a decision engine with a writer bolted on at the end, and the industry has spent two years serving both halves from the same component because that component was the only one available. Someone was going to notice. That there is now a vendor, a training method and a LangChain integration attached to the observation is less interesting than the observation.

Instrument your agent and find out what fraction of its model calls return something you immediately parse down to a single value. If that number is two thirds, you already have two workloads — and you were always going to have to split them, whoever ends up selling the second one.

Sources: LangChain, "Building a harness with Jev" by Sydney Runkle and Hunter Lovell, 17 September 2026 — the harness framing, the question types and the routing and guardrail use cases. TypeSafe AI, "Introducing System One Models & Jev" — the System One definition, the parallel sampler, RLCD, the 70–500ms latency range, the $0.042 per million input tokens with free output, the 193.6× and 444.6× workflow figures, the zero type-error claim, and the methodology caveats quoted in section 7, all of which are TypeSafe's own. A practical guide to Jev — the SDK surface, the 255-option ceiling on Choice, and the 64k total and 32k state context limits. MindStudio, "RLCD vs RLHF" — the absence of any independent verification. The decomposition argument, the case for calibration over speed, the shape-versus-correctness distinction and every recommendation are mine. For the cost mechanics this sits on top of see every file your agent reads stays on the bill, and for the threshold argument see Anthropic's agent oversight numbers.

Related Posts

One Refactor Cost $14.29. Every File the Agent Read Stayed on the BillSeptember 2, 2026

Confidence Is Not a Control: 700 Enterprises on Their Own AgentsSeptember 12, 2026

The Multi-Model Stack: How Startups Are Cutting AI Coding Costs by 85% in 2026July 8, 2026