Jev benchmarks, in context
Read the headline numbers, inspect the test conditions, and separate a provider claim from an independent result.
Headline result
Reported test setup
Same task and input
Same baseline and rules
Official headline ratios. Not Jev Tracker measurements.
Speed / Official claim
Up to193.6×
TypeSafe’s workflow evaluation headline, not a result for every task.
The provider describes these gains as being at the high end of expected real-world gains. Not our measurement.
Exact model/configuration behind this headline ratio: not independently reconstructed by Jev Tracker.
Cost / Official claim
Up to444.6×
TypeSafe’s workflow evaluation headline, not a universal price discount.
A workflow comparison does not predict your bill. Jev Tracker has not reproduced this ratio.
Exact model/configuration behind this headline ratio: not independently reconstructed by Jev Tracker.
Published conditions on the left. Our interpretation on the right.
| Condition | What the source says | How to read it |
|---|---|---|
| Tasks | Security incidents, agent trace observability, invoice processing and customer service. View source | Four bounded workflows, not a broad test of chat, writing or general intelligence. |
| Aggregation | Each plotted model configuration averages accuracy, cost and time equally across the four workflows. View source | An aggregate can hide differences between tasks. It is not a median visitor experience. |
| Reference answers | Consensus labels average GPT-6 Astra and Claude Fable 5.1 at high thinking; other models use default reasoning. View source | Accuracy here means agreement with the reference. Unequal reasoning settings matter when interpreting speed. |
| Decision harness | LLM comparisons use a wrapper that returns structured decisions and probabilities. View source | That output contract can add work. Do not treat it as an ordinary chat-response comparison. |
| Timing context | The launch post describes runs from a West Coast laptop, generally near service locations. View source | Network location is part of elapsed time. Local results are not a latency guarantee for every region. |
We have not verified the exact model pair behind the peak ratios, individual run timings, input sizes or comparable sample counts.
Keep three questions separate: was the output well formed, did it match a reference answer, and did the whole task succeed?
See decisions inside real examplesA concrete example / Customer service
The official workflow reads a conversation and account information, then combines model judgments with application rules. Its published examples include disagreements and a case where all compared models miss the reference.
A selected example is not an error-rate sample. It helps explain the failure mode, not how often it happens.
A repeated claim does not become an independent test.
Vercel attributes the same headline ratios to TypeSafe. A second publisher repeating a claim is not a second measurement.
How to compare results fairly.
Not necessarily. Count input preparation, network time, downstream actions, retries and failed decisions. Compare completion quality before comparing elapsed time.
No. A workflow’s cost depends on its inputs, number of calls, output contract and retries. Published unit pricing and the cost of a successful task answer different questions.
A value can fit the allowed format and still be the wrong choice. Evaluate answer quality against labeled examples, then examine what happens when the application acts on a mistake.
The same task and inputs, exact model versions, reasoning settings, location, concurrency, timing boundaries and success criteria. Report failures and variation, not just the most favorable run.
Original evaluations and integration coverage. Source-check times are shown in UTC.