Jev benchmarks, in context
Read the headline numbers, inspect the test conditions, and separate a provider claim from an independent result.
Headline result
Reported test setup
Same task and input
Same baseline and rules
Official headline ratios. Not Jev Tracker measurements.
Speed / Official claim
Up to193.6×
TypeSafe’s workflow evaluation headline, not a result for every task.
The provider describes these gains as being at the high end of expected real-world gains. Not our measurement.
Exact model/configuration behind this headline ratio: not independently reconstructed by Jev Tracker.
Cost / Official claim
Up to444.6×
TypeSafe’s workflow evaluation headline, not a universal price discount.
A workflow comparison does not predict your bill. Jev Tracker has not reproduced this ratio.
Exact model/configuration behind this headline ratio: not independently reconstructed by Jev Tracker.
Published conditions on the left. Our interpretation on the right.
| Condition | What the source says | How to read it |
|---|---|---|
| Tasks | Security incidents, agent trace observability, invoice processing and customer service. View source | Four bounded workflows, not a broad test of chat, writing or general intelligence. |
| Aggregation | Each plotted model configuration averages accuracy, cost and time equally across the four workflows. View source | An aggregate can hide differences between tasks. It is not a median visitor experience. |
| Reference answers | Consensus labels average GPT-6 Astra and Claude Fable 5.1 at high thinking; other models use default reasoning. View source | Accuracy here means agreement with the reference. Unequal reasoning settings matter when interpreting speed. |
| Decision harness | LLM comparisons use a wrapper that returns structured decisions and probabilities. View source | That output contract can add work. Do not treat it as an ordinary chat-response comparison. |
| Timing context | The launch post describes runs from a West Coast laptop, generally near service locations. View source | Network location is part of elapsed time. Local results are not a latency guarantee for every region. |
For this edition, we have not reconstructed the exact peak-ratio model pair, run-level timings, input sizes or a matched sample denominator. These are gaps in our verification, not a claim that the provider has never published them.
Keep three questions separate: was the output well formed, did it match a reference answer, and did the whole task succeed?
See decisions inside real examplesA concrete example / Customer service
The official workflow reads a conversation and account information, then combines model judgments with application rules. Its published examples include disagreements and a case where all compared models miss the reference.
A selected example is not an error-rate sample. It helps explain the failure mode, not how often it happens.
A repeated claim does not become an independent test.
Vercel attributes the same headline ratios to TypeSafe. A second publisher repeating a claim is not a second measurement.
How to compare results fairly.
Not necessarily. Count input preparation, network time, downstream actions, retries and failed decisions. Compare completion quality before comparing elapsed time.
No. A workflow’s cost depends on its inputs, number of calls, output contract and retries. Published unit pricing and the cost of a successful task answer different questions.
A value can fit the allowed format and still be the wrong choice. Evaluate answer quality against labeled examples, then examine what happens when the application acts on a mistake.
The same task and inputs, exact model versions, reasoning settings, location, concurrency, timing boundaries and success criteria. Report failures and variation, not just the most favorable run.
“Source checked” records a publication check, not a successful reproduction. Undated sources keep their publication date unspecified. All check times below are UTC.