Contents

Jev is suited to sorting large volumes of items quickly and cheaply, while large-model judges are better for overall assessments that draw on background knowledge or require written explanations. In a comparison of 150 projects, Jev and Claude Fable shared five of their top 10 picks.

Key Takeaways#

  • Jev costs $0.042 per million input tokens, with no output charges and a measured response time of about 300 milliseconds.
  • Among the top 10 selected from the same 150 projects, Jev and Fable shared five picks, while other Claude models and Fable shared six to seven.
  • Jev is suited to judging facts stated in text, but not to assessments requiring background knowledge or holistic judgment.
  • Large models report “confidence” in generated text, while Jev returns calibrated probabilities that can be used as thresholds.
  • In a workflow that uses Jev to narrow the candidates before a large model evaluates the remainder, the initial screening of 150 projects took nine seconds and cost $0.01.

What Each Approach Does Best#

Jev is suited to narrow judgments, such as whether a passage contains a specific fact or a message asks for a money transfer. It returns typed answers and probabilities for each option. These numerical outputs let you build workflows that, for example, defer a decision when the probability falls between 0.4 and 0.6.

Large-model judges excel at comparisons involving multiple criteria and at explaining their evaluations. They are better suited to questions with vague criteria, such as “Is this design good?”, or questions requiring background knowledge unavailable in the text alone. However, their generated “confidence” is not necessarily a calibrated probability.

How the 150 Projects Were Evaluated#

The experiment asked Jev and large-model judges, including Claude Fable, to select the top 10 from 150 Jev-related projects. Jev received only the text for each project and answered eight questions checking whether particular information was present, followed by cross-checks from other perspectives. Weighting and ranking were handled in code.

To compare the leading candidates, Jev evaluated pairs of projects, then answered again with the cards in reverse order. This was because Jev tended to favor the second card by an average of 0.14 in head-to-head comparisons. Even after controlling the comparison conditions, it shared only five picks with Fable.

Interpreting the Numbers#

The full evaluation of all 150 projects took Jev 530 calls, about $0.027, and 30 seconds. Claude Fable took about $0.95 and 105 seconds. Other Claude models shared six to seven of their top 10 picks with Fable, and the Claude models also differed among themselves.

This gap cannot be eliminated simply by refining Jev’s questions. Jev readily picked up measurements and thresholds stated in READMEs, while Fable assessed each project’s significance within its ecosystem. If that significance was not stated in the text, Jev had no basis for judging it. The top-10 overlap is also specific to these 150 projects, not a figure that applies directly to every selection task.

Choosing Which to Use#

Consider the following questions in order to decide which approach to use.

  • Can the question be answered literally from the text at hand? If so, Jev is a good fit.
  • Will the same judgment be repeated thousands or tens of thousands of times? The more frequent the task, the more valuable Jev’s low cost and short response time become.
  • Do you want to set thresholds or defer decisions? Jev is a good fit for numerical control.
  • Do you need background knowledge, trade-offs among multiple criteria, or explanations? If so, use a large-model judge.

“Does this message ask for a money transfer?” suits Jev, while “Is this proposal the best overall?” suits a large model. Counting, calculation, handling dates, and multistep reasoning are also tasks that are difficult to entrust to Jev.

Using Them in Two Stages#

To keep costs down, use Jev to screen candidates broadly, then have a large model evaluate only those that pass. In the experiment, the initial screening processed 150 projects for $0.01 in nine seconds. This suggests a way to reduce the cost and time of having a large model read every item.

Another approach is to have a large model create evaluation rules once, then let Jev handle repeated judgments. This uses the large model’s judgment to define the rules and Jev’s probabilities and low per-call cost for daily checks. Before adopting this approach, evaluate a small sample of your own text against your own criteria to check accuracy.

Next Step#

Choose one question you want to automate, check whether its answer depends solely on information in the text, then try a small-scale evaluation with Jev.

Source: https://jevbooks.com/zh/jev-vs-llm-judge/

Contents