Contents

Jev makes it easier to separate small agent decisions from large LLMs and delegate them to code or specialized models. The design focus shifts from “which LLM to use” to “how to divide decisions and automate them safely.”

Key Points#

  • Jev is a model intended to return typed decisions from an input state rather than generate text.
  • Separating quick decisions, such as tool selection and retry checks, from large LLMs may reduce latency and costs.
  • Use policies or code to check permissions rather than authorizing actions solely on Jev’s output.
  • Start in shadow mode, comparing Jev’s decisions with existing ones, and examine errors and confidence patterns before automating.

Divide Decisions Among Code, Jev, and LLMs#

Agents repeatedly make quick decisions such as “which tool to use,” “whether to retry,” and “whether to ask a human,” alongside complex reasoning. Delegating all of these to a large LLM adds up in processing time and cost.

The division of responsibilities is simple. Use regular code for deterministic decisions, Jev for narrowly scoped decisions involving some ambiguity, and large LLMs for complex reasoning across multiple sources of information. Jev fills the gap between code and LLMs with what TypeSafe calls “smart if-statements.”

Narrow Down Tools Up Front#

As the number of available tools grows, passing every candidate to the LLM each time becomes unwieldy. Using Jev to narrow the candidates before the LLM makes the final decision reduces the scope the LLM needs to consider.

Start by extracting a single decision, such as which_tool(), and use Jev’s Choice to select candidates. Measure routing accuracy and processing time, then expand once you have confirmed the benefits. This makes the impact of changes easier to track.

Combine Safety with Policies#

Design the system so that file modifications, shell commands, and external API calls are not authorized solely by Jev’s decisions. After receiving Jev’s output and confidence, use the harness’s policies to check permissions and action details, and ask a human when the conditions are not met.

For example, an initial threshold scheme for testing might automate decisions with confidence of 0.95 or above, apply additional checks from 0.70 to below 0.95, and ask a human below that. These numbers are not ready-to-use thresholds. Measure how confidence corresponds to accuracy on real tasks and adjust them.

Compare Decisions in Shadow Mode#

Rather than acting on Jev’s decisions from the start, have it evaluate the same cases in the background while existing rules or human operators handle them. Accumulating results lets you examine error types and the relationship between confidence and accuracy.

Begin by choosing one decision to evaluate and recording both existing decisions and Jev’s output. Once you have enough evaluation data, define what can be automated while retaining human review where needed. Shadow mode is not a Jev-specific feature; it is a general approach to introducing a new decision model into production.

Next Step#

Identify the decisions within your agent and choose one, such as which_tool() or should_retry(). Compare existing results with Jev’s output in shadow mode, examine the relationship between accuracy and confidence, then define what your policies will allow.

Source: https://note.com/npaka/n/ne4547ea3c6f5

Contents