Contents
A browser agent adds a book to a shopping cart. The page shows the requested item, and the task is complete. Yet the agent keeps clicking. Another copy of the same book lands in the cart.
The next action was valid in itself. The mistake was deciding to take another action.
One of Jev’s design patterns addresses this problem with an independent check. Without showing the checker the primary decision’s question or answer, it asks whether the page already shows the requested outcome. Code then decides whether execution should stop.
This example captures the appeal of Jev’s 16 design patterns ↗. These patterns are designed for models that return probabilities rather than text. They explain how to define questions, interpret uncertain answers, and keep application behavior under the control of code.
They are also useful when using conventional LLMs as classifiers, validators, or action selectors. The central question is the same: how can a model’s uncertain assessment lead to reliable behavior?
Probabilities Are Inputs to Decisions#
Jev is TypeSafe AI’s System One model. According to the documentation, it takes a state and typed questions and returns probabilities in roughly 300 milliseconds. It does not generate the text an application needs.
The question types include:
- noul: The probability of “yes.” A value near 0.5 means the model cannot distinguish between the alternatives, rather than that the property is present to a moderate degree.
- choice: A selection from the supplied candidates. An
otheroption provides an escape route when none of the candidates fits. - score: A probability-weighted average of levels defined for a particular situation.
The application remains responsible for making decisions. It determines what to ask, which answers permit an action, and what to do when an answer is missing.
The figures below come from the explanatory pages for Jev’s individual patterns. They are examples from those projects, not guarantees of performance in another application.
1. Decide What the Model Should Judge#
The first four patterns define the boundary between model assessment and application logic. Many prevent problems before any probabilities come back.
1. Viewfinder: Keep the State to the Minimum Needed#
What goes wrong: Sending the entire conversation makes judgment harder. In the documentation’s example, the input reached 40,000 tokens by the 200th edit, and accuracy fell from 0.84 on the first edit to 0.52.
How to prevent it: First define the unit being judged, then include only the context needed for that judgment. To check an edit, send the relevant change and necessary surrounding text, rather than the entire editing history.
2. Menu: Enumerate the Candidates#
What goes wrong: Asking a model to generate a CSS selector can produce nonexistent IDs, matches to multiple elements, or invalid syntax. The application cannot validate the proposed target until it receives it.
How to prevent it: Enumerate valid targets in code and assign them identifiers. Have the model select an identifier, and include “none of the above” so it can decline when the candidates are inadequate.
3. Question bank: Turn Rules into Questions#
What goes wrong: Repeatedly asking a large model to interpret an entire policy takes time and produces inconsistent decisions. In the documentation’s example, each edit cost about two seconds and one cent.
How to prevent it: Convert each rule into a specific question about an observable fact. Ask “Does this contain X?” rather than “Is this good?” Leave counting and date comparisons to code.
4. Ghostwriter: Separate Judgment from Generation#
What goes wrong: A model that returns only probabilities cannot write text to enter into a form. Moving the entire workflow to a large model that generates text increased the time per step from about 300 milliseconds to two seconds in this example.
How to prevent it: Use Jev to select actions and targets. Call a text-generation model only when the selected action requires text, and check its output before using it.
These patterns draw a useful distinction. Selecting an existing option and creating new content are separate jobs. Giving each its own interface makes both easier to check.
2. Structure Questions and Checks#
Even a short question can be poorly designed. The next four patterns address problems being obscured by averaging, unnecessary sequential calls, and missing stopping conditions.
5. Questionnaire: Ask Independent Questions Together#
What goes wrong: With a broad question such as “Does this look like machine-written text?”, one clear flaw can be buried among otherwise unobjectionable passages. In one example, a paragraph containing a detectable problem received a score of 0.52.
How to prevent it: Ask independent questions about specific flaws in a single request. Splitting the check into five types of flaws increased detections from 58 to 63 across 80 examples, without increasing false positives in that evaluation.
6. Prefetch: Ask Branch Questions in Advance#
What goes wrong: Asking for an action and then making another call to ask for its target creates avoidable latency. Two calls of about 300 milliseconds each mean roughly 600 milliseconds of waiting for the model per step.
How to prevent it: Send the questions for possible branches together. After selecting an action, read only the answer for that branch. Unused answers must not serve as grounds for authorizing additional actions.
7. Blind review: Cross-Check Independently#
What goes wrong: An agent may keep acting after completing its task. Clicking again and adding a second copy of the same book to the cart is one example.
How to prevent it: Include an independent verification question in the request. Provide observable facts from the page while withholding the primary decision’s question and answer, and have the checker assess completion independently.
8. Funnel: Route Cases in Stages#
What goes wrong: Sending every case to a large model wastes resources on easy decisions. In the documentation’s example, reviewing 200 files took five minutes and several dollars, even though 190 files had no problems.
How to prevent it: Process cases through code rules, then Jev, then a large model or a human. Stop as soon as a stage reaches a decision, reserving expensive checks for cases where uncertainty remains.
The practical lesson is to use model processing where it can change the outcome. Handle facts that can be determined conclusively with deterministic checks. Route difficult judgments to progressively more expensive reviews.
3. Give Probabilities a Clear Meaning#
A probability alone does not determine what action to take. A value of 0.62 might be enough to justify a comment but insufficient to justify an automatic rewrite.
The next four patterns make that interpretation explicit.
9. Traffic light: Set Thresholds for Each Action#
What goes wrong: Using one threshold for different actions produces unsuitable trade-offs. With a setting that blocked anything above 0.5, 12 of the 14 detections were false positives. Raising the threshold to 0.9, however, missed a real problem scored at 0.62.
How to prevent it: Set thresholds according to the action and its consequences. This example uses 0.8 for rewrites and 0.5 for comments, requiring stronger evidence for actions that are harder to undo.
10. Grey zone: Define a Range for Deferring Judgment#
What goes wrong: Some inputs do not produce enough separation to support a decision. In one example where screenshots were mixed into the input, all ten answers fell between 0.48 and 0.53.
How to prevent it: Treat a central range, such as 0.4–0.6, as “defer judgment.” Record the count for monitoring, but do not shift thresholds to force these answers into a “yes” or “no” branch.
11. Thermostat: Use Hysteresis#
What goes wrong: With a single threshold, small probability fluctuations can make the state oscillate. Setting a combat-detection threshold at 0.7 caused repeated state changes as values moved through 0.68, 0.72, 0.69, and 0.71.
How to prevent it: Use different thresholds for entering and leaving a state. For example, enter the state above 0.8, leave it below 0.6, and retain the previous state in between.
12. Scorecard: Combine Multiple Assessments#
What goes wrong: Asking broadly about a command’s risk on a scale from 0 to 2 can produce an answer of 1 almost every time. That result reveals little about which behavior creates the risk.
How to prevent it: Ask about individual facts, such as deletion, sending data externally, or downloading and executing content. Combine the answers in code using rules that the person responsible for the application can review.
Do not assume a probability threshold has the same meaning across different questions. Each question needs evidence from representative inputs, and each action needs a policy for handling uncertainty.
4. Handle Failures and Measure Performance in Operation#
Even well-designed questions can receive malformed responses. A once-useful question can also lose its value as inputs change.
The final four patterns address these operational problems.
13. Inspection: Validate the Contract#
What goes wrong: Filling missing answers with zero makes a broken response look harmless. In the documentation’s case study, incomplete responses passed through unchecked, and monitoring failed to detect the fault.
How to prevent it: Validate the answer count, probability ranges, and consistency of selections. Represent invalid output as “no answer,” distinct from a valid answer with a low probability.
14. Fail open, fail closed: Configure Failure Behavior#
What goes wrong: An outage can remove protection or bring work to a complete halt. During a 30-minute outage, an editor hook allowed actions through without checking them, while a build server rejected every command.
How to prevent it: Decide in advance which is more costly: interrupting work or allowing unchecked actions. Make that behavior an explicit configuration setting, rather than an accidental consequence of exception handling.
15. Flight recorder: Record and Replay#
What goes wrong: If logs contain only the final decision, you cannot evaluate the effect of changing a threshold. Assessing the impact of raising a deletion threshold from 0.8 to 0.9 requires the original probabilities.
How to prevent it: Make the decision logic a pure function. Record the raw probabilities, decision, previous decision, and input fingerprint so decisions can be replayed under a different policy.
16. Check-up: Calibrate with Real Measurements#
What goes wrong: An inaccurate question can remain in use for months. Of 14 questions, one about the amount of commentary had a precision of only about 20% and went unnoticed for three months.
How to prevent it: Evaluate each question on at least 20 real inputs, measuring its trigger rate and median and maximum probabilities. Check whether its judgments are actually correct, and retire questions that produce little beyond noise.
Replay and calibration serve different purposes. Replay shows what happens when you change the policy applied to recorded answers. Calibration checks whether those answers were useful in the first place.
The Three Patterns I Would Adopt First for LLM-Based Decisions#
For anyone already using an LLM to approve, reject, or classify inputs, I would prioritize these three:
1. Question bank. Replace a broad request such as “Check this for compliance” with questions about identifiable conditions. This makes it clear which parts code can handle and which require model judgment. 2. Grey zone. Treat uncertainty as an explicit application state. Forcing a binary choice hides uncertainty. An explicit “defer judgment” outcome lets you route a case for review or postpone an action. 3. Flight recorder. Preserve the evidence needed to revisit decisions. Without raw answers and replayable logic, threshold tuning becomes guesswork.
Menu is particularly useful when selecting from known actions or targets. Combined with Question bank, it shows why asking one yes/no question per candidate can be a sound design. Code defines the options, and the model evaluates them.
Be aware, however, that individual yes/no answers do not automatically form a mutually exclusive choice. Several candidates may receive high probabilities, or none may meet the conditions. Code must define how to handle both cases.
This week, choose one decision in your application. Rewrite it as a question about an observable fact, define what happens when the answer is uncertain or missing, and record the raw probabilities. Then, before using that decision to control an action with significant consequences, review at least 20 real inputs against Jev’s patterns ↗.
Contents