Contents

Harness² is a framework that improves an agent’s execution environment for each task and carries those lessons forward to subsequent tasks. It keeps model weights fixed while revising prompts, tool settings, and other components, with benchmarks to measure the effects.

Key Points#

  • Improves the execution environment at two levels: per-task refinement and continual learning.
  • Tests candidates, compares their behavior, and recomposes settings without ground-truth data.
  • Supports OpenCode and Codex execution environments, along with JobBench, LAB, and WorkBuddy-Bench.
  • A small demo runs without model credentials, but benchmarks that use models incur API costs.
  • Environments without network restrictions need safeguards against access to benchmark answers and grading criteria before evaluation.

What Harness² Improves#

An agent’s “harness” is the configuration and tooling used to run the model. Harness² can edit eight components: system prompts, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins.

These settings determine what information the model receives, which tools it can use, and how it approaches its work. The project aims to adapt the surrounding infrastructure to each task without retraining the model itself.

Two Levels of Improvement#

Per-task refinement follows a “Propose–Probe–Compose” process. First, it proposes harness changes and runs them on the same task to examine their effects. It then compares behavior across runs and recomposes the settings. It uses differences between runs as evidence rather than feedback from ground-truth data.

The second level carries lessons from refinement into subsequent tasks. It saves the adopted harness and lessons supported by execution results for use in later improvements. The design accumulates knowledge across tasks instead of starting configuration from scratch each time.

Choosing Parallel or Sequential Mode#

Set the number of recursions with --k. With --k 1, Propose–Probe–Compose runs once. With --k K --mode parallel, it tests general-purpose and domain-specific candidates to produce multiple configurations. With --k K --mode sequential, it refines the harness step by step, passing the improved harness and evidence from each step to the next.

Parallel mode suits a broad search across candidates, while sequential mode suits progressively refining a single configuration. Reported experiments with GLM-5 and Gemini 3.5 Flash outperformed the baseline harness, with gains of up to 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 points in LAB’s partial-pass rate. These figures apply to the published experimental conditions; other models or tasks may not see the same gains.

Start with the Demo#

To try it out, use Linux, Python 3.12, and uv. After cloning the repository, create a virtual environment and run the demo. The demo lets you try both recursion modes without model credentials.

snippet.txt
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2

Running benchmarks requires additional setup, including Node.js 20 or later and the Google Cloud CLI. After configuring a Vertex AI project and model, you can run a single LAB task, for example, as follows.

snippet.txt
harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

Runs that use models incur API costs. To help control costs, first limit the scope with an option such as --max-tasks 1 and use --dry-run to check task selection without calling the model.

What to Check Before Evaluation#

Keep conditions consistent across refinement runs, scoring, and reporting. Differences in the benchmark, execution environment, scope, --k, mode, or seed undermine the comparison. When rerunning with different conditions, use a separate output location. Results are normally saved under runs/, and rerunning the same command resumes from where it left off.

Also, network access in the execution environment is unrestricted, and Codex runs with its sandbox disabled. Because it may be able to access public benchmark answers and grading criteria, network restrictions must be configured before evaluation.

Next Step#

Open google-research/harness2 ↗, run harness2-demo --output runs/demo --k 2 on Linux, and review the parallel and sequential results.

Source: https://github.com/google-research/harness2

Contents