Contents

Preparations to win ISUCON2026 changed direction when a minimal agent setup outperformed a harness built over several weeks. What helped was making evaluations reliable and adding instructions only where repeated failures demonstrated a need.

Key Takeaways#

  • Across 159 evaluations totaling 674 hours, reliable tests proved more useful than detailed instructions for tackling the competition.
  • On ISUCON13, the minimal setup surpassed the elaborate harness’s best eight-hour score in under an hour.
  • The final setup retained operational controls while leaving most optimization decisions to the agent.
  • Restart checks, benchmark isolation, and repeated measurements revealed failures hidden behind impressive scores.

What I Needed the Harness to Do#

ISUCON is a competition to improve application performance within a time limit. My goal was to let a Claude Code session handle eight hours of optimization without human intervention.

I divided the system into a competition agent, a harness containing its instructions and tools, and an evaluation runner. The runner recreated past competition environments on AWS, launched the agent, collected logs, and deleted the infrastructure afterward. It also tested whether the application still worked correctly after the servers restarted.

Preparation ran from August 3 to October 30, 2026. The central question was whether each addition improved the valid final score. Simply keeping the agent running for eight hours was not enough.

Build the Evaluation Before the Strategy#

The first one-hour test exposed a basic recovery problem. After reaching 4,957 points, the agent broke the application and spent the remaining 45 minutes unable to recover. It had worked for 48 minutes without Git, so its first commit saved an already broken state.

I addressed this operationally. Deployments required a commit, benchmarks ran through an HTTP portal, and the agent no longer had direct access to the benchmark machine. Each run used a disposable directory and isolated configuration so that personal instructions, tools, and memory would not change conditions between tests.

The evaluation runner itself also needed validation. One run stalled at the initial setup wizard. Another ended because the runner mistakenly sent /exit. Later, a benchmark running in the background masked the agent’s lack of progress. Monitoring needed to track the agent’s actual progress, and all load generation had to stop before restart validation.

Why More Instructions Lost#

The elaborate full setup defined setup, exploration, convergence, and freeze phases. It also provided metric summaries, optimization playbooks, and structured records linking hypotheses to decisions. The minimal comparison setup, bare, retained the operational infrastructure while removing much of this guidance.

On September 22, bare reached 697,671 points on ISUCON13 after one hour. Until then, full’s best eight-hour score had been 688,906. Bare’s final score was 873,282. These runs did not establish a universal performance ratio, but they made the cost of instructions worth investigating.

Full spent 50–70 minutes on setup and used 51% of its output tokens during exploration on recordkeeping. Its instructions encouraged making small changes sequentially. Bare, meanwhile, rewrote 2,309 lines and raised the score from roughly 3,300 to 175,000 within 12 minutes, then wrote measurement scripts to answer the questions it needed to resolve. On October 5, I stopped developing full and closed 22 issues.

Add Controls Based on Observed Failures#

The production setup, lite, started from bare. Instead of restoring the previous workflow, I added measures addressing failures found during evaluation.

  • Resume when an API error ends a turn, but avoid repeatedly prompting during legitimate waits.
  • Require the agent to read the supplied competition materials at startup.
  • Prompt for restart validation and a final benchmark as the time limit approaches.
  • When progress stalls, have an auditor subagent suggest overlooked directions.

I expanded the auditor’s checklist based on missed opportunities in past runs. Each item described a general optimization consideration without naming the original problem. Its role was to broaden the search; the competition agent remained responsible for validating suggestions.

Adding documentation risked recreating full’s overhead. Every new reporting requirement therefore had to improve decisions or reduce failures.

Validate Outcomes, Not Peak Scores#

Scores were highly noisy. In one run, 58 benchmarks of the same commit produced a 21% score spread. A late personal best could therefore reflect variation rather than improvement. Repeated measurements of an unchanged implementation made the two easier to distinguish.

Evaluation flaws could also steer the agent in the wrong direction. The agent interpreted a failure caused by the benchmark server’s file descriptor limit as an application bottleneck. In another comparison, a score gap initially attributed to the harness was explained by network round-trip times of 0.18 versus 0.38 milliseconds.

The biggest validity issue was persistence. An application that stored data only in memory passed the restart test because the test reinitialized the data before checking anything. I reviewed 114 runs across seven problems, added checks after restart but before initialization, and randomized the restart order. Earlier results also needed qualification because the model might be able to reproduce details of past problems. Unseen and custom problems provided additional supporting evidence.

The Next Step#

Before adding another instruction, run your current harness and a minimal setup on the same task, then compare valid final outcomes, time to the first improvement, and tokens spent on recordkeeping.

Source: https://zenn.dev/luvtechno/articles/isucon2026-preparation

Contents