Contents

Having AI agents write code only scales so far if people still review and fix every line. Lauren Tan’s approach focuses less on adding model instructions and more on building a development environment that prevents mistakes and lets agents verify their own work.

Key Takeaways#

  • Build trust through codebase mechanisms such as types and dependencies, rather than prompts.
  • Turn agent mistakes into lint rules and CI checks that prevent them from recurring.
  • Provide separate verification CLIs to check correctness and playbooks to improve quality.
  • Clean up bad examples and ad hoc fixes, assuming agents will imitate existing code.
  • Start automation with small verification tasks, and provide a path back to a human when things fail.

Making Trust Systematic#

Drawing on her experience shipping 2,000 PRs a month on SpaceX’s AI team, Lauren Tan discusses “Trust Engineering.” The point is to build an environment where agents are less likely to make mistakes and can catch their own errors, rather than constantly monitoring them.

The mechanisms that support trust have a clear order of priority. The strongest approach is to make invalid states unrepresentable through type systems and data structures. Next come linters, compilers, and CI to catch problems. Written procedures, natural-language rules, and human reviews serve as supporting measures.

Simply adding explanatory text may not prevent the same mistake from happening again. Wherever possible, encode recurring problems into code or validation rules.

Preventing Recurring Mistakes#

Agents tend to use nearby code as a model. If old workarounds and stopgap implementations remain, agents may spread them as correct patterns. Maintaining a codebase means considering what agents will imitate.

Start by choosing one mistake an agent keeps making. If it misuses a particular API, for example, detect that with a lint rule. If the issue is inconsistent data, revisit types and data structures to make invalid values harder to pass in the first place. Even when a fix takes time, automating detection first can keep the problem from spreading.

The approach behind “Dune,” a framework Lauren introduces, is to reduce technical debt, narrow the set of recommended patterns, and turn discovered anti-patterns into lint rules. She also presents a design that bans comments entirely, but I would start by checking whether comments are being used to justify workarounds, rather than applying that policy directly to my team’s codebase.

Tools for Self-Verification#

An agent reporting “I fixed it” is not evidence that the fix is correct. A verification CLI that follows consistent procedures for startup, testing, and measurement lets agents and people check results the same way.

“ControlGlass,” introduced in the video, is one example: it uses the Chrome DevTools Protocol (CDP) to help interact with apps, capture traces, and take heap snapshots. Start by making existing testing and measurement tools callable from a CLI, with results that machines can evaluate. Dedicated tools can come later.

Treat correctness checks and quality improvements separately. Tests and measurements check for functional and performance problems. Put code design guidance and debugging procedures into playbooks for agents. “Pystack,” also introduced in the video, illustrates the idea of turning senior engineers’ workflows into skills.

Expanding Automation Gradually#

Once reliable verification is in place, events such as bug reports can automatically trigger agents. An agent might receive a report, reproduce the issue, develop a fix, and open a PR with test and measurement results attached. Lauren calls this surrounding automation the “Outer Loop.”

There is no need to delegate everything through PR creation at once. Initially, limit the scope to producing reproduction steps and a failing test from a bug report. Once results are consistent, move on to developing fixes, then connect PR creation as the final step. Make sure failures can be detected and a human can take over at each stage.

Focusing solely on 2,000 PRs a month could mean overlooking increased review workload or defects. Alongside PR counts, record verification pass rates, reasons changes are sent back, and defects found after fixes, then adjust the scope of automation.

The Next Step#

Choose one mistake an agent has repeated recently and check whether a lint rule or type constraint could prevent it. Start by adding a detection rule and making failures visible in CI.

Source: https://zenn.dev/dragon1208/articles/506d19d4adb0c4

Contents