Contents

In SRE Holdings’ trial, a Claude Code QA team raised 68 findings, of which 9 became tickets after review. Eight were bugs found only by the AI team. The most effective control was the process for rejecting incorrect findings, but four missed bugs showed that PASS verdicts also needed separate checks.

Key Takeaways#

  • Across 6 rounds and 146 test cases, the AI team found 8 bugs that other testers had not reported.
  • Of the 68 findings, 20 were false positives, including 5 that had passed AI verification.
  • A skeptical verifier helped reject incorrect findings, but repeating the tester’s observation method risked reproducing the same mistakes.
  • Rejecting false positives did not prevent incorrect PASS verdicts. Actual values also needed explicit checks against expected results.

Define Each Agent’s Role#

The trial combined predefined test cases and exploratory testing on two internal business SaaS applications. It was a small operational trial and does not demonstrate that AI can replace a QA team.

Claude Code worked with the following division of roles.

  • Leader: Reads specifications, prepares test cases and exploratory testing goals, assigns work, and consolidates findings.
  • Tester: Operates a visible Chrome window, runs assigned cases, and collects screenshots.
  • Verifier: Challenges each finding using a different observation method and returns CONFIRMED, REJECTED, or NEEDS-INFO.
  • Human PM: Provides specifications, handles login, reviews plans, and decides which findings become tickets.

Each tester had a separate Chrome instance. Separate browsers do not guarantee independent tests, however, because accounts may share organizational data or usage limits.

Filter Findings Before Creating Tickets#

The team introduced four review stages between observation and ticket creation. Each stage addressed a different source of unnecessary work.

| Stage | Action | Result | |---|---|---| | Tester reports | Collect steps, screenshots, expected behavior, and self-check results | 68 findings | | Leader review | Remove duplicates, specification questions, and known issues | 22 removed, leaving 46 | | Independent verification | Challenge findings and test alternative explanations | 15 rejected | | Human review | Check remaining findings against specifications and project history | 22 not filed, 9 converted to tickets |

The 59 findings that did not become tickets were not all false positives. Some duplicated existing work or described behavior the project already accepted. Confirmed false positives totaled 20: the 15 rejected during verification plus 5 stopped during human review, or about 29% of all findings.

Require testers to explain how they ruled out operator error. When specifications are unclear, have them submit a question rather than declare a defect.

Change the Verification Method#

Verification has little value if it simply repeats the same mistaken observation. The trial revealed several patterns behind confidently reported but incorrect findings.

  • A button appeared unresponsive. A helper tool selected a hidden duplicate button, clicked coordinates (0,0), and reported success. The tool was changed to stop when no visible target was available.
  • An error appeared to be missing. Both the tester and verifier missed a toast notification that disappeared after two seconds. Screenshots immediately after the action and confirmation of the exact message were needed.
  • A counter appeared to decrease twice. Another tester was using the same organization’s shared quota. The product behaved correctly; parallel testing affected the observation.
  • A row appeared disabled. Its color came from the pointer hovering over it. Moving the pointer away changed the visual evidence.

Require fresh screenshots, clear reproduction steps, and results such as “reproduced 3 out of 5 times” for every finding. Also require any unresolved alternative explanations. An empty field must not be interpreted as proof that every possibility has been ruled out.

Audit PASS Verdicts Separately#

The AI team found important defects, including incorrect statistical calculations and search filters that were not applied. It also missed four bugs within the test scope. In one report, a value was 14.5 times the corresponding value calculated on another screen, yet the case received a PASS because the expected fields were displayed.

The verifier reviewed reported problems. No one systematically challenged PASS verdicts. A review of the cases from rounds with missed bugs found that none had checked the values themselves.

Write expected results as observable comparisons. Replace “the amount field is displayed” with “the amount matches the corresponding value on the comparison screen.” Assign someone to review evidence for PASS verdicts involving calculations or business rules, including boundary cases such as zero amounts.

State the Constraints#

This setup required ongoing maintenance, human login, and processing large numbers of screenshots. According to the source, building and refining the kit took 24 working days and 140 commits. No measured cost savings or execution-time comparisons were provided.

Define operational boundaries before running the agents.

  • Use a test environment and require explicit target URLs and browser connection endpoints.
  • Keep login and credentials under human control.
  • Require approval for creating tickets, deleting data, changing permissions, and changing settings.
  • Isolate shared test data wherever possible, and run tests involving shared counters sequentially.

Next Step#

Test one feature with this workflow and assign a reviewer. Before accepting the results, have that reviewer check both the reported defects and the evidence supporting PASS verdicts involving calculations.

Source: https://zenn.dev/sre_holdings/articles/d4acb079680a79

Contents