Contents
All Claude needed to do was scroll an iOS timeline 300 points and stop. It moved a finger across the simulator, paused at the endpoint, then released.
The timeline moved 1,407 points.
Screenshots were causing another problem, too. A developer building a personal Mastodon client had been asking Claude to check implementation changes in the simulator. Loading screen after screen appeared to be consuming a substantial share of its context.
An investigation of 99 sessions led to a change in workflow: read interface elements and application state as text, use images only for visual checks, and give the agent more predictable tools for interacting with the screen and dragging.
Across four benchmark tasks, the number of images loaded fell from 27 to 1, and context growth decreased by 43%. Execution time, however, increased by 34%.
That tradeoff is what makes this case useful. Reducing screenshots saved a substantial amount of context. But it did not make every operation faster.
What the Session Logs Revealed#
The investigation found that 2,125 simulator images had been loaded into context across 99 sessions. These included screenshots and images generated from recorded video frames.
The standard screenshot tool returned images resized to 920 × 2,000 pixels, consuming roughly 2,500 tokens each. The session with the most images contained 176. Its context reached approximately 770,000 tokens, with images accounting for more than half.
The logs also revealed avoidable inefficiencies in screen interactions.
- More than 900 Bash calls immediately after taps consisted solely of a 1–5-second
sleep. - Of 892 swipes and drags, 165, or 18%, were followed immediately by a screenshot without waiting.
Screenshots taken immediately sometimes captured a transition in progress. Claude would choose coordinates from that image, but the screen had changed by the time it actually tapped.
Analyzing recorded video frames was outside the scope of these improvements. Images remain useful for investigating screen transitions. The focus was routine interactions: finding buttons, opening settings, and checking where scrolling had stopped.
Reading Coordinates and State as Text#
The main addition was a debug-only URL that instructed the app to write a screen dump to a file.
The dump included accessibility element types, labels, and center coordinates, along with application state such as the timeline’s contentOffset.y. Instead of finding tabs in an image, Claude could read a description like this:
画面: 静止済み（0.3秒）
ボタン "ホーム" 選択中 中心=(158,822)
ボタン "通知" 中心=(244,822)
状態: タイムライン
contentOffset.y: -116.0This directly answered two questions: where to tap, and what happened after the interaction.
Tap coordinates used the center of each element’s visible portion. This matters for partially visible rows, because the center of a row’s full frame can lie offscreen or underneath a bar.
In this implementation, UIKit did not populate the required labels unless accessibility support was enabled. Setting ApplicationAccessibilityEnabled in the simulator’s com.apple.Accessibility preferences and restarting the app made button and tab names available.
Because generating accessibility elements can add processing overhead, the developer removes this setting when measuring animation stutter.
External tools such as AXe and idb can also read interface elements. In this case, building the dump into the app was a good fit because the developer wanted access to internal state without adding dependencies.
Checking After Motion Stops#
Switching to text alone does not solve the problem of stale coordinates. The dump also needed a way to wait for the interface to stop moving.
The first approach compared screenshots, taking about three seconds per dump. Instead, the implementation checked the app’s internal animation and scrolling state, waiting until there had been no relevant motion for 250 milliseconds. This brought typical dump times down to about one second.
One detail complicated the check. The _UILiquidLensView in the iOS 26 tab bar retained a position-tracking animation. Including that animation meant the screen was never considered settled.
The solution excluded infinitely repeating animations and animations with calculated durations exceeding 60 seconds. It also imposed a five-second timeout.
This is a practical approximation, not a guarantee that the screen has finished loading. Images can arrive after motion stops. The agent still needs to check that the expected elements are present and request another dump if necessary.
Live timeline updates required a separate control. A debug URL temporarily held back new posts so rows would not shift between inspection and tapping. Releasing the hold inserted the pending posts.
Keeping Screenshots for Visual Checks#
Text can identify an icon element. It cannot tell you whether the icon rendered correctly.
In the new workflow, visual checks used screenshots captured with xcrun simctl io, resized to match the device’s width in points, and saved as JPEGs. On the tested device, that width was 402 pixels.
At this default size, one image pixel corresponded to one point in tap coordinates. A full-screen image required about 470 tokens, roughly one-fifth of the reported consumption of the standard screenshot tool. Cropping to a small icon reduced that to about 50 tokens.
Coordinates in cropped images require care. Their origin is the top-left corner of the crop, so the crop’s offset must be added before tapping.
The practical division is simple: use text to check the location and state of interaction targets. Use images to answer questions about appearance.
Fixing Dragging Before Pursuing Precision#
The 1,407-point scroll resulted from moving a finger 300 points at roughly 3,000 points per second. The end of the movement path repeated the same coordinates to represent a pause before lifting the finger.
But that pause did not reliably eliminate momentum.
Further experiments revealed a clearer pattern. Moving 20 points every 32 milliseconds for a total of 200 points caused the list to scroll 416 points, even with a 300-millisecond pause at the endpoint.
Adding tiny movements at the endpoint changed the result. Alternating the final position by 0.5 points every 16 milliseconds made a 200-point finger movement stop after roughly 180 points of scrolling.
The working hypothesis was that touch_path did not send consecutive identical coordinates as movement events. UIKit appeared to retain the velocity from the last actual movement. Tiny movements at the endpoint were thought to send new events with a much lower velocity.
This mechanism was inferred from observed behavior, not confirmed by inspecting the tool’s implementation.
The completed drag helper moves about 20 points every 32 milliseconds, then adds 16 tiny movements near the endpoint. In one measurement, moving the finger 320 points scrolled the list 300 points.
The initial movement absorbed before scrolling began varied from 10 to 80 points. Precise positioning therefore required reading contentOffset.y after the interaction and correcting the remaining distance.
What the Benchmarks Actually Measured#
The comparison used four tasks:
- Open the third attachment in a dummy image post, swipe once in the viewer, and close it.
- Change the time display setting, then restore it.
- Scroll the timeline exactly 300 points and stop.
- Visually inspect the icon in the top notification and save a screenshot.
The supplied report identifies the test agent as Claude Code using Opus 5.5. Each condition was scheduled to run three times on one Mac and one simulator device. Timing excluded subagent startup time.
| Task | Previous Time | New Time | Previous Image Count | New Image Count | |---|---:|---:|---:|---:| | Image viewer | 18 sec | 46 sec | 5 | 0 | | Settings | 36 sec | 53 sec | 13 | 0 | | Precise scrolling | 47 sec | 28 sec | 5 | 0 | | Notification check | 18 sec | 32 sec | 4 | 1 |
The headline totals are sums of the median values for each task, rather than measurements of a single end-to-end sequence.
| Metric | Previous Workflow | New Workflow | |---|---:|---:| | Images loaded | 27 | 1 | | Estimated image tokens | Approx. 66,000 | Approx. 50 | | Context growth | Approx. 107,000 tokens | Approx. 61,000 tokens | | Interaction time | 119 sec | 159 sec |
There is an important limitation. In four of five attempts, the notification check using the previous workflow stopped with an API-side error immediately after loading the notification screenshot. The cause was unknown, so this condition used the result from the only successful run.
These results support the decision made for this project. They are not a general performance estimate applicable to other apps or machines.
Why Less Context Took More Time#
The new workflow was slower on the image viewer, settings, and notification tasks.
During the benchmarks, dumps took 1.6–5 seconds because they waited for motion to stop. Screenshots returned in about 0.4 seconds. Observed model decision time also increased from roughly 1–2 seconds to 4–5 seconds. The reason was unclear.
The previous workflow sometimes finished faster precisely because it checked too early. If another screenshot quickly corrected stale screen information, the task could finish sooner overall, despite more images and occasional retries.
For precise scrolling, the tradeoff reversed. The new workflow finished in 28 seconds versus 47 seconds previously, requiring only one drag in all three runs. The previous workflow needed two or three drags.
Improve the Interface Before Choosing a Smaller Model#
The developer initially considered delegating simulator interactions to Haiku to conserve Opus’s context.
In early tests with the previous tools, three of the four tasks required retries. One scroll stopped after just 80 points, while another run changed the wrong setting. With the new tools, Haiku succeeded in all eight preliminary trials.
The sample was small. Even so, it suggested that tool design was worth revisiting before choosing a model.
Once screenshots were largely eliminated, direct operation by Opus added only about 14,000–16,000 tokens per task. Delegation took 3–18 seconds for startup and loading instructions, and added a separate reporting step. The developer continued with direct operation and documented the workflow in AGENTS.md.
To try this in your own project, start by choosing one recurring simulator task. Add a text dump containing the visible controls and one measurable state value, then compare context growth, completion time, and accuracy against the screenshot-based workflow. Keep screenshots for checks that actually require looking at the screen.
Source: https://zenn.dev/eai/articles/dont-let-agents-see-the-ios-sim-screen
Contents