DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to replay and export phone agent runs for analysis with Esp32 Iphone Agent

Review each step of an agent run and export screenshots and actions for another agent to analyze.

Authorization notice. Esp32 Iphone Agent is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forA developer or tester who needs to see why a phone agent tapped the wrong thing or got stuck, and wants the evidence in a form another agent can read.

The problemDebugging an AI phone agent is hard when the run is only a live screen. Without a record of each action, failures are guesswork.

The outcomeA replayable run on the phone with taps and swipes drawn on top, plus an exported trace of screenshots, actions and annotations.

What you need first

  • An ESP32-S3 board, tested with a Seeed Studio XIAO ESP32S3 (plain version, 8 MB PSRAM)
  • An iPhone on iOS 18 or later with cellular data
  • An OpenAI-compatible vision model endpoint and API key
  • A Mac with Xcode, needed once to flash the board and build the app

Steps

  1. Complete a run of the agent on the iPhone so it is saved on the phone.
  2. Open Runs in the app and select the run you want to examine.
  3. Replay the run step by step to see the taps and swipes drawn on the screen.
  4. Tap ⋯ and choose Export trace to get the screenshots, actions and saved annotations.
  5. Pass the exported trace to another agent and ask it to explain each failed step.

Prompt for your agent

Using the repo onehumanbeing/esp32-iphone-agent, take the exported trace from <run folder> and list every step where the taps or swipes did not change the screen. Expected output: a table of step number, action and whether the screen changed. Verify that each row matches a screenshot in the trace.

Caveat. Export is done from the app on the phone; the README's batch export and USB copying steps are in docs/run-data-export.md and were not checked here.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page