# AI use cases for Esp32 Iphone Agent (onehumanbeing/esp32-iphone-agent) Repo: [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | Use cases: 5 | Page: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/ An ESP32-S3 board that acts as an AirPlay screen mirror and USB mouse and keyboard, paired with an iPhone app that lets a vision model see and operate the phone. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Have an AI model complete a task on a real iPhone: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/run-phone-tasks-with-ai-vision/ - Replay and export phone agent runs for analysis: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/export-agent-runs-for-review/ - Start saved phone tasks from Siri or Shortcuts: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/start-saved-phone-tasks-hands-free/ - Edit the planner and actor prompts for a custom workflow: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/adapt-agent-prompts-to-own-workflow/ - Test the ESP32 board over BLE from a Mac: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/debug-board-link-without-phone-app/ ## How to have an AI model complete a task on a real iPhone with Esp32 Iphone Agent Repo: [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | Difficulty: Intermediate | Time: 2 hours | Basis: From the README Give a plain-language goal and let a vision model tap, swipe and type on the phone until done. **Who it is for:** A developer or tinkerer who wants an AI model to run routine phone tasks, such as setting alarms or changing settings, on a physical iPhone with no computer attached at run time. **The problem:** Phone automation usually needs a computer, a developer account or a jailbreak. The phone is still the only real device the task must run on. **The outcome:** The agent reads the mirrored screen, plans and acts through the USB HID hands, and reports when the goal is finished. ### Steps 1. Install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then run `arduino-cli lib install "NimBLE-Arduino@2.5.1"`. 2. Plug the board into the Mac and run `hardware/flash.sh`. 3. Run `cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig`, set DEVELOPMENT_TEAM and IA_BUNDLE_ID, then run `open ios/IPhoneAgent.xcodeproj` and launch the IPhoneAgent scheme on the iPhone. 4. In the app, complete setup: pair the board, join its Wi-Fi and enable Screen Mirroring to ESP32 Agent, turn on AssistiveTouch, then enter your model endpoint and key and choose a vision model. 5. Type a goal such as "Open Settings and go to Wi-Fi" and press Start. Watch the run and stop it from the Dynamic Island if needed. ### Prompt to paste into your agent ```text Using the repo onehumanbeing/esp32-iphone-agent, guide me through the four setup steps in its README for the iPhone at . Expected output: a checklist showing which of the four setup steps are complete. Verify that AssistiveTouch is on and the pointer calibration has finished before any task is started. ``` ### Caveat The app needs a Mac with Xcode once, and the phone's Wi-Fi goes to the board's hotspot during runs, so the model is reached over cellular data. Risk, credentials: The model endpoint key is stored in the iOS Keychain, so protect the key and the phone that holds it. Risk, dual-use: The agent acts with real hands on the phone, so only automate devices and accounts you own or are allowed to use. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/run-phone-tasks-with-ai-vision/ ## How to replay and export phone agent runs for analysis with Esp32 Iphone Agent Repo: [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | Difficulty: Beginner | Time: 30 minutes | Basis: From the README Review each step of an agent run and export screenshots and actions for another agent to analyze. **Who it is for:** A developer or tester who needs to see why a phone agent tapped the wrong thing or got stuck, and wants the evidence in a form another agent can read. **The problem:** Debugging an AI phone agent is hard when the run is only a live screen. Without a record of each action, failures are guesswork. **The outcome:** A replayable run on the phone with taps and swipes drawn on top, plus an exported trace of screenshots, actions and annotations. ### Steps 1. Complete a run of the agent on the iPhone so it is saved on the phone. 2. Open Runs in the app and select the run you want to examine. 3. Replay the run step by step to see the taps and swipes drawn on the screen. 4. Tap ⋯ and choose Export trace to get the screenshots, actions and saved annotations. 5. Pass the exported trace to another agent and ask it to explain each failed step. ### Prompt to paste into your agent ```text Using the repo onehumanbeing/esp32-iphone-agent, take the exported trace from and list every step where the taps or swipes did not change the screen. Expected output: a table of step number, action and whether the screen changed. Verify that each row matches a screenshot in the trace. ``` ### Caveat Export is done from the app on the phone; the README's batch export and USB copying steps are in docs/run-data-export.md and were not checked here. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/export-agent-runs-for-review/ ## How to start saved phone tasks from Siri or Shortcuts with Esp32 Iphone Agent Repo: [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Trigger a saved agent task with a voice command or a shortcut instead of opening the app. **Who it is for:** A user who already runs the agent on their own iPhone and wants to start a repeated task without opening the app each time. **The problem:** Opening the app and typing the goal again for a routine task is slow and repetitive. **The outcome:** A saved task starts from Siri, Shortcuts, a widget or Control Center, and the run appears on the Dynamic Island. ### Steps 1. Finish the one-time setup in the app so the board, mirroring and AssistiveTouch are working. 2. Save a task in the app with the goal you want to repeat. 3. Add the saved task to Shortcuts, a widget or Control Center, as the app supports. 4. Start the task from Siri or the chosen control. 5. Watch the Dynamic Island and stop the run from there if it goes wrong. ### Prompt to paste into your agent ```text Using the repo onehumanbeing/esp32-iphone-agent, check that a saved task named can be started from Shortcuts on the iPhone at . Expected output: a yes or no with the Shortcuts action used. Verify the Dynamic Island shows the run after it starts. ``` ### Caveat The README says these controls can run saved tasks but does not describe the exact Shortcuts setup, so follow the app screens for that step. Risk, dual-use: A task started from Siri or a widget runs without a fresh prompt, so check the saved goal before you trigger it. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/start-saved-phone-tasks-hands-free/ ## How to edit the planner and actor prompts for a custom workflow with Esp32 Iphone Agent Repo: [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | Difficulty: Advanced | Time: 2 hours 30 minutes | Basis: From the README Change the plain-text prompts in the repo and rebuild the app to tune how the agent behaves. **Who it is for:** A developer who finds the agent misreads a certain app or keeps taking the wrong step, and wants to tune its instructions directly. **The problem:** A generic agent prompt may not fit a specific app, so the agent repeats the same mistake. **The outcome:** Updated planner and actor prompts bundled into a rebuilt app that behaves more closely to the target workflow. ### Steps 1. Clone the repo and open the prompts/ folder. 2. Edit the planner and actor prompt text for the workflow you need. 3. Run `open ios/IPhoneAgent.xcodeproj` and rebuild the IPhoneAgent scheme. 4. Install the rebuilt app on the iPhone and run the same goal again. 5. Compare the new run in Runs against the old one to confirm the change. ### Prompt to paste into your agent ```text Using the repo onehumanbeing/esp32-iphone-agent, read the planner and actor prompts in prompts/ and propose one edit for . Expected output: the changed prompt text as a diff. Verify that the edit keeps the existing action names so the app still parses the model reply. ``` ### Caveat The README says prompts are bundled with the app and rebuilt, but it does not describe how the app parses replies, so test each edit on a real run. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/adapt-agent-prompts-to-own-workflow/ ## How to test the ESP32 board over BLE from a Mac with Esp32 Iphone Agent Repo: [onehumanbeing/esp32-iphone-agent](https://github.com/onehumanbeing/esp32-iphone-agent) | Difficulty: Advanced | Time: 1 hour 30 minutes | Basis: From the README Play the app's role from a Mac to check the board's BLE link, hotspot and HID output. **Who it is for:** A firmware developer who needs to check the board without the phone app running. **The problem:** When the agent fails, it is hard to tell whether the board, the BLE link or the phone app is at fault. **The outcome:** The board is exercised from a Mac over BLE, which isolates firmware problems from app problems. ### Steps 1. Install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then run `arduino-cli lib install "NimBLE-Arduino@2.5.1"`. 2. Run `hardware/flash.sh` with the board plugged into the Mac. 3. Read hardware/tools/README.md and the BLE protocol in docs/ble-protocol.md. 4. Run `hardware/tools/board_test.py` to play the app over BLE from the Mac. 5. Record which board command fails and compare it to the app's behavior. ### Prompt to paste into your agent ```text Using the repo onehumanbeing/esp32-iphone-agent, run hardware/tools/board_test.py against the board at and summarize each BLE command it sends. Expected output: a list of commands with pass or fail. Verify the protocol names match docs/ble-protocol.md. ``` ### Caveat The README says the test harness plays the app from a Mac, but it does not list the board_test.py arguments, so read its help output before running it. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/onehumanbeing-esp32-iphone-agent/debug-board-link-without-phone-app/