DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to drive an app by tapping its on-screen text with Better AI Chat

Let the AI find and tap buttons by their text in any app, after you grant Accessibility access.

Reverse engineering. The AI taps and types inside other apps, so it may act on a screen you did not expect, and you should keep Build mode confirmations on for any app that handles money or accounts.

Who it is forA user with a repetitive multi-step task in a phone app, such as filling a form, who wants the AI to perform the taps.

The problemTapping through the same screens by hand is slow, and tapping by fixed coordinates breaks when the layout shifts.

The outcomeThe AI completes a short sequence of taps and text entries in one app, and you confirm the result with a screenshot.

What you need first

  • An Android phone that can install an APK from outside the Play Store
  • A browser on the phone or computer to download the APK from GitHub Releases
  • An API key from a supported provider (OpenAI-compatible, Anthropic Claude, or Google Gemini)
  • For building from source: JDK 17 and Android SDK with compileSdk 36
  • Optional: Shizuku for the run_shell and app management tools, and Accessibility and Usage access toggles for full UI automation

Steps

  1. In Settings, turn on the Accessibility and Usage access toggles.
  2. Open the target app and start a chat in Build mode.
  3. Ask the AI to find the target button by its text. It uses ua_find_text or screen_ocr to inspect the screen, then ua_tap_text to tap it.
  4. For text entry or swipes, let the AI use ua_type or ua_swipe. Approve each confirmation prompt in Build mode.
  5. Ask the AI to run take_screenshot or screen_ocr to verify the final screen.

Prompt for your agent

Using the repo Verlintas/BetterAIChat, with Accessibility and Usage access enabled, ask the Agent to open <target app> and tap the button labeled <button text>. Expected output: a screenshot or OCR result showing the next screen. Verify the screen changed as described and that each tap was confirmed.

Caveat. The README says this path needs no root and no Shizuku, but the AI can only tap what the Accessibility service reports on screen, so results depend on each app's layout.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for Verlintas/BetterAIChat and "Drive an app by tapping its on-screen text"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer