# AI use cases for UI Mate (Tencent/UI-Mate) Repo: [Tencent/UI-Mate](https://github.com/Tencent/UI-Mate) | Use cases: 3 | Page: https://dedupedai.com/use-cases/tencent-ui-mate/ UI-Mate is a foundation GUI agent that observes a live desktop screen and acts through keyboard and mouse, with an OpenAI-compatible desktop client and a demonstration-guided mode. This file holds every use case of this repo. Each one starts with a "## " heading and ends with its URL. - Test a GUI agent on one screenshot locally: https://dedupedai.com/use-cases/tencent-ui-mate/test-gui-agent-on-one-screenshot/ - Benchmark a GUI agent on OSWorkerBench tasks: https://dedupedai.com/use-cases/tencent-ui-mate/benchmark-agent-on-osworkerbench/ - Teach an agent a recurring office task by showing it once: https://dedupedai.com/use-cases/tencent-ui-mate/teach-agent-a-workflow-by-demonstration/ ## How to test a GUI agent on one screenshot locally with UI Mate Repo: [Tencent/UI-Mate](https://github.com/Tencent/UI-Mate) | Difficulty: Intermediate | Time: 1 hour | Basis: From the README Serve UI Mate with vLLM and get a predicted next action for a screenshot you provide. **Who it is for:** A developer or QA engineer who wants to see how a computer-use agent reads a screen before wiring it into a larger workflow. **The problem:** Before running an agent on real work, you need to know whether it can read a specific screen and pick a sensible first move. **The outcome:** You get a predicted action for your own screenshot and instruction, with the screenshot and prediction written to outputs/. ### Steps 1. Run `pip install openai pillow` in the repository environment. 2. Run `vllm serve /path/to/UI-Mate-27B --trust-remote-code --served-model-name UI_Mate --port 8000 --tensor-parallel-size 2 --gpu-memory-utilization 0.85 --mm-encoder-tp-mode data --chat-template-content-format openai --limit-mm-per-prompt '{"image":6,"video":0}'`. (inferred, check the README) 3. Run `curl -s http://127.0.0.1:8000/v1/models` and confirm the served model name. 4. Run `python examples/run_agent.py --image /path/to/screen.png --instruction "" --base-url http://127.0.0.1:8000/v1`. (inferred, check the README) 5. Open the files written to `outputs/` and check the predicted position and action. ### Prompt to paste into your agent ```text Using the Tencent/UI-Mate repository, serve UI-Mate-27B with vLLM on port 8000, then run examples/run_agent.py with the screenshot at and the instruction . Report the predicted action and where the screenshot and prediction were saved in outputs/. Verify that the model name returned by /v1/models is UI_Mate before running. ``` ### Caveat The README example depends on a multi-GPU vLLM setup, so a single small machine may not run the 27B checkpoint without changes to these flags. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/tencent-ui-mate/test-gui-agent-on-one-screenshot/ ## How to benchmark a GUI agent on OSWorkerBench tasks with UI Mate Repo: [Tencent/UI-Mate](https://github.com/Tencent/UI-Mate) | Difficulty: Advanced | Time: 2 hours | Basis: From the README Load the 100 office tasks and 33 plus 45 demonstrations from Hugging Face to measure an agent. **Who it is for:** A researcher or evaluation engineer who needs a shared office workflow benchmark to compare computer-use agents. **The problem:** Comparing agents on ad hoc tasks gives results that are hard to reproduce, and office workflows with long horizons are hard to test by hand. **The outcome:** You have the task set and demonstration set loaded in Python, ready to feed into the evaluation harness in osworker_bench/. ### Steps 1. Run `pip install datasets` so the Hugging Face loader is available. (inferred, check the README) 2. Open the `osworker_bench/` folder in the repository and read its evaluation guide. 3. Run `python` and then `from datasets import load_dataset` followed by `tasks = load_dataset("tencent/OSWorkerBench")`. 4. Run `demonstrations = load_dataset("tencent/OSWorkerBench", "demonstrations")` to load the 3,989 captioned GUI-action steps. 5. Follow the end-to-end evaluation guide in the dataset viewer to run the harness with your agent. ### Prompt to paste into your agent ```text Use the Tencent/UI-Mate repository and its osworker_bench/ folder to load the OSWorkerBench tasks with load_dataset("tencent/OSWorkerBench") and the demonstrations configuration. Report the number of tasks and demonstration steps you loaded, and confirm the default configuration returns 100 tasks before running any evaluation. ``` ### Caveat The README describes the evaluation harness but the loading snippet alone does not run an agent, so the full evaluation still needs a served model and the steps in osworker_bench/. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/tencent-ui-mate/benchmark-agent-on-osworkerbench/ ## How to teach an agent a recurring office task by showing it once with UI Mate Repo: [Tencent/UI-Mate](https://github.com/Tencent/UI-Mate) | Difficulty: Advanced | Time: 3 hours | Basis: From the README Record one successful run, attach it, and have UI-Mate-democua-27B reuse the procedure on a new task. **Who it is for:** An operations lead or power user who repeats the same multi-app office procedure with different files or data each time. **The problem:** Some procedures depend on personal tools, file layouts, and naming rules that are easier to show than to describe in one instruction. **The outcome:** The agent follows the demonstrated procedure where it fits the new task and re-plans from the live screen when layouts or data differ. ### Steps 1. Record one successful run of the procedure with the UI Mate desktop application's demonstration capture. 2. Edit or attach the saved demonstration in the desktop application so it is linked to the new task. 3. Serve `UI-Mate-democua-27B` from its Hugging Face checkpoint using the same vLLM pattern as the general model, with a served model name you choose. 4. Start the run with the new task instruction and the attached demonstration, and use pause or interjection if the agent drifts. 5. Review the session and demonstration artifacts to confirm each subtask completed. ### Prompt to paste into your agent ```text Using the Tencent/UI-Mate repository, attach the demonstration at to a run of UI-Mate-democua-27B on the new task . Report which subtasks the agent completed and whether it re-planned from the live screen anywhere. Verify that the session artifacts show a step-by-step record before you accept the result. ``` ### Caveat The README says the demonstration is advice rather than a script, so results depend on how closely the new task matches the recorded one, and the desktop application setup steps are not in the README excerpt. Risk, credentials: Demonstration recordings capture screenshots and keystrokes, so they can expose passwords or tokens that appear on screen; redact or avoid recording them. Dual-use notice: only use this on software, sites and systems you own or are authorized to test. Disclaimer: Third-party repos we don't own or vouch for. AI-generated use cases can be wrong: review everything and run at your own risk. Full text: https://dedupedai.com/disclaimer/ URL: https://dedupedai.com/use-cases/tencent-ui-mate/teach-agent-a-workflow-by-demonstration/