DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to test a GUI agent on one screenshot locally with UI Mate

Serve UI Mate with vLLM and get a predicted next action for a screenshot you provide.

Short answer: Run pip install openai pillow in the repository environment; then run vllm serve /path/to/UI-Mate-27B --trust-remote-code --served-model-name UI_Mate --port 8000 --tensor-parallel-size 2 --gpu-memory-utilization 0.85 --mm-encoder-tp-mode data --chat-template-content-format openai --limit-mm-per-prompt '{"image":6,"video":0}'; run curl -s http://127.0.0.1:8000/v1/models and confirm the served model name. All steps.

Authorization notice. UI Mate is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forA developer or QA engineer who wants to see how a computer-use agent reads a screen before wiring it into a larger workflow.

The problemBefore running an agent on real work, you need to know whether it can read a specific screen and pick a sensible first move.

The outcomeYou get a predicted action for your own screenshot and instruction, with the screenshot and prediction written to outputs/.

What you need first

  • A GPU server able to serve the UI-Mate-27B checkpoint with vLLM (the README example uses two-way tensor parallelism)
  • Python with pip, so you can install openai and pillow
  • A clone of the Tencent/UI-Mate repository and the model weights from Hugging Face
  • A sandboxed or test desktop, since the agent controls real keyboard and mouse input

Steps

  1. Run pip install openai pillow in the repository environment.
  2. Run vllm serve /path/to/UI-Mate-27B --trust-remote-code --served-model-name UI_Mate --port 8000 --tensor-parallel-size 2 --gpu-memory-utilization 0.85 --mm-encoder-tp-mode data --chat-template-content-format openai --limit-mm-per-prompt '{"image":6,"video":0}'. (inferred, check the README)
  3. Run curl -s http://127.0.0.1:8000/v1/models and confirm the served model name.
  4. Run python examples/run_agent.py --image /path/to/screen.png --instruction "<your instruction>" --base-url http://127.0.0.1:8000/v1. (inferred, check the README)
  5. Open the files written to outputs/ and check the predicted position and action.

Prompt for your agent

Using the Tencent/UI-Mate repository, serve UI-Mate-27B with vLLM on port 8000, then run examples/run_agent.py with the screenshot at <screenshot path> and the instruction <instruction>. Report the predicted action and where the screenshot and prediction were saved in outputs/. Verify that the model name returned by /v1/models is UI_Mate before running.

Caveat. The README example depends on a multi-GPU vLLM setup, so a single small machine may not run the 27B checkpoint without changes to these flags.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for Tencent/UI-Mate and "Test a GUI agent on one screenshot locally"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer