DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to benchmark a GUI agent on OSWorkerBench tasks with UI Mate

Load the 100 office tasks and 33 plus 45 demonstrations from Hugging Face to measure an agent.

Short answer: Run pip install datasets so the Hugging Face loader is available; then open the osworker_bench/ folder in the repository and read its evaluation guide; run python and then from datasets import load_dataset followed by tasks = load_dataset("tencent/OSWorkerBench"). All steps.

Authorization notice. UI Mate is dual-use. Only use it on software, sites and systems you own or have explicit permission to test.

Who it is forA researcher or evaluation engineer who needs a shared office workflow benchmark to compare computer-use agents.

The problemComparing agents on ad hoc tasks gives results that are hard to reproduce, and office workflows with long horizons are hard to test by hand.

The outcomeYou have the task set and demonstration set loaded in Python, ready to feed into the evaluation harness in osworker_bench/.

What you need first

  • A GPU server able to serve the UI-Mate-27B checkpoint with vLLM (the README example uses two-way tensor parallelism)
  • Python with pip, so you can install openai and pillow
  • A clone of the Tencent/UI-Mate repository and the model weights from Hugging Face
  • A sandboxed or test desktop, since the agent controls real keyboard and mouse input

Steps

  1. Run pip install datasets so the Hugging Face loader is available. (inferred, check the README)
  2. Open the osworker_bench/ folder in the repository and read its evaluation guide.
  3. Run python and then from datasets import load_dataset followed by tasks = load_dataset("tencent/OSWorkerBench").
  4. Run demonstrations = load_dataset("tencent/OSWorkerBench", "demonstrations") to load the 3,989 captioned GUI-action steps.
  5. Follow the end-to-end evaluation guide in the dataset viewer to run the harness with your agent.

Prompt for your agent

Use the Tencent/UI-Mate repository and its osworker_bench/ folder to load the OSWorkerBench tasks with load_dataset("tencent/OSWorkerBench") and the demonstrations configuration. Report the number of tasks and demonstration steps you loaded, and confirm the default configuration returns 100 tasks before running any evaluation.

Caveat. The README describes the evaluation harness but the loading snippet alone does not run an agent, so the full evaluation still needs a served model and the steps in osworker_bench/.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for Tencent/UI-Mate and "Benchmark a GUI agent on OSWorkerBench tasks"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer