DEDUPED Hot AI repos, turned into use cases Get the Monday 5

How to deploy and watch an inference server on a LAN GPU box with ReadyLLM

Add a remote machine over SSH, start an engine there, and watch its GPU and VRAM live.

Short answer: Run cd backend then pip install -r requirements.txt, then uvicorn app.main:app --host 127.0.0.1 --port 8000; then in a second terminal, run cd frontend then npm install && npm run dev, and open http://localhost:3000; on Settings, add the remote machine over SSH with a key or password, confirm the OS, set the engine path and model directory, and click Test connection. All steps.

Credentials. Target settings are saved to ~/.model-deploy-assistant/targets.json, so limit who can read that file on a shared machine.

Remote script. One-click engine install can download, compile, or pip-install software on the remote machine, so check what it will run before you confirm.

Who it is forA team lead or ops engineer with a GPU server on the local network who needs to serve a model to others.

The problemStarting and watching an inference service on a remote GPU machine usually means juggling SSH sessions and separate monitoring tools.

The outcomeA model served by llama.cpp, vLLM, or SGLang on the remote target, with live GPU, VRAM, throughput, and latency curves in the browser.

What you need first

  • Python 3.9+ and Node.js 18+ on the control machine
  • A target machine with llama.cpp, vLLM, or ComfyUI already installed, or installable through the tool
  • Network or SSH access to any remote target you want to manage

Steps

  1. Run cd backend then pip install -r requirements.txt, then uvicorn app.main:app --host 127.0.0.1 --port 8000
  2. In a second terminal, run cd frontend then npm install && npm run dev, and open http://localhost:3000
  3. On Settings, add the remote machine over SSH with a key or password, confirm the OS, set the engine path and model directory, and click Test connection
  4. On Model Store, download a model and confirm integrity after the download finishes
  5. On Deploy, start the inference service and adjust runtime parameters, then open Monitor to watch the live curves

Prompt for your agent

Using the repo htobty/ReadyLLM, start the backend and frontend, add the remote SSH target <target-name> at <host-address>, and deploy a downloaded model with the engine the target has. Report the service status and the current GPU and VRAM readings from Monitor, and verify the service responds before finishing.

Caveat. vLLM does not run natively on Windows, so Windows targets must use WSL2 for it.

Use this in your AI agent

Ask your agent

Using the Deduped MCP, get the setup steps for htobty/ReadyLLM and "Deploy and watch an inference server on a LAN GPU box"

Add Deduped to Claude Code once

claude mcp add --transport http deduped https://dedupedai.com/mcp

Cursor, ChatGPT, others

Share: X LinkedIn Hacker News Email Copy link

Third-party repos we don't own. AI-written use cases can be wrong. Disclaimer