DEDUPED Hot AI repos, turned into use cases Get 5 a day

How to run typed extraction on your own GPU to keep data in-house with Type LLM

Serve Qwen3.8-27B with SGLang and point Type LLM at it so documents never leave your server.

Who it is forA team that cannot send internal documents to a hosted API and already has a GPU server.

The problemHosted services may not be acceptable for confidential records, but free-text output still needs schema guarantees.

The outcomeType LLM sends the same typed requests to a local SGLang endpoint and returns typed results from your own model.

What you need first

  • Python with pip installed
  • A TypeLLM API key from typellm.ai (includes $5 free credit), or a GPU server running SGLang with a compatible model such as Qwen3.8-27B
  • Documents or text to process, and a list of the fields you want back

Steps

  1. Follow the Qwen3.8-27B SGLang deployment guide linked in the README to start the server with prefix caching enabled
  2. Run pip install -U typellm
  3. Create TypeLLMClient("http://127.0.0.1:30000", model="Qwen/Qwen3.8-27B") with your server URL
  4. Call generate() with your questions and context as in the README example
  5. Check response.result against a few documents you already know the answers to

Prompt for your agent

Set up TypeLLM/TypeLLM against an SGLang server at <server_url> serving Qwen/Qwen3.8-27B. Run TypeLLMClient.generate() on <document_path> with the questions in <questions_file> and return response.result. Verify that the model returned values for every field and no value falls outside its enum.

Caveat. The README notes that on your own server no service limits apply, and a thinking field without a thinking_budget may reason until the context is full, so set a budget for thinking fields.

Share: X LinkedIn Hacker News Email Copy link

Disclaimer

Deduped lists open-source repositories that belong to other people. We do not own, maintain, audit, or vouch for any of them, and a repo appearing here is not an endorsement or a safety check. The use cases, steps, and prompts are written by AI and may be incomplete, outdated, or just wrong. Where a use case is marked "Seen in the wild", we link to public posts where someone described something similar, but we have not verified those posts and the use case itself is still our AI's own write-up. Use all of this at your own risk. Only run code, commands, or agents on systems and accounts you are authorized to use, and read the code and every command before you run it, especially anything that touches credentials, files, or the network. Nothing here is professional, security, legal, or financial advice. All trademarks and project names belong to their respective owners, and their use here does not imply affiliation. Content is provided as is, without warranties of any kind. Found a mistake, a safety problem, or something you want removed? Email [email protected] and we will look at it promptly. Full disclaimer page