Overview
Local language models on my own hardware: an RTX 3070 in an always-on server and an RTX 5080 in my gaming PC. Nothing talks to a model directly. Everything goes through one small broker that holds the queue, a key per client, and which jobs each client is allowed to ask for.
Why I built it
I wanted my chat bot and my other projects to use AI without paying per request, without anything leaving the house, and without turning my gaming PC into a server.
How it works
One broker, two GPUs
The broker speaks the same OpenAI-style API that Ollama does, so any script can use it. The 5080 is the first choice for every request. When the PC is off, or I am using the GPU myself, the job falls back to the 3070 on a smaller model, and the answer says where it ran.
The gaming PC runs a small agent that connects out to the broker and fetches jobs over that connection, so the PC never has to accept a connection from anywhere.
Roles, not models
A client asks for a role — general, chat, translate or persona — and the broker picks the model for whichever GPU is free. Each client key only gets the roles it has been given, so the bot in production can translate and do character voices, and nothing else.
Facts from real sources
Models invent. For questions, the model calls read-only lookups instead: F1 results, Wikipedia, game wikis, and a local full-text index of the whole Mass Effect wiki (4,314 articles in SQLite). It never picks a URL itself.
Limits from a load log
Every request is logged with its role, GPU, wait and duration — counts only, never who asked or what they wrote. Limits get chosen from that log, not guessed up front.
What I learned
Measure models on your own GPU, not by reviews. Some results from the test rounds:
| Question | Without lookups | With lookups |
|---|---|---|
| Valheim’s first five bosses | invented boss names, 25 s | right, 6.6 s |
| Current F1 championship leader | a confident wrong guess | right, 3.1 s |
| Next F1 race | “I don’t know” | right, 4.2 s |
- A 9B model “won” the first round on the 3070. Repeating every prompt three times showed the win was luck; Gemma 4 12B got the F1 points and the logic question right 3 of 3.
- An “uncensored” model was tested against its original on rough but harmless chat. The original never refused either, so it added nothing and was removed.
- In a blind test of character voices, the larger model on the 5080 was picked 7 times out of 9.
What’s next
Image understanding, a thinking mode for slow but careful answers, and longer jobs that run on the 5080 overnight.