Self-Hosted LLM Stack on GPU

Using commercial LLMs in professional contexts raises privacy, cost and external service dependency concerns. This project was born to build a completely local LLM stack, using available homelab hardware, maintaining adequate performance for daily use.

Ollama as the model runtime and integration point, Open WebUI as conversational interface, VS Code integration via Continue for coding support. The setup separates local and cloud profiles, exposed only through Cloudflare Zero Trust with authentication.

The main VM has the NVIDIA A10 24 GB VRAM in passthrough — sufficient for models up to 14B parameters in FP16 or 30B+ in Q4 quantization. For secondary workloads and testing, the AMD RX580 is used via llama.cpp with Vulkan backend.

Main models in use:

- deepseek-r1:14b — reasoning and complex tasks

- deepseek-coder:6.7b — code completion and review

- llama3.1:8b — daily use, fast response

- mistral:7b — Italian, summarization, writing

Open WebUI is exposed via Cloudflare Access with OTP — accessible from any authenticated device without VPN. The Ollama API stays on localhost and is never directly exposed.

VS Code + Continue uses a local profile (LAN) for daily coding and a remote profile (via Tunnel) when working offsite. Local profile latency is <100ms per token with deepseek-coder:6.7b.