Project case study
Self-Hosted AI Inference on K3s and NVIDIA GPU
A home-lab AI inference platform using K3s, NVIDIA GPU scheduling, Ollama, Open WebUI, ArgoCD, MetalLB, NFS, and persistent Kubernetes workloads.
This project explores GPU-accelerated AI inference on a home-lab Kubernetes environment. It runs Ollama and Open WebUI on K3s with an NVIDIA GeForce RTX 3060 worker and persistent storage for model and application data.
Platform design
The cluster uses the NVIDIA Container Runtime, NVIDIA Device Plugin, and a Kubernetes RuntimeClass to expose GPU resources to scheduled workloads. Deployments request GPU resources explicitly, while readiness and liveness probes protect the user-facing services from incomplete startup states.
MetalLB provides local load-balancer addresses. NFS-backed persistent volumes retain application state and downloaded models. ArgoCD supports repeatable, Git-driven deployment changes.
Operational work
The useful engineering work sits around the model server: GPU runtime initialization, device scheduling, DNS, TLS certificates, persistent-volume provisioning, networking, and health checks. Those constraints turn a local LLM demo into an infrastructure project that can be diagnosed and maintained.
The wider home lab has included Cilium and Hubble for networking and visibility, NGINX Gateway, Kubernetes Gateway API, cert-manager, Cloudflare Tunnel, and a dedicated GPU worker.