A local LLM deployment is simpler to operate than most teams expect and more complex to tune for production than most vendors admit. This guide covers hardware selection through production monitoring, based on deployments across Ollama, vLLM, and llama.cpp in production environments.
Hardware selection
VRAM is the binding constraint. A 7B parameter model quantized to Q4_K_M requires approximately 4.5GB VRAM. A 13B model at the same quantization requires approximately 8GB. A 70B model requires either multiple GPUs or a CPU inference approach with llama.cpp. For production inference serving multiple concurrent users, VRAM determines throughput -- more VRAM means more concurrent context windows, not just larger models.
For single-user or developer workloads: a consumer GPU with 8 to 16GB VRAM runs 7B to 13B models well. For production serving: NVIDIA A10G (24GB) handles 13B models at production throughput. NVIDIA A100 (80GB) handles 70B models or multiple smaller models simultaneously. CPU inference with llama.cpp is viable for batch processing where latency tolerance is 10 seconds or more per query.
Inference server choice
Ollama for local development and single-user deployments: simple model management, OpenAI-compatible API, minimal configuration. vLLM for production multi-user serving: PagedAttention for efficient VRAM use, continuous batching for throughput, structured output support. llama.cpp for CPU inference and environments without GPU hardware.
Production monitoring
Track four metrics: time-to-first-token (TTFT) for latency, tokens per second for throughput, VRAM utilization to detect memory pressure before OOM, and queue depth to detect when you need to scale. Alert when TTFT exceeds your SLA target or when queue depth stays above zero for more than 60 seconds.