vLLM and Quantization

Self-hosting open-weights LLMs in 2026 is a different problem than it was in 2023. Llama 3.x and Qwen 2.x match or beat hosted GPT-3.5-class models on most tasks, vLLM is the de facto serving stack for GPUs, and four quantization formats (GGUF, AWQ, GPTQ, FP8) cover almost every deployment from a Mac mini to an H100 cluster. This page covers what each layer does, how to launch a serving stack, and how to choose between them.

1. vLLM Architecture

vLLM (Kwon et al., SOSP 2023) brought two ideas to LLM serving that the rest of the field then copied:

• PagedAttention: the KV cache is allocated in fixed-size blocks (like OS virtual memory pages) rather than as one contiguous tensor per sequence. This eliminates internal fragmentation and lets vLLM serve far more concurrent sequences from the same GPU memory.
• Continuous batching: instead of static batches of N sequences that all start and finish together (and waste compute when one finishes early), vLLM swaps new sequences into a slot the moment an old one completes. Average GPU utilization on a steady stream of requests goes from ~30% (static batching) to ~80%+.

Add to that: tensor parallelism (split a model across multiple GPUs), pipeline parallelism (split layers across nodes), chunked prefill (overlap prompt processing with generation of other requests), prefix caching (reuse KV state for shared system prompts), and speculative decoding (a small draft model proposes tokens, the big model verifies in parallel).

2. Launching vLLM with Llama 3 or Qwen 2

# Single-GPU launch — Llama 3.1 8B Instruct, fp16, on one A10G or 4090.
pip install "vllm>=0.6"

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching
  

# Larger model, two GPUs with tensor parallelism — Qwen 2.5 72B AWQ on 2x A100 80GB.
vllm serve Qwen/Qwen2.5-72B-Instruct-AWQ \
  --tensor-parallel-size 2 \
  --quantization awq \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --enable-chunked-prefill
  

Useful flags worth knowing:

• --max-num-seqs N — concurrency cap per GPU; raise for throughput, lower for tail latency.
• --max-model-len — KV cache scales with this; setting it conservatively buys you more concurrent requests.
• --swap-space N — GiB of CPU RAM vLLM may use to swap KV blocks under pressure.
• --dtype bfloat16 — default on Ampere+; float16 is fine on older GPUs.

3. OpenAI-Compatible API

vLLM exposes the OpenAI Chat Completions API on the same port. Any client that talks to OpenAI talks to vLLM unchanged — point base_url at your server.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a SQL query to find duplicate emails in users."}],
    temperature=0.2,
    max_tokens=400,
)
print(resp.choices[0].message.content)
  

Tool use (function calling) works too if you launch with --enable-auto-tool-choice --tool-call-parser llama3_json (or hermes, mistral, depending on the model's tuning).

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser llama3_json
  

4. Quantization Formats: GGUF, AWQ, GPTQ, FP8

5. Picking a Format by Hardware

• Mac (M-series), small CPU box → GGUF Q4_K_M via llama.cpp / Ollama. There is no GPU to feed; quality is acceptable; tooling is the friendliest.
• Single consumer GPU (3090 / 4090, 24 GB) → GGUF (llama.cpp) for chat, or AWQ/GPTQ in vLLM if you need throughput and OpenAI-compatible API.
• A10G / L4 (24 GB) on AWS/GCP → AWQ in vLLM. AWQ kernels are well-optimized for Ampere; FP8 is emulated and slower than AWQ here.
• A100 80 GB → AWQ for memory savings, or bf16 for max quality if the model fits. FP8 emulation is not a win.
• H100 / H200 → FP8 (native hardware support); near-lossless quality with the throughput of an int8 model.
• L40S / 4090 with FP8 hardware → FP8 if the kernels are mature for your model; AWQ otherwise.

6. Throughput vs Quality Numbers

Approximate, drawn from public vLLM and llama.cpp benchmarks on Llama 3.1 8B, A100 80GB, batch 32, prompt 512 / generation 256:

Quality differences below ~1 percentage point are often invisible in domain-specific tasks; for general benchmarks they matter at the margins. Always re-run your own eval suite (see RAG Evaluation) — published deltas are starting points, not destinations.

7. Speculative Decoding

A small "draft" model proposes k tokens; the target model verifies them in a single forward pass. Accepted tokens are kept; rejected ones force a fallback. Net result: 1.5-3x lower latency at identical quality (verification is exact).

# Llama 3.1 70B as target, 8B as draft.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --speculative-model meta-llama/Llama-3.1-8B-Instruct \
  --num-speculative-tokens 5 \
  --use-v2-block-manager
  

Speculation pays off most for code, structured output, and other token-predictable workloads where the small model agrees with the big one often. On free-form creative writing the win is smaller.

8. vLLM vs TGI vs SGLang vs llama.cpp

9. Production Tuning

• Pin --max-model-len to what you actually need. A 128k-context model launched with 128k will reserve KV cache for it and drop your concurrency through the floor.
• Enable --enable-prefix-caching. System prompts and few-shot examples are reused across requests; prefix caching avoids re-prefilling them.
• Enable --enable-chunked-prefill. Long prompts no longer stall short-prompt requests behind them.
• Watch tail latency, not just average. Continuous batching trades P50 for higher P99 under load — set --max-num-seqs to enforce a ceiling.
• One model per process. Multi-tenant vLLM with different models is possible (LoRA adapters) but mixing base models in one process is not.
• Serve metrics on /metrics (Prometheus exposition). Track vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds.
• Cap request size at the gateway. A single 100k-token prompt can trash KV cache for everyone else; reject pathological inputs upstream.
• Quantize, then re-eval. The published "near-lossless" quants are near-lossless on average; your specific task may regress more.

Common Interview Questions:

↑ Back to Top