What is vLLM

What is vLLM? A Practical Guide with Python Examples

Fast, memory-efficient inference and serving for large language models

---

1. What is vLLM?

vLLM is an open-source library for fast LLM inference and serving. It was originally built at UC Berkeley and is now widely used to run models like Llama, Mistral, Qwen, and many others efficiently, both for local experimentation and production-grade serving.

Its core innovation is PagedAttention, a memory management technique inspired by virtual memory paging in operating systems. Instead of allocating one large, contiguous block of GPU memory for each sequence's KV cache (which wastes a lot of memory due to fragmentation), vLLM splits the KV cache into small fixed-size blocks and manages them dynamically — similar to how an OS manages pages of RAM.

The result: vLLM can serve far more concurrent requests on the same GPU compared to naive implementations, with significantly higher throughput.

2. Why people use it

• High throughput — continuous batching keeps the GPU busy instead of idling between requests.
• Efficient memory usage — PagedAttention reduces KV-cache waste, so you fit more sequences in memory.
• OpenAI-compatible API server — you can drop it in as a replacement for OpenAI's API in many existing apps.
• Broad model support — works with most popular Hugging Face transformer architectures.
• Quantization support — AWQ, GPTQ, and other formats to reduce memory footprint further.

3. Installing vLLM

pip install vllm

A CUDA-capable GPU is recommended (vLLM also has CPU and ROCm support, but GPU is the typical use case).

4. Basic offline inference example

The simplest way to use vLLM is through its Python API for offline batch generation:

from vllm import LLM, SamplingParams

# Load a model (downloads from Hugging Face if not cached locally)
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

# Define generation settings
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=200
)

# A batch of prompts — vLLM processes these efficiently together
prompts = [
    "Explain what a neural network is in one paragraph.",
    "Write a short poem about the ocean.",
    "What is the capital of Japan?",
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print("Prompt:", output.prompt)
    print("Response:", output.outputs[0].text)
    print("-" * 40)

5. Running vLLM as an API server

vLLM also ships an OpenAI-compatible server, so you can serve a model over HTTP and call it just like the OpenAI API:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --port 8000

Then call it from Python using the standard openai client library:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"   # vLLM doesn't require a real key by default
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "user", "content": "Give me three tips for learning Python."}
    ],
    temperature=0.7,
)

print(response.choices[0].message.content)

6. Key configuration options worth knowing

llm = LLM(
    model="mistralai/Mistral-7B-Instruct-v0.3",
    tensor_parallel_size=2,       # split model across 2 GPUs
    gpu_memory_utilization=0.9,   # fraction of GPU memory vLLM can use
    max_model_len=8192,           # max context length
    quantization="awq",           # use a quantized checkpoint
    dtype="bfloat16",             # numeric precision
)

• tensor_parallel_size — splits the model across multiple GPUs for larger models.
• gpu_memory_utilization — controls how aggressively vLLM reserves GPU memory for the KV cache.
• quantization — lets you run compressed model weights to save memory and increase speed.

7. When to use vLLM

vLLM shines when you need to serve an LLM to multiple users or handle many requests concurrently — chatbots, internal tools, RAG pipelines, or any production API. For single one-off prompts or very light experimentation, a simpler library like Hugging Face transformers may be enough. But once concurrency and throughput matter, vLLM is one of the most popular choices in the open-source ecosystem.

---

Note: Always check the official vLLM documentation for the latest supported models and configuration options, as the library evolves quickly.