If you’ve been running AI features in your Python app, you’ve probably used the OpenAI API. It’s reliable, well-documented, and expensive at scale. Now imagine running the same API interface on your own GPU hardware — or a cloud GPU rental — and cutting your inference costs by 90%.
That’s the promise of vLLM. And with vLLM 0.16.0, the open-source LLM serving engine just got its most significant update yet: a WebSocket-based Realtime API that mirrors OpenAI’s voice and streaming interfaces, plus a 30.8% end-to-end throughput improvement from async scheduling and pipeline parallelism.
In this tutorial, you will learn to:
- Understand what vLLM is and how PagedAttention works
- Set up vLLM 0.16.0 as a drop-in OpenAI API replacement on your own GPU
- Use the new WebSocket Realtime API with the OpenAI Python SDK
- Calculate when self-hosting beats the OpenAI API on cost
- Compare vLLM against Ollama, SGLang, and TensorRT-LLM
Page Contents
What is vLLM?
vLLM (Virtual LLM, official docs) is an open-source inference serving engine developed at UC Berkeley’s Sky Computing Lab. It uses a technique called PagedAttention — inspired by virtual memory paging in operating systems — to manage the KV cache memory of running models with far greater efficiency than naive approaches.
The result: vLLM delivers 14–24× higher throughput compared to HuggingFace Transformers on the same hardware, with significantly lower latency. It was built for production use from day one.
vLLM 0.16.0: What’s New
vLLM 0.16.0 (released February 25, 2026) is the largest vLLM release to date — 440 commits from 203 contributors. The headline features:
WebSocket-based Realtime API
The standout addition: a WebSocket Realtime API that is API-compatible with OpenAI’s Realtime API. This means if you have an application using OpenAI’s voice streaming, text completion streaming, or session-based API, you can point it at a vLLM server instead — with no code changes in most cases.
# Instead of OpenAI's API:
from openai import OpenAI
client = OpenAI(api_key="...", base_url="https://api.openai.com/v1")
# Point to your vLLM server instead:
client = OpenAI(
api_key="dummy-key",
base_url="http://localhost:8000/v1" # your vLLM instance
)
# Same API — zero code changes in most cases
completion = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Hello!"}]
)
For the Realtime WebSocket API specifically:
# Start vLLM with realtime API enabled vllm serve meta-llama/Llama-3.3-70B-Instruct --enable-realtime-api --port 8000 # Connect via OpenAI SDK (WebSocket-based) # The vLLM server now accepts OpenAI's realtime client SDK connections
30.8% Throughput Improvement
The second major improvement: async scheduling with pipeline parallelism is now fully supported. This delivers:
- 30.8% end-to-end throughput improvement on benchmark workloads
- 31.8% TPOT (Time Per Output Token) improvement
Pipeline parallelism splits the model across multiple GPUs (or multiple stages on the same GPU), so different layers of the model process different tokens simultaneously. Async scheduling prevents pipeline bubbles — the idle gaps where one GPU finishes a stage and waits for the next. The combination gives vLLM a significant real-world speedup on multi-GPU setups.
Setting Up vLLM as an OpenAI Drop-in Replacement
Option 1: Single GPU (Simplest)
# Install vLLM pip install vllm # Start the server (Llama 3.3 70B on a single A100) vllm serve meta-llama/Llama-3.3-70B-Instruct --host 0.0.0.0 --port 8000 --tensor-parallel-size 1 # Server is now live at http://localhost:8000 # OpenAI-compatible endpoint: http://localhost:8000/v1
Option 2: Multi-GPU (For Larger Models)
# Llama 3.3 70B needs 2x80GB GPUs minimum # Use tensor parallelism to split across GPUs vllm serve meta-llama/Llama-3.3-70B-Instruct --host 0.0.0.0 --port 8000 --tensor-parallel-size 2 --gpu-memory-utilization 0.9 --max-num-batched-tokens 8192
Connect Your App
import openai
# Connect to local vLLM
client = openai.OpenAI(
base_url="http://localhost:8000/v1",
api_key="dummy" # vLLM doesn't require real auth by default
)
# Streaming chat completion
stream = client.chat.completions.create(
model="meta-llama/Llama-3.3-70B-Instruct",
messages=[{"role": "user", "content": "Explain vLLM in one sentence."}],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
Cost Comparison: Self-Hosted vLLM vs OpenAI API
Here’s the real question: when does self-hosting make financial sense? Based on 2026 pricing data (OpenAI API pricing, Lambda Labs cloud GPU rental):
| Setup | Cost Profile | Best For |
|---|---|---|
| OpenAI API (GPT-4.5) | $6.90/M tokens output | Low-to-medium volume, no infra management |
| Anthropic API (Claude) | $9.86/M tokens output | Premium quality, safety-focused workloads |
| Open-weight APIs (DeepInfra) | ~$0.12/M tokens | Budget-sensitive, moderate scale |
| Self-hosted vLLM (cloud GPU rental) | ~$0.50–2/M tokens (amortized) | High-volume, predictable workloads |
| Self-hosted vLLM (owned GPU) | Hardware + electricity only | Very high volume, long-term deployment |
The crossover point for self-hosting is roughly 100M tokens per month for a mid-range application. At that scale, renting a cloud GPU (e.g., Lambda Labs, Modal, Paperspace) becomes cheaper than paying OpenAI’s API fees.
Scenario: 500M tokens/month output ───────────────────────────────────────── OpenAI GPT-4.5: 500 × $6.90 = $3,450/month Self-hosted vLLM: 500 × $0.80 = ~$400/month (cloud GPU rental) Savings: ≈ $3,000/month
For a production application doing 500M tokens/month, self-hosting vLLM can save $30,000+ per year. The caveat: self-hosting adds operational complexity — GPU provisioning, model downloading, serving optimization, and incident response all fall on your team.
vLLM vs the Alternatives
| Feature | vLLM | Ollama | SGLang | TensorRT-LLM |
|---|---|---|---|---|
| OpenAI API compat | Yes | Yes | Yes | Partial |
| Multi-GPU parallelism | TP + PP | Limited | Yes | Yes |
| Throughput | Excellent | Good | Excellent | Best-in-class |
| Ease of setup | Easy | Easiest | Medium | Hard |
| WebSocket Realtime API | Yes (0.16+) | No | Yes | No |
| Best for | General production | Local dev / prototyping | Complex routing | Max performance |
Summary
- vLLM 0.16.0: largest release yet — 440 commits, 203 contributors, WebSocket Realtime API now available
- 30.8% throughput boost from async scheduling + pipeline parallelism — significant for multi-GPU production setups
- Drop-in OpenAI API replacement — point your existing OpenAI SDK app at vLLM with one line change
- Cost break-even: ~100M tokens/month; above that, self-hosted vLLM is significantly cheaper
- Best for: production AI features at scale, teams with GPU access, applications that need data privacy (no API calls leaving your infrastructure)
If you’re building AI features in Python and haven’t evaluated self-hosting yet, vLLM 0.16.0 with its new WebSocket Realtime API is the best excuse to take a second look. Your existing OpenAI SDK code might work with zero changes.
