I use both every day. Ollama runs on my laptop and on the GPU workstation in the office, for development and for my own tools. vLLM serves a chat model from a single 96 GB GPU for a freelance project, first under Docker Compose and now in a small Kubernetes cluster. This post covers what each one is good at, how to install them, and the things that cost me time on the vLLM side.

vLLM vs Ollama at a glance

#

What Ollama is

#

Ollama is the easiest way I know to get a model answering on your own machine. It grew out of llama.cpp and runs quantised GGUF models, which is why an 8B model fits in about 5 GB and runs happily on a MacBook. You pull a model by name, it loads the first time you call it, and it drops out of memory after five minutes with nothing to do, so it shares a machine politely with everything else you have running.

That politeness is the trade-off. Ollama is built around one person at a time. It can answer a few requests in parallel, but every parallel slot costs memory, and it was never meant to batch dozens of users onto one GPU.

What vLLM is

#

vLLM is a serving engine, and it’s built for exactly that batching. Two ideas carry most of the weight. PagedAttention stores each request’s KV cache (the attention state a model builds up as it reads and writes tokens) in small fixed-size blocks, so GPU memory isn’t lost to padding and fragmentation. Continuous batching lets new requests join the batch between generation steps instead of waiting for the current batch to finish. Between them, one GPU running vLLM serves far more concurrent users than the same GPU working through requests one at a time.

The cost is that vLLM behaves as if it owns the card. It claims its share of GPU memory when it starts, fills whatever the weights don’t need with KV cache, and holds all of it whether anyone is calling or not. It also wants full Hugging Face weights or a GPU-friendly quantisation like FP8, not the GGUF files Ollama uses.

One API, two servers

#

Because both expose the OpenAI chat completions API, my code doesn’t know which one it’s talking to. The URL and model name come from the environment:

import os
from openai import OpenAI

# Laptop:     LLM_URL=http://localhost:11434/v1  LLM_MODEL=qwen3:8b
# GPU server: LLM_URL=http://localhost:8000/v1   LLM_MODEL=qwen3-8b
client = OpenAI(base_url=os.environ["LLM_URL"], api_key="not-needed")

reply = client.chat.completions.create(
    model=os.environ["LLM_MODEL"],
    messages=[{"role": "user", "content": "Say hello in Korean."}],
)
print(reply.choices[0].message.content)

On my laptop, which has no big GPU, the app points at Ollama. On the server it points at vLLM. Nothing else changes, and the app can be developed without the GPU box switched on at all. On the vLLM side, --served-model-name gives the model a short name, so clients don’t have to send the full Hugging Face path.

One caution, because it’s easy to forget: the same model on both doesn’t give the same answers. Ollama’s qwen3:8b is a 4-bit quantisation, while on vLLM you’d usually run 16-bit or FP8 weights. A prompt tuned against one can behave differently on the other, so anything you’re measuring, whether that’s answer quality, refusals or whether the JSON parses, has to be checked against the server production will actually use.

How to install Ollama

#

On a Mac or Windows, download the app from ollama.com. On Linux:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:8b

That’s the whole job. The API is now at http://localhost:11434, and it only listens on localhost unless you set OLLAMA_HOST, which is what you want. Mine runs on the office workstation bound to 127.0.0.1, and I reach it from anywhere through a tunnel (how I set that up with Cloudflare WARP).

How to install vLLM

#

On Linux with an NVIDIA GPU and a recent driver:

python -m venv .venv && source .venv/bin/activate
pip install vllm
vllm serve Qwen/Qwen3-8B --served-model-name qwen3-8b --max-model-len 16384

The first run downloads the weights from Hugging Face. Once the log says Application startup complete, curl localhost:8000/v1/models lists the model, and the Python above works against it.

Running vLLM in Docker

#

For anything that has to stay up, I use the official image rather than a pip install, mostly so CUDA, PyTorch and vLLM come pinned together. A generic Compose service:

services:
  vllm:
    image: vllm/vllm-openai:v0.30.0   # pin a tag; the image is about 30 GB
    restart: unless-stopped
    ipc: host                          # PyTorch needs shared memory
    command: >
      --model Qwen/Qwen3-8B
      --served-model-name qwen3-8b
      --gpu-memory-utilization 0.85
      --max-model-len 16384
    volumes:
      - ${HOME}/.cache/huggingface:/root/.cache/huggingface
    ports:
      - "127.0.0.1:8000:8000"          # reach it over ssh -L, not the LAN
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 10s
      timeout: 5s
      retries: 90
      start_period: 120s

Pin the tag. vLLM moves quickly, flags and defaults change between releases, and you want to be the one who decides when that happens. ipc: host (or a generous shm_size) is in vLLM’s own Docker instructions. The long start_period is the first of the things below.

What bit me running vLLM

#

Startup takes minutes. Loading the weights, compiling and capturing CUDA graphs took 190 seconds on my server, and three to five minutes is normal for a large model. A health check written for a web app will kill it before it’s ready. In Compose that means a long start_period; in Kubernetes it means a startup probe. It also means every restart is a few minutes with no model, so restarts shouldn’t be casual.

It takes the GPU memory you give it, at startup. --gpu-memory-utilization is the share of the card vLLM may use, 0.92 by default in recent releases, and whatever the weights don’t need becomes KV cache. That’s fine when vLLM is alone on the card. Mine isn’t, because smaller models run beside it, so I set 0.70 on the 96 GB card. The model, a 35B mixture-of-experts in FP8, loaded at 47.7 GiB and left 15 GiB for KV cache, which is plenty at a 16k context. If vLLM refuses to start because the maximum sequence length won’t fit in the KV cache, lower --max-model-len before you go looking for a bigger GPU.

LoRA slots cost memory up front. vLLM can serve one base model with several LoRA adapters and switch between them per request, by model name, which Ollama can’t really do. But it reserves room for --max-loras adapters at --max-lora-rank when it starts, and on a mixture-of-experts model that reservation covers the experts too, so it adds up fast. Four adapters at rank 64 didn’t fit at 0.70. Two did.

The default sequence limit didn’t suit a hybrid model. The model I serve mixes Mamba-style layers in with attention, and each sequence needs its own cache block for those layers. At vLLM’s default --max-num-seqs on that card it failed to start; --max-num-seqs 256 fixed it. If a newer architecture won’t start, check this flag before anything more exotic.

A Kubernetes Service named vllm crashes vLLM. This one’s my favourite. Kubernetes injects environment variables for every Service in the namespace, named after the Service, so a Service called vllm hands every pod VLLM_PORT=tcp://<cluster-ip>:8000. vLLM reads VLLM_PORT as one of its own settings, couldn’t make sense of a URL there, and died. After loading the whole model, naturally, so every attempt cost three minutes. vLLM’s docs warn about this, which I found afterwards. The fix is enableServiceLinks: false on the pod, or a Service with a different name.

Here’s the shape of a Deployment that works, cut down to the parts that matter:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 1
  strategy:
    type: Recreate               # the old pod has to free the GPU first
  selector:
    matchLabels: {app: vllm}
  template:
    metadata:
      labels: {app: vllm}
    spec:
      enableServiceLinks: false  # no VLLM_PORT=tcp://... from the Service
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.30.0
          args: ["--model=Qwen/Qwen3-8B", "--served-model-name=qwen3-8b"]
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: 32Gi       # loading needs far more host RAM than serving
          volumeMounts:
            - {name: shm, mountPath: /dev/shm}
          startupProbe:          # don't judge it until the model is loaded
            httpGet: {path: /health, port: 8000}
            initialDelaySeconds: 120
            periodSeconds: 10
            failureThreshold: 90
          livenessProbe:         # a restart costs a full reload, so be patient
            httpGet: {path: /health, port: 8000}
            periodSeconds: 30
            timeoutSeconds: 5
            failureThreshold: 10
      volumes:
        - name: shm
          emptyDir: {medium: Memory, sizeLimit: 16Gi}

The probes are the part worth copying. The startup probe waits two minutes, then gives the model another fifteen to come up before Kubernetes calls it dead. The liveness probe only restarts it after about five minutes of failures, because /health can be slow on a busy engine and a restart costs a full reload anyway. Recreate matters on a single GPU: a rolling update starts the new pod first, and it sits there waiting for a GPU the old pod still holds. The memory limit looks generous because it is; mine settles at about 7 GiB of host RAM once the weights are on the GPU, but the load itself needs a lot more.

What to know about Ollama

#

Fewer surprises, mostly because it does less.

The first request after a quiet spell is slow, because the model was unloaded and has to load again. OLLAMA_KEEP_ALIVE sets how long a model stays in memory after its last request; raise it if that pause annoys you, or set it to -1 on a machine that does nothing else.

The context window depends on how much GPU memory you have. Under 24 GiB, Ollama defaults to 4k tokens, far less than most models support, and a prompt that doesn’t fit is trimmed quietly rather than refused. If answers seem to ignore the start of a long prompt, set OLLAMA_CONTEXT_LENGTH (or num_ctx per request) to what you actually need.

If you forward a remote Ollama to your laptop, don’t put it on 11434. That’s Ollama’s default port, and if a local Ollama ever starts, the two race for it and you can end up talking to the wrong one without noticing. I forward mine to 11435.

And it isn’t a multi-user server. It’s fine for an internal tool a handful of people use; once real concurrency turns up, that’s the point to move to vLLM.

How much GPU memory you need

#

Rough sizes for the weights alone:

That’s about 2 bytes per parameter at 16-bit, 1 at 8-bit, and a little over half a byte at Q4_K_M. On top of the weights goes the KV cache, which grows with context length and with how many requests are in flight. That second part is why the same model needs much more memory on vLLM serving fifty users than on Ollama serving you.

Which one to use

#

If you want a model on your own machine, install Ollama and stop there; you’ll have something answering in ten minutes. If you have a GPU server and users, or an app, that will call it all day, use vLLM, accept the slow start and the memory it takes, and get the probes right. Plenty of setups end up with both, mine included: Ollama where the code gets written, vLLM where the requests land.

There are other options. SGLang is the usual alternative to vLLM, but I haven’t run it in production, so I’ve left it out rather than guess. And if the model has to run on a phone, that’s llama.cpp again, which I wrote about in running an LLM on-device in React Native.