vLLM vs LMDeploy vs Ollama: what happens as concurrency rises

Why we ran this benchmark

The most common question when choosing an LLM serving engine is “Which one is fastest?” But there’s another question that matters just as much in practice: “How many people will use it at the same time?”

vLLM, LMDeploy and Ollama all provide an OpenAI-compatible API, but they handle requests differently under the hood. In this post, we measured all three engines on the same GPU (RTX 5090), the same model (Qwen3-8B) and the same benchmark tool, at three concurrency levels: 8, 32 and 64. The goal was not to find out which one is “generally fast,” but to confirm with data which engine wins at which traffic level.

Benchmark setup

Hardware

• gcube workload, GPU: NVIDIA GeForce RTX 5090 (32 GB, compute capability 12.0)
• Model: Qwen3-8B (Qwen/Qwen3-8B for vLLM/LMDeploy, the qwen3:8b tag for Ollama; same weights)

Engine versions

Install commands (essentials only)

# System packages
apt-get install -y build-essential curl pciutils

# Ollama
curl -fsSL <https://ollama.com/install.sh> | sh
ollama serve &                 # start in the background
ollama pull qwen3:8b

# vLLM (serving + benchmark client)
python -m venv /opt/venv-vllm
/opt/venv-vllm/bin/pip install vllm==0.26.0

# LMDeploy
python -m venv /opt/venv-lmdeploy
/opt/venv-lmdeploy/bin/pip install lmdeploy==0.14.0

Why we used a single measurement tool

All three engines ship their own benchmark scripts, but different scripts measure in subtly different ways (warm-up handling, token counting and so on), which makes the numbers hard to compare directly. So we used vLLM’s built-in vllm bench serve as a common load generator and hit each engine’s OpenAI-compatible /v1/chat/completions endpoint the same way.

Methodology

Every test used the following fixed conditions.

vllm bench serve \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --base-url <http://127.0.0.1>:{PORT} \
  --model {MODEL_NAME} \
  --dataset-name random \
  --random-input-len 128 --random-output-len 128 \
  --max-concurrency {8|32|64} \
  --num-prompts {N}

Practical tip: Ollama exposes model names in its API as tags like qwen3:8b, which conflicts with benchmark tools that load the tokenizer from a Hugging Face repository ID. Specify --model qwen3:8b (for API requests) and --tokenizer Qwen/Qwen3-8B (for tokenizing) separately to fix this.

Results: throughput (req/s)

Three things stand out.

1. At concurrency 8, LMDeploy leads. At low concurrency, the TurboMind backend has less overhead.
2. As concurrency rises, vLLM pulls ahead. vLLM’s continuous batching gains more as requests pile up, so from 32 onward vLLM clearly leads, and the gap widens at 64.
3. Ollama barely changes as concurrency rises. Going from 8 → 32 → 64, it only moves from 0.14 → 0.31 → 0.30 req/s. It plateaus after 32, and at 64 it’s actually slightly lower than at 32.

The third point is the key one. For ease of local use, Ollama defaults to OLLAMA_NUM_PARALLEL=1, so it effectively processes requests one at a time. Even when the benchmark tool pushes requests at concurrency 64, the amount the server actually processes in parallel barely increases.

Results: time to first token (TTFT)

A metric that affects perceived speed as much as throughput, or even more, is TTFT (time to first token: the time from a request until the first token comes back). Lining up the three engines at concurrency 8 makes the difference even starker.

Ollama’s mean TTFT is about 30 seconds, and its P99 climbs to 75 seconds. That means if 8 people send requests at the same time, the users at the back of the queue may wait close to a minute for a response. This is a difference users feel far more directly than any throughput figure.

Which one to use when

Since the three engines have different characters, the conclusion isn’t “which is best” but “which fits the situation.”

• Ollama — Local development, personal experiments, and low-traffic environments with 1–2 concurrent users. Its biggest advantage, the easiest installation and model switching, is a real strength here.
• LMDeploy — Services with low concurrency (around 10 concurrent users or fewer in our test) that are sensitive to response latency. Its TTFT advantage at low concurrency translates directly into perceived quality.
• vLLM — Production serving with many concurrent users. Thanks to continuous batching, its throughput lead grows as concurrency rises.

On gcube, you can spin GPUs up and down per workload in an instant, so you can reproduce a comparison like this yourself within minutes of deploying a workload. We recommend first estimating which concurrency range your service falls into, then choosing the engine that fits that range.

All figures in this post were measured directly on a gcube RTX 5090 workload.