Online FP8 quantization in vLLM on an RTX 5090: when should you use it? — Qwen3-8B measured head to head

Introduction

In our three-way LLM serving benchmark, we compared vLLM, LMDeploy and Ollama. This time, we stayed on vLLM and pitted BF16 against online FP8 quantization. Rather than taking the conventional wisdom that “FP8 is faster” at face value, we checked it on a gcube RTX 5090 container and confirmed the results were reproducible across two independent runs.

The bottom line first: FP8 isn’t always faster. With few concurrent requests, its time per output token (TPOT) is actually worse than BF16’s. FP8’s real advantage — more room for the KV cache — only shows up once concurrency climbs.

Why we ran this experiment

Our earlier attempt at online FP8 quantization (Gemma 3 27B, 58 GB in BF16) failed with an OOM on a 32 GB GPU. The cause is simple. vLLM’s online FP8 conversion needs the original BF16 layers and the converted FP8 layers to sit in VRAM at the same time for a moment, so it gets risky once the model takes up more than 60% of GPU memory.

With that lesson in mind, we scaled down to Qwen3-8B (16 GB in BF16) and compared BF16 and FP8 head to head under identical conditions.

Environment

Practical tip: With this vLLM version, loading online FP8 without enforce_eager=True makes VLLM_COMPILE mode initialization run until it times out and dies. This option matters more than compile-time optimizations.

1. VRAM and memory footprint

Total VRAM usage is almost the same, because as long as there’s a gpu_memory_utilization cap, the KV cache fills whatever memory is left. FP8’s real effect is that the KV cache grows by as much as the model shrinks — in other words, you gain headroom to handle more requests at once.

2. Throughput benchmark — the winner flips with concurrency

We ran vllm bench serve at concurrency 8, 32 and 64, with 128 input and 128 output tokens. (At concurrency 8, we sent warm-up requests first and then measured, to avoid cold-start distortion.)

The pattern is clear.

• FP8 always wins on TTFT. The weights are smaller, so the load and compute path is shorter.
• At concurrency 8, FP8 clearly loses on TPOT (per-token generation speed). With small batches, the dequantization overhead of the FP8 kernels outweighs any throughput gain.
• From concurrency 32 the gap disappears, and at 64 FP8 starts to pull ahead on throughput too. The larger KV cache lets it run bigger batches.

We reproduced these results in two independent runs on a gcube RTX 5090. The VRAM and KV cache figures matched exactly, throughput was within 1%, and TTFT/TPOT were within 4%. The pattern of FP8 losing at low concurrency appeared in both runs — it’s not a fluke but a real characteristic of vLLM’s CUTLASS FP8 kernels.

3. Quality — does it really drop?

We sent seven prompts (code generation, math, Korean marketing copy, a haiku and more) to both BF16 and FP8 at temperature 0 and compared the outputs.

• The outputs weren’t 100% identical token for token (expected with quantization), but the reasoning and final answers were logically consistent.
• The only place the wording diverged was creative prompts that vary a lot anyway, such as marketing copy, and even there it’s hard to say quantization was the cause.

Conclusion — when to use FP8

Online FP8 quantization isn’t a switch that “makes everything faster once you turn it on.” It’s an optimization you decide on based on your real traffic pattern (concurrency).

---

All figures in this post were measured on a gcube RTX 5090 container, and we repeated the same procedure in a separate container to verify reproducibility.