LLM Inference Benchmarks: vLLM vs SGLang vs TensorRT-LLM on a Single L40S#

Choosing an inference engine for large language model serving is a decision with direct impact on latency, throughput, and infrastructure cost. This post presents a head-to-head benchmark of three production-grade engines — vLLM, SGLang, and TensorRT-LLM — across four real-world workload scenarios on a single NVIDIA L40S GPU. I specifically chose the L40S because it is a practical, mid-range GPU that is less commonly featured in benchmarks — most published comparisons default to the H100, which may not reflect the hardware teams are actually deploying on. All benchmarks use Nvidia AIPerf for reproducible, standardized measurement.

TL;DR — Summary and Recommendations#

Engine Selection Guide#

Key Takeaways#

1. TensorRT-LLM excels at high concurrency and long-context workloads, but requires more complex deployment (engine compilation, model conversion) and shows higher ITL at high load.

2. vLLM provides the best low-concurrency latency and the most accessible deployment experience. Its quantization support makes AWQ/GPTQ models perform 43% better than FP8 — a significant finding for teams using vLLM.

3. SGLang delivers the most consistent per-token streaming with the lowest ITL, making it ideal for interactive applications where typing-speed output matters.

4. MoE models are a game-changer. The Qwen3-30B-A3B model running on SGLang outperforms every dense model configuration across all engines — by a wide margin. Teams should evaluate MoE alternatives before scaling GPU infrastructure.

5. Quantization format matters more than expected on vLLM. AWQ and GPTQ outperform FP8 by 43% on throughput, suggesting that 4-bit quantization is the practical default for vLLM deployments.

Test Environment and Methodology#

All experiments run on a single machine with:

• GPU: NVIDIA L40S 48 GB
• Model: Qwen3-32B-FP8 (and variants for quantization/MoE tests)
• Benchmark tool: AIPerf v0.11.0
• Protocol: OpenAI-compatible /v1/chat/completions with streaming enabled
• Warmup: 30 requests excluded from results
• Profiling: 300 requests per run
• Input: Synthetic prompts with fixed token lengths (no variance)

Required Software#

• CUDA Toolkit 13 — required by FlashInfer, which SGLang depends on
• NVIDIA-SMI with CUDA 13 driver
• python3-dev — needed for building native extensions

Key Metrics#

Experiment 1: Concurrency Scaling#

Config: Qwen3-32B-FP8, ISL = 1024, OSL = 256, Concurrency = 8 → 32 → 64

This experiment measures how each engine handles increasing concurrent request load — the most common production scaling scenario.

Analysis#

TensorRT-LLM dominates throughput at scale. At concurrency 64, it delivers 342 tok/s — a 37% advantage over vLLM (245 tok/s) and SGLang (250 tok/s). Its throughput continues to increase from c32 to c64, while both vLLM and SGLang plateau.

vLLM has the best low-concurrency latency. At c8, vLLM achieves an average TTFT of just 834 ms — 2.4× faster than SGLang (2,029 ms) and 3× faster than TensorRT-LLM (2,541 ms). This makes vLLM the best choice for low-traffic, latency-sensitive workloads.

SGLang offers the best ITL consistency. Across all concurrency levels, SGLang maintains the lowest inter-token latency: 60.6 → 76.6 → 77.8 ms. This translates to the smoothest token streaming experience for end users.

TensorRT-LLM's TTFT scales most gracefully under load. At c32, TensorRT-LLM's TTFT is 4,554 ms — less than half of vLLM's 11,125 ms and SGLang's 12,332 ms. At c64, the gap narrows but TensorRT-LLM still leads at 11,738 ms vs ~40,500 ms for the other two.

All three engines hit a throughput ceiling around c32–c64. vLLM and SGLang show negligible throughput gain from c32 to c64 (< 1% increase), while TTFT roughly doubles. TensorRT-LLM still extracts meaningful throughput gain at c64 (+18% over c32), but at the cost of significantly higher ITL (124.4 ms).

Experiment 2: Long-Context Performance#

Config: Qwen3-32B-FP8, ISL = 8000, OSL = 512, Concurrency = 8

Long-context workloads (document summarization, RAG with large context windows) stress the prefill pipeline disproportionately. This experiment uses an 8× larger input sequence to reveal engine-level differences in attention computation and memory management.

Analysis#

TensorRT-LLM is the clear winner for long-context workloads. It delivers nearly 2× the throughput (75.9 tok/s vs ~40 tok/s) and an astonishing 7.5× faster average TTFT compared to vLLM and SGLang.

The TTFT gap is the most striking result in this entire benchmark suite: TensorRT-LLM processes 8,000 input tokens and returns the first output token in 8.6 seconds, while vLLM needs 63.8 seconds and SGLang needs 68.6 seconds. This suggests TensorRT-LLM's fused attention kernels and optimized KV-cache allocation provide a substantial advantage when prefill dominates request time.

vLLM has a slight edge over SGLang in both throughput (41.0 vs 38.2 tok/s) and TTFT (63.8s vs 68.6s), though both are significantly slower than TensorRT-LLM.

TensorRT-LLM trades TTFT for higher per-token latency. Its ITL of 88.4 ms is higher than vLLM (68.0 ms) and SGLang (72.6 ms), indicating that while it processes the input much faster, it generates each subsequent token slightly slower. For long-context summarization use cases, the prefill speedup far outweighs this difference.

Experiment 3: Quantization Comparison#

Config: vLLM, Concurrency = 32, ISL = 1024, OSL = 256

This experiment isolates the impact of quantization format by holding the inference engine constant (vLLM) and varying only the model checkpoint: FP8 vs AWQ vs GPTQ.

Analysis#

AWQ and GPTQ dramatically outperform FP8 on vLLM. Both 4-bit quantization methods deliver 43% higher throughput (349 vs 244 tok/s) and 4× faster TTFT (2.7s vs 11.1s) compared to FP8.

AWQ and GPTQ are virtually identical in performance. Throughput differs by less than 0.5%, and latency metrics are within noise margins. This indicates that vLLM's INT4 kernel implementations for both formats are equally optimized.

FP8's worse performance on vLLM is notable. While FP8 preserves more model precision (8-bit vs 4-bit), it uses significantly more memory per parameter, reducing the batch sizes vLLM can maintain at c32. The 4-bit methods leave more VRAM for KV-cache, enabling better batching and pipelining.

TTFT p99 gap is particularly significant. FP8's worst-case first-token latency is 25.8s versus ~11.5s for AWQ/GPTQ — a 2.2× difference that directly impacts tail latency SLOs.

Experiment 4: MoE Model — Qwen3-30B-A3B#

Config: SGLang, Concurrency = 32, ISL = 1024, OSL = 256

Mixture-of-Experts (MoE) models activate only a subset of parameters per token, promising higher throughput with lower compute. This experiment compares the dense Qwen3-32B-FP8 against the MoE Qwen3-30B-A3B-FP8, both served by SGLang.

Analysis#

The MoE model delivers a transformative performance advantage. Qwen3-30B-A3B activates only 3B parameters per token out of its total 30B, resulting in:

• 3.6× higher request throughput (3.51 vs 0.97 req/s)
• 3.6× higher token throughput (897 vs 249 tok/s)
• 15.8× faster average TTFT (779 ms vs 12,332 ms)
• 2.5× lower inter-token latency (31.1 ms vs 76.6 ms)

These are the largest performance differences observed across all experiments. The MoE architecture fundamentally changes the compute profile — with only 3B active parameters, the model effectively runs like a much smaller dense model while maintaining output quality comparable to the full 32B dense variant.

Sub-second TTFT at c32 is a standout result. The MoE model achieves 779 ms average TTFT even at concurrency 32 — faster than the dense model manages at concurrency 8 with any engine. Combined with a p99 TTFT of just 1,303 ms, this makes the MoE model viable for real-time, latency-sensitive applications at high concurrency.

Reproducing These Benchmarks#

All benchmark configurations, run scripts, and raw result JSON files are available in the accompanying repository. Each framework has its own workspace directory with a run_*.py script and a README.md with step-by-step instructions.

# Example: Run SGLang benchmark with Qwen3-32B-FP8 at concurrency 32
cd sglang-workspace
python run_sglang.py --model Qwen/Qwen3-32B-FP8 --concurrency 32 \
    --isl 1024 --osl 256 --requests 300

The raw AIPerf JSON exports are stored in aiperf-workspace/results/ and contain full statistical distributions (avg, p1–p99, min, max, std) for every metric — far more detail than the summary tables presented here.

TensorRT-LLM: Workarounds Applied#

Two issues required workarounds when benchmarking TensorRT-LLM:

1. Token counting mismatch. AIPerf was not counting tokens correctly when targeting TensorRT-LLM's OpenAI-compatible endpoint. The fix was to pass --use-server-token-count and --ignore_eos to the aiperf profile command, which delegates token counting to the server and prevents early termination on EOS tokens.

2. FP8 checkpoint incompatibility. The pre-quantized Qwen3-32B-FP8 checkpoint did not load correctly in TensorRT-LLM due to a configuration issue. To work around this, the tensorrt-llm-workspace/run_trtllm.py script was extended to first download the unquantized Qwen3-32B model and then quantize it to FP8 at build time.

If you run into questions or issues while reproducing these benchmarks, please open an issue on GitHub.