# Table of Contents - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) - [page](#page) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/README.md "Edit this page") ## Welcome to vLLM[¶](#welcome-to-vllm "Permanent link") [![vLLM Light](https://docs.vllm.ai/en/assets/logos/vllm-logo-text-light.png)](https://docs.vllm.ai/en/assets/logos/vllm-logo-text-light.png) [![vLLM Dark](https://docs.vllm.ai/en/assets/logos/vllm-logo-text-dark.png)](https://docs.vllm.ai/en/assets/logos/vllm-logo-text-dark.png) **Easy, fast, and cheap LLM serving for everyone** [Star](https://github.com/vllm-project/vllm) [Watch](https://github.com/vllm-project/vllm/subscription) [Fork](https://github.com/vllm-project/vllm/fork) vLLM is a fast and easy-to-use library for LLM inference and serving. Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu/) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors. Where to get started with vLLM depends on the type of user. If you are looking to: - Run open-source models on vLLM, we recommend starting with the [Quickstart Guide](https://docs.vllm.ai/en/getting_started/quickstart/) - Build applications with vLLM, we recommend starting with the [User Guide](https://docs.vllm.ai/en/usage/) - Build vLLM, we recommend starting with [Developer Guide](https://docs.vllm.ai/en/contributing/) For information about the development of vLLM, see: - [Roadmap](https://roadmap.vllm.ai/) - [Releases](https://github.com/vllm-project/vllm/releases) vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html) - Continuous batching of incoming requests, chunked prefill, prefix caching - Fast and flexible model execution with piecewise and full CUDA/HIP graphs - Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html) - Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton - Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL - Speculative decoding including n-gram, suffix, EAGLE, DFlash - Automatic kernel generation and graph-level transformations using torch.compile - Disaggregated prefill, decode, and encode vLLM is flexible and easy to use with: - Seamless integration with popular Hugging Face models - High-throughput serving with various decoding algorithms, including _parallel sampling_, _beam search_, and more - Tensor, pipeline, data, expert, and context parallelism for distributed inference - Streaming outputs - Generation of structured outputs using xgrammar or guidance - Tool calling and reasoning parsers - OpenAI-compatible API server, plus Anthropic Messages API and gRPC support - Efficient multi-LoRA support for dense and MoE layers - Support for NVIDIA GPUs, AMD GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more. vLLM seamlessly supports 200+ model architectures on HuggingFace, including: - Decoder-only LLMs (e.g., Llama, Qwen, Gemma) - Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS) - Hybrid attention and state-space models (e.g., Mamba, Qwen3.5) - Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral) - Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT) - Reward and classification models (e.g., Qwen-Math) Find the full list of supported models [here](https://docs.vllm.ai/en/models/supported_models/). For more information, check out the following: - [vLLM announcing blog post](https://blog.vllm.ai/2023/06/20/vllm.html) (intro to PagedAttention) - [vLLM paper](https://arxiv.org/abs/2309.06180) (SOSP 2023) - [How continuous batching enables 23x throughput in LLM inference while reducing p50 latency](https://www.anyscale.com/blog/continuous-batching-llm-inference) by Cade Daniel et al. - [vLLM Meetups](https://docs.vllm.ai/en/community/meetups/) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/README.md "Edit this page") The vllm command-line tool is used to run and manage vLLM models. You can start by viewing the help message with: `[](#__codelineno-0-1)vllm --help` Available Commands: `[](#__codelineno-1-1)vllm {chat,complete,serve,launch,bench,collect-env,run-batch}` ## serve[¶](#serve "Permanent link") Starts the vLLM OpenAI Compatible API server. Start with a model: `[](#__codelineno-2-1)vllm serve meta-llama/Llama-2-7b-hf` Specify the port: `[](#__codelineno-3-1)vllm serve meta-llama/Llama-2-7b-hf --port 8100` Serve over a Unix domain socket: `[](#__codelineno-4-1)vllm serve meta-llama/Llama-2-7b-hf --uds /tmp/vllm.sock` Check with --help for more options: `[](#__codelineno-5-1)# To list all flags [](#__codelineno-5-2)vllm serve --help=all [](#__codelineno-5-3)[](#__codelineno-5-4)# To view an argument group [](#__codelineno-5-5)vllm serve --help=ModelConfig [](#__codelineno-5-6)[](#__codelineno-5-7)# To view a single argument [](#__codelineno-5-8)vllm serve --help=max-num-seqs [](#__codelineno-5-9)[](#__codelineno-5-10)# To search by keyword or flag name [](#__codelineno-5-11)vllm serve --help=max` See [vllm serve](https://docs.vllm.ai/en/latest/serve/) for the full reference of all available arguments. ## launch[¶](#launch "Permanent link") Launch individual vLLM components. `[](#__codelineno-6-1)# Launch the rendering server component [](#__codelineno-6-2)vllm launch render meta-llama/Llama-3.2-1B-Instruct [](#__codelineno-6-3)[](#__codelineno-6-4)# Inspect all available flags for the render component [](#__codelineno-6-5)vllm launch render --help=all` See [vllm launch render](https://docs.vllm.ai/en/latest/launch/render/) for the current launch component reference. ## chat[¶](#chat "Permanent link") Generate chat completions via the running API server. `[](#__codelineno-7-1)# Directly connect to localhost API without arguments [](#__codelineno-7-2)vllm chat [](#__codelineno-7-3)[](#__codelineno-7-4)# Specify API url [](#__codelineno-7-5)vllm chat --url http://{vllm-serve-host}:{vllm-serve-port}/v1 [](#__codelineno-7-6)[](#__codelineno-7-7)# Quick chat with a single prompt [](#__codelineno-7-8)vllm chat --quick "hi"` See [vllm chat](https://docs.vllm.ai/en/latest/chat/) for the full reference of all available arguments. ## complete[¶](#complete "Permanent link") Generate text completions based on the given prompt via the running API server. `[](#__codelineno-8-1)# Directly connect to localhost API without arguments [](#__codelineno-8-2)vllm complete [](#__codelineno-8-3)[](#__codelineno-8-4)# Specify API url [](#__codelineno-8-5)vllm complete --url http://{vllm-serve-host}:{vllm-serve-port}/v1 [](#__codelineno-8-6)[](#__codelineno-8-7)# Quick complete with a single prompt [](#__codelineno-8-8)vllm complete --quick "The future of AI is"` See [vllm complete](https://docs.vllm.ai/en/latest/complete/) for the full reference of all available arguments. ## bench[¶](#bench "Permanent link") Run benchmark tests for latency online serving throughput and offline inference throughput. To use benchmark commands, please install with extra dependencies using `pip install vllm[bench]`. Available Commands: `[](#__codelineno-9-1)vllm bench {latency, serve, throughput}` ### latency[¶](#latency "Permanent link") Benchmark the latency of a single batch of requests. `[](#__codelineno-10-1)vllm bench latency \ [](#__codelineno-10-2) --model meta-llama/Llama-3.2-1B-Instruct \ [](#__codelineno-10-3) --input-len 32 \ [](#__codelineno-10-4) --output-len 1 \ [](#__codelineno-10-5) --enforce-eager \ [](#__codelineno-10-6) --load-format dummy` See [vllm bench latency](https://docs.vllm.ai/en/latest/bench/latency/) for the full reference of all available arguments. ### serve[¶](#serve_1 "Permanent link") Benchmark the online serving throughput. `[](#__codelineno-11-1)vllm bench serve \ [](#__codelineno-11-2) --model meta-llama/Llama-3.2-1B-Instruct \ [](#__codelineno-11-3) --host server-host \ [](#__codelineno-11-4) --port server-port \ [](#__codelineno-11-5) --random-input-len 32 \ [](#__codelineno-11-6) --random-output-len 4 \ [](#__codelineno-11-7) --num-prompts 5` See [vllm bench serve](https://docs.vllm.ai/en/latest/bench/serve/) for the full reference of all available arguments. ### throughput[¶](#throughput "Permanent link") Benchmark offline inference throughput. `[](#__codelineno-12-1)vllm bench throughput \ [](#__codelineno-12-2) --model meta-llama/Llama-3.2-1B-Instruct \ [](#__codelineno-12-3) --input-len 32 \ [](#__codelineno-12-4) --output-len 1 \ [](#__codelineno-12-5) --enforce-eager \ [](#__codelineno-12-6) --load-format dummy` See [vllm bench throughput](https://docs.vllm.ai/en/latest/bench/throughput/) for the full reference of all available arguments. ## collect-env[¶](#collect-env "Permanent link") Start collecting environment information. `[](#__codelineno-13-1)vllm collect-env` ## run-batch[¶](#run-batch "Permanent link") Run batch prompts and write results to file. Running with a local file: `[](#__codelineno-14-1)vllm run-batch \ [](#__codelineno-14-2) -i features/openai_batch/openai_example_batch.jsonl \ [](#__codelineno-14-3) -o results.jsonl \ [](#__codelineno-14-4) --model meta-llama/Meta-Llama-3-8B-Instruct` Using remote file: `[](#__codelineno-15-1)vllm run-batch \ [](#__codelineno-15-2) -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl \ [](#__codelineno-15-3) -o results.jsonl \ [](#__codelineno-15-4) --model meta-llama/Meta-Llama-3-8B-Instruct` See [vllm run-batch](https://docs.vllm.ai/en/latest/run-batch/) for the full reference of all available arguments. ## More Help[¶](#more-help "Permanent link") For detailed options of any subcommand, use: `[](#__codelineno-16-1)vllm --help` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/api/README.md "Edit this page") ## Configuration[¶](#configuration "Permanent link") API documentation for vLLM's configuration classes. - [vllm.config.ModelConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.ModelConfig " ModelConfig") - [vllm.config.CacheConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.CacheConfig " CacheConfig") - [vllm.config.LoadConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.LoadConfig " LoadConfig") - [vllm.config.ParallelConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.ParallelConfig " ParallelConfig") - [vllm.config.SchedulerConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.SchedulerConfig " SchedulerConfig") - [vllm.config.DeviceConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.DeviceConfig " DeviceConfig") - [vllm.config.SpeculativeConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.SpeculativeConfig " SpeculativeConfig") - [vllm.config.LoRAConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.LoRAConfig " LoRAConfig") - [vllm.config.MultiModalConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.MultiModalConfig " MultiModalConfig") - [vllm.config.PoolerConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.PoolerConfig " PoolerConfig") - [vllm.config.StructuredOutputsConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.StructuredOutputsConfig " StructuredOutputsConfig") - [vllm.config.ProfilerConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.ProfilerConfig " ProfilerConfig") - [vllm.config.ObservabilityConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.ObservabilityConfig " ObservabilityConfig") - [vllm.config.KVTransferConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.KVTransferConfig " KVTransferConfig") - [vllm.config.CompilationConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.CompilationConfig " CompilationConfig") - [vllm.config.VllmConfig](https://docs.vllm.ai/en/latest/vllm/config/#vllm.config.VllmConfig " VllmConfig") ## Offline Inference[¶](#offline-inference "Permanent link") LLM Class. - [vllm.LLM](https://docs.vllm.ai/en/latest/vllm/#vllm.LLM " LLM") Prompt schema for LLM APIs. - [vllm.inputs.llm](https://docs.vllm.ai/en/latest/vllm/inputs/llm/#vllm.inputs.llm " vllm.inputs.llm") ## vLLM Engines[¶](#vllm-engines "Permanent link") Engine classes for offline and online inference. - [vllm.LLMEngine](https://docs.vllm.ai/en/latest/vllm/#vllm.LLMEngine " LLMEngine = V1LLMEngine module-attribute ") - [vllm.AsyncLLMEngine](https://docs.vllm.ai/en/latest/vllm/#vllm.AsyncLLMEngine " AsyncLLMEngine = AsyncLLM module-attribute ") ## Inference Parameters[¶](#inference-parameters "Permanent link") Inference parameters for vLLM APIs. - [vllm.SamplingParams](https://docs.vllm.ai/en/latest/vllm/#vllm.SamplingParams " SamplingParams") - [vllm.PoolingParams](https://docs.vllm.ai/en/latest/vllm/#vllm.PoolingParams " PoolingParams") ## Multi-Modality[¶](#multi-modality "Permanent link") vLLM provides experimental support for multi-modal models through the [vllm.multimodal](https://docs.vllm.ai/en/latest/vllm/multimodal/#vllm.multimodal " vllm.multimodal") package. Multi-modal inputs can be passed alongside text and token prompts to [supported models](https://docs.vllm.ai/en/models/supported_models/#list-of-multimodal-language-models) via the `multi_modal_data` field in [vllm.inputs.PromptType](https://docs.vllm.ai/en/latest/vllm/inputs/#vllm.inputs.PromptType " PromptType = DecoderOnlyPrompt | EncoderDecoderPrompt module-attribute "). Looking to add your own multi-modal model? Please follow the instructions listed [here](https://docs.vllm.ai/en/contributing/model/multimodal/). - [vllm.multimodal.MULTIMODAL\_REGISTRY](https://docs.vllm.ai/en/latest/vllm/multimodal/#vllm.multimodal.MULTIMODAL_REGISTRY " MULTIMODAL_REGISTRY = MultiModalRegistry() module-attribute ") ### Internal data structures[¶](#internal-data-structures "Permanent link") - [vllm.multimodal.inputs.PlaceholderRange](https://docs.vllm.ai/en/latest/vllm/multimodal/inputs/#vllm.multimodal.inputs.PlaceholderRange " PlaceholderRange dataclass ") - [vllm.multimodal.inputs.NestedTensors](https://docs.vllm.ai/en/latest/vllm/multimodal/inputs/#vllm.multimodal.inputs.NestedTensors " NestedTensors = Union[list['NestedTensors'], list['torch.Tensor'], 'torch.Tensor', tuple['torch.Tensor', ...]] module-attribute ") - [vllm.multimodal.inputs.MultiModalFieldElem](https://docs.vllm.ai/en/latest/vllm/multimodal/inputs/#vllm.multimodal.inputs.MultiModalFieldElem " MultiModalFieldElem dataclass ") - [vllm.multimodal.inputs.MultiModalFieldConfig](https://docs.vllm.ai/en/latest/vllm/multimodal/inputs/#vllm.multimodal.inputs.MultiModalFieldConfig " MultiModalFieldConfig dataclass ") - [vllm.multimodal.inputs.MultiModalKwargsItem](https://docs.vllm.ai/en/latest/vllm/multimodal/inputs/#vllm.multimodal.inputs.MultiModalKwargsItem " MultiModalKwargsItem") - [vllm.multimodal.inputs.MultiModalKwargsItems](https://docs.vllm.ai/en/latest/vllm/multimodal/inputs/#vllm.multimodal.inputs.MultiModalKwargsItems " MultiModalKwargsItems") ### Data Parsing[¶](#data-parsing "Permanent link") - [vllm.multimodal.parse](https://docs.vllm.ai/en/latest/vllm/multimodal/parse/#vllm.multimodal.parse " vllm.multimodal.parse") ### Data Processing[¶](#data-processing "Permanent link") - [vllm.multimodal.processing](https://docs.vllm.ai/en/latest/vllm/multimodal/processing/#vllm.multimodal.processing " vllm.multimodal.processing") ### Registry[¶](#registry "Permanent link") - [vllm.multimodal.registry](https://docs.vllm.ai/en/latest/vllm/multimodal/registry/#vllm.multimodal.registry " vllm.multimodal.registry") ## Model Development[¶](#model-development "Permanent link") - [vllm.model\_executor.models.interfaces\_base](https://docs.vllm.ai/en/latest/vllm/model_executor/models/interfaces_base/#vllm.model_executor.models.interfaces_base " vllm.model_executor.models.interfaces_base") - [vllm.model\_executor.models.interfaces](https://docs.vllm.ai/en/latest/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces " vllm.model_executor.models.interfaces") - [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/latest/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters") --- # page - [Home](https://docs.vllm.ai/en/) - [User Guide](https://docs.vllm.ai/en/usage/) - [Developer Guide](https://docs.vllm.ai/en/contributing/) - [Benchmarking](https://docs.vllm.ai/en/benchmarking/) - [API Reference](https://docs.vllm.ai/en/api/) - [CLI Reference](https://docs.vllm.ai/en/latest/) - [Community](https://docs.vllm.ai/en/community/contact_us/) 1. [Home](https://docs.vllm.ai/en/) 2. [CLI Reference](https://docs.vllm.ai/en/latest/) [](https://github.com/vllm-project/vllm/edit/main/docs/cli/chat.md "Edit this page") ## Arguments[¶](#arguments "Permanent link") #### `--url`[¶](#-url "Permanent link") url of the running OpenAI-Compatible RESTful API server Default: `http://localhost:8000/v1` #### `--model-name`[¶](#-model-name "Permanent link") The model name used in prompt completion, default to the first model in list models API call. #### `--api-key`[¶](#-api-key "Permanent link") API key for OpenAI services. If provided, this api key will overwrite the api key obtained through environment variables. It is important to note that this option only applies to the OpenAI-compatible API endpoints and NOT other endpoints that may be present in the server. See the security guide in the vLLM docs for more details. #### `--system-prompt`[¶](#-system-prompt "Permanent link") The system prompt to be added to the chat template, used for models that support system prompts. #### `-q`, `--quick`[¶](#-q-quick "Permanent link") Send a single prompt as MESSAGE and print the response, then exit. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/benchmarking/sweeps.md "Edit this page") `vllm bench sweep` is a suite of commands designed to run benchmarks across multiple configurations and compare them by visualizing the results. ## Online Benchmark[¶](#online-benchmark "Permanent link") ### Basic[¶](#basic "Permanent link") `vllm bench sweep serve` starts `vllm serve` and iteratively runs `vllm bench serve` for each server configuration. Tip If you only need to run benchmarks for a single server configuration, consider using [GuideLLM](https://github.com/vllm-project/guidellm), an established performance benchmarking framework with live progress updates and automatic report generation. It is also more flexible than `vllm bench serve` in terms of dataset loading, request formatting, and workload patterns. Follow these steps to run the script: 1. Construct the base command to `vllm serve`, and pass it to the `--serve-cmd` option. 2. Construct the base command to `vllm bench serve`, and pass it to the `--bench-cmd` option. 3. (Optional) If you would like to vary the settings of `vllm serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--serve-params`. - Example: Tuning `--max-num-seqs` and `--max-num-batched-tokens`: `[](#__codelineno-0-1)[ [](#__codelineno-0-2) { [](#__codelineno-0-3) "max_num_seqs": 32, [](#__codelineno-0-4) "max_num_batched_tokens": 1024 [](#__codelineno-0-5) }, [](#__codelineno-0-6) { [](#__codelineno-0-7) "max_num_seqs": 64, [](#__codelineno-0-8) "max_num_batched_tokens": 1024 [](#__codelineno-0-9) }, [](#__codelineno-0-10) { [](#__codelineno-0-11) "max_num_seqs": 64, [](#__codelineno-0-12) "max_num_batched_tokens": 2048 [](#__codelineno-0-13) }, [](#__codelineno-0-14) { [](#__codelineno-0-15) "max_num_seqs": 128, [](#__codelineno-0-16) "max_num_batched_tokens": 2048 [](#__codelineno-0-17) }, [](#__codelineno-0-18) { [](#__codelineno-0-19) "max_num_seqs": 128, [](#__codelineno-0-20) "max_num_batched_tokens": 4096 [](#__codelineno-0-21) }, [](#__codelineno-0-22) { [](#__codelineno-0-23) "max_num_seqs": 256, [](#__codelineno-0-24) "max_num_batched_tokens": 4096 [](#__codelineno-0-25) } [](#__codelineno-0-26)]` 4. (Optional) If you would like to vary the settings of `vllm bench serve`, create a new JSON file and populate it with the parameter combinations you want to test. Pass the file path to `--bench-params`. - Example: Using different input/output lengths for random dataset: `[](#__codelineno-1-1)[ [](#__codelineno-1-2) { [](#__codelineno-1-3) "_benchmark_name": "scenario_A", [](#__codelineno-1-4) "random_input_len": 128, [](#__codelineno-1-5) "random_output_len": 32 [](#__codelineno-1-6) }, [](#__codelineno-1-7) { [](#__codelineno-1-8) "_benchmark_name": "scenario_B", [](#__codelineno-1-9) "random_input_len": 256, [](#__codelineno-1-10) "random_output_len": 64 [](#__codelineno-1-11) }, [](#__codelineno-1-12) { [](#__codelineno-1-13) "_benchmark_name": "scenario_C", [](#__codelineno-1-14) "random_input_len": 512, [](#__codelineno-1-15) "random_output_len": 128 [](#__codelineno-1-16) } [](#__codelineno-1-17)]` 5. Set `--output-dir` and optionally `--experiment-name` to control where to save the results. Example command: `[](#__codelineno-2-1)vllm bench sweep serve \ [](#__codelineno-2-2) --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \ [](#__codelineno-2-3) --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json' \ [](#__codelineno-2-4) --serve-params benchmarks/serve_hparams.json \ [](#__codelineno-2-5) --bench-params benchmarks/bench_hparams.json \ [](#__codelineno-2-6) --output-dir benchmarks/results \ [](#__codelineno-2-7) --experiment-name demo` By default, each parameter combination is benchmarked 3 times to make the results more reliable. You can adjust the number of runs by setting `--num-runs`. Important If both `--serve-params` and `--bench-params` are passed, the script will iterate over the Cartesian product between them. You can use `--dry-run` to preview the commands to be run. We only start the server once for each `--serve-params`, and keep it running for multiple `--bench-params`. Between each benchmark run, we call all `/reset_*_cache` endpoints to get a clean slate for the next run. In case you are using a custom `--serve-cmd`, you can override the commands used for resetting the state by setting `--after-bench-cmd`. Note You should set `_benchmark_name` to provide a human-readable name for parameter combinations involving many variables. This becomes mandatory if the file name would otherwise exceed the maximum path length allowed by the filesystem. Tip You can use the `--resume` option to continue the parameter sweep if an unexpected error occurs, e.g., timeout when connecting to HF Hub. ### Workload Explorer[¶](#workload-explorer "Permanent link") `vllm bench sweep serve_workload` is a variant of `vllm bench sweep serve` that explores different workload levels in order to find the tradeoff between latency and throughput. The results can also be [visualized](#visualization) to determine the feasible SLAs. The workload can be expressed in terms of request rate or concurrency (choose using `--workload-var`). Example command: `[](#__codelineno-3-1)vllm bench sweep serve_workload \ [](#__codelineno-3-2) --serve-cmd 'vllm serve meta-llama/Llama-2-7b-chat-hf' \ [](#__codelineno-3-3) --bench-cmd 'vllm bench serve --model meta-llama/Llama-2-7b-chat-hf --backend vllm --endpoint /v1/completions --dataset-name sharegpt --dataset-path benchmarks/ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100' \ [](#__codelineno-3-4) --workload-var max_concurrency \ [](#__codelineno-3-5) --serve-params benchmarks/serve_hparams.json \ [](#__codelineno-3-6) --bench-params benchmarks/bench_hparams.json \ [](#__codelineno-3-7) --num-runs 1 \ [](#__codelineno-3-8) --output-dir benchmarks/results \ [](#__codelineno-3-9) --experiment-name demo` The algorithm for exploring different workload levels can be summarized as follows: 1. Run the benchmark by sending requests one at a time (serial inference, lowest workload). This results in the lowest possible latency and throughput. 2. Run the benchmark by sending all requests at once (batch inference, highest workload). This results in the highest possible latency and throughput. 3. Estimate the value of `workload_var` corresponding to Step 2. 4. Run the benchmark over intermediate values of `workload_var` uniformly using the remaining iterations. You can override the number of iterations in the algorithm by setting `--workload-iters`. Tip This is our equivalent of [GuideLLM's `--profile sweep`](https://github.com/vllm-project/guidellm/blob/v0.5.3/src/guidellm/benchmark/profiles.py#L575). In general, `--workload-var max_concurrency` produces more reliable results because it directly controls the workload imposed on the vLLM engine. Nevertheless, we default to `--workload-var request_rate` to maintain similar behavior as GuideLLM. ## Startup Benchmark[¶](#startup-benchmark "Permanent link") `vllm bench sweep startup` runs `vllm bench startup` across parameter combinations to compare cold/warm startup time for different engine settings. Follow these steps to run the script: 1. (Optional) Construct the base command to `vllm bench startup`, and pass it to `--startup-cmd` (default: `vllm bench startup`). 2. (Optional) Reuse a `--serve-params` JSON from `vllm bench sweep serve` to vary engine settings. Only parameters supported by `vllm bench startup` are applied. 3. (Optional) Create a `--startup-params` JSON to vary startup-specific options like iteration counts. 4. Determine where you want to save the results, and pass that to `--output-dir`. Example `--serve-params`: `[](#__codelineno-4-1)[ [](#__codelineno-4-2) { [](#__codelineno-4-3) "_benchmark_name": "tp1", [](#__codelineno-4-4) "model": "Qwen/Qwen3-0.6B", [](#__codelineno-4-5) "tensor_parallel_size": 1, [](#__codelineno-4-6) "gpu_memory_utilization": 0.9 [](#__codelineno-4-7) }, [](#__codelineno-4-8) { [](#__codelineno-4-9) "_benchmark_name": "tp2", [](#__codelineno-4-10) "model": "Qwen/Qwen3-0.6B", [](#__codelineno-4-11) "tensor_parallel_size": 2, [](#__codelineno-4-12) "gpu_memory_utilization": 0.9 [](#__codelineno-4-13) } [](#__codelineno-4-14)]` Example `--startup-params`: `[](#__codelineno-5-1)[ [](#__codelineno-5-2) { [](#__codelineno-5-3) "_benchmark_name": "qwen3-0.6", [](#__codelineno-5-4) "num_iters_cold": 2, [](#__codelineno-5-5) "num_iters_warmup": 1, [](#__codelineno-5-6) "num_iters_warm": 2 [](#__codelineno-5-7) } [](#__codelineno-5-8)]` Example command: `[](#__codelineno-6-1)vllm bench sweep startup \ [](#__codelineno-6-2) --startup-cmd 'vllm bench startup --model Qwen/Qwen3-0.6B' \ [](#__codelineno-6-3) --serve-params benchmarks/serve_hparams.json \ [](#__codelineno-6-4) --startup-params benchmarks/startup_hparams.json \ [](#__codelineno-6-5) --output-dir benchmarks/results \ [](#__codelineno-6-6) --experiment-name demo` Important By default, unsupported parameters in `--serve-params` or `--startup-params` are ignored with a warning. Use `--strict-params` to fail fast on unknown keys. ## Visualization[¶](#visualization "Permanent link") ### Basic[¶](#basic_1 "Permanent link") `vllm bench sweep plot` can be used to plot performance curves from parameter sweep results. Control the variables to plot via `--var-x` and `--var-y`, optionally applying `--filter-by` and `--bin-by` to the values. The plot is organized according to `--fig-by`, `--row-by`, `--col-by`, and `--curve-by`. Example commands for visualizing [Workload Explorer](#workload-explorer) results: `[](#__codelineno-7-1)EXPERIMENT_DIR=${1:-"benchmarks/results/demo"} [](#__codelineno-7-2)[](#__codelineno-7-3)# Latency increases as the workload increases [](#__codelineno-7-4)vllm bench sweep plot $EXPERIMENT_DIR \ [](#__codelineno-7-5) --var-x max_concurrency \ [](#__codelineno-7-6) --var-y median_ttft_ms \ [](#__codelineno-7-7) --col-by _benchmark_name \ [](#__codelineno-7-8) --curve-by max_num_seqs,max_num_batched_tokens \ [](#__codelineno-7-9) --fig-name latency_curve [](#__codelineno-7-10)[](#__codelineno-7-11)# Throughput saturates as workload increases [](#__codelineno-7-12)vllm bench sweep plot $EXPERIMENT_DIR \ [](#__codelineno-7-13) --var-x max_concurrency \ [](#__codelineno-7-14) --var-y total_token_throughput \ [](#__codelineno-7-15) --col-by _benchmark_name \ [](#__codelineno-7-16) --curve-by max_num_seqs,max_num_batched_tokens \ [](#__codelineno-7-17) --fig-name throughput_curve [](#__codelineno-7-18)[](#__codelineno-7-19)# Tradeoff between latency and throughput [](#__codelineno-7-20)vllm bench sweep plot $EXPERIMENT_DIR \ [](#__codelineno-7-21) --var-x total_token_throughput \ [](#__codelineno-7-22) --var-y median_ttft_ms \ [](#__codelineno-7-23) --col-by _benchmark_name \ [](#__codelineno-7-24) --curve-by max_num_seqs,max_num_batched_tokens \ [](#__codelineno-7-25) --fig-name latency_throughput` Tip You can use `--dry-run` to preview the figures to be plotted. ### Pareto chart[¶](#pareto-chart "Permanent link") `vllm bench sweep plot_pareto` helps pick configurations that balance per-user and per-GPU throughput. Higher concurrency or batch size can raise GPU efficiency (per-GPU), but can add per user latency; lower concurrency improves per-user rate but underutilizes GPUs; The Pareto frontier shows the best achievable pairs across your runs. - x-axis: tokens/s/user = `output_throughput` ÷ concurrency (`--user-count-var`, default `max_concurrency`, fallback `max_concurrent_requests`). - y-axis: tokens/s/GPU = `output_throughput` ÷ GPU count (`--gpu-count-var` if set; else gpu\_count is TP×PP\*DP). - Output: a single figure at `OUTPUT_DIR/pareto/PARETO.png`. - Show the configuration used in each data point `--label-by` (default: `max_concurrency,gpu_count`). Example: `[](#__codelineno-8-1)EXPERIMENT_DIR=${1:-"benchmarks/results/demo"} [](#__codelineno-8-2)[](#__codelineno-8-3)vllm bench sweep plot_pareto $EXPERIMENT_DIR \ [](#__codelineno-8-4) --label-by max_concurrency,tensor_parallel_size,pipeline_parallel_size` Tip You can use `--dry-run` to preview the figures to be plotted. --- # page - [Home](https://docs.vllm.ai/en/) - [User Guide](https://docs.vllm.ai/en/usage/) - [Developer Guide](https://docs.vllm.ai/en/contributing/) - [Benchmarking](https://docs.vllm.ai/en/benchmarking/) - [API Reference](https://docs.vllm.ai/en/api/) - [CLI Reference](https://docs.vllm.ai/en/latest/) - [Community](https://docs.vllm.ai/en/community/contact_us/) 1. [Home](https://docs.vllm.ai/en/) 2. [CLI Reference](https://docs.vllm.ai/en/latest/) [](https://github.com/vllm-project/vllm/edit/main/docs/cli/complete.md "Edit this page") ## Arguments[¶](#arguments "Permanent link") #### `--url`[¶](#-url "Permanent link") url of the running OpenAI-Compatible RESTful API server Default: `http://localhost:8000/v1` #### `--model-name`[¶](#-model-name "Permanent link") The model name used in prompt completion, default to the first model in list models API call. #### `--api-key`[¶](#-api-key "Permanent link") API key for OpenAI services. If provided, this api key will overwrite the api key obtained through environment variables. It is important to note that this option only applies to the OpenAI-compatible API endpoints and NOT other endpoints that may be present in the server. See the security guide in the vLLM docs for more details. #### `--max-tokens`[¶](#-max-tokens "Permanent link") Maximum number of tokens to generate per output sequence. #### `-q`, `--quick`[¶](#-q-quick "Permanent link") Send a single prompt and print the completion output, then exit. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/benchmarking/cli.md "Edit this page") This section guides you through running benchmark tests with the extensive datasets supported on vLLM. It's a living document, updated as new features and datasets become available. Tip The benchmarks described on this page are mainly for evaluating specific vLLM features as well as regression testing. For benchmarking production vLLM servers, we recommend [GuideLLM](https://github.com/vllm-project/guidellm), an established performance benchmarking framework with live progress updates and automatic report generation. It is also more flexible than `vllm bench serve` in terms of dataset loading, request formatting, and workload patterns. ## Dataset Overview[¶](#dataset-overview "Permanent link") Dataset Online Offline Data Path ShareGPT ✅ ✅ `wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json` ShareGPT4V (Image) ✅ ✅ `wget https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/resolve/main/sharegpt4v_instruct_gpt4-vision_cap100k.json` Note that the images need to be downloaded separately. For example, to download COCO's 2017 Train images: `wget http://images.cocodataset.org/zips/train2017.zip` ShareGPT4Video (Video) ✅ ✅ `git clone https://huggingface.co/datasets/ShareGPT4Video/ShareGPT4Video` BurstGPT ✅ ✅ `wget https://github.com/HPMLL/BurstGPT/releases/download/v1.1/BurstGPT_without_fails_2.csv` Sonnet (deprecated) ✅ ✅ Local file: `benchmarks/sonnet.txt` Random ✅ ✅ `synthetic` RandomMultiModal (Image/Video) ✅ ✅ `synthetic` RandomForReranking ✅ ✅ `synthetic` Prefix Repetition ✅ ✅ `synthetic` HuggingFace-VisionArena ✅ ✅ `lmarena-ai/VisionArena-Chat` HuggingFace-MMVU ✅ ✅ `yale-nlp/MMVU` HuggingFace-InstructCoder ✅ ✅ `likaixin/InstructCoder` HuggingFace-AIMO ✅ ✅ `AI-MO/aimo-validation-aime`, `AI-MO/NuminaMath-1.5`, `AI-MO/NuminaMath-CoT` HuggingFace-Other ✅ ✅ `lmms-lab/LLaVA-OneVision-Data`, `Aeala/ShareGPT_Vicuna_unfiltered` HuggingFace-MTBench ✅ ✅ `philschmid/mt-bench` HuggingFace-HumanEval ✅ ✅ `openai/openai_humaneval` HuggingFace-GSM8K ✅ ✅ `openai/gsm8k` HuggingFace-Blazedit ✅ ✅ `vdaita/edit_5k_char`, `vdaita/edit_10k_char` HuggingFace-ASR ✅ ✅ `openslr/librispeech_asr`, `facebook/voxpopuli`, `LIUM/tedlium`, `edinburghcstr/ami`, `speechcolab/gigaspeech`, `kensho/spgispeech` Spec Bench ✅ ✅ `wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl` SPEED-Bench ✅ ✅ `curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \| python3 -` Custom ✅ ✅ Local file: `data.jsonl` Custom Audio ✅ ✅ Local file: `audio_data.jsonl` Custom Image ✅ ✅ Local file: `image_data.jsonl` Legend: - ✅ - supported - 🟡 - Partial support - 🚧 - to be supported Note HuggingFace dataset's `dataset-name` should be set to `hf`. For local `dataset-path`, please set `hf-name` to its Hugging Face ID like `[](#__codelineno-0-1)--dataset-path /datasets/VisionArena-Chat/ --hf-name lmarena-ai/VisionArena-Chat` ## Examples[¶](#examples "Permanent link") ### 🚀 Online Benchmark[¶](#online-benchmark "Permanent link") Show more First start serving your model: `[](#__codelineno-1-1)vllm serve NousResearch/Hermes-3-Llama-3.1-8B` Then run the benchmarking script: `[](#__codelineno-2-1)# download dataset [](#__codelineno-2-2)# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-2-3)vllm bench serve \ [](#__codelineno-2-4) --backend vllm \ [](#__codelineno-2-5) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-2-6) --endpoint /v1/completions \ [](#__codelineno-2-7) --dataset-name sharegpt \ [](#__codelineno-2-8) --dataset-path /ShareGPT_V3_unfiltered_cleaned_split.json \ [](#__codelineno-2-9) --num-prompts 10` If successful, you will see the following output: `[](#__codelineno-3-1)============ Serving Benchmark Result ============ [](#__codelineno-3-2)Successful requests: 10 [](#__codelineno-3-3)Benchmark duration (s): 5.78 [](#__codelineno-3-4)Total input tokens: 1369 [](#__codelineno-3-5)Total generated tokens: 2212 [](#__codelineno-3-6)Request throughput (req/s): 1.73 [](#__codelineno-3-7)Output token throughput (tok/s): 382.89 [](#__codelineno-3-8)Total token throughput (tok/s): 619.85 [](#__codelineno-3-9)---------------Time to First Token---------------- [](#__codelineno-3-10)Mean TTFT (ms): 71.54 [](#__codelineno-3-11)Median TTFT (ms): 73.88 [](#__codelineno-3-12)P99 TTFT (ms): 79.49 [](#__codelineno-3-13)-----Time per Output Token (excl. 1st token)------ [](#__codelineno-3-14)Mean TPOT (ms): 7.91 [](#__codelineno-3-15)Median TPOT (ms): 7.96 [](#__codelineno-3-16)P99 TPOT (ms): 8.03 [](#__codelineno-3-17)---------------Inter-token Latency---------------- [](#__codelineno-3-18)Mean ITL (ms): 7.74 [](#__codelineno-3-19)Median ITL (ms): 7.70 [](#__codelineno-3-20)P99 ITL (ms): 8.39 [](#__codelineno-3-21)==================================================` #### Results Visualization[¶](#results-visualization "Permanent link") The `--plot-timeline` and `--plot-dataset-stats` can be used to generate respectively the requests completion timeline and dataset prompt and output tokens statistics, which can be useful for debugging purpose or for deeper analysis. `[](#__codelineno-4-1)vllm bench serve \ [](#__codelineno-4-2) --backend vllm \ [](#__codelineno-4-3) --model meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-4-4) --endpoint /v1/completions \ [](#__codelineno-4-5) --dataset-name sharegpt \ [](#__codelineno-4-6) --dataset-path /ShareGPT_V3_unfiltered_cleaned_split.json \ [](#__codelineno-4-7) --num-prompts 100 \ [](#__codelineno-4-8) --plot-timeline \ [](#__codelineno-4-9) --timeline-itl-thresholds 2,5 \ [](#__codelineno-4-10) --plot-dataset-stats \ [](#__codelineno-4-11) --save-result` ##### Interactive Timeline[¶](#interactive-timeline "Permanent link") The generated timeline is an interactive visualization in the form of an HTML file that can be rendered in most browsers. To customize the ITL color thresholds, one can use `--timeline-itl-thresholds` flag (default: 25ms, 50ms) Example output: ##### Dataset statistics[¶](#dataset-statistics "Permanent link") The generated figure shows the input prompt and output tokens distribution. Example output: [![Dataset Statistics](https://docs.vllm.ai/en/assets/contributing/vllm_bench_serve_dataset_stats.png)](https://docs.vllm.ai/en/assets/contributing/vllm_bench_serve_dataset_stats.png) #### Custom Dataset[¶](#custom-dataset "Permanent link") If the dataset you want to benchmark is not supported yet in vLLM, even then you can benchmark on it using [`CustomDataset`](https://docs.vllm.ai/en/api/vllm/benchmarks/datasets/datasets/#vllm.benchmarks.datasets.datasets.CustomDataset " CustomDataset"). At inference time, use the option `--dataset-name custom`. Your data needs to be in the `.jsonl` format and needs to have "prompt" field per entry, e.g., data.jsonl `[](#__codelineno-5-1){"prompt": "What is the capital of India?"} [](#__codelineno-5-2){"prompt": "What is the capital of Iran?"} [](#__codelineno-5-3){"prompt": "What is the capital of China?"}` `[](#__codelineno-6-1)# start server [](#__codelineno-6-2)vllm serve meta-llama/Llama-3.1-8B-Instruct` `[](#__codelineno-7-1)# run benchmarking script [](#__codelineno-7-2)vllm bench serve --port 9001 --save-result --save-detailed \ [](#__codelineno-7-3) --backend vllm \ [](#__codelineno-7-4) --model meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-7-5) --endpoint /v1/completions \ [](#__codelineno-7-6) --dataset-name custom \ [](#__codelineno-7-7) --dataset-path \ [](#__codelineno-7-8) --custom-skip-chat-template \ [](#__codelineno-7-9) --num-prompts 80 \ [](#__codelineno-7-10) --max-concurrency 1 \ [](#__codelineno-7-11) --temperature=0.3 \ [](#__codelineno-7-12) --top-p=0.75 \ [](#__codelineno-7-13) --result-dir "./log/"` You can skip applying chat template if your data already has it by using `--custom-skip-chat-template`. #### Custom Audio Dataset[¶](#custom-audio-dataset "Permanent link") If the audio dataset you want to benchmark is not supported yet in vLLM, then you can benchmark on it using [`CustomAudioDataset`](https://docs.vllm.ai/en/api/vllm/benchmarks/datasets/datasets/#vllm.benchmarks.datasets.datasets.CustomAudioDataset " CustomAudioDataset"). At inference time, use the option `--dataset-name custom_audio`. Your data needs to be in the `.jsonl` format and needs to have "prompt" and "audio" fields per entry, e.g., `audio_data.jsonl`: `[](#__codelineno-8-1){"prompt": "What does this audio say?", "audio": "/path/to/audio_1.wav"} [](#__codelineno-8-2){"prompt": "Transcribe the audio.", "audio": "/path/to/audio_2.wav"}` - **Supported models:** The [`CustomAudioDataset`](https://docs.vllm.ai/en/api/vllm/benchmarks/datasets/datasets/#vllm.benchmarks.datasets.datasets.CustomAudioDataset " CustomAudioDataset") class supports two types of audio models: ASR models (e.g. Whisper) which do not require a "prompt" field; and multimodal audio-text chat models (e.g. Qwen2-Audio). Since these model types require different arguments at inference, we are giving two examples. - **Example 1: Whisper** Whisper is a dedicated ASR encoder-decoder model, so it uses `--backend openai-audio` and `--endpoint /v1/audio/transcriptions`. `[](#__codelineno-9-1)# start server [](#__codelineno-9-2)vllm serve openai/whisper-tiny` `[](#__codelineno-10-1)vllm bench serve \ [](#__codelineno-10-2) --model openai/whisper-tiny \ [](#__codelineno-10-3) --backend openai-audio \ [](#__codelineno-10-4) --endpoint /v1/audio/transcriptions \ [](#__codelineno-10-5) --dataset-name custom_audio \ [](#__codelineno-10-6) --dataset-path audio_data.jsonl \ [](#__codelineno-10-7) --no-oversample \ [](#__codelineno-10-8) --custom-output-len 256 \ [](#__codelineno-10-9) --save-result \ [](#__codelineno-10-10) --save-detailed \ [](#__codelineno-10-11) --result-filename whisper_bench.json` - **Example 2: Qwen2-Audio** Qwen2-Audio is a multimodal chat model that can do ASR and speech analysis, so it uses `--backend openai-chat`, and `--endpoint /v1/chat/completions`. It also requires `--enable-multimodal-chat` to enable multimodal chat transformation. `[](#__codelineno-11-1)vllm bench serve \ [](#__codelineno-11-2) --model Qwen/Qwen2-Audio-7B-Instruct \ [](#__codelineno-11-3) --backend openai-chat \ [](#__codelineno-11-4) --endpoint /v1/chat/completions \ [](#__codelineno-11-5) --dataset-name custom_audio \ [](#__codelineno-11-6) --dataset-path audio_data.jsonl \ [](#__codelineno-11-7) --no-oversample \ [](#__codelineno-11-8) --custom-output-len 256 \ [](#__codelineno-11-9) --enable-multimodal-chat \ [](#__codelineno-11-10) --save-result \ [](#__codelineno-11-11) --save-detailed \ [](#__codelineno-11-12) --result-filename qwen_bench.json` #### Custom Image Dataset[¶](#custom-image-dataset "Permanent link") If the image dataset you want to benchmark is not supported yet in vLLM, then you can benchmark on it using [`CustomImageDataset`](https://docs.vllm.ai/en/api/vllm/benchmarks/datasets/datasets/#vllm.benchmarks.datasets.datasets.CustomImageDataset " CustomImageDataset"). At inference time, use the option `--dataset-name custom_image`. Your data needs to be in the `.jsonl` format and can use "prompt" and "image\_files" fields per entry, e.g., `image_data.jsonl`: `[](#__codelineno-12-1){"prompt": "How many animals are present in the given image?", "image_files": ["/path/to/image/folder/horsepony.jpg"]} [](#__codelineno-12-2){"prompt": "What colour is the bird shown in the image?", "image_files": ["/path/to/image/folder/flycatcher.jpeg"]}` Every image listed in "image\_files" is added to the request in the listed order after the prompt text. To preserve an interleaved order of text and images, use a "content" field with OpenAI-compatible content parts: `[](#__codelineno-13-1){"content": [{"type": "text", "text": "Compare "}, {"type": "image", "image": "/path/to/image/folder/chart_a.png"}, {"type": "text", "text": " with "}, {"type": "image_url", "image_url": {"url": "/path/to/image/folder/chart_b.png"}}]}` The "image" shorthand accepts the same values as "image\_files". The "image\_url" field accepts either an OpenAI-style object with a "url" field or a URL string. By default, image references are sent to the serving endpoint as provided, with local image paths converted to `file://` URLs. If the benchmark client should load local and HTTP(S) images before sending requests, pass `--custom-ensure-client-side-data` to encode them as base64 data URLs on the client side. Existing `data:image/...` URLs are already self-contained and are kept unchanged. `[](#__codelineno-14-1)# need a model with vision capability here [](#__codelineno-14-2)vllm serve Qwen/Qwen2-VL-7B-Instruct` `[](#__codelineno-15-1)# run benchmarking script [](#__codelineno-15-2)vllm bench serve --save-result --save-detailed \ [](#__codelineno-15-3) --backend openai-chat \ [](#__codelineno-15-4) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-15-5) --endpoint /v1/chat/completions \ [](#__codelineno-15-6) --dataset-name custom_image \ [](#__codelineno-15-7) --dataset-path \ [](#__codelineno-15-8) --custom-ensure-client-side-data` Note that we need to use the `openai-chat` backend and `/v1/chat/completions` endpoint for multimodal inputs. #### VisionArena Benchmark for Vision Language Models[¶](#visionarena-benchmark-for-vision-language-models "Permanent link") `[](#__codelineno-16-1)# need a model with vision capability here [](#__codelineno-16-2)vllm serve Qwen/Qwen2-VL-7B-Instruct` `[](#__codelineno-17-1)vllm bench serve \ [](#__codelineno-17-2) --backend openai-chat \ [](#__codelineno-17-3) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-17-4) --endpoint /v1/chat/completions \ [](#__codelineno-17-5) --dataset-name hf \ [](#__codelineno-17-6) --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-17-7) --hf-split train \ [](#__codelineno-17-8) --num-prompts 1000` #### InstructCoder Benchmark with Speculative Decoding[¶](#instructcoder-benchmark-with-speculative-decoding "Permanent link") `[](#__codelineno-18-1)vllm serve meta-llama/Meta-Llama-3-8B-Instruct \ [](#__codelineno-18-2) --speculative-config $'{"method": "ngram", [](#__codelineno-18-3) "num_speculative_tokens": 5, "prompt_lookup_max": 5, [](#__codelineno-18-4) "prompt_lookup_min": 2}'` `[](#__codelineno-19-1)vllm bench serve \ [](#__codelineno-19-2) --model meta-llama/Meta-Llama-3-8B-Instruct \ [](#__codelineno-19-3) --dataset-name hf \ [](#__codelineno-19-4) --dataset-path likaixin/InstructCoder \ [](#__codelineno-19-5) --num-prompts 2048` #### Spec Bench Benchmark with Speculative Decoding[¶](#spec-bench-benchmark-with-speculative-decoding "Permanent link") `[](#__codelineno-20-1)vllm serve meta-llama/Meta-Llama-3-8B-Instruct \ [](#__codelineno-20-2) --speculative-config $'{"method": "ngram", [](#__codelineno-20-3) "num_speculative_tokens": 5, "prompt_lookup_max": 5, [](#__codelineno-20-4) "prompt_lookup_min": 2}'` [SpecBench dataset](https://github.com/hemingkx/Spec-Bench) Run all categories: `[](#__codelineno-21-1)# Download the dataset using: [](#__codelineno-21-2)# wget https://raw.githubusercontent.com/hemingkx/Spec-Bench/refs/heads/main/data/spec_bench/question.jsonl [](#__codelineno-21-3)[](#__codelineno-21-4)vllm bench serve \ [](#__codelineno-21-5) --model meta-llama/Meta-Llama-3-8B-Instruct \ [](#__codelineno-21-6) --dataset-name spec_bench \ [](#__codelineno-21-7) --dataset-path "/data/spec_bench/question.jsonl" \ [](#__codelineno-21-8) --num-prompts -1` Available categories include `[writing, roleplay, reasoning, math, coding, extraction, stem, humanities, translation, summarization, qa, math_reasoning, rag]`. Run only a specific category like "summarization": `[](#__codelineno-22-1)vllm bench serve \ [](#__codelineno-22-2) --model meta-llama/Meta-Llama-3-8B-Instruct \ [](#__codelineno-22-3) --dataset-name spec_bench \ [](#__codelineno-22-4) --dataset-path "/data/spec_bench/question.jsonl" \ [](#__codelineno-22-5) --num-prompts -1 [](#__codelineno-22-6) --spec-bench-category "summarization"` #### SPEED-Bench Benchmark with Speculative Decoding[¶](#speed-bench-benchmark-with-speculative-decoding "Permanent link") [SPEED-Bench](https://huggingface.co/datasets/nvidia/SPEED-Bench) is a unified and diverse dataset for speculative decoding, supporting acceptance rate and length measurements using the Qualitative split and throughput measurements using the Throughput splits in 5 configuration of input sequence length (1k, 2k, 8k, 16k, 32k). Note This dataset is governed by the [NVIDIA Evaluation Dataset License Agreement](https://huggingface.co/datasets/nvidia/SPEED-Bench/blob/main/License.pdf). For each dataset a user elects to use, the user is responsible for checking if the dataset license is fit for the intended purpose. The `prepare.py` script automatically fetches data from all the source datasets. First, download the dataset to a folder, using this one liner: `[](#__codelineno-23-1)curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py \| python3 -` The command supports also the following arguments: - `--config`: download only a subset of the dataset: `qualitative`, `throughput_1k`, `throughput_2k`, `throughput_8k`, `throughput_16k` and `throughput_32k`. By default, it will download all subsets. - `--output_dir`: download to a specified folder. By default, it will download to the current directory. Start a server with speculative decoding: `[](#__codelineno-24-1)vllm serve meta-llama/Llama-3.3-70B-Instruct \ [](#__codelineno-24-2) --speculative-config $'{"method": "eagle3", [](#__codelineno-24-3) "num_speculative_tokens": 3, [](#__codelineno-24-4) "model": "nvidia/Llama-3.3-70B-Instruct-Eagle3"}'` Run all categories in the Qualitative split: `[](#__codelineno-25-1)vllm bench serve \ [](#__codelineno-25-2) --model meta-llama/Llama-3.3-70B-Instruct \ [](#__codelineno-25-3) --dataset-name speed_bench \ [](#__codelineno-25-4) --dataset-path "/data/speed_bench" \ [](#__codelineno-25-5) --num-prompts -1` Available categories include `[writing, roleplay, reasoning, math, coding, stem, humanities, multilingual, summarization, qa, rag]`. Run only a specific category like "multilingual": `[](#__codelineno-26-1)vllm bench serve \ [](#__codelineno-26-2) --model meta-llama/Llama-3.3-70B-Instruct \ [](#__codelineno-26-3) --dataset-name speed_bench \ [](#__codelineno-26-4) --dataset-path "/data/speed_bench" \ [](#__codelineno-26-5) --num-prompts -1 [](#__codelineno-26-6) --speed-bench-category "multilingual"` Run all categories in the Throughput split (2k ISL): `[](#__codelineno-27-1)vllm bench serve \ [](#__codelineno-27-2) --model meta-llama/Llama-3.3-70B-Instruct \ [](#__codelineno-27-3) --dataset-name speed_bench \ [](#__codelineno-27-4) --speed-bench-dataset-subset throughput_2k [](#__codelineno-27-5) --dataset-path "/data/speed_bench/" \ [](#__codelineno-27-6) --num-prompts -1` Available categories include `[high_entropy, mixed, low_entropy]`, where high entropy data contains unstructued data such as creative writing while low entropy data contains more structured data such as coding, more details are in the dataset card. #### Other HuggingFaceDataset Examples[¶](#other-huggingfacedataset-examples "Permanent link") `[](#__codelineno-28-1)vllm serve Qwen/Qwen2-VL-7B-Instruct` `lmms-lab/LLaVA-OneVision-Data`: `[](#__codelineno-29-1)vllm bench serve \ [](#__codelineno-29-2) --backend openai-chat \ [](#__codelineno-29-3) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-29-4) --endpoint /v1/chat/completions \ [](#__codelineno-29-5) --dataset-name hf \ [](#__codelineno-29-6) --dataset-path lmms-lab/LLaVA-OneVision-Data \ [](#__codelineno-29-7) --hf-split train \ [](#__codelineno-29-8) --hf-subset "chart2text(cauldron)" \ [](#__codelineno-29-9) --num-prompts 10` `Aeala/ShareGPT_Vicuna_unfiltered`: `[](#__codelineno-30-1)vllm bench serve \ [](#__codelineno-30-2) --backend openai-chat \ [](#__codelineno-30-3) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-30-4) --endpoint /v1/chat/completions \ [](#__codelineno-30-5) --dataset-name hf \ [](#__codelineno-30-6) --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \ [](#__codelineno-30-7) --hf-split train \ [](#__codelineno-30-8) --num-prompts 10` `AI-MO/aimo-validation-aime`: `[](#__codelineno-31-1)vllm bench serve \ [](#__codelineno-31-2) --model Qwen/QwQ-32B \ [](#__codelineno-31-3) --dataset-name hf \ [](#__codelineno-31-4) --dataset-path AI-MO/aimo-validation-aime \ [](#__codelineno-31-5) --num-prompts 10 \ [](#__codelineno-31-6) --seed 42` `philschmid/mt-bench`: `[](#__codelineno-32-1)vllm bench serve \ [](#__codelineno-32-2) --model Qwen/QwQ-32B \ [](#__codelineno-32-3) --dataset-name hf \ [](#__codelineno-32-4) --dataset-path philschmid/mt-bench \ [](#__codelineno-32-5) --num-prompts 80` `openai/openai_humaneval`: `[](#__codelineno-33-1)vllm bench serve \ [](#__codelineno-33-2) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-33-3) --dataset-name hf \ [](#__codelineno-33-4) --dataset-path openai/openai_humaneval \ [](#__codelineno-33-5) --num-prompts 80` `openai/gsm8k`: `[](#__codelineno-34-1)vllm bench serve \ [](#__codelineno-34-2) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-34-3) --dataset-name hf \ [](#__codelineno-34-4) --dataset-path openai/gsm8k \ [](#__codelineno-34-5) --num-prompts 80` `vdaita/edit_5k_char` or `vdaita/edit_10k_char`: `[](#__codelineno-35-1)vllm bench serve \ [](#__codelineno-35-2) --model Qwen/QwQ-32B \ [](#__codelineno-35-3) --dataset-name hf \ [](#__codelineno-35-4) --dataset-path vdaita/edit_5k_char \ [](#__codelineno-35-5) --num-prompts 90 \ [](#__codelineno-35-6) --blazedit-min-distance 0.01 \ [](#__codelineno-35-7) --blazedit-max-distance 0.99` `openslr/librispeech_asr`, `facebook/voxpopuli`, `LIUM/tedlium`, `edinburghcstr/ami`, `speechcolab/gigaspeech`, `kensho/spgispeech` `[](#__codelineno-36-1)vllm bench serve \ [](#__codelineno-36-2) --model openai/whisper-large-v3-turbo \ [](#__codelineno-36-3) --backend openai-audio \ [](#__codelineno-36-4) --dataset-name hf \ [](#__codelineno-36-5) --dataset-path facebook/voxpopuli --hf-subset en --hf-split test --no-stream --trust-remote-code \ [](#__codelineno-36-6) --num-prompts 99999999 \ [](#__codelineno-36-7) --no-oversample \ [](#__codelineno-36-8) --endpoint /v1/audio/transcriptions \ [](#__codelineno-36-9) --ready-check-timeout-sec 600 \ [](#__codelineno-36-10) --save-result \ [](#__codelineno-36-11) --max-concurrency 512` #### Running With Sampling Parameters[¶](#running-with-sampling-parameters "Permanent link") When using OpenAI-compatible backends such as `vllm`, optional sampling parameters can be specified. Example client command: `[](#__codelineno-37-1)vllm bench serve \ [](#__codelineno-37-2) --backend vllm \ [](#__codelineno-37-3) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-37-4) --endpoint /v1/completions \ [](#__codelineno-37-5) --dataset-name sharegpt \ [](#__codelineno-37-6) --dataset-path /ShareGPT_V3_unfiltered_cleaned_split.json \ [](#__codelineno-37-7) --top-k 10 \ [](#__codelineno-37-8) --top-p 0.9 \ [](#__codelineno-37-9) --temperature 0.5 \ [](#__codelineno-37-10) --num-prompts 10` #### Running With Ramp-Up Request Rate[¶](#running-with-ramp-up-request-rate "Permanent link") The benchmark tool also supports ramping up the request rate over the duration of the benchmark run. This can be useful for stress testing the server or finding the maximum throughput that it can handle, given some latency budget. Two ramp-up strategies are supported: - `linear`: Increases the request rate linearly from a start value to an end value. - `exponential`: Increases the request rate exponentially. The following arguments can be used to control the ramp-up: - `--ramp-up-strategy`: The ramp-up strategy to use (`linear` or `exponential`). - `--ramp-up-start-rps`: The request rate at the beginning of the benchmark. - `--ramp-up-end-rps`: The request rate at the end of the benchmark. #### Load Pattern Configuration[¶](#load-pattern-configuration "Permanent link") vLLM's benchmark serving script provides sophisticated load pattern simulation capabilities through three key parameters that control request generation and concurrency behavior: ##### Load Pattern Control Parameters[¶](#load-pattern-control-parameters "Permanent link") - `--request-rate`: Controls the target request generation rate (requests per second). Set to `inf` for maximum throughput testing or finite values for controlled load simulation. - `--burstiness`: Controls traffic variability using a Gamma distribution (range: > 0). Lower values create bursty traffic, higher values create uniform traffic. - `--max-concurrency`: Limits concurrent outstanding requests. If this argument is not provided, concurrency is unlimited. Set a value to simulate backpressure. These parameters work together to create realistic load patterns with carefully chosen defaults. The `--request-rate` parameter defaults to `inf` (infinite), which sends all requests immediately for maximum throughput testing. When set to finite values, it uses either a Poisson process (default `--burstiness=1.0`) or Gamma distribution for realistic request timing. The `--burstiness` parameter only takes effect when `--request-rate` is not infinite - a value of 1.0 creates natural Poisson traffic, while lower values (0.1-0.5) create bursty patterns and higher values (2.0-5.0) create uniform spacing. The `--max-concurrency` parameter defaults to `None` (unlimited) but can be set to simulate real-world constraints where a load balancer or API gateway limits concurrent connections. When combined, these parameters allow you to simulate everything from unrestricted stress testing (`--request-rate=inf`) to production-like scenarios with realistic arrival patterns and resource constraints. The `--burstiness` parameter mathematically controls request arrival patterns using a Gamma distribution where: - Shape parameter: `burstiness` value - Coefficient of Variation (CV): \\(\\frac{1}{\\sqrt{burstiness}}\\) - Traffic characteristics: - `burstiness = 0.1`: Highly bursty traffic (CV ≈ 3.16) - stress testing - `burstiness = 1.0`: Natural Poisson traffic (CV = 1.0) - realistic simulation - `burstiness = 5.0`: Uniform traffic (CV ≈ 0.45) - controlled load testing [![Load Pattern Examples](https://docs.vllm.ai/en/assets/contributing/load-pattern-examples.png)](https://docs.vllm.ai/en/assets/contributing/load-pattern-examples.png) _Figure: Load pattern examples for each use case. Top row: Request arrival timelines showing cumulative requests over time. Bottom row: Inter-arrival time distributions showing traffic variability patterns. Each column represents a different use case with its specific parameter settings and resulting traffic characteristics._ Load Pattern Recommendations by Use Case: Use Case Burstiness Request Rate Max Concurrency Description Maximum Throughput N/A Infinite Limited **Most common**: Simulates load balancer/gateway limits with unlimited user demand Realistic Testing 1.0 Moderate (5-20) Infinite Natural Poisson traffic patterns for baseline performance Stress Testing 0.1-0.5 High (20-100) Infinite Challenging burst patterns to test resilience Latency Profiling 2.0-5.0 Low (1-10) Infinite Uniform load for consistent timing analysis Capacity Planning 1.0 Variable Limited Test resource limits with realistic constraints SLA Validation 1.0 Target rate SLA limit Production-like constraints for compliance testing These load patterns help evaluate different aspects of your vLLM deployment, from basic performance characteristics to resilience under challenging traffic conditions. The **Maximum Throughput** pattern (`--request-rate=inf --max-concurrency=`) is the most commonly used configuration for production benchmarking. This simulates real-world deployment architectures where: - Users send requests as fast as they can (infinite rate) - A load balancer or API gateway controls the maximum concurrent connections - The system operates at its concurrency limit, revealing true throughput capacity - `--burstiness` has no effect since request timing is not controlled when rate is infinite This pattern helps determine optimal concurrency settings for your production load balancer configuration. To effectively configure load patterns, especially for **Capacity Planning** and **SLA Validation** use cases, you need to understand your system's resource limits. During startup, vLLM reports KV cache configuration that directly impacts your load testing parameters: `[](#__codelineno-38-1)GPU KV cache size: 15,728,640 tokens [](#__codelineno-38-2)Maximum concurrency for 8,192 tokens per request: 1920` Where: - GPU KV cache size: Total tokens that can be cached across all concurrent requests - Maximum concurrency: Theoretical maximum concurrent requests for the given `max_model_len` - Calculation: `max_concurrency = kv_cache_size / max_model_len` Using KV cache metrics for load pattern configuration: - For Capacity Planning: Set `--max-concurrency` to 80-90% of the reported maximum to test realistic resource constraints - For SLA Validation: Use the reported maximum as your SLA limit to ensure compliance testing matches production capacity - For Realistic Testing: Monitor memory usage when approaching theoretical limits to understand sustainable request rates - Request rate guidance: Use the KV cache size to estimate sustainable request rates for your specific workload and sequence lengths ### 📈 Offline Throughput Benchmark[¶](#offline-throughput-benchmark "Permanent link") Show more `[](#__codelineno-39-1)vllm bench throughput \ [](#__codelineno-39-2) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-39-3) --dataset-name sonnet \ [](#__codelineno-39-4) --dataset-path vllm/benchmarks/sonnet.txt \ [](#__codelineno-39-5) --num-prompts 10` If successful, you will see the following output `[](#__codelineno-40-1)Throughput: 7.15 requests/s, 4656.00 total tokens/s, 1072.15 output tokens/s [](#__codelineno-40-2)Total num prompt tokens: 5014 [](#__codelineno-40-3)Total num output tokens: 1500` #### VisionArena Benchmark for Vision Language Models[¶](#visionarena-benchmark-for-vision-language-models_1 "Permanent link") `[](#__codelineno-41-1)vllm bench throughput \ [](#__codelineno-41-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-41-3) --backend vllm-chat \ [](#__codelineno-41-4) --dataset-name hf \ [](#__codelineno-41-5) --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-41-6) --num-prompts 1000 \ [](#__codelineno-41-7) --hf-split train` The `num prompt tokens` now includes image token counts `[](#__codelineno-42-1)Throughput: 2.55 requests/s, 4036.92 total tokens/s, 326.90 output tokens/s [](#__codelineno-42-2)Total num prompt tokens: 14527 [](#__codelineno-42-3)Total num output tokens: 1280` #### InstructCoder Benchmark with Speculative Decoding[¶](#instructcoder-benchmark-with-speculative-decoding_1 "Permanent link") `[](#__codelineno-43-1)VLLM_WORKER_MULTIPROC_METHOD=spawn \ [](#__codelineno-43-2)vllm bench throughput \ [](#__codelineno-43-3) --dataset-name=hf \ [](#__codelineno-43-4) --dataset-path=likaixin/InstructCoder \ [](#__codelineno-43-5) --model=meta-llama/Meta-Llama-3-8B-Instruct \ [](#__codelineno-43-6) --input-len=1000 \ [](#__codelineno-43-7) --output-len=100 \ [](#__codelineno-43-8) --num-prompts=2048 \ [](#__codelineno-43-9) --async-engine \ [](#__codelineno-43-10) --speculative-config $'{"method": "ngram", [](#__codelineno-43-11) "num_speculative_tokens": 5, "prompt_lookup_max": 5, [](#__codelineno-43-12) "prompt_lookup_min": 2}'` `[](#__codelineno-44-1)Throughput: 104.77 requests/s, 23836.22 total tokens/s, 10477.10 output tokens/s [](#__codelineno-44-2)Total num prompt tokens: 261136 [](#__codelineno-44-3)Total num output tokens: 204800` #### Other HuggingFaceDataset Examples[¶](#other-huggingfacedataset-examples_1 "Permanent link") `lmms-lab/LLaVA-OneVision-Data`: `[](#__codelineno-45-1)vllm bench throughput \ [](#__codelineno-45-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-45-3) --backend vllm-chat \ [](#__codelineno-45-4) --dataset-name hf \ [](#__codelineno-45-5) --dataset-path lmms-lab/LLaVA-OneVision-Data \ [](#__codelineno-45-6) --hf-split train \ [](#__codelineno-45-7) --hf-subset "chart2text(cauldron)" \ [](#__codelineno-45-8) --num-prompts 10` `Aeala/ShareGPT_Vicuna_unfiltered`: `[](#__codelineno-46-1)vllm bench throughput \ [](#__codelineno-46-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-46-3) --backend vllm-chat \ [](#__codelineno-46-4) --dataset-name hf \ [](#__codelineno-46-5) --dataset-path Aeala/ShareGPT_Vicuna_unfiltered \ [](#__codelineno-46-6) --hf-split train \ [](#__codelineno-46-7) --num-prompts 10` `AI-MO/aimo-validation-aime`: `[](#__codelineno-47-1)vllm bench throughput \ [](#__codelineno-47-2) --model Qwen/QwQ-32B \ [](#__codelineno-47-3) --backend vllm \ [](#__codelineno-47-4) --dataset-name hf \ [](#__codelineno-47-5) --dataset-path AI-MO/aimo-validation-aime \ [](#__codelineno-47-6) --hf-split train \ [](#__codelineno-47-7) --num-prompts 10` Benchmark with LoRA adapters: `[](#__codelineno-48-1)# download dataset [](#__codelineno-48-2)# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-48-3)vllm bench throughput \ [](#__codelineno-48-4) --model meta-llama/Llama-2-7b-hf \ [](#__codelineno-48-5) --backend vllm \ [](#__codelineno-48-6) --dataset_path /ShareGPT_V3_unfiltered_cleaned_split.json \ [](#__codelineno-48-7) --dataset_name sharegpt \ [](#__codelineno-48-8) --num-prompts 10 \ [](#__codelineno-48-9) --max-loras 2 \ [](#__codelineno-48-10) --max-lora-rank 8 \ [](#__codelineno-48-11) --enable-lora \ [](#__codelineno-48-12) --lora-path yard1/llama-2-7b-sql-lora-test` #### Synthetic Random Multimodal (random-mm)[¶](#synthetic-random-multimodal-random-mm "Permanent link") Generate synthetic multimodal inputs for offline throughput testing without external datasets. Use `--backend vllm-chat` so that image tokens are counted correctly. `[](#__codelineno-49-1)vllm bench throughput \ [](#__codelineno-49-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-49-3) --backend vllm-chat \ [](#__codelineno-49-4) --dataset-name random-mm \ [](#__codelineno-49-5) --num-prompts 100 \ [](#__codelineno-49-6) --random-input-len 300 \ [](#__codelineno-49-7) --random-output-len 40 \ [](#__codelineno-49-8) --random-mm-base-items-per-request 2 \ [](#__codelineno-49-9) --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \ [](#__codelineno-49-10) --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}'` ### 🛠️ Structured Output Benchmark[¶](#structured-output-benchmark "Permanent link") Show more Benchmark the performance of structured output generation (JSON, grammar, regex). #### Server Setup[¶](#server-setup "Permanent link") `[](#__codelineno-50-1)vllm serve NousResearch/Hermes-3-Llama-3.1-8B` #### JSON Schema Benchmark[¶](#json-schema-benchmark "Permanent link") `[](#__codelineno-51-1)python3 benchmarks/benchmark_serving_structured_output.py \ [](#__codelineno-51-2) --backend vllm \ [](#__codelineno-51-3) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-51-4) --dataset json \ [](#__codelineno-51-5) --structured-output-ratio 1.0 \ [](#__codelineno-51-6) --request-rate 10 \ [](#__codelineno-51-7) --num-prompts 1000` #### Grammar-based Generation Benchmark[¶](#grammar-based-generation-benchmark "Permanent link") `[](#__codelineno-52-1)python3 benchmarks/benchmark_serving_structured_output.py \ [](#__codelineno-52-2) --backend vllm \ [](#__codelineno-52-3) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-52-4) --dataset grammar \ [](#__codelineno-52-5) --structure-type grammar \ [](#__codelineno-52-6) --request-rate 10 \ [](#__codelineno-52-7) --num-prompts 1000` #### Regex-based Generation Benchmark[¶](#regex-based-generation-benchmark "Permanent link") `[](#__codelineno-53-1)python3 benchmarks/benchmark_serving_structured_output.py \ [](#__codelineno-53-2) --backend vllm \ [](#__codelineno-53-3) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-53-4) --dataset regex \ [](#__codelineno-53-5) --request-rate 10 \ [](#__codelineno-53-6) --num-prompts 1000` #### Choice-based Generation Benchmark[¶](#choice-based-generation-benchmark "Permanent link") `[](#__codelineno-54-1)python3 benchmarks/benchmark_serving_structured_output.py \ [](#__codelineno-54-2) --backend vllm \ [](#__codelineno-54-3) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-54-4) --dataset choice \ [](#__codelineno-54-5) --request-rate 10 \ [](#__codelineno-54-6) --num-prompts 1000` #### XGrammar Benchmark Dataset[¶](#xgrammar-benchmark-dataset "Permanent link") `[](#__codelineno-55-1)python3 benchmarks/benchmark_serving_structured_output.py \ [](#__codelineno-55-2) --backend vllm \ [](#__codelineno-55-3) --model NousResearch/Hermes-3-Llama-3.1-8B \ [](#__codelineno-55-4) --dataset xgrammar_bench \ [](#__codelineno-55-5) --request-rate 10 \ [](#__codelineno-55-6) --num-prompts 1000` ### 📚 Long Document QA Benchmark[¶](#long-document-qa-benchmark "Permanent link") Show more Benchmark the performance of long document question-answering with prefix caching. #### Basic Long Document QA Test[¶](#basic-long-document-qa-test "Permanent link") `[](#__codelineno-56-1)python3 benchmarks/benchmark_long_document_qa_throughput.py \ [](#__codelineno-56-2) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-56-3) --enable-prefix-caching \ [](#__codelineno-56-4) --num-documents 16 \ [](#__codelineno-56-5) --document-length 2000 \ [](#__codelineno-56-6) --output-len 50 \ [](#__codelineno-56-7) --repeat-count 5` #### Different Repeat Modes[¶](#different-repeat-modes "Permanent link") `[](#__codelineno-57-1)# Random mode (default) - shuffle prompts randomly [](#__codelineno-57-2)python3 benchmarks/benchmark_long_document_qa_throughput.py \ [](#__codelineno-57-3) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-57-4) --enable-prefix-caching \ [](#__codelineno-57-5) --num-documents 8 \ [](#__codelineno-57-6) --document-length 3000 \ [](#__codelineno-57-7) --repeat-count 3 \ [](#__codelineno-57-8) --repeat-mode random [](#__codelineno-57-9)[](#__codelineno-57-10)# Tile mode - repeat entire prompt list in sequence [](#__codelineno-57-11)python3 benchmarks/benchmark_long_document_qa_throughput.py \ [](#__codelineno-57-12) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-57-13) --enable-prefix-caching \ [](#__codelineno-57-14) --num-documents 8 \ [](#__codelineno-57-15) --document-length 3000 \ [](#__codelineno-57-16) --repeat-count 3 \ [](#__codelineno-57-17) --repeat-mode tile [](#__codelineno-57-18)[](#__codelineno-57-19)# Interleave mode - repeat each prompt consecutively [](#__codelineno-57-20)python3 benchmarks/benchmark_long_document_qa_throughput.py \ [](#__codelineno-57-21) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-57-22) --enable-prefix-caching \ [](#__codelineno-57-23) --num-documents 8 \ [](#__codelineno-57-24) --document-length 3000 \ [](#__codelineno-57-25) --repeat-count 3 \ [](#__codelineno-57-26) --repeat-mode interleave` ### 🗂️ Prefix Caching Benchmark[¶](#prefix-caching-benchmark "Permanent link") Show more Benchmark the efficiency of automatic prefix caching. #### Fixed Prompt with Prefix Caching[¶](#fixed-prompt-with-prefix-caching "Permanent link") `[](#__codelineno-58-1)python3 benchmarks/benchmark_prefix_caching.py \ [](#__codelineno-58-2) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-58-3) --enable-prefix-caching \ [](#__codelineno-58-4) --num-prompts 1 \ [](#__codelineno-58-5) --repeat-count 100 \ [](#__codelineno-58-6) --input-length-range 128:256` #### ShareGPT Dataset with Prefix Caching[¶](#sharegpt-dataset-with-prefix-caching "Permanent link") `[](#__codelineno-59-1)# download dataset [](#__codelineno-59-2)# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-59-3)[](#__codelineno-59-4)python3 benchmarks/benchmark_prefix_caching.py \ [](#__codelineno-59-5) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-59-6) --dataset-path /path/ShareGPT_V3_unfiltered_cleaned_split.json \ [](#__codelineno-59-7) --enable-prefix-caching \ [](#__codelineno-59-8) --num-prompts 20 \ [](#__codelineno-59-9) --repeat-count 5 \ [](#__codelineno-59-10) --input-length-range 128:256` ##### Prefix Repetition Dataset[¶](#prefix-repetition-dataset "Permanent link") `[](#__codelineno-60-1)vllm bench serve \ [](#__codelineno-60-2) --backend openai \ [](#__codelineno-60-3) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-60-4) --dataset-name prefix_repetition \ [](#__codelineno-60-5) --num-prompts 100 \ [](#__codelineno-60-6) --prefix-repetition-prefix-len 512 \ [](#__codelineno-60-7) --prefix-repetition-suffix-len 128 \ [](#__codelineno-60-8) --prefix-repetition-num-prefixes 5 \ [](#__codelineno-60-9) --prefix-repetition-output-len 128` ### Replay Timed Traces[¶](#replay-timed-traces "Permanent link") Show more Example of how to run traces which have timing information with them. #### Running MoonshotAI traces[¶](#running-moonshotai-traces "Permanent link") Start the server: `[](#__codelineno-61-1)vllm serve Qwen/Qwen3.5-2B \ [](#__codelineno-61-2)--host 127.0.0.1 --port 8000` Run the benchmark: `[](#__codelineno-62-1)# Download an example trace [](#__codelineno-62-2)# curl -L -o conversation_trace.jsonl \ [](#__codelineno-62-3)#https://raw.githubusercontent.com/kvcache-ai/Mooncake/main/FAST25-release/traces/conversation_trace.jsonl [](#__codelineno-62-4)[](#__codelineno-62-5)vllm bench serve --model Qwen/Qwen3.5-2B \ [](#__codelineno-62-6)--dataset-name=timed_trace --num-prompts 100 --host 127.0.0.1 \ [](#__codelineno-62-7)--port 8000 --dataset-path ./conversation_trace.jsonl \ [](#__codelineno-62-8)--ignore-eos --self-timed --timed-trace-chunk-hash-size 512 \ [](#__codelineno-62-9)--timed-trace-sec-multiplier 0.001` This will replay the first 100 lines from the trace file `conversation.jsonl`. ### 🧪 Hashing Benchmarks[¶](#hashing-benchmarks "Permanent link") Show more Two helper scripts live in `benchmarks/` to compare hashing options used by prefix caching and related utilities. They are standalone (no server required) and help choose a hash algorithm before enabling prefix caching in production. - `benchmarks/benchmark_hash.py`: Micro-benchmark that measures per-call latency of three implementations on a representative `(bytes, tuple[int])` payload. `[](#__codelineno-63-1)python benchmarks/benchmark_hash.py --iterations 20000 --seed 42` - `benchmarks/benchmark_prefix_block_hash.py`: End-to-end block hashing benchmark that runs the full prefix-cache hash pipeline (`hash_block_tokens`) across many fake blocks and reports throughput. `[](#__codelineno-64-1)python benchmarks/benchmark_prefix_block_hash.py --num-blocks 20000 --block-size 32 --trials 5` Supported algorithms: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor`. Install optional deps to exercise all variants: `[](#__codelineno-65-1)uv pip install xxhash cbor2` If an algorithm’s dependency is missing, the script will skip it and continue. ### ⚡ Request Prioritization Benchmark[¶](#request-prioritization-benchmark "Permanent link") Show more Benchmark the performance of request prioritization in vLLM. #### Basic Prioritization Test[¶](#basic-prioritization-test "Permanent link") `[](#__codelineno-66-1)python3 benchmarks/benchmark_prioritization.py \ [](#__codelineno-66-2) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-66-3) --input-len 128 \ [](#__codelineno-66-4) --output-len 64 \ [](#__codelineno-66-5) --num-prompts 100 \ [](#__codelineno-66-6) --scheduling-policy priority` #### Multiple Sequences per Prompt[¶](#multiple-sequences-per-prompt "Permanent link") `[](#__codelineno-67-1)python3 benchmarks/benchmark_prioritization.py \ [](#__codelineno-67-2) --model meta-llama/Llama-2-7b-chat-hf \ [](#__codelineno-67-3) --input-len 128 \ [](#__codelineno-67-4) --output-len 64 \ [](#__codelineno-67-5) --num-prompts 100 \ [](#__codelineno-67-6) --scheduling-policy priority \ [](#__codelineno-67-7) --n 2` ### 👁️ Multi-Modal Benchmark[¶](#multi-modal-benchmark "Permanent link") Show more Benchmark the performance of multi-modal requests in vLLM. #### Images (ShareGPT4V)[¶](#images-sharegpt4v "Permanent link") Start vLLM: `[](#__codelineno-68-1)vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ [](#__codelineno-68-2) --dtype bfloat16 \ [](#__codelineno-68-3) --limit-mm-per-prompt '{"image": 1}' \ [](#__codelineno-68-4) --allowed-local-media-path /path/to/sharegpt4v/images` Send requests with images: `[](#__codelineno-69-1)vllm bench serve \ [](#__codelineno-69-2) --backend openai-chat \ [](#__codelineno-69-3) --model Qwen/Qwen2.5-VL-7B-Instruct \ [](#__codelineno-69-4) --dataset-name sharegpt \ [](#__codelineno-69-5) --dataset-path /path/to/ShareGPT4V/sharegpt4v_instruct_gpt4-vision_cap100k.json \ [](#__codelineno-69-6) --num-prompts 100 \ [](#__codelineno-69-7) --save-result \ [](#__codelineno-69-8) --result-dir ~/vllm_benchmark_results \ [](#__codelineno-69-9) --save-detailed \ [](#__codelineno-69-10) --endpoint /v1/chat/completions` #### Videos (ShareGPT4Video)[¶](#videos-sharegpt4video "Permanent link") Start vLLM: `[](#__codelineno-70-1)vllm serve Qwen/Qwen2.5-VL-7B-Instruct \ [](#__codelineno-70-2) --dtype bfloat16 \ [](#__codelineno-70-3) --limit-mm-per-prompt '{"video": 1}' \ [](#__codelineno-70-4) --allowed-local-media-path /path/to/sharegpt4video/videos` Send requests with videos: `[](#__codelineno-71-1)vllm bench serve \ [](#__codelineno-71-2) --backend openai-chat \ [](#__codelineno-71-3) --model Qwen/Qwen2.5-VL-7B-Instruct \ [](#__codelineno-71-4) --dataset-name sharegpt \ [](#__codelineno-71-5) --dataset-path /path/to/ShareGPT4Video/llava_v1_5_mix665k_with_video_chatgpt72k_share4video28k.json \ [](#__codelineno-71-6) --num-prompts 100 \ [](#__codelineno-71-7) --save-result \ [](#__codelineno-71-8) --result-dir ~/vllm_benchmark_results \ [](#__codelineno-71-9) --save-detailed \ [](#__codelineno-71-10) --endpoint /v1/chat/completions` #### Synthetic Random Images (random-mm)[¶](#synthetic-random-images-random-mm "Permanent link") Generate synthetic image inputs alongside random text prompts to stress-test vision models without external datasets. Notes: - For online benchmarks, use `--backend openai-chat` with endpoint `/v1/chat/completions`. - For offline benchmarks, use `--backend vllm-chat` (see [Offline Throughput Benchmark](#-offline-throughput-benchmark) for an example). Start the server (example): `[](#__codelineno-72-1)vllm serve Qwen/Qwen2.5-VL-3B-Instruct \ [](#__codelineno-72-2) --dtype bfloat16 \ [](#__codelineno-72-3) --max-model-len 16384 \ [](#__codelineno-72-4) --limit-mm-per-prompt '{"image": 3, "video": 0}' \ [](#__codelineno-72-5) --mm-processor-kwargs max_pixels=1003520` Benchmark. It is recommended to use the flag `--ignore-eos` to simulate real responses. You can set the size of the output via the arg `random-output-len`. Ex.1: Fixed number of items and a single image resolution, enforcing generation of approx 40 tokens: `[](#__codelineno-73-1)vllm bench serve \ [](#__codelineno-73-2) --backend openai-chat \ [](#__codelineno-73-3) --model Qwen/Qwen2.5-VL-3B-Instruct \ [](#__codelineno-73-4) --endpoint /v1/chat/completions \ [](#__codelineno-73-5) --dataset-name random-mm \ [](#__codelineno-73-6) --num-prompts 100 \ [](#__codelineno-73-7) --max-concurrency 10 \ [](#__codelineno-73-8) --random-prefix-len 25 \ [](#__codelineno-73-9) --random-input-len 300 \ [](#__codelineno-73-10) --random-output-len 40 \ [](#__codelineno-73-11) --random-range-ratio 0.2 \ [](#__codelineno-73-12) --random-mm-base-items-per-request 2 \ [](#__codelineno-73-13) --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \ [](#__codelineno-73-14) --random-mm-bucket-config '{(224, 224, 1): 1.0}' \ [](#__codelineno-73-15) --request-rate inf \ [](#__codelineno-73-16) --ignore-eos \ [](#__codelineno-73-17) --seed 42` The number of items per request can be controlled by passing multiple image buckets: `[](#__codelineno-74-1) --random-mm-base-items-per-request 2 \ [](#__codelineno-74-2) --random-mm-num-mm-items-range-ratio 0.5 \ [](#__codelineno-74-3) --random-mm-limit-mm-per-prompt '{"image": 4, "video": 0}' \ [](#__codelineno-74-4) --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}' \` Flags specific to `random-mm`: - `--random-mm-base-items-per-request`: base number of multimodal items per request. - `--random-mm-num-mm-items-range-ratio`: vary item count uniformly in the closed integer range \[floor(n·(1−r)), ceil(n·(1+r))\]. Set r=0 to keep it fixed; r=1 allows 0 items. - `--random-mm-limit-mm-per-prompt`: per-modality hard caps, e.g. '{"image": 3, "video": 0}'. - `--random-mm-bucket-config`: dict mapping (H, W, T) → probability. Entries with probability 0 are removed; remaining probabilities are renormalized to sum to 1. Use T=1 for images. Set any T>1 for videos (video sampling not yet supported). Behavioral notes: - If the requested base item count cannot be satisfied under the provided per-prompt limits, the tool raises an error rather than silently clamping. How sampling works: - Determine per-request item count k by sampling uniformly from the integer range defined by `--random-mm-base-items-per-request` and `--random-mm-num-mm-items-range-ratio`, then clamp k to at most the sum of per-modality limits. - For each of the k items, sample a bucket (H, W, T) according to the normalized probabilities in `--random-mm-bucket-config`, while tracking how many items of each modality have been added. - If a modality (e.g., image) reaches its limit from `--random-mm-limit-mm-per-prompt`, all buckets of that modality are excluded and the remaining bucket probabilities are renormalized before continuing. This should be seen as an edge case, and if this behavior can be avoided by setting `--random-mm-limit-mm-per-prompt` to a large number. Note that this might result in errors due to engine config `--limit-mm-per-prompt`. - The resulting request contains synthetic image data in `multi_modal_data` (OpenAI Chat format). When `random-mm` is used with the OpenAI Chat backend, prompts remain text and MM content is attached via `multi_modal_data`. ### 🔬 Multimodal Processor Benchmark[¶](#multimodal-processor-benchmark "Permanent link") Benchmark per-stage latency of the multimodal (MM) input processor pipeline, including the encoder forward pass. This is useful for profiling preprocessing bottlenecks in vision-language models. Show more The benchmark measures the following stages for each request: Stage Description `get_mm_hashes_secs` Time spent hashing multimodal inputs `get_cache_missing_items_secs` Time spent looking up the processor cache `apply_hf_processor_secs` Time spent in the HuggingFace processor `merge_mm_kwargs_secs` Time spent merging multimodal kwargs `apply_prompt_updates_secs` Time spent updating prompt tokens `preprocessor_total_secs` Total preprocessing time `encoder_forward_secs` Time spent in the encoder model forward pass `num_encoder_calls` Number of encoder invocations per request The benchmark also reports end-to-end latency (TTFT + decode time) per request. Use `--metric-percentiles` to select which percentiles to report (default: p99) and `--output-json` to save results. #### Basic Example with Synthetic Data (random-mm)[¶](#basic-example-with-synthetic-data-random-mm "Permanent link") `[](#__codelineno-75-1)vllm bench mm-processor \ [](#__codelineno-75-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-75-3) --dataset-name random-mm \ [](#__codelineno-75-4) --num-prompts 50 \ [](#__codelineno-75-5) --random-input-len 300 \ [](#__codelineno-75-6) --random-output-len 40 \ [](#__codelineno-75-7) --random-mm-base-items-per-request 2 \ [](#__codelineno-75-8) --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \ [](#__codelineno-75-9) --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}'` #### Using a HuggingFace Dataset[¶](#using-a-huggingface-dataset "Permanent link") `[](#__codelineno-76-1)vllm bench mm-processor \ [](#__codelineno-76-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-76-3) --dataset-name hf \ [](#__codelineno-76-4) --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-76-5) --hf-split train \ [](#__codelineno-76-6) --num-prompts 100` #### Warmup, Custom Percentiles, and JSON Output[¶](#warmup-custom-percentiles-and-json-output "Permanent link") `[](#__codelineno-77-1)vllm bench mm-processor \ [](#__codelineno-77-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-77-3) --dataset-name random-mm \ [](#__codelineno-77-4) --num-prompts 200 \ [](#__codelineno-77-5) --num-warmups 5 \ [](#__codelineno-77-6) --random-input-len 300 \ [](#__codelineno-77-7) --random-output-len 40 \ [](#__codelineno-77-8) --random-mm-base-items-per-request 1 \ [](#__codelineno-77-9) --metric-percentiles 50,90,95,99 \ [](#__codelineno-77-10) --output-json results.json` See [`vllm bench mm-processor`](https://docs.vllm.ai/en/cli/bench/mm_processor/) for the full argument reference. ### Embedding Benchmark[¶](#embedding-benchmark "Permanent link") Benchmark the performance of embedding requests in vLLM. Show more #### Text Embeddings[¶](#text-embeddings "Permanent link") Unlike generative models which use Completions API or Chat Completions API, you should set `--backend openai-embeddings` and `--endpoint /v1/embeddings` to use the Embeddings API. You can use any text dataset to benchmark the model, such as ShareGPT. Start the server: `[](#__codelineno-78-1)vllm serve jinaai/jina-embeddings-v3 --trust-remote-code` Run the benchmark: `[](#__codelineno-79-1)# download dataset [](#__codelineno-79-2)# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-79-3)vllm bench serve \ [](#__codelineno-79-4) --model jinaai/jina-embeddings-v3 \ [](#__codelineno-79-5) --backend openai-embeddings \ [](#__codelineno-79-6) --endpoint /v1/embeddings \ [](#__codelineno-79-7) --dataset-name sharegpt \ [](#__codelineno-79-8) --dataset-path /ShareGPT_V3_unfiltered_cleaned_split.json` #### Multi-modal Embeddings[¶](#multi-modal-embeddings "Permanent link") Unlike generative models which use Completions API or Chat Completions API, you should set `--endpoint /v1/embeddings` to use the Embeddings API. The backend to use depends on the model: - CLIP: `--backend openai-embeddings-clip` - VLM2Vec: `--backend openai-embeddings-vlm2vec` For other models, please add your own implementation inside [vllm/benchmarks/lib/endpoint\_request\_func.py](https://github.com/vllm-project/vllm/blob/main/vllm/benchmarks/lib/endpoint_request_func.py) to match the expected instruction format. You can use any text or multi-modal dataset to benchmark the model, as long as the model supports it. For example, you can use ShareGPT and VisionArena to benchmark vision-language embeddings. Serve and benchmark CLIP: `[](#__codelineno-80-1)# Run this in another process [](#__codelineno-80-2)vllm serve openai/clip-vit-base-patch32 [](#__codelineno-80-3)[](#__codelineno-80-4)# Run these one by one after the server is up [](#__codelineno-80-5)# download dataset [](#__codelineno-80-6)# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-80-7)vllm bench serve \ [](#__codelineno-80-8) --model openai/clip-vit-base-patch32 \ [](#__codelineno-80-9) --backend openai-embeddings-clip \ [](#__codelineno-80-10) --endpoint /v1/embeddings \ [](#__codelineno-80-11) --dataset-name sharegpt \ [](#__codelineno-80-12) --dataset-path /ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-80-13)[](#__codelineno-80-14)vllm bench serve \ [](#__codelineno-80-15) --model openai/clip-vit-base-patch32 \ [](#__codelineno-80-16) --backend openai-embeddings-clip \ [](#__codelineno-80-17) --endpoint /v1/embeddings \ [](#__codelineno-80-18) --dataset-name hf \ [](#__codelineno-80-19) --dataset-path lmarena-ai/VisionArena-Chat` Serve and benchmark VLM2Vec: `[](#__codelineno-81-1)# Run this in another process [](#__codelineno-81-2)vllm serve TIGER-Lab/VLM2Vec-Full --runner pooling \ [](#__codelineno-81-3) --trust-remote-code \ [](#__codelineno-81-4) --chat-template examples/template_vlm2vec_phi3v.jinja [](#__codelineno-81-5)[](#__codelineno-81-6)# Run these one by one after the server is up [](#__codelineno-81-7)# download dataset [](#__codelineno-81-8)# wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-81-9)vllm bench serve \ [](#__codelineno-81-10) --model TIGER-Lab/VLM2Vec-Full \ [](#__codelineno-81-11) --backend openai-embeddings-vlm2vec \ [](#__codelineno-81-12) --endpoint /v1/embeddings \ [](#__codelineno-81-13) --dataset-name sharegpt \ [](#__codelineno-81-14) --dataset-path /ShareGPT_V3_unfiltered_cleaned_split.json [](#__codelineno-81-15)[](#__codelineno-81-16)vllm bench serve \ [](#__codelineno-81-17) --model TIGER-Lab/VLM2Vec-Full \ [](#__codelineno-81-18) --backend openai-embeddings-vlm2vec \ [](#__codelineno-81-19) --endpoint /v1/embeddings \ [](#__codelineno-81-20) --dataset-name hf \ [](#__codelineno-81-21) --dataset-path lmarena-ai/VisionArena-Chat` ### Reranker Benchmark[¶](#reranker-benchmark "Permanent link") Benchmark the performance of rerank requests in vLLM. Show more Unlike generative models which use Completions API or Chat Completions API, you should set `--backend vllm-rerank` and `--endpoint /v1/rerank` to use the Reranker API. For reranking, the only supported dataset is `--dataset-name random-rerank` Start the server: `[](#__codelineno-82-1)vllm serve BAAI/bge-reranker-v2-m3` Run the benchmark: `[](#__codelineno-83-1)vllm bench serve \ [](#__codelineno-83-2) --model BAAI/bge-reranker-v2-m3 \ [](#__codelineno-83-3) --backend vllm-rerank \ [](#__codelineno-83-4) --endpoint /v1/rerank \ [](#__codelineno-83-5) --dataset-name random-rerank \ [](#__codelineno-83-6) --tokenizer BAAI/bge-reranker-v2-m3 \ [](#__codelineno-83-7) --random-input-len 512 \ [](#__codelineno-83-8) --num-prompts 10 \ [](#__codelineno-83-9) --random-batch-size 5` For reranker models, this will create `num_prompts / random_batch_size` requests with `random_batch_size` "documents" where each one has close to `random_input_len` tokens. In the example above, this results in 2 rerank requests with 5 "documents" each where each document has close to 512 tokens. Please note that the `/v1/rerank` is also supported by embedding models. So if you're running with an embedding model, also set `--no_reranker`. Because in this case the query is treated as an individual prompt by the server, here we send `random_batch_size - 1` documents to account for the extra prompt which is the query. The token accounting to report the throughput numbers correctly is also adjusted. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/run-batch.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. #### `--enable-log-requests`, `--no-enable-log-requests`[¶](#-enable-log-requests-no-enable-log-requests "Permanent link") Enable logging request information, dependent on log level: - INFO: Request ID, parameters and LoRA request. - DEBUG: Prompt inputs (e.g: text, token IDs). You can set the minimum log level via `VLLM_LOGGING_LEVEL`. Default: `False` ### BatchFrontend[¶](#batchfrontend "Permanent link") Arguments for the batch runner frontend. #### `--lora-modules`[¶](#-lora-modules "Permanent link") #### `--chat-template`[¶](#-chat-template "Permanent link") #### `--chat-template-content-format`[¶](#-chat-template-content-format "Permanent link") Possible choices: `auto`, `openai`, `string` Default: `auto` #### `--trust-request-chat-template`, `--no-trust-request-chat-template`[¶](#-trust-request-chat-template-no-trust-request-chat-template "Permanent link") Default: `False` #### `--default-chat-template-kwargs`[¶](#-default-chat-template-kwargs "Permanent link") : Should either be a valid JSON string or JSON keys passed individually. #### `--response-role`[¶](#-response-role "Permanent link") Default: `assistant` #### `--return-tokens-as-token-ids`, `--no-return-tokens-as-token-ids`[¶](#-return-tokens-as-token-ids-no-return-tokens-as-token-ids "Permanent link") Default: `False` #### `--enable-auto-tool-choice`, `--no-enable-auto-tool-choice`[¶](#-enable-auto-tool-choice-no-enable-auto-tool-choice "Permanent link") Default: `False` #### `--exclude-tools-when-tool-choice-none`, `--no-exclude-tools-when-tool-choice-none`[¶](#-exclude-tools-when-tool-choice-none-no-exclude-tools-when-tool-choice-none "Permanent link") Default: `False` #### `--tool-call-parser`[¶](#-tool-call-parser "Permanent link") #### `--tool-parser-plugin`[¶](#-tool-parser-plugin "Permanent link") Default: `""` #### `--tool-server`[¶](#-tool-server "Permanent link") #### `--log-config-file`[¶](#-log-config-file "Permanent link") #### `--max-log-len`[¶](#-max-log-len "Permanent link") #### `--enable-prompt-tokens-details`, `--no-enable-prompt-tokens-details`[¶](#-enable-prompt-tokens-details-no-enable-prompt-tokens-details "Permanent link") Default: `False` #### `--enable-server-load-tracking`, `--no-enable-server-load-tracking`[¶](#-enable-server-load-tracking-no-enable-server-load-tracking "Permanent link") Default: `False` #### `--enable-force-include-usage`, `--no-enable-force-include-usage`[¶](#-enable-force-include-usage-no-enable-force-include-usage "Permanent link") Default: `False` #### `--enable-tokenizer-info-endpoint`, `--no-enable-tokenizer-info-endpoint`[¶](#-enable-tokenizer-info-endpoint-no-enable-tokenizer-info-endpoint "Permanent link") Default: `False` #### `--enable-log-outputs`, `--no-enable-log-outputs`[¶](#-enable-log-outputs-no-enable-log-outputs "Permanent link") Default: `False` #### `--enable-log-deltas`, `--no-enable-log-deltas`[¶](#-enable-log-deltas-no-enable-log-deltas "Permanent link") Default: `True` #### `--log-error-stack`, `--no-log-error-stack`[¶](#-log-error-stack-no-log-error-stack "Permanent link") Default: `False` #### `--tokens-only`, `--no-tokens-only`[¶](#-tokens-only-no-tokens-only "Permanent link") Default: `False` #### `--fingerprint-mode`[¶](#-fingerprint-mode "Permanent link") Possible choices: `custom`, `full`, `hash`, `none` Default: `full` #### `--fingerprint-value`[¶](#-fingerprint-value "Permanent link") #### `-i`, `--input-file`[¶](#-i-input-file "Permanent link") The path or url to a single input file. Currently supports local file paths, or the http protocol (http or https). If a URL is specified, the file should be available via HTTP GET. #### `-o`, `--output-file`[¶](#-o-output-file "Permanent link") The path or url to a single output file. Currently supports local file paths, or web (http or https) urls. If a URL is specified, the file should be available via HTTP PUT. #### `--output-tmp-dir`[¶](#-output-tmp-dir "Permanent link") The directory to store the output file before uploading it to the output URL. #### `--enable-metrics`[¶](#-enable-metrics "Permanent link") Enable Prometheus metrics Default: `False` #### `--host`[¶](#-host "Permanent link") Host name for the Prometheus metrics server (only needed if enable-metrics is set). #### `--port`[¶](#-port "Permanent link") Port number for the Prometheus metrics server (only needed if enable-metrics is set). Default: `8000` #### `--url`[¶](#-url "Permanent link") \[DEPRECATED\] Host name for the Prometheus metrics server (only needed if enable-metrics is set). Use --host instead. Default: `0.0.0.0` ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [Benchmarking](https://docs.vllm.ai/en/latest/) [](https://github.com/vllm-project/vllm/edit/main/docs/benchmarking/README.md "Edit this page") vLLM provides comprehensive benchmarking tools for performance testing and evaluation: - **[Benchmark CLI](https://docs.vllm.ai/en/latest/cli/)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing. - **[Parameter Sweeps](https://docs.vllm.ai/en/latest/sweeps/)**: Automate `vllm bench` runs for multiple configurations, useful for [optimization and tuning](https://docs.vllm.ai/en/configuration/optimization/). - **[Performance Dashboard](https://docs.vllm.ai/en/latest/dashboard/)**: Automated CI that publishes benchmarks on each commit. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/benchmarking/dashboard.md "Edit this page") The performance dashboard is used to confirm whether new changes improve/degrade performance under various workloads. It is updated by triggering benchmark runs on every commit with both the `perf-benchmarks` and `ready` labels, and when a PR is merged into vLLM. The results are automatically published to the public [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm). ## Manually Trigger the benchmark[¶](#manually-trigger-the-benchmark "Permanent link") Use [vllm-ci-test-repo images](https://gallery.ecr.aws/q9t5s3a7/vllm-ci-test-repo) with vLLM benchmark suite. For x86 CPU environment, please use the image with "-cpu" postfix. For AArch64 CPU environment, please use the image with "-arm64-cpu" postfix. Here is an example for docker run command for CPU. For GPUs skip setting the `ON_CPU` env var. `[](#__codelineno-0-1)export VLLM_COMMIT=7f42dc20bb2800d09faa72b26f25d54e26f1b694 # use full commit hash from the main branch [](#__codelineno-0-2)export HF_TOKEN= [](#__codelineno-0-3)if [[ "$(uname -m)" == aarch64 || "$(uname -m)" == arm64 ]]; then [](#__codelineno-0-4) IMG_SUFFIX="arm64-cpu" [](#__codelineno-0-5)else [](#__codelineno-0-6) IMG_SUFFIX="cpu" [](#__codelineno-0-7)fi [](#__codelineno-0-8)docker run -it --entrypoint /bin/bash -v /data/huggingface:/root/.cache/huggingface -e HF_TOKEN=$HF_TOKEN -e ON_CPU=1 --shm-size=16g --name vllm-cpu-ci public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:${VLLM_COMMIT}-${IMG_SUFFIX}` Then, run below command inside the docker instance. `[](#__codelineno-1-1)bash .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh` When run, benchmark script generates results under **benchmark/results** folder, along with the benchmark\_results.md and benchmark\_results.json. ### Runtime environment variables[¶](#runtime-environment-variables "Permanent link") - `ON_CPU`: set the value to '1' on Intel® Xeon® and Arm® Neoverse™ Processors. Default value is 0. - `SERVING_JSON`: JSON file to use for the serving tests. Default value is empty string (use default file). - `LATENCY_JSON`: JSON file to use for the latency tests. Default value is empty string (use default file). - `THROUGHPUT_JSON`: JSON file to use for the throughout tests. Default value is empty string (use default file). - `REMOTE_HOST`: IP for the remote vLLM service to benchmark. Default value is empty string. - `REMOTE_PORT`: Port for the remote vLLM service to benchmark. Default value is empty string. - `PROMPTS_PER_CONCURRENCY`: Multiplier to compute `num_prompts` for serving tests (`num_prompts = max_concurrency × value`). Overrides JSON `num_prompts`. Default is NULL. - `ENABLE_ADAPTIVE_CONCURRENCY`: set the value to '1' to enable adaptive SLA-based concurrency search after the static serving max\_concurrency sweep. Default value is 0. - `SLA_TTFT_MS`: default TTFT SLA threshold in milliseconds for adaptive concurrency search. Default value is 3000. - `SLA_TPOT_MS`: default TPOT SLA threshold in milliseconds for adaptive concurrency search. Default value is 100. - `ADAPTIVE_MAX_PROBES`: maximum number of extra adaptive search probes. Default value is 8. - `ADAPTIVE_MAX_CONCURRENCY`: maximum allowed concurrency during adaptive search. Default value is 1024. ### Visualization[¶](#visualization "Permanent link") The `convert-results-json-to-markdown.py` helps you put the benchmarking results inside a markdown table with real benchmarking results. You can find the result presented as a table inside the `buildkite/performance-benchmark` job page. If you do not see the table, please wait till the benchmark finish running. The json version of the table (together with the json version of the benchmark) will be also attached to the markdown file. The raw benchmarking results (in the format of json files) are in the `Artifacts` tab of the benchmarking. #### Performance Results Comparison[¶](#performance-results-comparison "Permanent link") The `compare-json-results.py` helps to compare benchmark results JSON files converted using `convert-results-json-to-markdown.py`. When run, benchmark script generates results under `benchmark/results` folder, along with the `benchmark_results.md` and `benchmark_results.json`. `compare-json-results.py` compares two `benchmark_results.json` files and provides performance ratio e.g. for Output Tput, Median TTFT and Median TPOT. If only one benchmark\_results.json is passed, `compare-json-results.py` compares different TP and PP configurations in the benchmark\_results.json instead. Here is an example using the script to compare result\_a and result\_b with max concurrency and qps for same Model, Dataset name, input/output length. `python3 compare-json-results.py -f results_a/benchmark_results.json -f results_b/benchmark_results.json` **_Output Tput (tok/s) — Model : \[ meta-llama/Llama-3.1-8B-Instruct \] , Dataset Name : \[ random \] , Input Len : \[ 2048.0 \] , Output Len : \[ 2048.0 \]_** \# of max concurrency qps results\_a/benchmark\_results.json results\_b/benchmark\_results.json perf\_ratio 0 12 inf 24.98 186.03 7.45 1 16 inf 25.49 246.92 9.69 2 24 inf 27.74 293.34 10.57 3 32 inf 28.61 306.69 10.72 **_compare-json-results.py – Command-Line Parameters_** compare-json-results.py provides configurable parameters to compare one or more benchmark\_results.json files and generate summary tables and plots. In most cases, users only need to specify --file to parse the desired benchmark results. Parameter Type Default Value Description `--file` `str` (appendable) _None_ Input JSON result file(s). Can be specified multiple times to compare multiple benchmark outputs. `--debug` `bool` `False` Enables debug mode. When set, prints all available information to aid troubleshooting and validation. `--plot` / `--no-plot` `bool` `True` Controls whether performance plots are generated. Use `--no-plot` to disable graph generation. `--xaxis` `str` `# of max concurrency.` Column name used as the X-axis in comparison plots (for example, concurrency or batch size). `--latency` `str` `p99` Latency aggregation method used for TTFT/TPOT. Supported values: `median` or `p99`. `--ttft-max-ms` `float` `3000.0` Reference upper bound (milliseconds) for TTFT plots, typically used to visualize SLA thresholds. `--tpot-max-ms` `float` `100.0` Reference upper bound (milliseconds) for TPOT plots, typically used to visualize SLA thresholds. **_Valid Max Concurrency Summary_** Based on the configured TTFT and TPOT SLA thresholds, compare-json-results.py computes the maximum valid concurrency for each benchmark result. The “Max # of max concurrency. (Both)” column represents the highest concurrency level that satisfies both TTFT and TPOT constraints simultaneously. This value is typically used in capacity planning and sizing guides. # Configuration Max # of max concurrency. (TTFT ≤ 10000 ms) Max # of max concurrency. (TPOT ≤ 100 ms) Max # of max concurrency. (Both) Output Tput @ Both (tok/s) TTFT @ Both (ms) TPOT @ Both (ms) 0 results-a 128.00 12.00 12.00 127.76 3000.82 93.24 1 results-b 128.00 32.00 32.00 371.42 2261.53 81.74 More information on the performance benchmarks and their parameters can be found in [Benchmark README](https://github.com/intel-ai-tce/vllm/blob/more_cpu_models/.buildkite/nightly-benchmarks/README.md) and [performance benchmark description](https://github.com/vllm-project/vllm/blob/main/.buildkite/performance-benchmarks/performance-benchmarks-descriptions.md). ## Continuous Benchmarking[¶](#continuous-benchmarking "Permanent link") The continuous benchmarking provides automated performance monitoring for vLLM across different models and GPU devices. This helps track vLLM's performance characteristics over time and identify any performance regressions or improvements. ### How It Works[¶](#how-it-works "Permanent link") The continuous benchmarking is triggered via a [GitHub workflow CI](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-benchmark.yml) in the PyTorch infrastructure repository, which runs automatically every 4 hours. The workflow executes three types of performance tests: - **Serving tests**: Measure request handling and API performance - **Throughput tests**: Evaluate token generation rates - **Latency tests**: Assess response time characteristics ### Benchmark Configuration[¶](#benchmark-configuration "Permanent link") The benchmarking currently runs on a predefined set of models configured in the [vllm-benchmarks directory](https://github.com/pytorch/pytorch-integration-testing/tree/main/vllm-benchmarks/benchmarks). To add new models for benchmarking: 1. Navigate to the appropriate GPU directory in the benchmarks configuration 2. Add your model specifications to the corresponding configuration files 3. The new models will be included in the next scheduled benchmark run --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/serve.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--headless`[¶](#-headless "Permanent link") Run in headless mode. See multi-node data parallel documentation for more details. Default: `False` #### `--api-server-count`, `-asc`[¶](#-api-server-count-asc "Permanent link") How many API server processes to run. Defaults to data\_parallel\_size if not specified. #### `--config`[¶](#-config "Permanent link") Read CLI options from a config file. Must be a YAML with the following options: https://docs.vllm.ai/en/latest/configuration/serve\_args.html #### `--grpc`[¶](#-grpc "Permanent link") Launch a gRPC server instead of the HTTP OpenAI-compatible server. Requires: pip install vllm\[grpc\]. Default: `False` #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. #### `--enable-log-requests`, `--no-enable-log-requests`[¶](#-enable-log-requests-no-enable-log-requests "Permanent link") Enable logging request information, dependent on log level: - INFO: Request ID, parameters and LoRA request. - DEBUG: Prompt inputs (e.g: text, token IDs). You can set the minimum log level via `VLLM_LOGGING_LEVEL`. Default: `False` ### Frontend[¶](#frontend "Permanent link") Arguments for the OpenAI-compatible frontend server. #### `--lora-modules`[¶](#-lora-modules "Permanent link") #### `--chat-template`[¶](#-chat-template "Permanent link") #### `--chat-template-content-format`[¶](#-chat-template-content-format "Permanent link") Possible choices: `auto`, `openai`, `string` Default: `auto` #### `--trust-request-chat-template`, `--no-trust-request-chat-template`[¶](#-trust-request-chat-template-no-trust-request-chat-template "Permanent link") Default: `False` #### `--default-chat-template-kwargs`[¶](#-default-chat-template-kwargs "Permanent link") : Should either be a valid JSON string or JSON keys passed individually. #### `--response-role`[¶](#-response-role "Permanent link") Default: `assistant` #### `--return-tokens-as-token-ids`, `--no-return-tokens-as-token-ids`[¶](#-return-tokens-as-token-ids-no-return-tokens-as-token-ids "Permanent link") Default: `False` #### `--enable-auto-tool-choice`, `--no-enable-auto-tool-choice`[¶](#-enable-auto-tool-choice-no-enable-auto-tool-choice "Permanent link") Default: `False` #### `--exclude-tools-when-tool-choice-none`, `--no-exclude-tools-when-tool-choice-none`[¶](#-exclude-tools-when-tool-choice-none-no-exclude-tools-when-tool-choice-none "Permanent link") Default: `False` #### `--tool-call-parser`[¶](#-tool-call-parser "Permanent link") #### `--tool-parser-plugin`[¶](#-tool-parser-plugin "Permanent link") Default: `""` #### `--tool-server`[¶](#-tool-server "Permanent link") #### `--log-config-file`[¶](#-log-config-file "Permanent link") #### `--max-log-len`[¶](#-max-log-len "Permanent link") #### `--enable-prompt-tokens-details`, `--no-enable-prompt-tokens-details`[¶](#-enable-prompt-tokens-details-no-enable-prompt-tokens-details "Permanent link") Default: `False` #### `--enable-server-load-tracking`, `--no-enable-server-load-tracking`[¶](#-enable-server-load-tracking-no-enable-server-load-tracking "Permanent link") Default: `False` #### `--enable-force-include-usage`, `--no-enable-force-include-usage`[¶](#-enable-force-include-usage-no-enable-force-include-usage "Permanent link") Default: `False` #### `--enable-tokenizer-info-endpoint`, `--no-enable-tokenizer-info-endpoint`[¶](#-enable-tokenizer-info-endpoint-no-enable-tokenizer-info-endpoint "Permanent link") Default: `False` #### `--enable-log-outputs`, `--no-enable-log-outputs`[¶](#-enable-log-outputs-no-enable-log-outputs "Permanent link") Default: `False` #### `--enable-log-deltas`, `--no-enable-log-deltas`[¶](#-enable-log-deltas-no-enable-log-deltas "Permanent link") Default: `True` #### `--log-error-stack`, `--no-log-error-stack`[¶](#-log-error-stack-no-log-error-stack "Permanent link") Default: `False` #### `--tokens-only`, `--no-tokens-only`[¶](#-tokens-only-no-tokens-only "Permanent link") Default: `False` #### `--fingerprint-mode`[¶](#-fingerprint-mode "Permanent link") Possible choices: `custom`, `full`, `hash`, `none` Default: `full` #### `--fingerprint-value`[¶](#-fingerprint-value "Permanent link") #### `--host`[¶](#-host "Permanent link") Host name. #### `--port`[¶](#-port "Permanent link") Port number. Default: `8000` #### `--data-parallel-supervisor-port`[¶](#-data-parallel-supervisor-port "Permanent link") HTTP port for aggregated health endpoints in multi-port external LB mode. Default: `9256` #### `--dp-supervisor-probe-interval-s`[¶](#-dp-supervisor-probe-interval-s "Permanent link") Seconds between aggregated health probes in multi-port external LB mode. Default: `5.0` #### `--dp-supervisor-probe-timeout-s`[¶](#-dp-supervisor-probe-timeout-s "Permanent link") Seconds to wait between retries when a child health probe fails with a connection error in multi-port external LB mode. Default: `5.0` #### `--dp-supervisor-probe-failure-threshold`[¶](#-dp-supervisor-probe-failure-threshold "Permanent link") Number of consecutive connection-error retries before a child health probe is declared failed in multi-port external LB mode. Default: `3` #### `--uds`[¶](#-uds "Permanent link") Unix domain socket path. If set, host and port arguments are ignored. #### `--uvicorn-log-level`[¶](#-uvicorn-log-level "Permanent link") Possible choices: `critical`, `debug`, `error`, `info`, `trace`, `warning` Log level for uvicorn. Default: `info` #### `--disable-uvicorn-access-log`, `--no-disable-uvicorn-access-log`[¶](#-disable-uvicorn-access-log-no-disable-uvicorn-access-log "Permanent link") Disable uvicorn access log. Default: `False` #### `--disable-access-log-for-endpoints`[¶](#-disable-access-log-for-endpoints "Permanent link") Comma-separated list of endpoint paths to exclude from uvicorn access logs. This is useful to reduce log noise from high-frequency endpoints like health checks. Example: "/health,/metrics,/ping". When set, access logs for requests to these paths will be suppressed while keeping logs for other endpoints. #### `--allow-credentials`, `--no-allow-credentials`[¶](#-allow-credentials-no-allow-credentials "Permanent link") Allow credentials. Default: `False` #### `--allowed-origins`[¶](#-allowed-origins "Permanent link") Allowed origins. Default: `['*']` #### `--allowed-methods`[¶](#-allowed-methods "Permanent link") Allowed methods. Default: `['*']` Allowed headers. Default: `['*']` #### `--api-key`[¶](#-api-key "Permanent link") If provided, the server will require one of these keys to be presented in the header. #### `--ssl-keyfile`[¶](#-ssl-keyfile "Permanent link") The file path to the SSL key file. #### `--ssl-certfile`[¶](#-ssl-certfile "Permanent link") The file path to the SSL cert file. #### `--ssl-ca-certs`[¶](#-ssl-ca-certs "Permanent link") The CA certificates file. #### `--enable-ssl-refresh`, `--no-enable-ssl-refresh`[¶](#-enable-ssl-refresh-no-enable-ssl-refresh "Permanent link") Refresh SSL Context when SSL certificate files change Default: `False` #### `--ssl-cert-reqs`[¶](#-ssl-cert-reqs "Permanent link") Whether client certificate is required (see stdlib ssl module's). Default: `0` #### `--ssl-ciphers`[¶](#-ssl-ciphers "Permanent link") SSL cipher suites for HTTPS (TLS 1.2 and below only). Example: 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305' #### `--root-path`[¶](#-root-path "Permanent link") FastAPI root\_path when app is behind a path based routing proxy. #### `--middleware`[¶](#-middleware "Permanent link") Additional ASGI middleware to apply to the app. We accept multiple --middleware arguments. The value should be an import path. If a function is provided, vLLM will add it to the server using `@app.middleware('http')`. If a class is provided, vLLM will add it to the server using `app.add_middleware()`. Default: `[]` If specified, API server will add X-Request-Id header to responses. Default: `False` #### `--disable-fastapi-docs`, `--no-disable-fastapi-docs`[¶](#-disable-fastapi-docs-no-disable-fastapi-docs "Permanent link") Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. Default: `False` #### `--h11-max-incomplete-event-size`[¶](#-h11-max-incomplete-event-size "Permanent link") Maximum size (bytes) of an incomplete HTTP event (header or body) for h11 parser. Helps mitigate header abuse. Default: 4194304 (4 MB). Default: `4194304` Maximum number of HTTP headers allowed in a request for h11 parser. Helps mitigate header abuse. Default: 256. Default: `256` #### `--enable-offline-docs`, `--no-enable-offline-docs`[¶](#-enable-offline-docs-no-enable-offline-docs "Permanent link") Enable offline FastAPI documentation for air-gapped environments. Uses vendored static assets bundled with vLLM. Default: `False` #### `--enable-flash-late-interaction`, `--no-enable-flash-late-interaction`[¶](#-enable-flash-late-interaction-no-enable-flash-late-interaction "Permanent link") If set, run pooling score MaxSim on GPU in the API server process. Can significantly improve late-interaction scoring performance. Default: `True` ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/latency.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--input-len`[¶](#-input-len "Permanent link") Default: `32` #### `--output-len`[¶](#-output-len "Permanent link") Default: `128` #### `--batch-size`[¶](#-batch-size "Permanent link") Default: `8` #### `--n`[¶](#-n "Permanent link") Number of generated sequences per prompt. Default: `1` #### `--use-beam-search`[¶](#-use-beam-search "Permanent link") Default: `False` #### `--num-iters-warmup`[¶](#-num-iters-warmup "Permanent link") Number of iterations to run for warmup. Default: `10` #### `--num-iters`[¶](#-num-iters "Permanent link") Number of iterations to run. Default: `30` #### `--profile`[¶](#-profile "Permanent link") profile the generation process of a single batch Default: `False` #### `--output-json`[¶](#-output-json "Permanent link") Path to save the latency results in JSON format. #### `--disable-detokenize`[¶](#-disable-detokenize "Permanent link") Do not detokenize responses (i.e. do not include detokenization time in the latency measurement) Default: `False` #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. Default: `False` #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/mm_processor.md "Edit this page") ## Overview[¶](#overview "Permanent link") `vllm bench mm-processor` profiles the multimodal input processor pipeline of vision-language models. It measures per-stage latency from the HuggingFace processor through to the encoder forward pass, helping you identify preprocessing bottlenecks and understand how different image resolutions or item counts affect end-to-end request time. The benchmark supports two data sources: synthetic random multimodal inputs (`random-mm`) and HuggingFace datasets (`hf`). Warmup requests are run before measurement to ensure stable results. ## Quick Start[¶](#quick-start "Permanent link") `[](#__codelineno-0-1)vllm bench mm-processor \ [](#__codelineno-0-2) --model Qwen/Qwen2-VL-7B-Instruct \ [](#__codelineno-0-3) --dataset-name random-mm \ [](#__codelineno-0-4) --num-prompts 50 \ [](#__codelineno-0-5) --random-input-len 300 \ [](#__codelineno-0-6) --random-output-len 40 \ [](#__codelineno-0-7) --random-mm-base-items-per-request 2 \ [](#__codelineno-0-8) --random-mm-limit-mm-per-prompt '{"image": 3, "video": 0}' \ [](#__codelineno-0-9) --random-mm-bucket-config '{(256, 256, 1): 0.7, (720, 1280, 1): 0.3}'` ## Measured Stages[¶](#measured-stages "Permanent link") Stage Description `get_mm_hashes_secs` Time spent hashing multimodal inputs `get_cache_missing_items_secs` Time spent looking up the processor cache `apply_hf_processor_secs` Time spent in the HuggingFace processor `merge_mm_kwargs_secs` Time spent merging multimodal kwargs `apply_prompt_updates_secs` Time spent updating prompt tokens `preprocessor_total_secs` Total preprocessing time `encoder_forward_secs` Time spent in the encoder model forward pass `num_encoder_calls` Number of encoder invocations per request The benchmark also reports end-to-end latency (TTFT + decode time) per request. Use `--metric-percentiles` to select which percentiles to report (default: p99) and `--output-json` to save results. For more examples (HF datasets, warmup, JSON output), see [Benchmarking CLI — Multimodal Processor Benchmark](https://docs.vllm.ai/en/benchmarking/cli/#multimodal-processor-benchmark). ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. #### `--dataset-name`[¶](#-dataset-name "Permanent link") Possible choices: `random-mm`, `hf` Name of the dataset to benchmark on. Defaults to 'random-mm'. Default: `random-mm` #### `--num-prompts`[¶](#-num-prompts "Permanent link") Number of prompts to process. Default: `10` #### `--num-warmups`[¶](#-num-warmups "Permanent link") Number of warmup prompts to process. Default: `1` #### `--random-input-len`[¶](#-random-input-len "Permanent link") Number of input tokens per request, used only for random sampling. Default: `1024` #### `--random-output-len`[¶](#-random-output-len "Permanent link") Number of output tokens per request, used only for random sampling. Default: `128` #### `--random-range-ratio`[¶](#-random-range-ratio "Permanent link") Range ratio for sampling input/output length, used only for random sampling. A single float applies to both ISL and OSL. A JSON dict like '{"input": 0.3, "output": 0.5}' sets them independently. Values must be in \[0, 1). Default: `0.0` #### `--random-prefix-len`[¶](#-random-prefix-len "Permanent link") Number of fixed prefix tokens before the random context in a request. The total input length is the sum of `random-prefix-len` and a random context length sampled from \[input\_len \* (1 - range\_ratio), input\_len \* (1 + range\_ratio)\]. Default: `0` #### `--random-batch-size`[¶](#-random-batch-size "Permanent link") Batch size for random sampling. Only used for embeddings benchmark. Default: `1` #### `--no-reranker`[¶](#-no-reranker "Permanent link") Whether the model supports reranking natively. Only used for reranker benchmark. Default: `False` #### `--random-mm-base-items-per-request`[¶](#-random-mm-base-items-per-request "Permanent link") Base number of multimodal items per request for random-mm. Actual per-request count is sampled around this base using --random-mm-num-mm-items-range-ratio. Default: `1` #### `--random-mm-num-mm-items-range-ratio`[¶](#-random-mm-num-mm-items-range-ratio "Permanent link") Range ratio r in \[0, 1\] for sampling items per request. We sample uniformly from the closed integer range \[floor(n_(1-r)), ceil(n_(1+r))\] where n is the base items per request. r=0 keeps it fixed; r=1 allows 0 items. The maximum is clamped to the sum of per-modality limits from --random-mm-limit-mm-per-prompt. An error is raised if the computed min exceeds the max. Default: `0.0` #### `--random-mm-limit-mm-per-prompt`[¶](#-random-mm-limit-mm-per-prompt "Permanent link") Per-modality hard caps for items attached per request, e.g. '{"image": 3, "video": 0}'. The sampled per-request item count is clamped to the sum of these limits. When a modality reaches its cap, its buckets are excluded and probabilities are renormalized.OBS.: Only image sampling is supported for now. Default: `{'image': 255, 'video': 1}` #### `--random-mm-bucket-config`[¶](#-random-mm-bucket-config "Permanent link") The bucket config is a dictionary mapping a multimodal itemsampling configuration to a probability.Currently allows for 2 modalities: images and videos. An bucket key is a tuple of (height, width, num\_frames)The value is the probability of sampling that specific item. Example: --random-mm-bucket-config {(256, 256, 1): 0.5, (720, 1280, 1): 0.4, (720, 1280, 16): 0.10} First item: images with resolution 256x256 w.p. 0.5Second item: images with resolution 720x1280 w.p. 0.4 Third item: videos with resolution 720x1280 and 16 frames w.p. 0.1OBS.: If the probabilities do not sum to 1, they are normalized.OBS bis.: Only image sampling is supported for now. Default: `{(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}` #### `--dataset-path`[¶](#-dataset-path "Permanent link") Path to the dataset file or HuggingFace dataset name (e.g., 'yale-nlp/MMVU', 'lmarena-ai/VisionArena-Chat'). #### `--hf-subset`[¶](#-hf-subset "Permanent link") Subset of the HuggingFace dataset (optional). #### `--hf-split`[¶](#-hf-split "Permanent link") Split of the HuggingFace dataset (e.g., 'train', 'test', 'validation'). #### `--output-len`[¶](#-output-len "Permanent link") Output length for each request. Overrides the default output lengths from the dataset. #### `--output-json`[¶](#-output-json "Permanent link") Path to save the benchmark results in JSON format. #### `--metric-percentiles`[¶](#-metric-percentiles "Permanent link") Comma-separated list of percentiles to calculate (e.g., '50,90,99'). Default: `99` #### `--disable-tqdm`[¶](#-disable-tqdm "Permanent link") Disable tqdm progress bar. Default: `False` ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/serve.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--trust-remote-code`[¶](#-trust-remote-code "Permanent link") Trust remote code from huggingface Default: `False` #### `--seed`[¶](#-seed "Permanent link") Default: `0` #### `--num-prompts`[¶](#-num-prompts "Permanent link") Number of prompts to process. Default: `1000` #### `--dataset-name`[¶](#-dataset-name "Permanent link") Possible choices: `sharegpt`, `burstgpt`, `sonnet`, `random`, `random-mm`, `random-rerank`, `hf`, `custom`, `custom_audio`, `custom_image`, `custom_mm`, `prefix_repetition`, `spec_bench`, `speed_bench`, `timed_trace` Name of the dataset to benchmark on. Default: `random` #### `--no-stream`[¶](#-no-stream "Permanent link") Do not load the dataset in streaming mode. Default: `False` #### `--dataset-path`[¶](#-dataset-path "Permanent link") Path to the sharegpt/sonnet dataset or the HF dataset ID if using HF dataset. #### `--no-oversample`[¶](#-no-oversample "Permanent link") Do not oversample if the dataset has fewer samples than num-prompts. Default: `False` #### `--skip-chat-template`[¶](#-skip-chat-template "Permanent link") Skip applying chat template to prompt for datasets that support it. Default: `False` #### `--enable-multimodal-chat`[¶](#-enable-multimodal-chat "Permanent link") Enable multimodal chat transformation for datasets that support it. Default: `False` #### `--disable-shuffle`[¶](#-disable-shuffle "Permanent link") Disable shuffling of dataset samples for deterministic ordering. Default: `False` #### `--label`[¶](#-label "Permanent link") The label (prefix) of the benchmark results. If not specified, the value of '--backend' will be used as the label. #### `--backend`[¶](#-backend "Permanent link") Possible choices: `vllm`, `openai`, `openai-chat`, `openai-audio`, `openai-embeddings`, `openai-embeddings-chat`, `openai-embeddings-clip`, `openai-embeddings-vlm2vec`, `infinity-embeddings`, `infinity-embeddings-clip`, `vllm-pooling`, `vllm-rerank` The type of backend or endpoint to use for the benchmark. Default: `openai` #### `--base-url`[¶](#-base-url "Permanent link") Server or API base url if not using http host and port. #### `--host`[¶](#-host "Permanent link") Default: `127.0.0.1` #### `--port`[¶](#-port "Permanent link") Default: `8000` #### `--endpoint`[¶](#-endpoint "Permanent link") API endpoint. Default: `/v1/completions` Key-value pairs (e.g, --header x-additional-info=0.3.3) for headers to be passed with each request. These headers override per backend constants and values set via environment variable, and will be overridden by other arguments (such as request ids). #### `--max-concurrency`[¶](#-max-concurrency "Permanent link") Maximum number of concurrent requests. This can be used to help simulate an environment where a higher level component is enforcing a maximum number of concurrent requests. While the --request-rate argument controls the rate at which requests are initiated, this argument will control how many are actually allowed to execute at a time. This means that when used in combination, the actual request rate may be lower than specified with --request-rate, if the server is not processing requests fast enough to keep up. #### `--model`[¶](#-model "Permanent link") Name of the model. If not specified, will fetch the first model from the server's /v1/models endpoint. #### `--input-len`[¶](#-input-len "Permanent link") General input length for datasets. Maps to dataset-specific input length arguments (e.g., --random-input-len, --sonnet-input-len). If not specified, uses dataset defaults. #### `--output-len`[¶](#-output-len "Permanent link") General output length for datasets. Maps to dataset-specific output length arguments (e.g., --random-output-len, --sonnet-output-len). If not specified, uses dataset defaults. #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the tokenizer, if not using the default tokenizer. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Tokenizer mode: ``- "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek_v32" will always use the tokenizer from `deepseek_v32`. - "qwen_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins.`` Default: `auto` #### `--use-beam-search`[¶](#-use-beam-search "Permanent link") Default: `False` #### `--logprobs`[¶](#-logprobs "Permanent link") Number of logprobs-per-token to compute & return as part of the request. If unspecified, then either (1) if beam search is disabled, no logprobs are computed & a single dummy logprob is returned for each token; or (2) if beam search is enabled 1 logprob per token is computed #### `--request-rate`[¶](#-request-rate "Permanent link") Number of requests per second. If this is inf, then all the requests are sent at time 0. Otherwise, we use Poisson process or gamma distribution to synthesize the request arrival times. Default: `inf` #### `--burstiness`[¶](#-burstiness "Permanent link") Burstiness factor of the request generation. Only take effect when request\_rate is not inf. Default value is 1, which follows Poisson process. Otherwise, the request intervals follow a gamma distribution. A lower burstiness value (0 < burstiness < 1) results in more bursty requests. A higher burstiness value (burstiness > 1) results in a more uniform arrival of requests. Default: `1.0` #### `--disable-tqdm`[¶](#-disable-tqdm "Permanent link") Specify to disable tqdm progress bar. Default: `False` #### `--num-warmups`[¶](#-num-warmups "Permanent link") Number of warmup requests. Default: `0` #### `--profile`[¶](#-profile "Permanent link") Use vLLM Profiling. --profiler-config must be provided on the server. Default: `False` #### `--save-result`[¶](#-save-result "Permanent link") Specify to save benchmark results to a json file Default: `False` #### `--save-detailed`[¶](#-save-detailed "Permanent link") When saving the results, whether to include per request information such as response, error, ttfts, tpots, etc. Default: `False` #### `--append-result`[¶](#-append-result "Permanent link") Append the benchmark result to the existing json file. Default: `False` #### `--metadata`[¶](#-metadata "Permanent link") Key-value pairs (e.g, --metadata version=0.3.3 tp=1) for metadata of this run to be saved in the result JSON file for record keeping purposes. #### `--result-dir`[¶](#-result-dir "Permanent link") Specify directory to save benchmark json results.If not specified, results are saved in the current directory. #### `--result-filename`[¶](#-result-filename "Permanent link") Specify the filename to save benchmark json results.If not specified, results will be saved in {label}-{args.request\_rate}qps-{base\_model\_id}-{current\_dt}.json format. #### `--ignore-eos`[¶](#-ignore-eos "Permanent link") Set ignore\_eos flag when sending the benchmark request.Warning: ignore\_eos is not supported in deepspeed\_mii and tgi. Default: `False` #### `--self-timed`, `--no-self-timed`[¶](#-self-timed-no-self-timed "Permanent link") Use timing information from the traces instead of the configuration. This is useful when replaying traces faithfully based on their timestamps. When unset, defaults to False, except for --dataset-name=timed\_trace where it defaults to True. Use --no-self-timed to force off. When off, user defined generation rates are used and in trace timing info is ignored. #### `--percentile-metrics`[¶](#-percentile-metrics "Permanent link") Comma-separated list of selected metrics to report percentiles. This argument specifies the metrics to report percentiles. Allowed metric names are "ttft", "tpot", "itl", "e2el". If not specified, defaults to "ttft,tpot,itl" for generative models and "e2el" for pooling models. #### `--metric-percentiles`[¶](#-metric-percentiles "Permanent link") Comma-separated list of percentiles for selected metrics. To report 25-th, 50-th, and 75-th percentiles, use "25,50,75". Default value is "99".Use "--percentile-metrics" to select metrics. Default: `99` #### `--goodput`[¶](#-goodput "Permanent link") Specify service level objectives for goodput as "KEY:VALUE" pairs, where the key is a metric name, and the value is in milliseconds. Multiple "KEY:VALUE" pairs can be provided, separated by spaces. Allowed request level metric names are "ttft", "tpot", "e2el". For more context on the definition of goodput, refer to DistServe paper: https://arxiv.org/pdf/2401.09670 and the blog: https://hao-ai-lab.github.io/blogs/distserve #### `--request-id-prefix`[¶](#-request-id-prefix "Permanent link") Specify the prefix of request id. Default: `bench-4cc2d129-` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name used in the API. If not specified, the model name will be the same as the `--model` argument. #### `--lora-modules`[¶](#-lora-modules "Permanent link") A subset of LoRA module names passed in when launching the server. For each request, the script chooses a LoRA module at random by default. Use --lora-assignment to control selection strategy. #### `--lora-assignment`[¶](#-lora-assignment "Permanent link") Possible choices: `random`, `round-robin` Strategy for assigning LoRA modules to requests. 'random' (default) selects a LoRA at random for each request. 'round-robin' cycles through LoRA modules deterministically. Default: `random` #### `--ramp-up-strategy`[¶](#-ramp-up-strategy "Permanent link") Possible choices: `linear`, `exponential` The ramp-up strategy. This would be used to ramp up the request rate from initial RPS to final RPS rate (specified by --ramp-up-start-rps and --ramp-up-end-rps.) over the duration of the benchmark. #### `--ramp-up-start-rps`[¶](#-ramp-up-start-rps "Permanent link") The starting request rate for ramp-up (RPS). Needs to be specified when --ramp-up-strategy is used. #### `--ramp-up-end-rps`[¶](#-ramp-up-end-rps "Permanent link") The ending request rate for ramp-up (RPS). Needs to be specified when --ramp-up-strategy is used. #### `--ready-check-timeout-sec`[¶](#-ready-check-timeout-sec "Permanent link") Maximum time to wait for the endpoint to become ready in seconds. Ready check will be skipped by default. Default: `0` #### `--chat-template-kwargs`[¶](#-chat-template-kwargs "Permanent link") A JSON string of kwargs forwarded to the tokenizer's apply\_chat\_template when a dataset renders prompts client-side (e.g. custom / speed\_bench). Example: '{"thinking": true}' to enable reasoning models. #### `--extra-body`[¶](#-extra-body "Permanent link") A JSON string representing extra body parameters to include in each request.Example: '{"chat\_template\_kwargs":{"enable\_thinking":false}}' #### `--skip-tokenizer-init`[¶](#-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer Default: `False` #### `--insecure`[¶](#-insecure "Permanent link") Disable SSL certificate verification. Use this option when connecting to servers with self-signed certificates. Default: `False` #### `--plot-timeline`[¶](#-plot-timeline "Permanent link") Generate an HTML timeline plot showing request execution. The plot will be saved alongside the results JSON file. Default: `False` #### `--timeline-itl-thresholds`[¶](#-timeline-itl-thresholds "Permanent link") ITL thresholds in milliseconds for timeline plot coloring. Specify two comma-separated values to categorize inter-token latencies into three groups: below first threshold (green), between thresholds (orange), and above second threshold (red). Default: `25,50` #### `--plot-dataset-stats`[¶](#-plot-dataset-stats "Permanent link") Generate a matplotlib figure with dataset statistics showing prompt tokens, output tokens, and combined token distributions. Default: `False` ### custom dataset options[¶](#custom-dataset-options "Permanent link") #### `--custom-output-len`[¶](#-custom-output-len "Permanent link") Number of output tokens per request. Unless it is set to -1, the value overrides potential output length loaded from the dataset. It is used only for custom dataset. Default: `256` #### `--custom-ensure-client-side-data`[¶](#-custom-ensure-client-side-data "Permanent link") Ensure custom dataset media is sent as client-side data instead of references. For custom\_image datasets, this loads local and HTTP(S) images on the benchmark client and encodes them as base64 data URLs. Existing data:image URLs are kept unchanged. Default: `False` ### spec bench dataset options[¶](#spec-bench-dataset-options "Permanent link") #### `--spec-bench-output-len`[¶](#-spec-bench-output-len "Permanent link") Num of output tokens per request, used only for spec bench dataset. Default: `256` #### `--spec-bench-category`[¶](#-spec-bench-category "Permanent link") Category for spec bench dataset. If None, use all categories. ### sonnet dataset options[¶](#sonnet-dataset-options "Permanent link") #### `--sonnet-input-len`[¶](#-sonnet-input-len "Permanent link") Number of input tokens per request, used only for sonnet dataset. Default: `550` #### `--sonnet-output-len`[¶](#-sonnet-output-len "Permanent link") Number of output tokens per request, used only for sonnet dataset. Default: `150` #### `--sonnet-prefix-len`[¶](#-sonnet-prefix-len "Permanent link") Number of prefix tokens per request, used only for sonnet dataset. Default: `200` ### sharegpt dataset options[¶](#sharegpt-dataset-options "Permanent link") #### `--sharegpt-output-len`[¶](#-sharegpt-output-len "Permanent link") Output length for each request. Overrides the output length from the ShareGPT dataset. ### timed-trace dataset options[¶](#timed-trace-dataset-options "Permanent link") #### `--timed-trace-chunk-hash-size`[¶](#-timed-trace-chunk-hash-size "Permanent link") Each hash tokens, if present, represent how many token hashes. For example in the Moonshot traces it is 512, while the Qwen/Alibaba has 16. Default: `16` #### `--timed-trace-sec-multiplier`[¶](#-timed-trace-sec-multiplier "Permanent link") What multiplier to use when converting timestamps to seconds. We will multiply timestamps by this. For exampleif the timestamps are in milliseconds, then pass 0.001.If they are already in seconds, then the default 1 is sufficient. Default: `1` #### `--timed-trace-label-timestamp`[¶](#-timed-trace-label-timestamp "Permanent link") What json label to use to index the timestamp in the trace. Default: `timestamp` #### `--timed-trace-label-input-length`[¶](#-timed-trace-label-input-length "Permanent link") What json label to use to index the input length field in the trace. Default: `input_length` #### `--timed-trace-label-output-length`[¶](#-timed-trace-label-output-length "Permanent link") What json label to use to index the output length field in the trace. Default: `output_length` #### `--timed-trace-label-hash-ids`[¶](#-timed-trace-label-hash-ids "Permanent link") What json label to use to index the hash ids for the input prompts. Default: `hash_ids` ### blazedit dataset options[¶](#blazedit-dataset-options "Permanent link") #### `--blazedit-min-distance`[¶](#-blazedit-min-distance "Permanent link") Minimum distance for blazedit dataset. Min: 0, Max: 1.0 Default: `0.0` #### `--blazedit-max-distance`[¶](#-blazedit-max-distance "Permanent link") Maximum distance for blazedit dataset. Min: 0, Max: 1.0 Default: `1.0` ### asr dataset options[¶](#asr-dataset-options "Permanent link") #### `--asr-max-audio-len-sec`[¶](#-asr-max-audio-len-sec "Permanent link") Maximum audio length in seconds for ASR dataset. Default: `inf` #### `--asr-min-audio-len-sec`[¶](#-asr-min-audio-len-sec "Permanent link") Minimum audio length in seconds for ASR dataset. Default: `0.0` ### random dataset options[¶](#random-dataset-options "Permanent link") #### `--random-input-len`[¶](#-random-input-len "Permanent link") Number of input tokens per request, used only for random sampling. Default: `1024` #### `--random-output-len`[¶](#-random-output-len "Permanent link") Number of output tokens per request, used only for random sampling. Default: `128` #### `--random-range-ratio`[¶](#-random-range-ratio "Permanent link") Range ratio for sampling input/output length, used only for random sampling. A single float applies to both ISL and OSL. A JSON dict like '{"input": 0.3, "output": 0.5}' sets them independently. Values must be in \[0, 1). Default: `0.0` #### `--random-prefix-len`[¶](#-random-prefix-len "Permanent link") Number of fixed prefix tokens before the random context in a request. The total input length is the sum of `random-prefix-len` and a random context length sampled from \[input\_len \* (1 - range\_ratio), input\_len \* (1 + range\_ratio)\]. Default: `0` #### `--random-batch-size`[¶](#-random-batch-size "Permanent link") Batch size for random sampling. Only used for embeddings benchmark. Default: `1` #### `--no-reranker`[¶](#-no-reranker "Permanent link") Whether the model supports reranking natively. Only used for reranker benchmark. Default: `False` ### random multimodal dataset options extended from random dataset[¶](#random-multimodal-dataset-options-extended-from-random-dataset "Permanent link") #### `--random-mm-base-items-per-request`[¶](#-random-mm-base-items-per-request "Permanent link") Base number of multimodal items per request for random-mm. Actual per-request count is sampled around this base using --random-mm-num-mm-items-range-ratio. Default: `1` #### `--random-mm-num-mm-items-range-ratio`[¶](#-random-mm-num-mm-items-range-ratio "Permanent link") Range ratio r in \[0, 1\] for sampling items per request. We sample uniformly from the closed integer range \[floor(n_(1-r)), ceil(n_(1+r))\] where n is the base items per request. r=0 keeps it fixed; r=1 allows 0 items. The maximum is clamped to the sum of per-modality limits from --random-mm-limit-mm-per-prompt. An error is raised if the computed min exceeds the max. Default: `0.0` #### `--random-mm-limit-mm-per-prompt`[¶](#-random-mm-limit-mm-per-prompt "Permanent link") Per-modality hard caps for items attached per request, e.g. '{"image": 3, "video": 0}'. The sampled per-request item count is clamped to the sum of these limits. When a modality reaches its cap, its buckets are excluded and probabilities are renormalized.OBS.: Only image sampling is supported for now. Default: `{'image': 255, 'video': 1}` #### `--random-mm-bucket-config`[¶](#-random-mm-bucket-config "Permanent link") The bucket config is a dictionary mapping a multimodal itemsampling configuration to a probability.Currently allows for 2 modalities: images and videos. An bucket key is a tuple of (height, width, num\_frames)The value is the probability of sampling that specific item. Example: --random-mm-bucket-config {(256, 256, 1): 0.5, (720, 1280, 1): 0.4, (720, 1280, 16): 0.10} First item: images with resolution 256x256 w.p. 0.5Second item: images with resolution 720x1280 w.p. 0.4 Third item: videos with resolution 720x1280 and 16 frames w.p. 0.1OBS.: If the probabilities do not sum to 1, they are normalized.OBS bis.: Only image sampling is supported for now. Default: `{(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}` ### hf dataset options[¶](#hf-dataset-options "Permanent link") #### `--hf-subset`[¶](#-hf-subset "Permanent link") Subset of the HF dataset. #### `--hf-split`[¶](#-hf-split "Permanent link") Split of the HF dataset. #### `--hf-name`[¶](#-hf-name "Permanent link") Name of the dataset on HuggingFace (e.g., 'lmarena-ai/VisionArena-Chat'). Specify this if your dataset-path is a local path. #### `--hf-output-len`[¶](#-hf-output-len "Permanent link") Output length for each request. Overrides the output lengths from the sampled HF dataset. ### prefix repetition dataset options[¶](#prefix-repetition-dataset-options "Permanent link") #### `--prefix-repetition-prefix-len`[¶](#-prefix-repetition-prefix-len "Permanent link") Number of prefix tokens per request, used only for prefix repetition dataset. Default: `256` #### `--prefix-repetition-suffix-len`[¶](#-prefix-repetition-suffix-len "Permanent link") Number of suffix tokens per request, used only for prefix repetition dataset. Total input length is prefix\_len + suffix\_len. Default: `256` #### `--prefix-repetition-num-prefixes`[¶](#-prefix-repetition-num-prefixes "Permanent link") Number of prefixes to generate, used only for prefix repetition dataset. Prompts per prefix is num\_requests // num\_prefixes. Default: `10` #### `--prefix-repetition-output-len`[¶](#-prefix-repetition-output-len "Permanent link") Number of output tokens per request, used only for prefix repetition dataset. Default: `128` ### speed bench dataset options[¶](#speed-bench-dataset-options "Permanent link") SPEED-Bench dataset: https://huggingface.co/datasets/nvidia/SPEED-Bench `` Download the dataset using: `curl -LsSf https://raw.githubusercontent.com/NVIDIA-NeMo/Skills/refs/heads/main/nemo_skills/dataset/speed-bench/prepare.py | python3 -` `` #### `--speed-bench-dataset-subset`[¶](#-speed-bench-dataset-subset "Permanent link") Possible choices: `throughput_2k`, `throughput_32k`, `qualitative`, `throughput_1k`, `throughput_16k`, `throughput_8k` Subset of the SPEED-Bench dataset. Default: `qualitative` #### `--speed-bench-output-len`[¶](#-speed-bench-output-len "Permanent link") Num of output tokens per request, used only for speed bench dataset. Default: `4096` #### `--speed-bench-category`[¶](#-speed-bench-category "Permanent link") Category for speed bench dataset. If None, use all categories. ### sampling parameters[¶](#sampling-parameters "Permanent link") #### `--top-p`[¶](#-top-p "Permanent link") Top-p sampling parameter. Only has effect on openai-compatible backends. #### `--top-k`[¶](#-top-k "Permanent link") Top-k sampling parameter. Only has effect on openai-compatible backends. #### `--min-p`[¶](#-min-p "Permanent link") Min-p sampling parameter. Only has effect on openai-compatible backends. #### `--temperature`[¶](#-temperature "Permanent link") Temperature sampling parameter. Only has effect on openai-compatible backends. #### `--frequency-penalty`[¶](#-frequency-penalty "Permanent link") Frequency penalty sampling parameter. Only has effect on openai-compatible backends. #### `--presence-penalty`[¶](#-presence-penalty "Permanent link") Presence penalty sampling parameter. Only has effect on openai-compatible backends. #### `--repetition-penalty`[¶](#-repetition-penalty "Permanent link") Repetition penalty sampling parameter. Only has effect on openai-compatible backends. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/throughput.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--backend`[¶](#-backend "Permanent link") Possible choices: `vllm`, `hf`, `mii`, `vllm-chat` Default: `vllm` #### `--dataset-name`[¶](#-dataset-name "Permanent link") Possible choices: `sharegpt`, `random`, `sonnet`, `burstgpt`, `hf`, `prefix_repetition`, `random-mm`, `random-rerank` Name of the dataset to benchmark on. Default: `sharegpt` #### `--dataset`[¶](#-dataset "Permanent link") Path to the ShareGPT dataset, will be deprecated in the next release. The dataset is expected to be a json in form of list\[dict\[..., conversations: list\[dict\[..., value: \]\]\]\] #### `--dataset-path`[¶](#-dataset-path "Permanent link") Path to the dataset #### `--input-len`[¶](#-input-len "Permanent link") Input prompt length for each request #### `--output-len`[¶](#-output-len "Permanent link") Output length for each request. Overrides the output length from the dataset. #### `--n`[¶](#-n "Permanent link") Number of generated sequences per prompt. Default: `1` #### `--num-prompts`[¶](#-num-prompts "Permanent link") Number of prompts to process. Default: `1000` #### `--num-warmups`[¶](#-num-warmups "Permanent link") Number of warmup prompts to process before the timed benchmark. Default: `0` #### `--hf-max-batch-size`[¶](#-hf-max-batch-size "Permanent link") Maximum batch size for HF backend. #### `--hf-enable-torch-compile`[¶](#-hf-enable-torch-compile "Permanent link") Enable Torch compile for HF backend. Default: `False` #### `--output-json`[¶](#-output-json "Permanent link") Path to save the throughput results in JSON format. #### `--async-engine`[¶](#-async-engine "Permanent link") Use vLLM async engine rather than LLM class. Default: `False` #### `--prequeue-requests`[¶](#-prequeue-requests "Permanent link") For the vLLM backends, enqueue all requests before allowing the scheduler to process them. This can improve benchmark reproducibility by removing overlap between request rendering and engine scheduling, but may reduce measured throughput. Request rendering is typically fast relative to scheduling and processing; the intended use case of this flag is multimodal benchmarks with time-consuming image rendering. Default: `False` #### `--disable-detokenize`[¶](#-disable-detokenize "Permanent link") Do not detokenize the response (i.e. do not include detokenization time in the measurement) Default: `False` #### `--lora-path`[¶](#-lora-path "Permanent link") Path to the lora adapters to use. This can be an absolute path, a relative path, or a Hugging Face model identifier. #### `--lora-assignment`[¶](#-lora-assignment "Permanent link") Possible choices: `random`, `round-robin` Strategy for assigning LoRA adapters to requests. 'random' (default) selects a LoRA at random for each request. 'round-robin' cycles through LoRAs deterministically. Default: `random` #### `--prefix-len`[¶](#-prefix-len "Permanent link") Number of fixed prefix tokens before the random context in a request (default: 0). Default: `0` #### `--hf-subset`[¶](#-hf-subset "Permanent link") Subset of the HF dataset. #### `--hf-split`[¶](#-hf-split "Permanent link") Split of the HF dataset. #### `--hf-name`[¶](#-hf-name "Permanent link") Name of the dataset on HuggingFace (e.g., 'lmms-lab/LLaVA-OneVision-Data'). Specify this when --dataset-path is a local filesystem path so the benchmark can identify the correct dataset class. #### `--profile`[¶](#-profile "Permanent link") Use vLLM Profiling. --profiler-config must be provided on the server. Default: `False` #### `--prefix-repetition-prefix-len`[¶](#-prefix-repetition-prefix-len "Permanent link") Number of prefix tokens per request, used only for prefix repetition dataset. #### `--prefix-repetition-suffix-len`[¶](#-prefix-repetition-suffix-len "Permanent link") Number of suffix tokens per request, used only for prefix repetition dataset. Total input length is prefix\_len + suffix\_len. #### `--prefix-repetition-num-prefixes`[¶](#-prefix-repetition-num-prefixes "Permanent link") Number of prefixes to generate, used only for prefix repetition dataset. Prompts per prefix is num\_requests // num\_prefixes. #### `--prefix-repetition-output-len`[¶](#-prefix-repetition-output-len "Permanent link") Number of output tokens per request, used only for prefix repetition dataset. #### `--random-input-len`[¶](#-random-input-len "Permanent link") Number of input tokens per request, used only for random sampling. Default: `1024` #### `--random-output-len`[¶](#-random-output-len "Permanent link") Number of output tokens per request, used only for random sampling. Default: `128` #### `--random-range-ratio`[¶](#-random-range-ratio "Permanent link") Range ratio for sampling input/output length, used only for random sampling. A single float applies to both ISL and OSL. A JSON dict like '{"input": 0.3, "output": 0.5}' sets them independently. Values must be in \[0, 1). Default: `0.0` #### `--random-prefix-len`[¶](#-random-prefix-len "Permanent link") Number of fixed prefix tokens before the random context in a request. The total input length is the sum of `random-prefix-len` and a random context length sampled from \[input\_len \* (1 - range\_ratio), input\_len \* (1 + range\_ratio)\]. Default: `0` #### `--random-batch-size`[¶](#-random-batch-size "Permanent link") Batch size for random sampling. Only used for embeddings benchmark. Default: `1` #### `--no-reranker`[¶](#-no-reranker "Permanent link") Whether the model supports reranking natively. Only used for reranker benchmark. Default: `False` #### `--random-mm-base-items-per-request`[¶](#-random-mm-base-items-per-request "Permanent link") Base number of multimodal items per request for random-mm. Actual per-request count is sampled around this base using --random-mm-num-mm-items-range-ratio. Default: `1` #### `--random-mm-num-mm-items-range-ratio`[¶](#-random-mm-num-mm-items-range-ratio "Permanent link") Range ratio r in \[0, 1\] for sampling items per request. We sample uniformly from the closed integer range \[floor(n_(1-r)), ceil(n_(1+r))\] where n is the base items per request. r=0 keeps it fixed; r=1 allows 0 items. The maximum is clamped to the sum of per-modality limits from --random-mm-limit-mm-per-prompt. An error is raised if the computed min exceeds the max. Default: `0.0` #### `--random-mm-limit-mm-per-prompt`[¶](#-random-mm-limit-mm-per-prompt "Permanent link") Per-modality hard caps for items attached per request, e.g. '{"image": 3, "video": 0}'. The sampled per-request item count is clamped to the sum of these limits. When a modality reaches its cap, its buckets are excluded and probabilities are renormalized.OBS.: Only image sampling is supported for now. Default: `{'image': 255, 'video': 1}` #### `--random-mm-bucket-config`[¶](#-random-mm-bucket-config "Permanent link") The bucket config is a dictionary mapping a multimodal itemsampling configuration to a probability.Currently allows for 2 modalities: images and videos. An bucket key is a tuple of (height, width, num\_frames)The value is the probability of sampling that specific item. Example: --random-mm-bucket-config {(256, 256, 1): 0.5, (720, 1280, 1): 0.4, (720, 1280, 16): 0.10} First item: images with resolution 256x256 w.p. 0.5Second item: images with resolution 720x1280 w.p. 0.4 Third item: videos with resolution 720x1280 and 16 frames w.p. 0.1OBS.: If the probabilities do not sum to 1, they are normalized.OBS bis.: Only image sampling is supported for now. Default: `{(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}` #### `--asr-min-audio-len-sec`[¶](#-asr-min-audio-len-sec "Permanent link") Minimum audio duration in seconds for ASR dataset filtering. Default: `0.0` #### `--asr-max-audio-len-sec`[¶](#-asr-max-audio-len-sec "Permanent link") Maximum audio duration in seconds for ASR dataset filtering. Default: `inf` #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. #### `--enable-log-requests`, `--no-enable-log-requests`[¶](#-enable-log-requests-no-enable-log-requests "Permanent link") Enable logging request information, dependent on log level: - INFO: Request ID, parameters and LoRA request. - DEBUG: Prompt inputs (e.g: text, token IDs). You can set the minimum log level via `VLLM_LOGGING_LEVEL`. Default: `False` ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [CLI Reference](https://docs.vllm.ai/en/latest/) 3. [vllm bench](https://docs.vllm.ai/en/latest/cli/latency/) [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/sweep/plot.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--fig-dir`[¶](#-fig-dir "Permanent link") The directory to save the figures, relative to `OUTPUT_DIR`. By default, the same directory is used. Default: `""` #### `--fig-by`[¶](#-fig-by "Permanent link") A comma-separated list of variables, such that a separate figure is created for each combination of these variables. Default: `""` #### `--row-by`[¶](#-row-by "Permanent link") A comma-separated list of variables, such that a separate row is created for each combination of these variables. Default: `""` #### `--col-by`[¶](#-col-by "Permanent link") A comma-separated list of variables, such that a separate column is created for each combination of these variables. Default: `""` #### `--curve-by`[¶](#-curve-by "Permanent link") A comma-separated list of variables, such that a separate curve is created for each combination of these variables. #### `--var-x`[¶](#-var-x "Permanent link") The variable for the x-axis. Default: `total_token_throughput` #### `--var-y`[¶](#-var-y "Permanent link") The variable for the y-axis Default: `median_ttft_ms` #### `--filter-by`[¶](#-filter-by "Permanent link") A comma-separated list of statements indicating values to filter by. This is useful to remove outliers. Example: `max_concurrency<1000,max_num_batched_tokens<=4096` means plot only the points where `max_concurrency` is less than 1000 and `max_num_batched_tokens` is no greater than 4096. Default: `""` #### `--bin-by`[¶](#-bin-by "Permanent link") A comma-separated list of statements indicating values to bin by. This is useful to avoid plotting points that are too close together. Example: `request_throughput%%1` means use a bin size of 1 for the `request_throughput` variable. Default: `""` #### `--scale-x`[¶](#-scale-x "Permanent link") The scale to use for the x-axis. Currently only accepts string values such as 'log' and 'sqrt'. See also: https://seaborn.pydata.org/generated/seaborn.objects.Plot.scale.html #### `--scale-y`[¶](#-scale-y "Permanent link") The scale to use for the y-axis. Currently only accepts string values such as 'log' and 'sqrt'. See also: https://seaborn.pydata.org/generated/seaborn.objects.Plot.scale.html #### `--fig-name`[¶](#-fig-name "Permanent link") Name prefix for the output figure file. Group data is always appended when present. Default: 'FIGURE'. Example: --fig-name my\_performance\_plot Default: `FIGURE` #### `--no-error-bars`[¶](#-no-error-bars "Permanent link") If set, disables error bars on the plot. By default, error bars are shown. Default: `False` #### `--fig-height`[¶](#-fig-height "Permanent link") Height of each subplot in inches. Default: 6.4 Default: `6.4` #### `--fig-dpi`[¶](#-fig-dpi "Permanent link") Resolution of the output figure in dots per inch. Default: 300 Default: `300` #### `--dry-run`[¶](#-dry-run "Permanent link") If set, prints the information about each figure to plot, then exits without drawing them. Default: `False` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [CLI Reference](https://docs.vllm.ai/en/latest/) 3. [vllm bench](https://docs.vllm.ai/en/latest/cli/latency/) [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/sweep/plot_pareto.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--user-count-var`[¶](#-user-count-var "Permanent link") Result key that stores concurrent user count. Falls back to max\_concurrent\_requests if missing. Default: `max_concurrency` #### `--gpu-count-var`[¶](#-gpu-count-var "Permanent link") Result key that stores GPU count. If not provided, falls back to num\_gpus/gpu\_count or tensor\_parallel\_size \* pipeline\_parallel\_size. #### `--label-by`[¶](#-label-by "Permanent link") Comma-separated list of fields to annotate on Pareto frontier points. Default: `max_concurrency,gpu_count` #### `--dry-run`[¶](#-dry-run "Permanent link") If set, prints the figures to plot without drawing them. Default: `False` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [CLI Reference](https://docs.vllm.ai/en/latest/) 3. [vllm bench](https://docs.vllm.ai/en/latest/cli/latency/) [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/sweep/serve.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--serve-cmd`[¶](#-serve-cmd "Permanent link") The command used to run the server: `vllm serve ...` #### `--bench-cmd`[¶](#-bench-cmd "Permanent link") The command used to run the benchmark: `vllm bench serve ...` #### `--after-bench-cmd`[¶](#-after-bench-cmd "Permanent link") After a benchmark run is complete, invoke this command instead of the default `ServerWrapper.clear_cache()`. #### `--show-stdout`[¶](#-show-stdout "Permanent link") If set, logs the standard output of subcommands. Useful for debugging but can be quite spammy. Default: `False` #### `--server-ready-timeout`[¶](#-server-ready-timeout "Permanent link") Timeout in seconds to wait for the server to become ready. Default: `300` #### `--serve-params`[¶](#-serve-params "Permanent link") Path to JSON file containing parameter combinations for the `vllm serve` command. Can be either a list of dicts or a dict where keys are benchmark names. If both `serve_params` and `bench_params` are given, this script will iterate over their Cartesian product. #### `--link-vars`[¶](#-link-vars "Permanent link") Comma-separated list of linked variables between serve and bench, e.g. max\_num\_seqs=max\_concurrency,max\_model\_len=random\_input\_len Default: `""` #### `--bench-params`[¶](#-bench-params "Permanent link") Path to JSON file containing parameter combinations for the `vllm bench serve` command. Can be either a list of dicts or a dict where keys are benchmark names. If both `serve_params` and `bench_params` are given, this script will iterate over their Cartesian product. #### `-o`, `--output-dir`[¶](#-o-output-dir "Permanent link") The main directory to which results are written. Default: `results` #### `-e`, `--experiment-name`[¶](#-e-experiment-name "Permanent link") The name of this experiment (defaults to current timestamp). Results will be stored under `output_dir/experiment_name`. #### `--num-runs`[¶](#-num-runs "Permanent link") Number of runs per parameter combination. Default: `3` #### `--dry-run`[¶](#-dry-run "Permanent link") If set, prints the commands to run, then exits without executing them. Default: `False` #### `--resume`[¶](#-resume "Permanent link") Resume a previous execution of this script, i.e., only run parameter combinations for which there are still no output files under `output_dir/experiment_name`. Default: `False` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [CLI Reference](https://docs.vllm.ai/en/latest/) 3. [vllm bench](https://docs.vllm.ai/en/latest/cli/latency/) [](https://github.com/vllm-project/vllm/edit/main/docs/cli/bench/sweep/serve_workload.md "Edit this page") ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--serve-cmd`[¶](#-serve-cmd "Permanent link") The command used to run the server: `vllm serve ...` #### `--bench-cmd`[¶](#-bench-cmd "Permanent link") The command used to run the benchmark: `vllm bench serve ...` #### `--after-bench-cmd`[¶](#-after-bench-cmd "Permanent link") After a benchmark run is complete, invoke this command instead of the default `ServerWrapper.clear_cache()`. #### `--show-stdout`[¶](#-show-stdout "Permanent link") If set, logs the standard output of subcommands. Useful for debugging but can be quite spammy. Default: `False` #### `--server-ready-timeout`[¶](#-server-ready-timeout "Permanent link") Timeout in seconds to wait for the server to become ready. Default: `300` #### `--serve-params`[¶](#-serve-params "Permanent link") Path to JSON file containing parameter combinations for the `vllm serve` command. Can be either a list of dicts or a dict where keys are benchmark names. If both `serve_params` and `bench_params` are given, this script will iterate over their Cartesian product. #### `--link-vars`[¶](#-link-vars "Permanent link") Comma-separated list of linked variables between serve and bench, e.g. max\_num\_seqs=max\_concurrency,max\_model\_len=random\_input\_len Default: `""` #### `--bench-params`[¶](#-bench-params "Permanent link") Path to JSON file containing parameter combinations for the `vllm bench serve` command. Can be either a list of dicts or a dict where keys are benchmark names. If both `serve_params` and `bench_params` are given, this script will iterate over their Cartesian product. #### `-o`, `--output-dir`[¶](#-o-output-dir "Permanent link") The main directory to which results are written. Default: `results` #### `-e`, `--experiment-name`[¶](#-e-experiment-name "Permanent link") The name of this experiment (defaults to current timestamp). Results will be stored under `output_dir/experiment_name`. #### `--num-runs`[¶](#-num-runs "Permanent link") Number of runs per parameter combination. Default: `3` #### `--dry-run`[¶](#-dry-run "Permanent link") If set, prints the commands to run, then exits without executing them. Default: `False` #### `--resume`[¶](#-resume "Permanent link") Resume a previous execution of this script, i.e., only run parameter combinations for which there are still no output files under `output_dir/experiment_name`. Default: `False` ### workload options[¶](#workload-options "Permanent link") #### `--workload-var`[¶](#-workload-var "Permanent link") Possible choices: `request_rate`, `max_concurrency` The variable to adjust in each iteration. Default: `request_rate` #### `--workload-iters`[¶](#-workload-iters "Permanent link") Number of workload levels to explore. This includes the first two iterations used to interpolate the value of `workload_var` for remaining iterations. Default: `10` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/cli/launch/render.md "Edit this page") ## Overview[¶](#overview "Permanent link") `vllm launch render` starts a GPU-less rendering server for preprocessing and postprocessing only. `[](#__codelineno-0-1)vllm launch render meta-llama/Llama-3.2-1B-Instruct --port 8100` This command reuses the standard serving parser, so model, frontend, networking, and related CLI options follow the same conventions as [`vllm serve`](https://docs.vllm.ai/en/latest/serve/). ## JSON CLI Arguments[¶](#json-cli-arguments "Permanent link") When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## Arguments[¶](#arguments "Permanent link") #### `--headless`[¶](#-headless "Permanent link") Run in headless mode. See multi-node data parallel documentation for more details. Default: `False` #### `--api-server-count`, `-asc`[¶](#-api-server-count-asc "Permanent link") How many API server processes to run. Defaults to data\_parallel\_size if not specified. #### `--config`[¶](#-config "Permanent link") Read CLI options from a config file. Must be a YAML with the following options: https://docs.vllm.ai/en/latest/configuration/serve\_args.html #### `--grpc`[¶](#-grpc "Permanent link") Launch a gRPC server instead of the HTTP OpenAI-compatible server. Requires: pip install vllm\[grpc\]. Default: `False` #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. #### `--enable-log-requests`, `--no-enable-log-requests`[¶](#-enable-log-requests-no-enable-log-requests "Permanent link") Enable logging request information, dependent on log level: - INFO: Request ID, parameters and LoRA request. - DEBUG: Prompt inputs (e.g: text, token IDs). You can set the minimum log level via `VLLM_LOGGING_LEVEL`. Default: `False` ### Frontend[¶](#frontend "Permanent link") Arguments for the OpenAI-compatible frontend server. #### `--lora-modules`[¶](#-lora-modules "Permanent link") #### `--chat-template`[¶](#-chat-template "Permanent link") #### `--chat-template-content-format`[¶](#-chat-template-content-format "Permanent link") Possible choices: `auto`, `openai`, `string` Default: `auto` #### `--trust-request-chat-template`, `--no-trust-request-chat-template`[¶](#-trust-request-chat-template-no-trust-request-chat-template "Permanent link") Default: `False` #### `--default-chat-template-kwargs`[¶](#-default-chat-template-kwargs "Permanent link") : Should either be a valid JSON string or JSON keys passed individually. #### `--response-role`[¶](#-response-role "Permanent link") Default: `assistant` #### `--return-tokens-as-token-ids`, `--no-return-tokens-as-token-ids`[¶](#-return-tokens-as-token-ids-no-return-tokens-as-token-ids "Permanent link") Default: `False` #### `--enable-auto-tool-choice`, `--no-enable-auto-tool-choice`[¶](#-enable-auto-tool-choice-no-enable-auto-tool-choice "Permanent link") Default: `False` #### `--exclude-tools-when-tool-choice-none`, `--no-exclude-tools-when-tool-choice-none`[¶](#-exclude-tools-when-tool-choice-none-no-exclude-tools-when-tool-choice-none "Permanent link") Default: `False` #### `--tool-call-parser`[¶](#-tool-call-parser "Permanent link") #### `--tool-parser-plugin`[¶](#-tool-parser-plugin "Permanent link") Default: `""` #### `--tool-server`[¶](#-tool-server "Permanent link") #### `--log-config-file`[¶](#-log-config-file "Permanent link") #### `--max-log-len`[¶](#-max-log-len "Permanent link") #### `--enable-prompt-tokens-details`, `--no-enable-prompt-tokens-details`[¶](#-enable-prompt-tokens-details-no-enable-prompt-tokens-details "Permanent link") Default: `False` #### `--enable-server-load-tracking`, `--no-enable-server-load-tracking`[¶](#-enable-server-load-tracking-no-enable-server-load-tracking "Permanent link") Default: `False` #### `--enable-force-include-usage`, `--no-enable-force-include-usage`[¶](#-enable-force-include-usage-no-enable-force-include-usage "Permanent link") Default: `False` #### `--enable-tokenizer-info-endpoint`, `--no-enable-tokenizer-info-endpoint`[¶](#-enable-tokenizer-info-endpoint-no-enable-tokenizer-info-endpoint "Permanent link") Default: `False` #### `--enable-log-outputs`, `--no-enable-log-outputs`[¶](#-enable-log-outputs-no-enable-log-outputs "Permanent link") Default: `False` #### `--enable-log-deltas`, `--no-enable-log-deltas`[¶](#-enable-log-deltas-no-enable-log-deltas "Permanent link") Default: `True` #### `--log-error-stack`, `--no-log-error-stack`[¶](#-log-error-stack-no-log-error-stack "Permanent link") Default: `False` #### `--tokens-only`, `--no-tokens-only`[¶](#-tokens-only-no-tokens-only "Permanent link") Default: `False` #### `--fingerprint-mode`[¶](#-fingerprint-mode "Permanent link") Possible choices: `custom`, `full`, `hash`, `none` Default: `full` #### `--fingerprint-value`[¶](#-fingerprint-value "Permanent link") #### `--host`[¶](#-host "Permanent link") Host name. #### `--port`[¶](#-port "Permanent link") Port number. Default: `8000` #### `--data-parallel-supervisor-port`[¶](#-data-parallel-supervisor-port "Permanent link") HTTP port for aggregated health endpoints in multi-port external LB mode. Default: `9256` #### `--dp-supervisor-probe-interval-s`[¶](#-dp-supervisor-probe-interval-s "Permanent link") Seconds between aggregated health probes in multi-port external LB mode. Default: `5.0` #### `--dp-supervisor-probe-timeout-s`[¶](#-dp-supervisor-probe-timeout-s "Permanent link") Seconds to wait between retries when a child health probe fails with a connection error in multi-port external LB mode. Default: `5.0` #### `--dp-supervisor-probe-failure-threshold`[¶](#-dp-supervisor-probe-failure-threshold "Permanent link") Number of consecutive connection-error retries before a child health probe is declared failed in multi-port external LB mode. Default: `3` #### `--uds`[¶](#-uds "Permanent link") Unix domain socket path. If set, host and port arguments are ignored. #### `--uvicorn-log-level`[¶](#-uvicorn-log-level "Permanent link") Possible choices: `critical`, `debug`, `error`, `info`, `trace`, `warning` Log level for uvicorn. Default: `info` #### `--disable-uvicorn-access-log`, `--no-disable-uvicorn-access-log`[¶](#-disable-uvicorn-access-log-no-disable-uvicorn-access-log "Permanent link") Disable uvicorn access log. Default: `False` #### `--disable-access-log-for-endpoints`[¶](#-disable-access-log-for-endpoints "Permanent link") Comma-separated list of endpoint paths to exclude from uvicorn access logs. This is useful to reduce log noise from high-frequency endpoints like health checks. Example: "/health,/metrics,/ping". When set, access logs for requests to these paths will be suppressed while keeping logs for other endpoints. #### `--allow-credentials`, `--no-allow-credentials`[¶](#-allow-credentials-no-allow-credentials "Permanent link") Allow credentials. Default: `False` #### `--allowed-origins`[¶](#-allowed-origins "Permanent link") Allowed origins. Default: `['*']` #### `--allowed-methods`[¶](#-allowed-methods "Permanent link") Allowed methods. Default: `['*']` Allowed headers. Default: `['*']` #### `--api-key`[¶](#-api-key "Permanent link") If provided, the server will require one of these keys to be presented in the header. #### `--ssl-keyfile`[¶](#-ssl-keyfile "Permanent link") The file path to the SSL key file. #### `--ssl-certfile`[¶](#-ssl-certfile "Permanent link") The file path to the SSL cert file. #### `--ssl-ca-certs`[¶](#-ssl-ca-certs "Permanent link") The CA certificates file. #### `--enable-ssl-refresh`, `--no-enable-ssl-refresh`[¶](#-enable-ssl-refresh-no-enable-ssl-refresh "Permanent link") Refresh SSL Context when SSL certificate files change Default: `False` #### `--ssl-cert-reqs`[¶](#-ssl-cert-reqs "Permanent link") Whether client certificate is required (see stdlib ssl module's). Default: `0` #### `--ssl-ciphers`[¶](#-ssl-ciphers "Permanent link") SSL cipher suites for HTTPS (TLS 1.2 and below only). Example: 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305' #### `--root-path`[¶](#-root-path "Permanent link") FastAPI root\_path when app is behind a path based routing proxy. #### `--middleware`[¶](#-middleware "Permanent link") Additional ASGI middleware to apply to the app. We accept multiple --middleware arguments. The value should be an import path. If a function is provided, vLLM will add it to the server using `@app.middleware('http')`. If a class is provided, vLLM will add it to the server using `app.add_middleware()`. Default: `[]` If specified, API server will add X-Request-Id header to responses. Default: `False` #### `--disable-fastapi-docs`, `--no-disable-fastapi-docs`[¶](#-disable-fastapi-docs-no-disable-fastapi-docs "Permanent link") Disable FastAPI's OpenAPI schema, Swagger UI, and ReDoc endpoint. Default: `False` #### `--h11-max-incomplete-event-size`[¶](#-h11-max-incomplete-event-size "Permanent link") Maximum size (bytes) of an incomplete HTTP event (header or body) for h11 parser. Helps mitigate header abuse. Default: 4194304 (4 MB). Default: `4194304` Maximum number of HTTP headers allowed in a request for h11 parser. Helps mitigate header abuse. Default: 256. Default: `256` #### `--enable-offline-docs`, `--no-enable-offline-docs`[¶](#-enable-offline-docs-no-enable-offline-docs "Permanent link") Enable offline FastAPI documentation for air-gapped environments. Uses vendored static assets bundled with vLLM. Default: `False` #### `--enable-flash-late-interaction`, `--no-enable-flash-late-interaction`[¶](#-enable-flash-late-interaction-no-enable-flash-late-interaction "Permanent link") If set, run pooling score MaxSim on GPU in the API server process. Can significantly improve late-interaction scoring performance. Default: `True` ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. --- # page - [Home](https://docs.vllm.ai/en/) - [User Guide](https://docs.vllm.ai/en/usage/) - [Developer Guide](https://docs.vllm.ai/en/contributing/) - [Benchmarking](https://docs.vllm.ai/en/benchmarking/) - [API Reference](https://docs.vllm.ai/en/api/) - [CLI Reference](https://docs.vllm.ai/en/cli/) - [Community](https://docs.vllm.ai/en/latest/community/) 1. [Home](https://docs.vllm.ai/en/) 2. [Community](https://docs.vllm.ai/en/latest/community/) [](https://github.com/vllm-project/vllm/edit/main/docs/community/contact_us.md "Edit this page") - For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm/issues) - For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai/) - For coordinating contributions and development, please use [Slack](https://slack.vllm.ai/) - For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature - For collaborations and partnerships, please contact us at [\[email protected\]](https://docs.vllm.ai/cdn-cgi/l/email-protection#afccc0c3c3cecdc0ddcedbc6c0c1efd9c3c3c281cec6) --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [Community](https://docs.vllm.ai/en/latest/contact_us/) [](https://github.com/vllm-project/vllm/edit/main/docs/community/meetups.md "Edit this page") We host regular meetups around the world. We will share the project updates from the vLLM team and have guest speakers from the industry to share their experience and insights. Please visit [vllm.ai/events](https://vllm.ai/events) to learn more. --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [Community](https://docs.vllm.ai/en/latest/contact_us/) [](https://github.com/vllm-project/vllm/edit/main/docs/community/sponsors.md "Edit this page") vLLM is a community project. Our compute resources for development and testing are supported by the following organizations. Thank you for your support! Please visit [vllm.ai/#sponsors](https://vllm.ai/#sponsors) to learn more. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/configuration/README.md "Edit this page") This section lists the most common options for running vLLM. There are three main levels of configuration, from highest priority to lowest priority: - [Request parameters](https://docs.vllm.ai/en/serving/online_serving/openai_compatible_server/#completions-api) and [input arguments](https://docs.vllm.ai/en/api/#inference-parameters) - [Engine arguments](https://docs.vllm.ai/en/latest/engine_args/) - [Environment variables](https://docs.vllm.ai/en/latest/env_vars/) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/configuration/conserving_memory.md "Edit this page") Large models might cause your machine to run out of memory (OOM). Here are some options that help alleviate this problem. ## Tensor Parallelism (TP)[¶](#tensor-parallelism-tp "Permanent link") Tensor parallelism (`tensor_parallel_size` option) can be used to split the model across multiple GPUs. The following code splits the model across 2 GPUs. `[](#__codelineno-0-1)from vllm import LLM [](#__codelineno-0-2)[](#__codelineno-0-3)llm = LLM(model="ibm-granite/granite-3.1-8b-instruct", tensor_parallel_size=2)` Warning To ensure that vLLM initializes CUDA correctly, you should avoid calling related functions (e.g. [torch.accelerator.set\_device\_index](https://pytorch.org/docs/stable/generated/torch.accelerator.set_device_index.html#torch.accelerator.set_device_index)) before initializing vLLM. Otherwise, you may run into an error like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`. To control which devices are used, please instead set the `CUDA_VISIBLE_DEVICES` environment variable. Note With tensor parallelism enabled, each process will read the whole model and split it into chunks, which makes the disk reading time even longer (proportional to the size of tensor parallelism). You can convert the model checkpoint to a sharded checkpoint using [examples/features/sharded\_state/load\_sharded\_state\_offline.py](https://github.com/vllm-project/vllm/blob/main/examples/features/sharded_state/load_sharded_state_offline.py). The conversion process might take some time, but later you can load the sharded checkpoint much faster. The model loading time should remain constant regardless of the size of tensor parallelism. ## Quantization[¶](#quantization "Permanent link") Quantized models take less memory at the cost of lower precision. Statically quantized models can be downloaded from HF Hub (some popular ones are available at [Red Hat AI](https://huggingface.co/RedHatAI)) and used directly without extra configuration. Dynamic quantization is also supported via the `quantization` option -- see [here](https://docs.vllm.ai/en/features/quantization/) for more details. ## Context length and batch size[¶](#context-length-and-batch-size "Permanent link") You can further reduce memory usage by limiting the context length of the model (`max_model_len` option) and the maximum batch size (`max_num_seqs` option). `[](#__codelineno-1-1)from vllm import LLM [](#__codelineno-1-2)[](#__codelineno-1-3)llm = LLM(model="adept/fuyu-8b", max_model_len=2048, max_num_seqs=2)` ## Reduce CUDA Graphs[¶](#reduce-cuda-graphs "Permanent link") By default, we optimize model inference using CUDA graphs which take up extra memory in the GPU. You can adjust `compilation_config` to achieve a better balance between inference speed and memory usage: Code `[](#__codelineno-2-1)from vllm import LLM [](#__codelineno-2-2)from vllm.config import CompilationConfig, CompilationMode [](#__codelineno-2-3)[](#__codelineno-2-4)llm = LLM( [](#__codelineno-2-5) model="meta-llama/Llama-3.1-8B-Instruct", [](#__codelineno-2-6) compilation_config=CompilationConfig( [](#__codelineno-2-7) mode=CompilationMode.VLLM_COMPILE, [](#__codelineno-2-8) # By default, it goes up to max_num_seqs [](#__codelineno-2-9) cudagraph_capture_sizes=[1, 2, 4, 8, 16], [](#__codelineno-2-10) ), [](#__codelineno-2-11))` You can disable graph capturing completely via the `enforce_eager` flag: `[](#__codelineno-3-1)from vllm import LLM [](#__codelineno-3-2)[](#__codelineno-3-3)llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", enforce_eager=True)` ## Adjust cache size[¶](#adjust-cache-size "Permanent link") If you run out of CPU RAM, try the following options: - (Multi-modal models only) you can set the size of multi-modal cache by setting `mm_processor_cache_gb` engine argument (default 4 GiB). - (CPU backend only) you can set the size of KV cache using `VLLM_CPU_KVCACHE_SPACE` environment variable (default 4 GiB). ## Multi-modal input limits[¶](#multi-modal-input-limits "Permanent link") You can allow a smaller number of multi-modal items per prompt to reduce the memory footprint of the model: `[](#__codelineno-4-1)from vllm import LLM [](#__codelineno-4-2)[](#__codelineno-4-3)# Accept up to 3 images and 1 video per prompt [](#__codelineno-4-4)llm = LLM( [](#__codelineno-4-5) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-4-6) limit_mm_per_prompt={"image": 3, "video": 1}, [](#__codelineno-4-7))` You can go a step further and disable unused modalities completely by setting its limit to zero. For example, if your application only accepts image input, there is no need to allocate any memory for videos. `[](#__codelineno-5-1)from vllm import LLM [](#__codelineno-5-2)[](#__codelineno-5-3)# Accept any number of images but no videos [](#__codelineno-5-4)llm = LLM( [](#__codelineno-5-5) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-5-6) limit_mm_per_prompt={"video": 0}, [](#__codelineno-5-7))` You can even run a multi-modal model for text-only inference: `[](#__codelineno-6-1)from vllm import LLM [](#__codelineno-6-2)[](#__codelineno-6-3)# Don't accept images. Just text. [](#__codelineno-6-4)llm = LLM( [](#__codelineno-6-5) model="google/gemma-3-27b-it", [](#__codelineno-6-6) limit_mm_per_prompt={"image": 0}, [](#__codelineno-6-7))` ### Configurable options[¶](#configurable-options "Permanent link") `limit_mm_per_prompt` also accepts configurable options per modality. In the configurable form, you still specify `count`, and you may optionally provide size hints that control how vLLM profiles and reserves memory for your multi‑modal inputs. This helps you tune memory for the actual media you expect, instead of the model’s absolute maxima. Configurable options by modality: - `image`: `{"count": int, "width": int, "height": int}` - `video`: `{"count": int, "num_frames": int, "width": int, "height": int}` - `audio`: `{"count": int, "length": int}` Details could be found in [`ImageDummyOptions`](https://docs.vllm.ai/en/api/vllm/config/multimodal/#vllm.config.multimodal.ImageDummyOptions " ImageDummyOptions"), [`VideoDummyOptions`](https://docs.vllm.ai/en/api/vllm/config/multimodal/#vllm.config.multimodal.VideoDummyOptions " VideoDummyOptions"), and [`AudioDummyOptions`](https://docs.vllm.ai/en/api/vllm/config/multimodal/#vllm.config.multimodal.AudioDummyOptions " AudioDummyOptions"). Examples: `[](#__codelineno-7-1)from vllm import LLM [](#__codelineno-7-2)[](#__codelineno-7-3)# Up to 5 images per prompt, profile with 512x512. [](#__codelineno-7-4)# Up to 1 video per prompt, profile with 32 frames at 640x640. [](#__codelineno-7-5)llm = LLM( [](#__codelineno-7-6) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-7-7) limit_mm_per_prompt={ [](#__codelineno-7-8) "image": {"count": 5, "width": 512, "height": 512}, [](#__codelineno-7-9) "video": {"count": 1, "num_frames": 32, "width": 640, "height": 640}, [](#__codelineno-7-10) }, [](#__codelineno-7-11))` For backward compatibility, passing an integer works as before and is interpreted as `{"count": }`. For example: - `limit_mm_per_prompt={"image": 5}` is equivalent to `limit_mm_per_prompt={"image": {"count": 5}}` - You can mix formats: `limit_mm_per_prompt={"image": 5, "video": {"count": 1, "num_frames": 32, "width": 640, "height": 640}}` Note - The size hints affect memory profiling only. They shape the dummy inputs used to compute reserved activation sizes. They do not change how inputs are actually processed at inference time. - If a hint exceeds what the model can accept, vLLM clamps it to the model's effective maximum and may log a warning. Warning These size hints currently only affect activation memory profiling. Encoder cache size is determined by the actual inputs at runtime and is not limited by these hints. ## Multi-modal processor arguments[¶](#multi-modal-processor-arguments "Permanent link") For certain models, you can adjust the multi-modal processor arguments to reduce the size of the processed multi-modal inputs, which in turn saves memory. Here are some examples: `[](#__codelineno-8-1)from vllm import LLM [](#__codelineno-8-2)[](#__codelineno-8-3)# Available for Qwen2-VL series models [](#__codelineno-8-4)llm = LLM( [](#__codelineno-8-5) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-8-6) mm_processor_kwargs={"max_pixels": 768 * 768}, # Default is 1280 * 28 * 28 [](#__codelineno-8-7)) [](#__codelineno-8-8)[](#__codelineno-8-9)# Available for InternVL series models [](#__codelineno-8-10)llm = LLM( [](#__codelineno-8-11) model="OpenGVLab/InternVL2-2B", [](#__codelineno-8-12) mm_processor_kwargs={"max_dynamic_patch": 4}, # Default is 12 [](#__codelineno-8-13))` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/configuration/engine_args.md "Edit this page") Engine arguments control the behavior of the vLLM engine. - For [offline inference](https://docs.vllm.ai/en/serving/offline_inference/), they are part of the arguments to [LLM](https://docs.vllm.ai/en/api/vllm/#vllm.LLM " LLM") class. - For [online serving](https://docs.vllm.ai/en/serving/online_serving/), they are part of the arguments to `vllm serve`. The engine argument classes, [EngineArgs](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.EngineArgs " EngineArgs dataclass ") and [AsyncEngineArgs](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.AsyncEngineArgs " AsyncEngineArgs dataclass "), are a combination of the configuration classes defined in [vllm.config](https://docs.vllm.ai/en/api/vllm/config/#vllm.config " vllm.config"). Therefore, if you are interested in developer documentation, we recommend looking at these configuration classes as they are the source of truth for types, defaults and docstrings. When passing JSON CLI arguments, the following sets of arguments are equivalent: - `--json-arg '{"key1": "value1", "key2": {"key3": "value2"}}'` - `--json-arg.key1 value1 --json-arg.key2.key3 value2` Additionally, list elements can be passed individually using `+`: - `--json-arg '{"key4": ["value3", "value4", "value5"]}'` - `--json-arg.key4+ value3 --json-arg.key4+='value4,value5'` ## [`EngineArgs`](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.EngineArgs " EngineArgs dataclass ")[¶](#engineargs "Permanent link") #### `--disable-log-stats`[¶](#-disable-log-stats "Permanent link") Disable logging statistics. Default: `False` #### `--aggregate-engine-logging`[¶](#-aggregate-engine-logging "Permanent link") Log aggregate rather than per-engine statistics when using data parallelism. Default: `False` #### `--fail-on-environ-validation`, `--no-fail-on-environ-validation`[¶](#-fail-on-environ-validation-no-fail-on-environ-validation "Permanent link") If set, the engine will raise an error if environment validation fails. Default: `False` #### `--shutdown-timeout`[¶](#-shutdown-timeout "Permanent link") Shutdown timeout in seconds. 0 = abort, >0 = wait. Default: `0` #### `--gdn-prefill-backend`[¶](#-gdn-prefill-backend "Permanent link") Possible choices: `flashinfer`, `triton`, `cutedsl` Select GDN prefill backend. ### ModelConfig[¶](#modelconfig "Permanent link") Configuration for the model. #### `--model`[¶](#-model "Permanent link") Name or path of the Hugging Face model to use. It is also used as the content for `model_name` tag in metrics output when `served_model_name` is not specified. Default: `Qwen/Qwen3-0.6B` #### `--runner`[¶](#-runner "Permanent link") Possible choices: `auto`, `draft`, `generate`, `pooling` The type of model runner to use. Each vLLM instance only supports one model runner, even if the same model can be used for multiple types. Default: `auto` #### `--convert`[¶](#-convert "Permanent link") Possible choices: `auto`, `classify`, `embed`, `none` Convert the model using adapters defined in [vllm.model\_executor.models.adapters](https://docs.vllm.ai/en/api/vllm/model_executor/models/adapters/#vllm.model_executor.models.adapters " vllm.model_executor.models.adapters"). The most common use case is to adapt a text generation model to be used for pooling tasks. Default: `auto` #### `--tokenizer`[¶](#-tokenizer "Permanent link") Name or path of the Hugging Face tokenizer to use. If unspecified, model name or path will be used. #### `--tokenizer-mode`[¶](#-tokenizer-mode "Permanent link") Possible choices: `auto`, `deepseek_v32`, `deepseek_v4`, `hf`, `mistral`, `slow` Tokenizer mode: - "auto" will use the tokenizer from `mistral_common` for Mistral models if available, otherwise it will use the "hf" tokenizer. - "hf" will use the fast tokenizer if available. - "slow" will always use the slow tokenizer. - "mistral" will always use the tokenizer from `mistral_common`. - "deepseek\_v32" will always use the tokenizer from `deepseek_v32`. - "deepseek\_v4" will always use the tokenizer from `deepseek_v4`. - "qwen\_vl" will always use the tokenizer from `qwen_vl`. - Other custom values can be supported via plugins. To swap the Rust BPE backend that powers HF fast tokenizers for the [fastokens](https://github.com/crusoecloud/fastokens) implementation, set `VLLM_USE_FASTOKENS=1` instead — that override applies to any mode that loads an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Default: `auto` #### `--trust-remote-code`, `--no-trust-remote-code`[¶](#-trust-remote-code-no-trust-remote-code "Permanent link") Trust remote code (e.g., from HuggingFace) when downloading the model and tokenizer. Default: `False` #### `--dtype`[¶](#-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float`, `float16`, `float32`, `half` Data type for model weights and activations: - "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. - "half" for FP16. Recommended for AWQ quantization. - "float16" is the same as "half". - "bfloat16" for a balance between precision and range. - "float" is shorthand for FP32 precision. - "float32" for FP32 precision. Default: `auto` #### `--seed`[¶](#-seed "Permanent link") Random seed for reproducibility. We must set the global seed because otherwise, different tensor parallel workers would sample different tokens, leading to inconsistent results. Default: `0` #### `--hf-config-path`[¶](#-hf-config-path "Permanent link") Name or path of the Hugging Face config to use. If unspecified, model name or path will be used. #### `--allowed-local-media-path`[¶](#-allowed-local-media-path "Permanent link") Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk. Should only be enabled in trusted environments. Default: `""` #### `--allowed-media-domains`[¶](#-allowed-media-domains "Permanent link") If set, only media URLs that belong to this domain can be used for multi-modal inputs. #### `--revision`[¶](#-revision "Permanent link") The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--code-revision`[¶](#-code-revision "Permanent link") The specific revision to use for the model code on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--tokenizer-revision`[¶](#-tokenizer-revision "Permanent link") The specific revision to use for the tokenizer on the Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. #### `--max-model-len`[¶](#-max-model-len "Permanent link") Model context length (prompt and output). If unspecified, will be automatically derived from the model config. When passing via `--max-model-len`, supports k/m/g/K/M/G in human-readable format. Examples: - 1k -> 1000 - 1K -> 1024 - 25.6k -> 25,600 - \-1 or 'auto' -> Automatically choose the maximum model length that fits in GPU memory. This will use the model's maximum context length if it fits, otherwise it will find the largest length that can be accommodated. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. Also accepts -1 or 'auto' as a special value for auto-detection. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600 - '-1' or 'auto' -> -1 (special value for auto-detection)` #### `--quantization`, `-q`[¶](#-quantization-q "Permanent link") Method used to quantize the weights. If `None`, we first check the `quantization_config` attribute in the model config file. If that is `None`, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. #### `--quantization-config`[¶](#-quantization-config "Permanent link") User-facing quantization configuration. Carries per-layer-kind specs (linear, moe) and ignore patterns; see :class:`QuantizationConfigArgs`. Auto-populated from the matching online shorthand when `quantization` is one of the values in `ONLINE_QUANT_SHORTHAND_NAMES`. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.QuantizationConfigArgs Should either be a valid JSON string or JSON keys passed individually. #### `--allow-deprecated-quantization`, `--no-allow-deprecated-quantization`[¶](#-allow-deprecated-quantization-no-allow-deprecated-quantization "Permanent link") Whether to allow deprecated quantization methods. Default: `False` #### `--enforce-eager`, `--no-enforce-eager`[¶](#-enforce-eager-no-enforce-eager "Permanent link") Whether to always use eager-mode PyTorch. If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility. Default: `False` #### `--enable-return-routed-experts`, `--no-enable-return-routed-experts`[¶](#-enable-return-routed-experts-no-enable-return-routed-experts "Permanent link") Whether to return routed experts. Default: `False` #### `--max-logprobs`[¶](#-max-logprobs "Permanent link") Maximum number of log probabilities to return when `logprobs` is specified in `SamplingParams`. The default value comes the default for the OpenAI Chat Completions API. -1 means no cap, i.e. all (output\_length \* vocab\_size) logprobs are allowed to be returned and it may cause OOM. Default: `20` #### `--logprobs-mode`[¶](#-logprobs-mode "Permanent link") Possible choices: `processed_logits`, `processed_logprobs`, `raw_logits`, `raw_logprobs` Indicates the content returned in the logprobs and prompt\_logprobs. Supported mode: 1) raw\_logprobs, 2) processed\_logprobs, 3) raw\_logits, 4) processed\_logits. Raw means the values before applying any logit processors, like bad words. Processed means the values after applying all processors, including temperature and top\_k/top\_p. Default: `raw_logprobs` #### `--use-fp64-gumbel`, `--no-use-fp64-gumbel`[¶](#-use-fp64-gumbel-no-use-fp64-gumbel "Permanent link") Whether to use FP64 (instead of FP32) random noise for Gumbel-max and equivalent exponential-race sampling. FP64 preserves lower-tail sampling events that fp32 uniform/exponential draws can truncate, at the cost of significantly lower throughput on most GPUs. Default: `False` #### `--disable-sliding-window`, `--no-disable-sliding-window`[¶](#-disable-sliding-window-no-disable-sliding-window "Permanent link") Whether to disable sliding window. If True, we will disable the sliding window functionality of the model, capping to sliding window size. If the model does not support sliding window, this argument is ignored. Default: `False` #### `--disable-cascade-attn`, `--no-disable-cascade-attn`[¶](#-disable-cascade-attn-no-disable-cascade-attn "Permanent link") Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be useful for preventing potential numerical issues. This defaults to True, so users must opt in to cascade attention by setting this to False. Even when this is set to False, cascade attention will only be used when the heuristic tells that it's beneficial. Default: `True` #### `--skip-tokenizer-init`, `--no-skip-tokenizer-init`[¶](#-skip-tokenizer-init-no-skip-tokenizer-init "Permanent link") Skip initialization of tokenizer and detokenizer. Expects valid `prompt_token_ids` and `None` for prompt from the input. The generated output will contain token ids. Default: `False` #### `--enable-prompt-embeds`, `--no-enable-prompt-embeds`[¶](#-enable-prompt-embeds-no-enable-prompt-embeds "Permanent link") If `True`, enables passing text embeddings as inputs via the `prompt_embeds` key. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--served-model-name`[¶](#-served-model-name "Permanent link") The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s) will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metrics tag will take the first one. #### `--config-format`[¶](#-config-format "Permanent link") Possible choices: `auto`, `hf`, `mistral` The format of the model config to load: - "auto" will try to load the config in hf format if available after trying to load in mistral format. - "hf" will load the config in hf format. - "mistral" will load the config in mistral format. Default: `auto` #### `--hf-token`[¶](#-hf-token "Permanent link") The token to use as HTTP bearer authorization for remote files . If `True`, will use the token generated when running `hf auth login` (stored in `~/.cache/huggingface/token`). #### `--hf-overrides`[¶](#-hf-overrides "Permanent link") If a dictionary, contains arguments to be forwarded to the Hugging Face config. If a callable, it is called to update the HuggingFace config. Default: `{}` #### `--pooler-config`[¶](#-pooler-config "Permanent link") Pooler config which controls the behaviour of output pooling in pooling models. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.PoolerConfig Should either be a valid JSON string or JSON keys passed individually. #### `--generation-config`[¶](#-generation-config "Permanent link") The folder path to the generation config. Defaults to `"auto"`, the generation config will be loaded from model path. If set to `"vllm"`, no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded from the specified folder path. If `max_new_tokens` is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Default: `auto` #### `--override-generation-config`[¶](#-override-generation-config "Permanent link") Overrides or sets generation config. e.g. `{"temperature": 0.5}`. If used with `--generation-config auto`, the override parameters will be merged with the default config from the model. If used with `--generation-config vllm`, only the override parameters are used. Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-sleep-mode`, `--no-enable-sleep-mode`[¶](#-enable-sleep-mode-no-enable-sleep-mode "Permanent link") Enable sleep mode for the engine (only cuda and hip platforms are supported). Default: `False` #### `--enable-cumem-allocator`, `--no-enable-cumem-allocator`[¶](#-enable-cumem-allocator-no-enable-cumem-allocator "Permanent link") Enable the custom cumem allocator to leverage advanced GPU memory allocation features such as multi-node NVLink support. Sleep mode automatically enables this allocator. Only cuda and hip platforms are supported. Default: `False` #### `--model-impl`[¶](#-model-impl "Permanent link") Possible choices: `auto`, `terratorch`, `transformers`, `vllm` Which implementation of the model to use: - "auto" will try to use the vLLM implementation, if it exists, and fall back to the Transformers implementation if no vLLM implementation is available. - "vllm" will use the vLLM model implementation. - "transformers" will use the Transformers model implementation. - "terratorch" will use the TerraTorch model implementation. Default: `auto` #### `--override-attention-dtype`[¶](#-override-attention-dtype "Permanent link") Override dtype for attention #### `--logits-processors`[¶](#-logits-processors "Permanent link") One or more logits processors' fully-qualified class names or class definitions #### `--io-processor-plugin`[¶](#-io-processor-plugin "Permanent link") IOProcessor plugin name to load at model startup #### `--renderer-num-workers`[¶](#-renderer-num-workers "Permanent link") Number of worker threads in the renderer thread pool. The pool is consumed by the async renderer path (e.g. the OpenAI-compatible API server started by `vllm serve`) to parallelize tokenization, chat template rendering, and multimodal preprocessing across concurrent requests. The offline `LLM` entrypoint uses the synchronous renderer path and processes prompts (including multimodal preprocessing) serially, so this setting has no effect there. Default: `1` ### LoadConfig[¶](#loadconfig "Permanent link") Configuration for loading the model weights. #### `--load-format`[¶](#-load-format "Permanent link") The format of the model weights to load. - "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. - "pt" will load the weights in the pytorch bin format. - "safetensors" will load the weights in the safetensors format. - "instanttensor" will load the Safetensors weights on CUDA devices using InstantTensor, which enables distributed loading with pipelined prefetching and fast direct I/O. - "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. - "dummy" will initialize the weights with random values, which is mainly for profiling. - "tensorizer" will use CoreWeave's tensorizer library for fast weight loading. See the Tensorize vLLM Model script in the Examples section for more information. - "runai\_streamer" will load the Safetensors weights using Run:ai Model Streamer. - "runai\_streamer\_sharded" will load weights from pre-sharded checkpoint files using Run:ai Model Streamer. - "bitsandbytes" will load the weights using bitsandbytes quantization. - "sharded\_state" will load weights from pre-sharded checkpoint files, supporting efficient loading of tensor-parallel models. - "gguf" will load weights from GGUF format files (details specified in https://github.com/ggml-org/ggml/blob/master/docs/gguf.md). - "mistral" will load weights from consolidated safetensors files used by Mistral models. - "modelexpress" will load weights using ModelExpress. - Other custom values can be supported via plugins. Default: `auto` #### `--download-dir`[¶](#-download-dir "Permanent link") Directory to download and load the weights, default to the default cache directory of Hugging Face. #### `--safetensors-load-strategy`[¶](#-safetensors-load-strategy "Permanent link") Specifies the loading strategy for safetensors weights. - None (default): Uses memory-mapped (lazy) loading. When an NFS filesystem is detected and the total checkpoint size fits within 90%%%% of available RAM, prefetching is enabled automatically. - "lazy": Weights are memory-mapped from the file. This enables on-demand loading and is highly efficient for models on local storage. Unlike the default (None), auto-prefetch on NFS is not performed. - "eager": The entire file is read into CPU memory upfront before loading. This is recommended for models on network filesystems (e.g., Lustre, NFS) as it avoids inefficient random reads, significantly speeding up model initialization. However, it uses more CPU RAM. - "prefetch": Checkpoint files are read into the OS page cache before workers load them, speeding up the model loading phase. Useful on network or high-latency storage. - "torchao": Weights are loaded in upfront and then reconstructed into torchao tensor subclasses. This is used when the checkpoint was quantized using torchao and saved using safetensors. Needs `torchao >= 0.14.0`. #### `--safetensors-prefetch-num-threads`[¶](#-safetensors-prefetch-num-threads "Permanent link") Number of worker threads used to prefetch safetensors checkpoint files into the OS page cache when safetensors prefetching is enabled. Default: `8` #### `--safetensors-prefetch-block-size`[¶](#-safetensors-prefetch-block-size "Permanent link") Read size in bytes for each safetensors checkpoint file prefetch. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` Default: `16777216` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load\_format. Default: `{}` #### `--ignore-patterns`[¶](#-ignore-patterns "Permanent link") The list of patterns to ignore when loading the model. Default to "original/\*_/_" to avoid repeated loading of llama's checkpoints. Default: `['original/**/*']` #### `--use-tqdm-on-load`, `--no-use-tqdm-on-load`[¶](#-use-tqdm-on-load-no-use-tqdm-on-load "Permanent link") Whether to enable tqdm for showing progress bar when loading model weights. Default: `True` #### `--pt-load-map-location`[¶](#-pt-load-map-location "Permanent link") The map location for loading pytorch checkpoint, to support loading checkpoints can only be loaded on certain devices like "cuda", this is equivalent to `{"": "cuda"}`. Another supported format is mapping from different devices like from GPU 1 to GPU 0: `{"cuda:1": "cuda:0"}`. Note that when passed from command line, the strings in dictionary need to be double quoted for json parsing. For more details, see the original doc for `map_location` parameter in [`torch.load`](https://pytorch.org/docs/stable/generated/torch.load.html#torch.load) parameter. Default: `cpu` ### AttentionConfig[¶](#attentionconfig "Permanent link") Configuration for attention mechanisms in vLLM. #### `--attention-backend`[¶](#-attention-backend "Permanent link") Attention backend to use. Use "auto" or None for automatic selection. ### MambaConfig[¶](#mambaconfig "Permanent link") Configuration for Mamba SSM backends. #### `--mamba-backend`[¶](#-mamba-backend "Permanent link") Mamba SSU backend to use. Default: `MambaBackendEnum.TRITON` #### `--enable-mamba-cache-stochastic-rounding`, `--no-enable-mamba-cache-stochastic-rounding`[¶](#-enable-mamba-cache-stochastic-rounding-no-enable-mamba-cache-stochastic-rounding "Permanent link") Enable stochastic rounding when writing SSM state to fp16 cache. Uses random bits to unbias the rounding error, which can improve numerical stability for long sequences. Default: `False` #### `--mamba-cache-philox-rounds`[¶](#-mamba-cache-philox-rounds "Permanent link") Number of Philox PRNG rounds for stochastic rounding random number generation. 0 uses the Triton default. Higher values improve randomness quality at the cost of compute. Default: `0` ### StructuredOutputsConfig[¶](#structuredoutputsconfig "Permanent link") Dataclass which contains structured outputs config for the engine. #### `--reasoning-parser`[¶](#-reasoning-parser "Permanent link") Select the reasoning parser depending on the model that you're using. This is used to parse the reasoning content into OpenAI API format. Default: `""` #### `--reasoning-parser-plugin`[¶](#-reasoning-parser-plugin "Permanent link") Path to a dynamically reasoning parser plugin that can be dynamically loaded and registered. Default: `""` ### ParallelConfig[¶](#parallelconfig "Permanent link") Configuration for the distributed execution. #### `--distributed-executor-backend`[¶](#-distributed-executor-backend "Permanent link") Possible choices: `external_launcher`, `mp`, `ray`, `uni` Backend to use for distributed model workers, either "ray" or "mp" (multiprocessing). If the product of pipeline\_parallel\_size and tensor\_parallel\_size is less than or equal to the number of GPUs available, "mp" will be used to keep processing on a single host. Otherwise, an error will be raised. To use "mp" you must also set nnodes, and to use "ray" you must manually set distributed\_executor\_backend to "ray". Note: [TPU](https://docs.vllm.ai/projects/tpu/en/latest/) platform only supports Ray for distributed inference. #### `--pipeline-parallel-size`, `-pp`[¶](#-pipeline-parallel-size-pp "Permanent link") Number of pipeline parallel groups. Default: `1` #### `--master-addr`[¶](#-master-addr "Permanent link") distributed master address for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `127.0.0.1` #### `--master-port`[¶](#-master-port "Permanent link") distributed master port for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `29501` #### `--nnodes`, `-n`[¶](#-nnodes-n "Permanent link") num of nodes for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `1` #### `--node-rank`, `-r`[¶](#-node-rank-r "Permanent link") distributed node rank for multi-node distributed inference when distributed\_executor\_backend is mp. Default: `0` #### `--distributed-timeout-seconds`[¶](#-distributed-timeout-seconds "Permanent link") Timeout in seconds for distributed operations (e.g., init\_process\_group). If set, this value is passed to torch.distributed.init\_process\_group as the timeout parameter. If None, PyTorch's default timeout is used (600s for NCCL). Increase this for multi-node setups where model downloads may be slow. #### `--cpu-distributed-timeout-seconds`[¶](#-cpu-distributed-timeout-seconds "Permanent link") Timeout (in seconds) for cpu communication groups. If None, PyTorch's default timeout is used (1800s for gloo). #### `--numa-bind`, `--no-numa-bind`[¶](#-numa-bind-no-numa-bind "Permanent link") Enable NUMA binding for GPU worker subprocesses. By default, workers are pinned to their GPU's NUMA-local CPUs and memory; on PCT-capable Xeons they also auto-bind to the SKU's PCT priority cores. Default: `False` #### `--numa-bind-nodes`[¶](#-numa-bind-nodes "Permanent link") NUMA node to bind each GPU worker to. Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the GPU-to-NUMA topology. The values are passed to `numactl --membind` and `--cpunodebind`, so they must be valid `numactl` NUMA node indices. #### `--numa-bind-cpus`[¶](#-numa-bind-cpus "Permanent link") Optional CPU lists to bind each GPU worker to. Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful for custom policies such as binding to PCT or other high-frequency cores. Each entry must use `numactl --physcpubind` CPU-list syntax, for example `"0-3"` or `"0,2,4-7"`. #### `--tensor-parallel-size`, `-tp`[¶](#-tensor-parallel-size-tp "Permanent link") Number of tensor parallel groups. Default: `1` #### `--decode-context-parallel-size`, `-dcp`[¶](#-decode-context-parallel-size-dcp "Permanent link") Number of decode context parallel groups, because the world size does not change by dcp, it simply reuse the GPUs of TP group, and tp\_size needs to be divisible by dcp\_size. Default: `1` #### `--dcp-comm-backend`[¶](#-dcp-comm-backend "Permanent link") Possible choices: `a2a`, `ag_rs` Communication backend for Decode Context Parallel (DCP). - "ag\_rs": AllGather + ReduceScatter (default, existing behavior) - "a2a": All-to-All exchange of partial outputs + LSE, then combine with Triton kernel. Reduces NCCL calls from 3 to 2 per layer for MLA models. Default: `ag_rs` #### `--dcp-kv-cache-interleave-size`[¶](#-dcp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP. dcp\_kv\_cache\_interleave\_size has been replaced by cp\_kv\_cache\_interleave\_size, and will be deprecated when PCP is fully supported. Default: `1` #### `--cp-kv-cache-interleave-size`[¶](#-cp-kv-cache-interleave-size "Permanent link") Interleave size of kv\_cache storage while using DCP or PCP. For `total_cp_rank = pcp_rank * dcp_world_size + dcp_rank`, and `total_cp_world_size = pcp_world_size * dcp_world_size`. store interleave\_size tokens on total\_cp\_rank i, then store next interleave\_size tokens on total\_cp\_rank i+1. Interleave\_size=1: token-level alignment, where token `i` is stored on total\_cp\_rank `i %% total_cp_world_size`. Interleave\_size=block\_size: block-level alignment, where tokens are first populated to the preceding ranks. Tokens are then stored in (rank i+1, block j) only after (rank i, block j) is fully occupied. Block\_size should be greater than or equal to cp\_kv\_cache\_interleave\_size. Block\_size should be divisible by cp\_kv\_cache\_interleave\_size. Default: `1` #### `--prefill-context-parallel-size`, `-pcp`[¶](#-prefill-context-parallel-size-pcp "Permanent link") Number of prefill context parallel groups. Default: `1` #### `--data-parallel-size`, `-dp`[¶](#-data-parallel-size-dp "Permanent link") Number of data parallel groups. MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. Default: `1` #### `--data-parallel-rank`, `-dpn`[¶](#-data-parallel-rank-dpn "Permanent link") Data parallel rank of this instance. When set, enables external load balancer mode for MoE data-parallel deployments. Unsupported for non-MoE models; launch independent vLLM instances instead. #### `--data-parallel-start-rank`, `-dpr`[¶](#-data-parallel-start-rank-dpr "Permanent link") Starting data parallel rank for secondary nodes. #### `--data-parallel-size-local`, `-dpl`[¶](#-data-parallel-size-local-dpl "Permanent link") Number of data parallel replicas to run on this node. #### `--data-parallel-address`, `-dpa`[¶](#-data-parallel-address-dpa "Permanent link") Address of data parallel cluster head-node. #### `--data-parallel-rpc-port`, `-dpp`[¶](#-data-parallel-rpc-port-dpp "Permanent link") Port for data parallel RPC communication. #### `--data-parallel-backend`, `-dpb`[¶](#-data-parallel-backend-dpb "Permanent link") Backend for data parallel, either "mp" or "ray". Default: `mp` #### `--data-parallel-hybrid-lb`, `--no-data-parallel-hybrid-lb`, `-dph`[¶](#-data-parallel-hybrid-lb-no-data-parallel-hybrid-lb-dph "Permanent link") Whether to use "hybrid" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. Enables running an AsyncLLM and API server on a "per-node" basis where vLLM load balances between local data parallel ranks, but an external LB balances between vLLM nodes/replicas. Set explicitly in conjunction with --data-parallel-start-rank. Default: `False` #### `--data-parallel-external-lb`, `--no-data-parallel-external-lb`, `-dpe`[¶](#-data-parallel-external-lb-no-data-parallel-external-lb-dpe "Permanent link") Whether to use "external" DP LB mode. Applies only to online serving and when data\_parallel\_size > 0. This is useful for a "one-pod-per-rank" wide-EP setup in Kubernetes. Supported only for MoE deployments; non-MoE models should use independent vLLM instances without --data-parallel-\* arguments. Set implicitly when --data-parallel-rank is provided explicitly to vllm serve. Default: `False` #### `--data-parallel-multi-port-external-lb`, `-dpm`[¶](#-data-parallel-multi-port-external-lb-dpm "Permanent link") Run a node-local supervisor that launches one external-LB API server per local data parallel rank and exposes aggregated health on a supervisor port. Default: `False` #### `--enable-expert-parallel`, `--no-enable-expert-parallel`, `-ep`[¶](#-enable-expert-parallel-no-enable-expert-parallel-ep "Permanent link") Use expert parallelism instead of tensor parallelism for MoE layers. Default: `False` #### `--enable-ep-weight-filter`, `--no-enable-ep-weight-filter`[¶](#-enable-ep-weight-filter-no-enable-ep-weight-filter "Permanent link") Skip non-local expert weights during model loading when expert parallelism is active. Each rank only reads its own expert shard from disk, which can drastically reduce storage I/O for MoE models with per-expert weight tensors (e.g. DeepSeek, Mixtral, Kimi-K2.5). Has no effect on 3D fused-expert checkpoints (e.g. GPT-OSS) or non-MoE models. Default: `False` #### `--all2all-backend`[¶](#-all2all-backend "Permanent link") Possible choices: `allgather_reducescatter`, `deepep_high_throughput`, `deepep_low_latency`, `flashinfer_all2allv`, `flashinfer_nvlink_one_sided`, `flashinfer_nvlink_two_sided`, `mori_high_throughput`, `mori_low_latency`, `naive`, `nixl_ep`, `pplx` All2All backend for MoE expert parallel communication. Available options: - "allgather\_reducescatter": All2all based on allgather and reducescatter - "deepep\_high\_throughput": Use deepep high-throughput kernels - "deepep\_low\_latency": Use deepep low-latency kernels - "mori\_high\_throughput": MoRI EP with InterNodeV1 for multi-node - "mori\_low\_latency": MoRI EP with InterNodeV1LL for multi-node - "nixl\_ep": Use nixl-ep kernels - "flashinfer\_nvlink\_two\_sided": Use flashinfer two-sided kernels for mnnvl - "flashinfer\_nvlink\_one\_sided": Use flashinfer high-throughput a2a kernels Default: `allgather_reducescatter` #### `--enable-dbo`, `--no-enable-dbo`[¶](#-enable-dbo-no-enable-dbo "Permanent link") Enable dual batch overlap for the model executor. Default: `False` #### `--ubatch-size`[¶](#-ubatch-size "Permanent link") Number of ubatch size. Default: `0` #### `--enable-elastic-ep`, `--no-enable-elastic-ep`[¶](#-enable-elastic-ep-no-enable-elastic-ep "Permanent link") Enable elastic expert parallelism with stateless NCCL groups for DP/EP. Default: `False` #### `--dbo-decode-token-threshold`[¶](#-dbo-decode-token-threshold "Permanent link") The threshold for dual batch overlap for batches only containing decodes. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `32` #### `--dbo-prefill-token-threshold`[¶](#-dbo-prefill-token-threshold "Permanent link") The threshold for dual batch overlap for batches that contain one or more prefills. If the number of tokens in the request is greater than this threshold, microbatching will be used. Otherwise, the request will be processed in a single batch. Default: `512` #### `--disable-nccl-for-dp-synchronization`, `--no-disable-nccl-for-dp-synchronization`[¶](#-disable-nccl-for-dp-synchronization-no-disable-nccl-for-dp-synchronization "Permanent link") Forces the dp synchronization logic in vllm/v1/worker/dp\_utils.py to use Gloo instead of NCCL for its all reduce. Defaults to True when async scheduling is enabled, False otherwise. #### `--enable-eplb`, `--no-enable-eplb`[¶](#-enable-eplb-no-enable-eplb "Permanent link") Enable expert parallelism load balancing for MoE layers. Default: `False` #### `--eplb-config`[¶](#-eplb-config "Permanent link") Expert parallelism configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.EPLBConfig Should either be a valid JSON string or JSON keys passed individually. Default: `EPLBConfig(window_size=1000, step_interval=3000, num_redundant_experts=0, log_balancedness=False, log_balancedness_interval=1, use_async=True, policy='default', communicator=None)` #### `--expert-placement-strategy`[¶](#-expert-placement-strategy "Permanent link") Possible choices: `linear`, `round_robin` The expert placement strategy for MoE layers: - "linear": Experts are placed in a contiguous manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 1\] and rank 1 will have experts \[2, 3\]. - "round\_robin": Experts are placed in a round-robin manner. For example, with 4 experts and 2 ranks, rank 0 will have experts \[0, 2\] and rank 1 will have experts \[1, 3\]. This strategy can help improve load balancing for grouped expert models with no redundant experts. Default: `linear` #### `--max-parallel-loading-workers`[¶](#-max-parallel-loading-workers "Permanent link") Maximum number of parallel loading workers when loading model sequentially in multiple batches. To avoid RAM OOM when using tensor parallel and large models. #### `--ray-workers-use-nsight`, `--no-ray-workers-use-nsight`[¶](#-ray-workers-use-nsight-no-ray-workers-use-nsight "Permanent link") Whether to profile Ray workers with nsight, see https://docs.ray.io/en/latest/ray-observability/user-guides/profiling.html#profiling-nsight-profiler. Default: `False` #### `--disable-custom-all-reduce`, `--no-disable-custom-all-reduce`[¶](#-disable-custom-all-reduce-no-disable-custom-all-reduce "Permanent link") Disable the custom all-reduce kernel and fall back to NCCL. Default: `False` #### `--worker-cls`[¶](#-worker-cls "Permanent link") The full name of the worker class to use. If "auto", the worker class will be determined based on the platform. Default: `auto` #### `--worker-extension-cls`[¶](#-worker-extension-cls "Permanent link") The full name of the worker extension class to use. The worker extension class is dynamically inherited by the worker class. This is used to inject new attributes and methods to the worker class for use in collective\_rpc calls. Default: `""` ### CacheConfig[¶](#cacheconfig "Permanent link") Configuration for the KV cache. #### `--block-size`[¶](#-block-size "Permanent link") Size of a contiguous cache block in number of tokens. Accepts None (meaning "use default"). After construction, always int. #### `--gpu-memory-utilization`[¶](#-gpu-memory-utilization "Permanent link") The fraction of GPU memory to be used for the model executor, which can range from 0 to 1. For example, a value of 0.5 would imply 50%% GPU memory utilization. If unspecified, will use the default value of 0.92. This is a per-instance limit, and only applies to the current vLLM instance. It does not matter if you have another vLLM instance running on the same GPU. For example, if you have two vLLM instances running on the same GPU, you can set the GPU memory utilization to 0.5 for each instance. Default: `0.92` #### `--kv-cache-memory-bytes`[¶](#-kv-cache-memory-bytes "Permanent link") Size of KV Cache per GPU in bytes. By default, this is set to None and vllm can automatically infer the kv cache size based on gpu\_memory\_utilization. However, users may want to manually specify the kv cache memory size. kv\_cache\_memory\_bytes allows more fine-grain control of how much memory gets used when compared with using gpu\_memory\_utilization. Note that kv\_cache\_memory\_bytes (when not-None) ignores gpu\_memory\_utilization Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--kv-cache-dtype`[¶](#-kv-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `fp8`, `fp8_ds_mla`, `fp8_e4m3`, `fp8_e5m2`, `fp8_inc`, `fp8_per_token_head`, `int8_per_token_head`, `nvfp4`, `turboquant_3bit_nc`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_k8v4` Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8\_e4m3) and fp8\_e5m2. ROCm (AMD GPU) supports fp8 (=fp8\_e4m3). Intel Gaudi (HPU) supports fp8 (using fp8\_inc). Some models (namely DeepSeekV3.2) default to fp8, set to bfloat16 to use bfloat16 instead, this is an invalid option for models that do not default to fp8. Default: `auto` #### `--num-gpu-blocks-override`[¶](#-num-gpu-blocks-override "Permanent link") Number of GPU blocks to use. This overrides the profiled `num_gpu_blocks` if specified. Does nothing if `None`. Used for testing preemption. #### `--enable-prefix-caching`, `--no-enable-prefix-caching`[¶](#-enable-prefix-caching-no-enable-prefix-caching "Permanent link") Whether to enable prefix caching. #### `--prefix-caching-hash-algo`[¶](#-prefix-caching-hash-algo "Permanent link") Possible choices: `sha256`, `sha256_cbor`, `xxhash`, `xxhash_cbor` Set the hash algorithm for prefix caching: - "sha256" uses Pickle for object serialization before hashing. This is the current default, as SHA256 is the most secure choice to avoid potential hash collisions. - "sha256\_cbor" provides a reproducible, cross-language compatible hash. It serializes objects using canonical CBOR and hashes them with SHA-256. - "xxhash" uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional `xxhash` package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. - "xxhash\_cbor" combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional `xxhash` package. Default: `sha256` #### `--calculate-kv-scales`, `--no-calculate-kv-scales`[¶](#-calculate-kv-scales-no-calculate-kv-scales "Permanent link") Deprecated: This option is deprecated and will be removed in v0.19. It enables dynamic calculation of `k_scale` and `v_scale` when kv\_cache\_dtype is fp8. If `False`, the scales will be loaded from the model checkpoint if available. Otherwise, the scales will default to 1.0. Default: `False` #### `--kv-cache-dtype-skip-layers`[¶](#-kv-cache-dtype-skip-layers "Permanent link") Layer patterns to skip KV cache quantization. Accepts layer indices (e.g., '0', '2', '4') or attention type names (e.g., 'sliding\_window'). Default: `[]` #### `--kv-sharing-fast-prefill`, `--no-kv-sharing-fast-prefill`[¶](#-kv-sharing-fast-prefill-no-kv-sharing-fast-prefill "Permanent link") This feature is work in progress and no prefill optimization takes place with this flag enabled currently. In some KV sharing setups, e.g. YOCO (https://arxiv.org/abs/2405.05254), some layers can skip tokens corresponding to prefill. This flag enables attention metadata for eligible layers to be overridden with metadata necessary for implementing this optimization in some models (e.g. Gemma3n) Default: `False` #### `--mamba-cache-dtype`[¶](#-mamba-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (both the conv as well as the ssm state). If set to 'auto', the data type will be inferred from the model config. Default: `auto` #### `--mamba-ssm-cache-dtype`[¶](#-mamba-ssm-cache-dtype "Permanent link") Possible choices: `auto`, `bfloat16`, `float16`, `float32` The data type to use for the Mamba cache (ssm state only, conv state will still be controlled by mamba\_cache\_dtype). If set to 'auto', the data type for the ssm state will be determined by mamba\_cache\_dtype. Default: `auto` #### `--mamba-block-size`[¶](#-mamba-block-size "Permanent link") Size of a contiguous cache block in number of tokens for mamba cache. Can be set only when prefix caching is enabled. Value must be a multiple of 8 to align with causal\_conv1d kernel. #### `--mamba-cache-mode`[¶](#-mamba-cache-mode "Permanent link") Possible choices: `align`, `all`, `none` The cache strategy for Mamba layers. - "none": set when prefix caching is disabled. - "all": cache the mamba state of all tokens at position i \* block\_size. This is the default behavior (for models that support it) when prefix caching is enabled. - "align": only cache the mamba state of the last token of each scheduler step and when the token is at position i \* block\_size. Default: `none` #### `--kv-offloading-size`[¶](#-kv-offloading-size "Permanent link") Size of the KV cache offloading buffer in GiB. When TP > 1, this is the total buffer size summed across all TP ranks. By default, this is set to None, which means no KV offloading is enabled. When set, vLLM will enable KV cache offloading to CPU using the kv\_offloading\_backend. #### `--kv-offloading-backend`[¶](#-kv-offloading-backend "Permanent link") Possible choices: `lmcache`, `native` The backend to use for KV cache offloading. Supported backends include 'native' (vLLM native CPU offloading), 'lmcache'. KV offloading is only activated when kv\_offloading\_size is set. Default: `native` ### OffloadConfig[¶](#offloadconfig "Permanent link") Configuration for model weight offloading to reduce GPU memory usage. #### `--offload-backend`[¶](#-offload-backend "Permanent link") Possible choices: `auto`, `prefetch`, `uva` The backend for weight offloading. Options: - "auto": Selects based on which sub-config has non-default values (prefetch if offload\_group\_size > 0, uva if cpu\_offload\_gb > 0). - "uva": UVA (Unified Virtual Addressing) zero-copy offloading. - "prefetch": Async prefetch with group-based layer offloading. Default: `auto` #### `--cpu-offload-gb`[¶](#-cpu-offload-gb "Permanent link") The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. This uses UVA (Unified Virtual Addressing) for zero-copy access. Default: `0` #### `--cpu-offload-params`[¶](#-cpu-offload-params "Permanent link") The set of parameter name segments to target for CPU offloading. Unmatched parameters are not offloaded. If this set is empty, parameters are offloaded non-selectively until the memory limit defined by `cpu_offload_gb` is reached. Examples: - For parameter name "mlp.experts.w2\_weight": - "experts" or "experts.w2\_weight" will match. - "expert" or "w2" will NOT match (must be exact segments). This allows distinguishing parameters like "w2\_weight" and "w2\_weight\_scale". Default: `set()` #### `--offload-group-size`[¶](#-offload-group-size "Permanent link") Group every N layers together. Offload last `offload_num_in_group` layers of each group. Default is 0 (disabled). Example: group\_size=8, num\_in\_group=2 offloads layers 6,7,14,15,22,23,... Unlike cpu\_offload\_gb, this uses explicit async prefetching to hide transfer latency. Default: `0` #### `--offload-num-in-group`[¶](#-offload-num-in-group "Permanent link") Number of layers to offload per group. Must be <= offload\_group\_size. Default is 1. Default: `1` #### `--offload-prefetch-step`[¶](#-offload-prefetch-step "Permanent link") Number of layers to prefetch ahead. Higher values hide more latency but use more GPU memory. Default is 1. Default: `1` #### `--offload-params`[¶](#-offload-params "Permanent link") The set of parameter name segments to target for prefetch offloading. Unmatched parameters are not offloaded. If this set is empty, ALL parameters of each offloaded layer are offloaded. Uses segment matching: "w13\_weight" matches "mlp.experts.w13\_weight" but not "mlp.experts.w13\_weight\_scale". Default: `set()` ### MultiModalConfig[¶](#multimodalconfig "Permanent link") Controls the behavior of multimodal models. #### `--language-model-only`, `--no-language-model-only`[¶](#-language-model-only-no-language-model-only "Permanent link") If True, disables all multimodal inputs by setting all modality limits to 0. Equivalent to setting `--limit-mm-per-prompt` to 0 for every modality. Default: `False` #### `--limit-mm-per-prompt`[¶](#-limit-mm-per-prompt "Permanent link") The maximum number of input items and options allowed per prompt for each modality. Defaults to 999 for each modality. Legacy format (count only): Configurable format (with options): {"video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}, "image": {"count": 5, "width": 512, "height": 512}} Mixed format (combining both): {"image": 16, "video": {"count": 1, "num\_frames": 32, "width": 512, "height": 512}} Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--enable-mm-embeds`, `--no-enable-mm-embeds`[¶](#-enable-mm-embeds-no-enable-mm-embeds "Permanent link") If `True`, enables passing multimodal embeddings: for `LLM` class, this refers to tensor inputs under `multi_modal_data`; for the OpenAI-compatible server, this refers to chat messages with content `"type": "*_embeds"`. When enabled with `--limit-mm-per-prompt` set to 0 for a modality, precomputed embeddings skip count validation for that modality, saving memory by not loading encoder modules while still enabling embeddings as an input. Limits greater than 0 still apply to embeddings. WARNING: The vLLM engine may crash if incorrect shape of embeddings is passed. Only enable this flag for trusted users! Default: `False` #### `--media-io-kwargs`[¶](#-media-io-kwargs "Permanent link") Additional args passed to process media inputs, keyed by modalities. For example, to set num\_frames for video, set `--media-io-kwargs '{"video": {"num_frames": 40} }'` Should either be a valid JSON string or JSON keys passed individually. Default: `{}` #### `--mm-processor-kwargs`[¶](#-mm-processor-kwargs "Permanent link") Arguments to be forwarded to the model's processor for multi-modal data, e.g., image processor. Overrides for the multi-modal processor obtained from `transformers.AutoProcessor.from_pretrained`. The available overrides depend on the model that is being run. For example, for Phi-3-Vision: `{"num_crops": 4}`. Should either be a valid JSON string or JSON keys passed individually. #### `--mm-processor-cache-gb`[¶](#-mm-processor-cache-gb "Permanent link") The size (in GiB) of the multi-modal processor cache, which is used to avoid re-processing past multi-modal inputs. This cache is duplicated for each API process and engine core process, resulting in a total memory usage of `mm_processor_cache_gb * (api_server_count + data_parallel_size)`. Set to `0` to disable this cache completely (not recommended). Default: `4` #### `--mm-processor-cache-type`[¶](#-mm-processor-cache-type "Permanent link") Possible choices: `lru`, `shm` Type of cache to use for the multi-modal preprocessor/mapper. If `shm`, use shared memory FIFO cache. If `lru`, use mirrored LRU cache. Default: `lru` #### `--mm-shm-cache-max-object-size-mb`[¶](#-mm-shm-cache-max-object-size-mb "Permanent link") Size limit (in MiB) for each object stored in the multi-modal processor shared memory cache. Only effective when `mm_processor_cache_type` is `"shm"`. Default: `128` #### `--mm-encoder-only`, `--no-mm-encoder-only`[¶](#-mm-encoder-only-no-mm-encoder-only "Permanent link") When enabled, skips the language component of the model. This is usually only valid in disaggregated Encoder process. Default: `False` #### `--mm-encoder-tp-mode`[¶](#-mm-encoder-tp-mode "Permanent link") Possible choices: `data`, `weights` Indicates how to optimize multi-modal encoder inference using tensor parallelism (TP). - `"weights"`: Within the same vLLM engine, split the weights of each layer across TP ranks. (default TP behavior) - `"data"`: Within the same vLLM engine, split the batched input data across TP ranks to process the data in parallel, while hosting the full weights on each TP rank. This batch-level DP is not to be confused with API request-level DP (which is controlled by `--data-parallel-size`). This is only supported on a per-model basis and falls back to `"weights"` if the encoder does not support DP. Default: `weights` #### `--mm-encoder-attn-backend`[¶](#-mm-encoder-attn-backend "Permanent link") Optional override for the multi-modal encoder attention backend when using vision transformers. Accepts any value from `vllm.v1.attention.backends.registry.AttentionBackendEnum` (e.g. `FLASH_ATTN`). #### `--mm-encoder-attn-dtype`[¶](#-mm-encoder-attn-dtype "Permanent link") Possible choices: `fp8`, `None` Optional dtype override for ViT encoder attention. Set to `"fp8"` to enable FP8 quantization via the FlashInfer cuDNN backend. When set to `"fp8"` without a scale file, dynamic scaling is used automatically. See docs/features/quantization/fp8\_vit\_attn.md for details. #### `--mm-encoder-fp8-scale-path`[¶](#-mm-encoder-fp8-scale-path "Permanent link") Path to a JSON file containing per-layer FP8 Q/K/V scales for ViT encoder attention. When provided (with `mm_encoder_attn_dtype="fp8"`), static scaling is used. When omitted, dynamic scaling is used. #### `--mm-encoder-fp8-scale-save-path`[¶](#-mm-encoder-fp8-scale-save-path "Permanent link") When set with dynamic FP8 scaling (`mm_encoder_attn_dtype="fp8"` and no `mm_encoder_fp8_scale_path`), saves the calibrated scales to this file after the amax history buffer is full. The saved file can then be used as `mm_encoder_fp8_scale_path` in subsequent runs. #### `--mm-encoder-fp8-scale-save-margin`[¶](#-mm-encoder-fp8-scale-save-margin "Permanent link") Safety margin multiplied onto scales when auto-saving. A value > 1 leaves headroom so that inputs with larger activations than the calibration set do not overflow FP8 range. Default 1.5. Default: `1.5` #### `--interleave-mm-strings`, `--no-interleave-mm-strings`[¶](#-interleave-mm-strings-no-interleave-mm-strings "Permanent link") Enable fully interleaved support for multimodal prompts, while using --chat-template-content-format=string. Default: `False` #### `--skip-mm-profiling`, `--no-skip-mm-profiling`[¶](#-skip-mm-profiling-no-skip-mm-profiling "Permanent link") When enabled, skips multimodal memory profiling and only profiles with language backbone model during engine initialization. This reduces engine startup time but shifts the responsibility to users for estimating the peak memory usage of the activation of multimodal encoder and embedding cache. Default: `False` #### `--video-pruning-rate`[¶](#-video-pruning-rate "Permanent link") Sets pruning rate for video pruning via Efficient Video Sampling. Value sits in range \[0;1) and determines fraction of media tokens from each video to be pruned. #### `--mm-tensor-ipc`[¶](#-mm-tensor-ipc "Permanent link") Possible choices: `direct_rpc`, `torch_shm` IPC (inter-process communication) method for multimodal tensors. - "direct\_rpc": Use msgspec serialization via RPC - "torch\_shm": Use torch.multiprocessing shared memory for zero-copy IPC Defaults to "direct\_rpc". Default: `direct_rpc` ### LoRAConfig[¶](#loraconfig "Permanent link") Configuration for LoRA. #### `--enable-lora`, `--no-enable-lora`[¶](#-enable-lora-no-enable-lora "Permanent link") If True, enable handling of LoRA adapters. #### `--max-loras`[¶](#-max-loras "Permanent link") Max number of LoRAs in a single batch. Default: `1` #### `--max-lora-rank`[¶](#-max-lora-rank "Permanent link") Possible choices: `1`, `8`, `16`, `32`, `64`, `128`, `256`, `320`, `512` Max LoRA rank. Default: `16` #### `--lora-dtype`[¶](#-lora-dtype "Permanent link") Data type for LoRA. If auto, will default to base model dtype. Default: `auto` #### `--enable-tower-connector-lora`, `--no-enable-tower-connector-lora`[¶](#-enable-tower-connector-lora-no-enable-tower-connector-lora "Permanent link") If `True`, LoRA support for the tower (vision encoder) and connector of multimodal models will be enabled. This is an experimental feature and currently only supports some MM models such as the Qwen VL series. The default is False. Default: `False` #### `--max-cpu-loras`[¶](#-max-cpu-loras "Permanent link") Maximum number of LoRAs to store in CPU memory. Must be >= than `max_loras`. #### `--fully-sharded-loras`, `--no-fully-sharded-loras`[¶](#-fully-sharded-loras-no-fully-sharded-loras "Permanent link") By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. Default: `False` #### `--lora-target-modules`[¶](#-lora-target-modules "Permanent link") Restrict LoRA to specific module suffixes (e.g., \["o\_proj", "qkv\_proj"\]). If None, all supported LoRA modules are used. This allows deployment-time control over which modules have LoRA applied, useful for performance tuning. #### `--default-mm-loras`[¶](#-default-mm-loras "Permanent link") Dictionary mapping specific modalities to LoRA model paths; this field is only applicable to multimodal models and should be leveraged when a model always expects a LoRA to be active when a given modality is present. Note that currently, if a request provides multiple additional modalities, each of which have their own LoRA, we do NOT apply default\_mm\_loras because we currently only support one lora adapter per prompt. When run in offline mode, the lora IDs for n modalities will be automatically assigned to 1-n with the names of the modalities in alphabetic order. Should either be a valid JSON string or JSON keys passed individually. #### `--specialize-active-lora`, `--no-specialize-active-lora`[¶](#-specialize-active-lora-no-specialize-active-lora "Permanent link") Whether to construct lora kernel grid by the number of active LoRA adapters. When set to True, separate cuda graphs will be captured for different counts of active LoRAs (powers of 2 up to max\_loras), which can improve performance for variable LoRA usage patterns at the cost of increased startup time and memory usage. Only takes effect when cudagraph\_specialize\_lora is True. Default: `False` #### `--enable-mixed-moe-lora-format`, `--no-enable-mixed-moe-lora-format`[¶](#-enable-mixed-moe-lora-format-no-enable-mixed-moe-lora-format "Permanent link") If True, force the engine to use the universal 2D MoE LoRA wrapper (`FusedMoEWithLoRA`) regardless of the model's `is_3d_moe_weight` flag, so that 2D-format and 3D-format MoE LoRA adapters can be served in the same deployment. Only meaningful forMoE models; ignored otherwise. Default False keeps the existing model-driven behavior. Default: `False` ### ObservabilityConfig[¶](#observabilityconfig "Permanent link") Configuration for observability - metrics and tracing. #### `--show-hidden-metrics-for-version`[¶](#-show-hidden-metrics-for-version "Permanent link") Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated metric has been hidden since the v0.7.0 release, you use `--show-hidden-metrics-for-version=0.7` as a temporary escape hatch while you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. #### `--otlp-traces-endpoint`[¶](#-otlp-traces-endpoint "Permanent link") Target URL to which OpenTelemetry traces will be sent. #### `--collect-detailed-traces`[¶](#-collect-detailed-traces "Permanent link") Possible choices: `all`, `model`, `worker`, `None`, `model,worker`, `model,all`, `worker,model`, `worker,all`, `all,model`, `all,worker` It makes sense to set this only if `--otlp-traces-endpoint` is set. If set, it will collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. Note that collecting detailed timing information for each request can be expensive. #### `--kv-cache-metrics`, `--no-kv-cache-metrics`[¶](#-kv-cache-metrics-no-kv-cache-metrics "Permanent link") Enable KV cache residency metrics (lifetime, idle time, reuse gaps). Uses sampling to minimize overhead. Requires log stats to be enabled (i.e., --disable-log-stats not set). Default: `False` #### `--kv-cache-metrics-sample`[¶](#-kv-cache-metrics-sample "Permanent link") Sampling rate for KV cache metrics (0.0, 1.0\]. Default 0.01 = 1%% of blocks. Default: `0.01` #### `--cudagraph-metrics`, `--no-cudagraph-metrics`[¶](#-cudagraph-metrics-no-cudagraph-metrics "Permanent link") Enable CUDA graph metrics (number of padded/unpadded tokens, runtime cudagraph dispatch modes, and their observed frequencies at every logging interval). Default: `False` #### `--enable-layerwise-nvtx-tracing`, `--no-enable-layerwise-nvtx-tracing`[¶](#-enable-layerwise-nvtx-tracing-no-enable-layerwise-nvtx-tracing "Permanent link") Enable layerwise NVTX tracing. This traces the execution of each layer or module in the model and attach information such as input/output shapes to nvtx range markers. Noted that this doesn't work with CUDA graphs enabled. Default: `False` #### `--enable-mfu-metrics`, `--no-enable-mfu-metrics`[¶](#-enable-mfu-metrics-no-enable-mfu-metrics "Permanent link") Enable Model FLOPs Utilization (MFU) metrics. Default: `False` #### `--enable-logging-iteration-details`, `--no-enable-logging-iteration-details`[¶](#-enable-logging-iteration-details-no-enable-logging-iteration-details "Permanent link") Enable detailed logging of iteration details. If set, vllm EngineCore will log iteration details This includes number of context/generation requests and tokens and the elapsed cpu time for the iteration. Default: `False` ### SchedulerConfig[¶](#schedulerconfig "Permanent link") Scheduler configuration. #### `--max-num-batched-tokens`[¶](#-max-num-batched-tokens "Permanent link") Maximum number of tokens that can be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. Parse human-readable integers like '1k', '2M', etc. Including decimal values with decimal multipliers. `Examples: - '1k' -> 1,000 - '1K' -> 1,024 - '25.6k' -> 25,600` #### `--max-num-seqs`[¶](#-max-num-seqs "Permanent link") Maximum number of sequences to be processed in a single iteration. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--max-num-partial-prefills`[¶](#-max-num-partial-prefills "Permanent link") For chunked prefill, the maximum number of sequences that can be partially prefilled concurrently. Default: `1` #### `--max-long-partial-prefills`[¶](#-max-long-partial-prefills "Permanent link") For chunked prefill, the maximum number of prompts longer than long\_prefill\_token\_threshold that will be prefilled concurrently. Setting this less than max\_num\_partial\_prefills will allow shorter prompts to jump the queue in front of longer prompts in some cases, improving latency. Default: `1` #### `--long-prefill-token-threshold`[¶](#-long-prefill-token-threshold "Permanent link") For chunked prefill, a request is considered long if the prompt is longer than this number of tokens. Default: `0` #### `--scheduling-policy`[¶](#-scheduling-policy "Permanent link") Possible choices: `fcfs`, `priority` The scheduling policy to use: - "fcfs" means first come first served, i.e. requests are handled in order of arrival. - "priority" means requests are handled based on given priority (lower value means earlier handling) and time of arrival deciding any ties). Default: `fcfs` #### `--enable-chunked-prefill`, `--no-enable-chunked-prefill`[¶](#-enable-chunked-prefill-no-enable-chunked-prefill "Permanent link") If True, prefill requests can be chunked based on the remaining `max_num_batched_tokens`. The default value here is mainly for convenience when testing. In real usage, this should be set in `EngineArgs.create_engine_config`. #### `--disable-chunked-mm-input`, `--no-disable-chunked-mm-input`[¶](#-disable-chunked-mm-input-no-disable-chunked-mm-input "Permanent link") If set to true and chunked prefill is enabled, we do not want to partially schedule a multimodal item. Only used in V1 This ensures that if a request has a mixed prompt (like text tokens TTTT followed by image tokens IIIIIIIIII) where only some image tokens can be scheduled (like TTTTIIIII, leaving IIIII), it will be scheduled as TTTT in one step and IIIIIIIIII in the next. Default: `False` #### `--scheduler-cls`[¶](#-scheduler-cls "Permanent link") The scheduler class to use. "vllm.v1.core.sched.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a class of form "mod.custom\_class". #### `--scheduler-reserve-full-isl`, `--no-scheduler-reserve-full-isl`[¶](#-scheduler-reserve-full-isl-no-scheduler-reserve-full-isl "Permanent link") If True, the scheduler checks whether the full input sequence length fits in the KV cache before admitting a new request, rather than only checking the first chunk. Prevents over-admission and KV cache thrashing with chunked prefill. Default: `True` #### `--disable-hybrid-kv-cache-manager`, `--no-disable-hybrid-kv-cache-manager`[¶](#-disable-hybrid-kv-cache-manager-no-disable-hybrid-kv-cache-manager "Permanent link") If set to True, KV cache manager will allocate the same size of KV cache for all attention layers even if there are multiple type of attention layers like full attention and sliding window attention. If set to None, the default value will be determined based on the environment and starting configuration. #### `--async-scheduling`, `--no-async-scheduling`[¶](#-async-scheduling-no-async-scheduling "Permanent link") If set to False, disable async scheduling. Async scheduling helps to avoid gaps in GPU utilization, leading to better latency and throughput. #### `--stream-interval`[¶](#-stream-interval "Permanent link") The interval (or buffer size) for streaming in terms of token length. A smaller value (1) makes streaming smoother by sending each token immediately, while a larger value (e.g., 10) reduces host overhead and may increase throughput by batching multiple tokens before sending. Default: `1` ### CompilationConfig[¶](#compilationconfig "Permanent link") Configuration for compilation. ``You must pass CompilationConfig to VLLMConfig constructor. VLLMConfig's post_init does further initialization. If used outside of the VLLMConfig, some fields will be left in an improper state. It contains PassConfig, which controls the custom fusion/transformation passes. The rest has three parts: - Top-level Compilation control: - [`mode`][vllm.config.CompilationConfig.mode] - [`debug_dump_path`][vllm.config.CompilationConfig.debug_dump_path] - [`cache_dir`][vllm.config.CompilationConfig.cache_dir] - [`backend`][vllm.config.CompilationConfig.backend] - [`custom_ops`][vllm.config.CompilationConfig.custom_ops] - [`splitting_ops`][vllm.config.CompilationConfig.splitting_ops] - [`compile_mm_encoder`][vllm.config.CompilationConfig.compile_mm_encoder] - CudaGraph capture: - [`cudagraph_mode`][vllm.config.CompilationConfig.cudagraph_mode] - [`cudagraph_capture_sizes`] [vllm.config.CompilationConfig.cudagraph_capture_sizes] - [`max_cudagraph_capture_size`] [vllm.config.CompilationConfig.max_cudagraph_capture_size] - [`cudagraph_num_of_warmups`] [vllm.config.CompilationConfig.cudagraph_num_of_warmups] - [`cudagraph_copy_inputs`] [vllm.config.CompilationConfig.cudagraph_copy_inputs] - Inductor compilation: - [`compile_sizes`][vllm.config.CompilationConfig.compile_sizes] - [`compile_ranges_endpoints`] [vllm.config.CompilationConfig.compile_ranges_endpoints] - [`inductor_compile_config`] [vllm.config.CompilationConfig.inductor_compile_config] - [`inductor_passes`][vllm.config.CompilationConfig.inductor_passes] - custom inductor passes Why we have different sizes for cudagraph and inductor: - cudagraph: a cudagraph captured for a specific size can only be used for the same size. We need to capture all the sizes we want to use. - inductor: a graph compiled by inductor for a general shape can be used for different sizes. Inductor can also compile for specific sizes, where it can have more information to optimize the graph with fully static shapes. However, we find the general shape compilation is sufficient for most cases. It might be beneficial to compile for certain small batchsizes, where inductor is good at optimizing.`` #### `--cudagraph-capture-sizes`[¶](#-cudagraph-capture-sizes "Permanent link") Sizes to capture cudagraph. - None (default): capture sizes are inferred from vllm config. - list\[int\]: capture sizes are specified as given. #### `--max-cudagraph-capture-size`[¶](#-max-cudagraph-capture-size "Permanent link") The maximum cudagraph capture size. If cudagraph\_capture\_sizes is specified, this will be set to the largest size in that list (or checked for consistency if specified). If cudagraph\_capture\_sizes is not specified, the list of sizes is generated automatically following the pattern: `[1, 2, 4] + list(range(8, 256, 8)) + list( range(256, max_cudagraph_capture_size + 1, 16))` If not specified, max\_cudagraph\_capture\_size is set to min(max\_num\_seqs\*2, 512) by default. This voids OOM in tight memory scenarios with small max\_num\_seqs, and prevents capture of many large graphs (>512) that would greatly increase startup time with limited performance benefit. ### KernelConfig[¶](#kernelconfig "Permanent link") Configuration for kernel selection and warmup behavior. #### `--ir-op-priority`[¶](#-ir-op-priority "Permanent link") vLLM IR op priority for dispatching/lowering during the forward pass. Platform defaults appended automatically during VllmConfig.**post\_init**. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.IrOpPriorityConfig Should either be a valid JSON string or JSON keys passed individually. Default: `IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[])` #### `--enable-flashinfer-autotune`, `--no-enable-flashinfer-autotune`[¶](#-enable-flashinfer-autotune-no-enable-flashinfer-autotune "Permanent link") If True, run FlashInfer autotuning during kernel warmup. #### `--moe-backend`[¶](#-moe-backend "Permanent link") Possible choices: `aiter`, `auto`, `cutlass`, `deep_gemm`, `deep_gemm_mega_moe`, `emulation`, `flashinfer_b12x`, `flashinfer_cutedsl`, `flashinfer_cutlass`, `flashinfer_trtllm`, `humming`, `marlin`, `triton`, `triton_unfused` Backend for MoE expert computation kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "triton": Use Triton-based fused MoE kernels - "deep\_gemm": Use DeepGEMM kernels (FP8 block-quantized only) - "deep\_gemm\_mega\_moe": Use DeepGEMM mega MoE kernels - "cutlass": Use vLLM CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TRTLLM-GEN kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_cutedsl": Use FlashInfer with CuteDSL kernels (FP4 only) - "flashinfer\_b12x": Use FlashInfer CuteDSL fused MoE for SM12x (RTX Pro 6000 / DGX Spark) - "marlin": Use Marlin kernels (weight-only quantization) - "humming": Use Humming Mixed Precision kernels - "triton\_unfused": Use Triton unfused MoE kernels - "aiter": Use AMD AITer kernels (ROCm only) - "emulation": use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. Default: `auto` #### `--linear-backend`[¶](#-linear-backend "Permanent link") Possible choices: `aiter`, `auto`, `conch`, `cutlass`, `deep_gemm`, `emulation`, `exllama`, `fbgemm`, `flashinfer_cudnn`, `flashinfer_cutlass`, `flashinfer_trtllm`, `machete`, `marlin`, `torch`, `triton` Backend for quantized linear layer GEMM kernels. Available options: - "auto": Automatically select the best backend based on model and hardware - "cutlass": Use CUTLASS-based kernels - "flashinfer\_cutlass": Use FlashInfer with CUTLASS kernels - "flashinfer\_trtllm": Use FlashInfer with TensorRT-LLM kernels - "flashinfer\_cudnn": Use FlashInfer with cuDNN kernels - "marlin": Use Marlin kernels - "triton": Use Triton-based kernels - "deep\_gemm": Use DeepGEMM kernels - "torch": Use PyTorch native scaled\_mm kernels - "aiter": Use AMD AITer kernels (ROCm only) - "machete": Use Machete kernels (mixed-precision) - "fbgemm": Use FBGEMM kernels - "conch": Use Conch mixed-precision kernels - "exllama": Use Exllama mixed-precision kernels - "emulation": Use slow dequant-to-BF16 emulation (for testing only) Default: `auto` ### VllmConfig[¶](#vllmconfig "Permanent link") Dataclass which contains all vllm-related configuration. This simplifies passing around the distinct configurations in the codebase. #### `--speculative-config`, `-sc`[¶](#-speculative-config-sc "Permanent link") Speculative decoding configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.SpeculativeConfig Should either be a valid JSON string or JSON keys passed individually. #### `--spec-method`[¶](#-spec-method "Permanent link") Possible choices: `custom_class`, `deepseek_mtp`, `dflash`, `draft_model`, `eagle`, `eagle3`, `ernie_mtp`, `exaone4_5_mtp`, `exaone_moe_mtp`, `extract_hidden_states`, `gemma4_mtp`, `glm4_moe_lite_mtp`, `glm4_moe_mtp`, `glm_ocr_mtp`, `hy_v3_mtp`, `longcat_flash_mtp`, `medusa`, `mimo_mtp`, `mimo_v2_mtp`, `mlp_speculator`, `mtp`, `nemotron_h_mtp`, `ngram`, `ngram_gpu`, `pangu_ultra_moe_mtp`, `qwen3_5_mtp`, `qwen3_next_mtp`, `step3p5_mtp`, `suffix`, `None` The name of the speculative method to use. If users provide and set the `model` param, the speculative method type will be detected automatically if possible, if `model` param is not provided, the method name must be provided. If using `ngram` method, the related configuration `prompt_lookup_max` and `prompt_lookup_min` should be considered. #### `--spec-model`[¶](#-spec-model "Permanent link") The name of the draft model, eagle head, or additional weights, if provided. #### `--spec-tokens`[¶](#-spec-tokens "Permanent link") The number of speculative tokens, if provided. It will default to the number in the draft model config if present, otherwise, it is required. #### `--kv-transfer-config`[¶](#-kv-transfer-config "Permanent link") The configurations for distributed KV cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kv-events-config`[¶](#-kv-events-config "Permanent link") The configurations for event publishing. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KVEventsConfig Should either be a valid JSON string or JSON keys passed individually. #### `--ec-transfer-config`[¶](#-ec-transfer-config "Permanent link") The configurations for distributed EC cache transfer. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ECTransferConfig Should either be a valid JSON string or JSON keys passed individually. #### `--compilation-config`, `-cc`[¶](#-compilation-config-cc "Permanent link") `torch.compile` and cudagraph capture configuration for the model. As a shorthand, one can append compilation arguments via -cc.parameter=argument such as `-cc.mode=3` (same as `-cc='{"mode":3}'`). You can specify the full compilation config like so: `{"mode": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]}` API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.CompilationConfig Should either be a valid JSON string or JSON keys passed individually. Default: `{'mode': None, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'ir_enable_torch_wrap': None, 'splitting_ops': None, 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': None, 'compile_ranges_endpoints': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': None, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': None, 'static_all_moe_layers': []}` #### `--attention-config`, `-ac`[¶](#-attention-config-ac "Permanent link") Attention configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.AttentionConfig Should either be a valid JSON string or JSON keys passed individually. Default: `AttentionConfig(backend=None, flash_attn_version=None, use_prefill_decode_attention=False, flash_attn_max_num_splits_for_cuda_graph=32, tq_max_kv_splits_for_cuda_graph=32, use_trtllm_attention=None, disable_flashinfer_q_quantization=False, mla_prefill_backend=None, use_prefill_query_quantization=False, use_fp4_indexer_cache=False, use_non_causal=False, flex_attn_block_m=None, flex_attn_block_n=None, flex_attn_q_block_size=None, flex_attn_kv_block_size=None)` #### `--reasoning-config`[¶](#-reasoning-config "Permanent link") The configurations for reasoning model. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ReasoningConfig Should either be a valid JSON string or JSON keys passed individually. #### `--kernel-config`[¶](#-kernel-config "Permanent link") Kernel configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.KernelConfig Should either be a valid JSON string or JSON keys passed individually. Default: `KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[], fused_add_rms_norm=[]), enable_flashinfer_autotune=None, moe_backend='auto', linear_backend='auto')` #### `--additional-config`[¶](#-additional-config "Permanent link") Additional config for specified platform. Different platforms may support different configs. Make sure the configs are valid for the platform you are using. Contents must be hashable. Default: `{}` #### `--structured-outputs-config`[¶](#-structured-outputs-config "Permanent link") Structured outputs configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.StructuredOutputsConfig Should either be a valid JSON string or JSON keys passed individually. Default: `StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False)` #### `--profiler-config`[¶](#-profiler-config "Permanent link") Profiling configuration. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.ProfilerConfig Should either be a valid JSON string or JSON keys passed individually. Default: `ProfilerConfig(profiler=None, torch_profiler_dir='', torch_profiler_with_stack=True, torch_profiler_with_flops=False, torch_profiler_use_gzip=True, torch_profiler_dump_cuda_time_total=True, torch_profiler_record_shapes=False, torch_profiler_with_memory=False, ignore_frontend=False, delay_iterations=0, max_iterations=0, warmup_iterations=0, active_iterations=5, wait_iterations=0)` #### `--optimization-level`[¶](#-optimization-level "Permanent link") The optimization level. These levels trade startup time cost for performance, with -O0 having the best startup time and -O3 having the best performance. -O2 is used by default. See OptimizationLevel for full description. Default: `2` #### `--performance-mode`[¶](#-performance-mode "Permanent link") Possible choices: `balanced`, `interactivity`, `throughput` Performance mode for runtime behavior, 'balanced' is the default. 'interactivity' favors low end-to-end per-request latency at small batch sizes (fine-grained CUDA graphs, latency-oriented kernels). 'throughput' favors aggregate tokens/sec at high concurrency (larger CUDA graphs, more aggressive batching, throughput-oriented kernels). Default: `balanced` #### `--weight-transfer-config`[¶](#-weight-transfer-config "Permanent link") The configurations for weight transfer during RL training. API docs: https://docs.vllm.ai/en/latest/api/vllm/config/#vllm.config.WeightTransferConfig Should either be a valid JSON string or JSON keys passed individually. ## [`AsyncEngineArgs`](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.AsyncEngineArgs " AsyncEngineArgs dataclass ")[¶](#asyncengineargs "Permanent link") #### `--enable-log-requests`, `--no-enable-log-requests`[¶](#-enable-log-requests-no-enable-log-requests "Permanent link") Enable logging request information, dependent on log level: - INFO: Request ID, parameters and LoRA request. - DEBUG: Prompt inputs (e.g: text, token IDs). You can set the minimum log level via `VLLM_LOGGING_LEVEL`. Default: `False` --- # page ```[](#__codelineno-0-1)logger = logging.getLogger(__name__) [](#__codelineno-0-2) [](#__codelineno-0-3)[](#__codelineno-0-4)def _resolve_rust_frontend_path() -> str | None: [](#__codelineno-0-5) """Resolve the Rust frontend binary path. [](#__codelineno-0-6) [](#__codelineno-0-7) Returns None if VLLM_USE_RUST_FRONTEND is not enabled. [](#__codelineno-0-8) When enabled, resolves VLLM_RUST_FRONTEND_PATH ("auto" by default) [](#__codelineno-0-9) to the actual binary path. [](#__codelineno-0-10) """ [](#__codelineno-0-11) use_rust = bool(int(os.environ.get("VLLM_USE_RUST_FRONTEND", "0"))) [](#__codelineno-0-12) raw = os.environ.get("VLLM_RUST_FRONTEND_PATH", "auto") [](#__codelineno-0-13) [](#__codelineno-0-14) if not use_rust: [](#__codelineno-0-15) if os.environ.get("VLLM_RUST_FRONTEND_PATH") is not None: [](#__codelineno-0-16) logger.warning( [](#__codelineno-0-17) "VLLM_RUST_FRONTEND_PATH is set but VLLM_USE_RUST_FRONTEND " [](#__codelineno-0-18) "is not enabled. The Rust frontend will not be used. " [](#__codelineno-0-19) "Set VLLM_USE_RUST_FRONTEND=1 to enable it." [](#__codelineno-0-20) ) [](#__codelineno-0-21) return None [](#__codelineno-0-22) [](#__codelineno-0-23) if raw.lower() in ("auto", "1", "true"): [](#__codelineno-0-24) pkg_dir = os.path.dirname(os.path.abspath(__file__)) [](#__codelineno-0-25) candidate = os.path.join(pkg_dir, "vllm-rs") [](#__codelineno-0-26) if os.path.isfile(candidate) and os.access(candidate, os.X_OK): [](#__codelineno-0-27) return candidate [](#__codelineno-0-28) [](#__codelineno-0-29) raise FileNotFoundError( [](#__codelineno-0-30) "VLLM_RUST_FRONTEND_PATH=auto but the vllm-rs binary was " [](#__codelineno-0-31) f"not found at {candidate}. " [](#__codelineno-0-32) "Build with setuptools-rust or set the path explicitly." [](#__codelineno-0-33) ) [](#__codelineno-0-34) return raw [](#__codelineno-0-35) [](#__codelineno-0-36)[](#__codelineno-0-37)environment_variables: dict[str, Callable[[], Any]] = { [](#__codelineno-0-38) # ================== Installation Time Env Vars ================== [](#__codelineno-0-39) # Target device of vLLM, supporting [cuda (by default), [](#__codelineno-0-40) # rocm, cpu] [](#__codelineno-0-41) "VLLM_TARGET_DEVICE": lambda: os.getenv("VLLM_TARGET_DEVICE", "cuda").lower(), [](#__codelineno-0-42) # Main CUDA version of vLLM. This follows PyTorch but can be overridden. [](#__codelineno-0-43) "VLLM_MAIN_CUDA_VERSION": lambda: ( [](#__codelineno-0-44) os.getenv("VLLM_MAIN_CUDA_VERSION", "").lower() or "13.0" [](#__codelineno-0-45) ), [](#__codelineno-0-46) # Controls PyTorch float32 matmul precision mode within vLLM workers. [](#__codelineno-0-47) # Valid options mirror torch.set_float32_matmul_precision [](#__codelineno-0-48) "VLLM_FLOAT32_MATMUL_PRECISION": env_with_choices( [](#__codelineno-0-49) "VLLM_FLOAT32_MATMUL_PRECISION", [](#__codelineno-0-50) "highest", [](#__codelineno-0-51) ["highest", "high", "medium"], [](#__codelineno-0-52) case_sensitive=False, [](#__codelineno-0-53) ), [](#__codelineno-0-54) # Enable batch-invariant mode: deterministic results regardless of [](#__codelineno-0-55) # batch composition. Requires NVIDIA GPU with compute capability >= 9.0. [](#__codelineno-0-56) "VLLM_BATCH_INVARIANT": lambda: bool(int(os.getenv("VLLM_BATCH_INVARIANT", "0"))), [](#__codelineno-0-57) # Use tensor descriptors for Q/K/V loads and output stores in the [](#__codelineno-0-58) # Triton unified-attention kernel. Enables HW 2D block reads on [](#__codelineno-0-59) # Intel Xe2/Xe3; the non-TD branch is dead-code-eliminated at Triton [](#__codelineno-0-60) # compile time so other platforms see no overhead. Tri-state override: [](#__codelineno-0-61) # unset (default) lets the `triton_attn` backend auto-select per [](#__codelineno-0-62) # platform (currently auto-enabled on XPU only); ``1`` forces TD on; [](#__codelineno-0-63) # ``0`` forces TD off. Useful for A/B benchmarking the TD path. [](#__codelineno-0-64) "VLLM_TRITON_ATTN_USE_TD": lambda: {"1": True, "0": False}.get( [](#__codelineno-0-65) os.getenv("VLLM_TRITON_ATTN_USE_TD", "").strip() [](#__codelineno-0-66) ), [](#__codelineno-0-67) # Maximum number of compilation jobs to run in parallel. [](#__codelineno-0-68) # By default this is the number of CPUs [](#__codelineno-0-69) "MAX_JOBS": lambda: os.getenv("MAX_JOBS", None), [](#__codelineno-0-70) # Number of threads to use for nvcc [](#__codelineno-0-71) # By default this is 1. [](#__codelineno-0-72) # If set, `MAX_JOBS` will be reduced to avoid oversubscribing the CPU. [](#__codelineno-0-73) "NVCC_THREADS": lambda: os.getenv("NVCC_THREADS", None), [](#__codelineno-0-74) # If set, vllm will use precompiled native binaries (*.so and vllm-rs). [](#__codelineno-0-75) "VLLM_USE_PRECOMPILED": lambda: ( [](#__codelineno-0-76) os.environ.get("VLLM_USE_PRECOMPILED", "").strip().lower() in ("1", "true") [](#__codelineno-0-77) or bool(os.environ.get("VLLM_PRECOMPILED_WHEEL_LOCATION")) [](#__codelineno-0-78) ), [](#__codelineno-0-79) # If set, vllm will use the precompiled Rust frontend binary (vllm-rs). [](#__codelineno-0-80) "VLLM_USE_PRECOMPILED_RUST": lambda: ( [](#__codelineno-0-81) os.environ.get("VLLM_USE_PRECOMPILED_RUST", "").strip().lower() in ("1", "true") [](#__codelineno-0-82) ), [](#__codelineno-0-83) # If set, skip adding +precompiled suffix to version string [](#__codelineno-0-84) "VLLM_SKIP_PRECOMPILED_VERSION_SUFFIX": lambda: bool( [](#__codelineno-0-85) int(os.environ.get("VLLM_SKIP_PRECOMPILED_VERSION_SUFFIX", "0")) [](#__codelineno-0-86) ), [](#__codelineno-0-87) # Used to mark that setup.py is running in a Docker build context, [](#__codelineno-0-88) # in order to force the use of precompiled binaries. [](#__codelineno-0-89) "VLLM_DOCKER_BUILD_CONTEXT": lambda: ( [](#__codelineno-0-90) os.environ.get("VLLM_DOCKER_BUILD_CONTEXT", "").strip().lower() in ("1", "true") [](#__codelineno-0-91) ), [](#__codelineno-0-92) # CMake build type [](#__codelineno-0-93) # If not set, defaults to "Debug" or "RelWithDebInfo" [](#__codelineno-0-94) # Available options: "Debug", "Release", "RelWithDebInfo" [](#__codelineno-0-95) "CMAKE_BUILD_TYPE": env_with_choices( [](#__codelineno-0-96) "CMAKE_BUILD_TYPE", None, ["Debug", "Release", "RelWithDebInfo"] [](#__codelineno-0-97) ), [](#__codelineno-0-98) # If set, vllm will print verbose logs during installation [](#__codelineno-0-99) "VERBOSE": lambda: bool(int(os.getenv("VERBOSE", "0"))), [](#__codelineno-0-100) # Root directory for vLLM configuration files [](#__codelineno-0-101) # Defaults to `~/.config/vllm` unless `XDG_CONFIG_HOME` is set [](#__codelineno-0-102) # Note that this not only affects how vllm finds its configuration files [](#__codelineno-0-103) # during runtime, but also affects how vllm installs its configuration [](#__codelineno-0-104) # files during **installation**. [](#__codelineno-0-105) "VLLM_CONFIG_ROOT": lambda: os.path.expanduser( [](#__codelineno-0-106) os.getenv( [](#__codelineno-0-107) "VLLM_CONFIG_ROOT", [](#__codelineno-0-108) os.path.join(get_default_config_root(), "vllm"), [](#__codelineno-0-109) ) [](#__codelineno-0-110) ), [](#__codelineno-0-111) # ================== Runtime Env Vars ================== [](#__codelineno-0-112) # Root directory for vLLM cache files [](#__codelineno-0-113) # Defaults to `~/.cache/vllm` unless `XDG_CACHE_HOME` is set [](#__codelineno-0-114) "VLLM_CACHE_ROOT": lambda: os.path.expanduser( [](#__codelineno-0-115) os.getenv( [](#__codelineno-0-116) "VLLM_CACHE_ROOT", [](#__codelineno-0-117) os.path.join(get_default_cache_root(), "vllm"), [](#__codelineno-0-118) ) [](#__codelineno-0-119) ), [](#__codelineno-0-120) # used in distributed environment to determine the ip address [](#__codelineno-0-121) # of the current node, when the node has multiple network interfaces. [](#__codelineno-0-122) # If you are using multi-node inference, you should set this differently [](#__codelineno-0-123) # on each node. [](#__codelineno-0-124) "VLLM_HOST_IP": lambda: os.getenv("VLLM_HOST_IP", ""), [](#__codelineno-0-125) # used in distributed environment to manually set the communication port [](#__codelineno-0-126) # Note: if VLLM_PORT is set, and some code asks for multiple ports, the [](#__codelineno-0-127) # VLLM_PORT will be used as the first port, and the rest will be generated [](#__codelineno-0-128) # by incrementing the VLLM_PORT value. [](#__codelineno-0-129) "VLLM_PORT": get_vllm_port, [](#__codelineno-0-130) # path used for ipc when the frontend api server is running in [](#__codelineno-0-131) # multi-processing mode to communicate with the backend engine process. [](#__codelineno-0-132) "VLLM_RPC_BASE_PATH": lambda: os.getenv( [](#__codelineno-0-133) "VLLM_RPC_BASE_PATH", tempfile.gettempdir() [](#__codelineno-0-134) ), [](#__codelineno-0-135) # If true, will load models from ModelScope instead of Hugging Face Hub. [](#__codelineno-0-136) # note that the value is true or false, not numbers [](#__codelineno-0-137) "VLLM_USE_MODELSCOPE": lambda: ( [](#__codelineno-0-138) os.environ.get("VLLM_USE_MODELSCOPE", "False").lower() == "true" [](#__codelineno-0-139) ), [](#__codelineno-0-140) # If true, replace the Rust BPE backend that powers HF fast tokenizers [](#__codelineno-0-141) # with the `fastokens` (https://github.com/crusoecloud/fastokens) shim. [](#__codelineno-0-142) # Applies to any tokenizer mode that loads an HF fast tokenizer [](#__codelineno-0-143) # (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). The `fastokens` [](#__codelineno-0-144) # Python package must be installed. [](#__codelineno-0-145) "VLLM_USE_FASTOKENS": lambda: bool(int(os.getenv("VLLM_USE_FASTOKENS", "0"))), [](#__codelineno-0-146) # Interval in seconds to log a warning message when the ring buffer is full [](#__codelineno-0-147) "VLLM_RINGBUFFER_WARNING_INTERVAL": lambda: int( [](#__codelineno-0-148) os.environ.get("VLLM_RINGBUFFER_WARNING_INTERVAL", "60") [](#__codelineno-0-149) ), [](#__codelineno-0-150) # path to cudatoolkit home directory, under which should be bin, include, [](#__codelineno-0-151) # and lib directories. [](#__codelineno-0-152) "CUDA_HOME": lambda: os.environ.get("CUDA_HOME", None), [](#__codelineno-0-153) # Path to the NCCL library file. It is needed because nccl>=2.19 brought [](#__codelineno-0-154) # by PyTorch contains a bug: https://github.com/NVIDIA/nccl/issues/1234 [](#__codelineno-0-155) "VLLM_NCCL_SO_PATH": lambda: os.environ.get("VLLM_NCCL_SO_PATH", None), [](#__codelineno-0-156) # when `VLLM_NCCL_SO_PATH` is not set, vllm will try to find the nccl [](#__codelineno-0-157) # library file in the locations specified by `LD_LIBRARY_PATH` [](#__codelineno-0-158) "LD_LIBRARY_PATH": lambda: os.environ.get("LD_LIBRARY_PATH", None), [](#__codelineno-0-159) # flag to control the chunk size (in MB) for sleeping memory allocations under ROCm [](#__codelineno-0-160) "VLLM_ROCM_SLEEP_MEM_CHUNK_SIZE": lambda: int( [](#__codelineno-0-161) os.environ.get("VLLM_ROCM_SLEEP_MEM_CHUNK_SIZE", "256") [](#__codelineno-0-162) ), [](#__codelineno-0-163) # Feature flag to enable/disable Inductor standalone compile. [](#__codelineno-0-164) # In torch <= 2.7 we ignore this flag; in torch >= 2.9 this is [](#__codelineno-0-165) # enabled by default. [](#__codelineno-0-166) "VLLM_USE_STANDALONE_COMPILE": lambda: ( [](#__codelineno-0-167) os.environ.get("VLLM_USE_STANDALONE_COMPILE", "1") == "1" [](#__codelineno-0-168) ), [](#__codelineno-0-169) # Inductor's pre-grad passes don't do anything for vLLM. [](#__codelineno-0-170) # The pre-grad passes get run even on cache-hit and negatively impact [](#__codelineno-0-171) # vllm cold compile times by O(1s) [](#__codelineno-0-172) # Can remove this after the following issue gets fixed [](#__codelineno-0-173) # TODO(luka): maybe_inplace requires this [](#__codelineno-0-174) # https://github.com/pytorch/pytorch/issues/174502 [](#__codelineno-0-175) "VLLM_ENABLE_PREGRAD_PASSES": lambda: ( [](#__codelineno-0-176) os.environ.get("VLLM_ENABLE_PREGRAD_PASSES", "1") == "1" [](#__codelineno-0-177) ), [](#__codelineno-0-178) # Experimental: breakable cudagraph does not rely on torch.compile [](#__codelineno-0-179) "VLLM_USE_BREAKABLE_CUDAGRAPH": lambda: ( [](#__codelineno-0-180) os.environ.get("VLLM_USE_BREAKABLE_CUDAGRAPH", "0") == "1" [](#__codelineno-0-181) ), [](#__codelineno-0-182) # Debug pattern matching inside custom passes. [](#__codelineno-0-183) # Should be set to the fx.Node name (e.g. 'getitem_34' or 'scaled_mm_3'). [](#__codelineno-0-184) "VLLM_PATTERN_MATCH_DEBUG": lambda: os.environ.get( [](#__codelineno-0-185) "VLLM_PATTERN_MATCH_DEBUG", None [](#__codelineno-0-186) ), [](#__codelineno-0-187) # Dump fx graphs to the given directory. [](#__codelineno-0-188) # It will override CompilationConfig.debug_dump_path if set. [](#__codelineno-0-189) "VLLM_DEBUG_DUMP_PATH": lambda: os.environ.get("VLLM_DEBUG_DUMP_PATH", None), [](#__codelineno-0-190) # Feature flag to enable/disable AOT compilation. This will ensure [](#__codelineno-0-191) # compilation is done in warmup phase and the compilation will be [](#__codelineno-0-192) # reused in subsequent calls. [](#__codelineno-0-193) "VLLM_USE_AOT_COMPILE": use_aot_compile, [](#__codelineno-0-194) # Feature flag to enable/disable bytecode in [](#__codelineno-0-195) # TorchCompileWithNoGuardsWrapper. [](#__codelineno-0-196) "VLLM_USE_BYTECODE_HOOK": lambda: bool( [](#__codelineno-0-197) int(os.environ.get("VLLM_USE_BYTECODE_HOOK", "1")) [](#__codelineno-0-198) ), [](#__codelineno-0-199) # Force vllm to always load AOT compiled models from disk. Failure [](#__codelineno-0-200) # to load will result in a hard error when this is enabled. [](#__codelineno-0-201) # Will be ignored when VLLM_USE_AOT_COMPILE is disabled. [](#__codelineno-0-202) "VLLM_FORCE_AOT_LOAD": lambda: os.environ.get("VLLM_FORCE_AOT_LOAD", "0") == "1", [](#__codelineno-0-203) # Enable loading compiled models directly from cached standalone compile artifacts [](#__codelineno-0-204) # without re-splitting graph modules. This reduces overhead during model [](#__codelineno-0-205) # loading by using reconstruct_serializable_fn_from_mega_artifact. [](#__codelineno-0-206) "VLLM_USE_MEGA_AOT_ARTIFACT": use_mega_aot_artifact, [](#__codelineno-0-207) # local rank of the process in the distributed setting, used to determine [](#__codelineno-0-208) # the GPU device id [](#__codelineno-0-209) "LOCAL_RANK": lambda: int(os.environ.get("LOCAL_RANK", "0")), [](#__codelineno-0-210) # used to control the visible devices in the distributed setting [](#__codelineno-0-211) "CUDA_VISIBLE_DEVICES": lambda: os.environ.get("CUDA_VISIBLE_DEVICES", None), [](#__codelineno-0-212) # timeout for each iteration in the engine [](#__codelineno-0-213) "VLLM_ENGINE_ITERATION_TIMEOUT_S": lambda: int( [](#__codelineno-0-214) os.environ.get("VLLM_ENGINE_ITERATION_TIMEOUT_S", "60") [](#__codelineno-0-215) ), [](#__codelineno-0-216) # Timeout in seconds for waiting for engine cores to become ready [](#__codelineno-0-217) # during startup. Default is 600 seconds (10 minutes). [](#__codelineno-0-218) "VLLM_ENGINE_READY_TIMEOUT_S": lambda: int( [](#__codelineno-0-219) os.environ.get("VLLM_ENGINE_READY_TIMEOUT_S", "600") [](#__codelineno-0-220) ), [](#__codelineno-0-221) # API key for vLLM API server [](#__codelineno-0-222) "VLLM_API_KEY": lambda: os.environ.get("VLLM_API_KEY", None), [](#__codelineno-0-223) # Whether to log responses from API Server for debugging [](#__codelineno-0-224) "VLLM_DEBUG_LOG_API_SERVER_RESPONSE": lambda: ( [](#__codelineno-0-225) os.environ.get("VLLM_DEBUG_LOG_API_SERVER_RESPONSE", "False").lower() == "true" [](#__codelineno-0-226) ), [](#__codelineno-0-227) # S3 access information, used for tensorizer to load model from S3 [](#__codelineno-0-228) "S3_ACCESS_KEY_ID": lambda: os.environ.get("S3_ACCESS_KEY_ID", None), [](#__codelineno-0-229) "S3_SECRET_ACCESS_KEY": lambda: os.environ.get("S3_SECRET_ACCESS_KEY", None), [](#__codelineno-0-230) "S3_ENDPOINT_URL": lambda: os.environ.get("S3_ENDPOINT_URL", None), [](#__codelineno-0-231) # Usage stats collection [](#__codelineno-0-232) "VLLM_USAGE_STATS_SERVER": lambda: os.environ.get( [](#__codelineno-0-233) "VLLM_USAGE_STATS_SERVER", "https://stats.vllm.ai" [](#__codelineno-0-234) ), [](#__codelineno-0-235) "VLLM_NO_USAGE_STATS": lambda: os.environ.get("VLLM_NO_USAGE_STATS", "0") == "1", [](#__codelineno-0-236) "VLLM_DO_NOT_TRACK": lambda: ( [](#__codelineno-0-237) ( [](#__codelineno-0-238) os.environ.get("VLLM_DO_NOT_TRACK", None) [](#__codelineno-0-239) or os.environ.get("DO_NOT_TRACK", None) [](#__codelineno-0-240) or "0" [](#__codelineno-0-241) ) [](#__codelineno-0-242) == "1" [](#__codelineno-0-243) ), [](#__codelineno-0-244) "VLLM_USAGE_SOURCE": lambda: os.environ.get("VLLM_USAGE_SOURCE", "production"), [](#__codelineno-0-245) # Logging configuration [](#__codelineno-0-246) # If set to 0, vllm will not configure logging [](#__codelineno-0-247) # If set to 1, vllm will configure logging using the default configuration [](#__codelineno-0-248) # or the configuration file specified by VLLM_LOGGING_CONFIG_PATH [](#__codelineno-0-249) "VLLM_CONFIGURE_LOGGING": lambda: bool( [](#__codelineno-0-250) int(os.getenv("VLLM_CONFIGURE_LOGGING", "1")) [](#__codelineno-0-251) ), [](#__codelineno-0-252) "VLLM_LOGGING_CONFIG_PATH": lambda: os.getenv("VLLM_LOGGING_CONFIG_PATH"), [](#__codelineno-0-253) # this is used for configuring the default logging level [](#__codelineno-0-254) "VLLM_LOGGING_LEVEL": lambda: os.getenv("VLLM_LOGGING_LEVEL", "INFO").upper(), [](#__codelineno-0-255) # this is used for configuring the default logging stream [](#__codelineno-0-256) "VLLM_LOGGING_STREAM": lambda: os.getenv("VLLM_LOGGING_STREAM", "ext://sys.stdout"), [](#__codelineno-0-257) # if set, VLLM_LOGGING_PREFIX will be prepended to all log messages [](#__codelineno-0-258) "VLLM_LOGGING_PREFIX": lambda: os.getenv("VLLM_LOGGING_PREFIX", ""), [](#__codelineno-0-259) # Controls colored logging output. Options: "auto" (default, colors when terminal), [](#__codelineno-0-260) # "1" (always use colors), "0" (never use colors) [](#__codelineno-0-261) "VLLM_LOGGING_COLOR": lambda: os.getenv("VLLM_LOGGING_COLOR", "auto"), [](#__codelineno-0-262) # Standard unix flag for disabling ANSI color codes [](#__codelineno-0-263) "NO_COLOR": lambda: os.getenv("NO_COLOR", "0") != "0", [](#__codelineno-0-264) # If set, vllm will log stats at this interval in seconds [](#__codelineno-0-265) # If not set, vllm will log stats every 10 seconds. [](#__codelineno-0-266) "VLLM_LOG_STATS_INTERVAL": lambda: ( [](#__codelineno-0-267) val [](#__codelineno-0-268) if (val := float(os.getenv("VLLM_LOG_STATS_INTERVAL", "10."))) > 0.0 [](#__codelineno-0-269) else 10.0 [](#__codelineno-0-270) ), [](#__codelineno-0-271) # Trace function calls [](#__codelineno-0-272) # If set to 1, vllm will trace function calls [](#__codelineno-0-273) # Useful for debugging [](#__codelineno-0-274) "VLLM_TRACE_FUNCTION": lambda: int(os.getenv("VLLM_TRACE_FUNCTION", "0")), [](#__codelineno-0-275) # Whether to use the FlashInfer top-k / top-p sampler on CUDA. Enabled [](#__codelineno-0-276) # by default when the hardware supports it — set to 0 to opt out [](#__codelineno-0-277) # explicitly, which forces the PyTorch-native (Triton for bs>=8) path. [](#__codelineno-0-278) "VLLM_USE_FLASHINFER_SAMPLER": lambda: ( [](#__codelineno-0-279) bool(int(os.environ["VLLM_USE_FLASHINFER_SAMPLER"])) [](#__codelineno-0-280) if "VLLM_USE_FLASHINFER_SAMPLER" in os.environ [](#__codelineno-0-281) else True [](#__codelineno-0-282) ), [](#__codelineno-0-283) # Pipeline stage partition strategy [](#__codelineno-0-284) "VLLM_PP_LAYER_PARTITION": lambda: os.getenv("VLLM_PP_LAYER_PARTITION", None), [](#__codelineno-0-285) # (CPU backend only) CPU key-value cache space. [](#__codelineno-0-286) # default is None and will be set as 4 GB [](#__codelineno-0-287) "VLLM_CPU_KVCACHE_SPACE": lambda: ( [](#__codelineno-0-288) int(os.getenv("VLLM_CPU_KVCACHE_SPACE", "0")) [](#__codelineno-0-289) if "VLLM_CPU_KVCACHE_SPACE" in os.environ [](#__codelineno-0-290) else None [](#__codelineno-0-291) ), [](#__codelineno-0-292) # (CPU backend only) CPU core ids bound by OpenMP threads, e.g., "0-31", [](#__codelineno-0-293) # "0,1,2", "0-31,33". CPU cores of different ranks are separated by '|'. [](#__codelineno-0-294) "VLLM_CPU_OMP_THREADS_BIND": lambda: os.getenv("VLLM_CPU_OMP_THREADS_BIND", "auto"), [](#__codelineno-0-295) # (CPU backend only) CPU cores not used by OMP threads . [](#__codelineno-0-296) # Those CPU cores will not be used by OMP threads of a rank. [](#__codelineno-0-297) "VLLM_CPU_NUM_OF_RESERVED_CPU": lambda: ( [](#__codelineno-0-298) int(os.getenv("VLLM_CPU_NUM_OF_RESERVED_CPU", "0")) [](#__codelineno-0-299) if "VLLM_CPU_NUM_OF_RESERVED_CPU" in os.environ [](#__codelineno-0-300) else None [](#__codelineno-0-301) ), [](#__codelineno-0-302) # (CPU backend only) whether to use SGL kernels, optimized for small batch. [](#__codelineno-0-303) "VLLM_CPU_SGL_KERNEL": lambda: bool(int(os.getenv("VLLM_CPU_SGL_KERNEL", "0"))), [](#__codelineno-0-304) # (CPU backend only) whether to enable attention spilt KV. [](#__codelineno-0-305) "VLLM_CPU_ATTN_SPLIT_KV": lambda: bool( [](#__codelineno-0-306) int(os.getenv("VLLM_CPU_ATTN_SPLIT_KV", "1")) [](#__codelineno-0-307) ), [](#__codelineno-0-308) # (Zen CPU backend) eagerly prepack weights into ZenDNN blocked layout [](#__codelineno-0-309) # at model load time. Eliminates per-inference layout conversion overhead. [](#__codelineno-0-310) "VLLM_ZENTORCH_WEIGHT_PREPACK": lambda: bool( [](#__codelineno-0-311) int(os.getenv("VLLM_ZENTORCH_WEIGHT_PREPACK", "1")) [](#__codelineno-0-312) ), [](#__codelineno-0-313) # (CPU backend only) whether to use SGLang INT4 W4A8 kernels for AWQ. [](#__codelineno-0-314) "VLLM_CPU_INT4_W4A8": lambda: bool(int(os.getenv("VLLM_CPU_INT4_W4A8", "1"))), [](#__codelineno-0-315) # If the env var is set, Ray Compiled Graph uses the specified [](#__codelineno-0-316) # channel type to communicate between workers belonging to [](#__codelineno-0-317) # different pipeline-parallel stages. [](#__codelineno-0-318) # Available options: [](#__codelineno-0-319) # - "auto": use the default channel type [](#__codelineno-0-320) # - "nccl": use NCCL for communication [](#__codelineno-0-321) # - "shm": use shared memory and gRPC for communication [](#__codelineno-0-322) "VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE": env_with_choices( [](#__codelineno-0-323) "VLLM_USE_RAY_COMPILED_DAG_CHANNEL_TYPE", "auto", ["auto", "nccl", "shm"] [](#__codelineno-0-324) ), [](#__codelineno-0-325) # If the env var is set, it enables GPU communication overlap [](#__codelineno-0-326) # (experimental feature) in Ray's Compiled Graph. [](#__codelineno-0-327) "VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM": lambda: bool( [](#__codelineno-0-328) int(os.getenv("VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM", "0")) [](#__codelineno-0-329) ), [](#__codelineno-0-330) # If the env var is set, it uses a Ray Communicator wrapping [](#__codelineno-0-331) # vLLM's pipeline parallelism communicator to interact with Ray's [](#__codelineno-0-332) # Compiled Graph. Otherwise, it uses Ray's NCCL communicator. [](#__codelineno-0-333) "VLLM_USE_RAY_WRAPPED_PP_COMM": lambda: bool( [](#__codelineno-0-334) int(os.getenv("VLLM_USE_RAY_WRAPPED_PP_COMM", "1")) [](#__codelineno-0-335) ), [](#__codelineno-0-336) # When True and distributed_executor_backend="ray", use RayExecutorV2 [](#__codelineno-0-337) # (MQ-based) instead of RayDistributedExecutor (compiled-graph backend). [](#__codelineno-0-338) "VLLM_USE_RAY_V2_EXECUTOR_BACKEND": lambda: bool( [](#__codelineno-0-339) int(os.getenv("VLLM_USE_RAY_V2_EXECUTOR_BACKEND", "1")) [](#__codelineno-0-340) ), [](#__codelineno-0-341) # When True, GroupCoordinator constructs its CPU/device subgroups via [](#__codelineno-0-342) # ``torch.distributed.split_group(backend=...)`` [](#__codelineno-0-343) # and ``init_distributed_environment`` initializes the default PG with [](#__codelineno-0-344) # mixed ``cpu:gloo,cuda:nccl`` backend + eager ``device_id`` binding. [](#__codelineno-0-345) "VLLM_DISTRIBUTED_USE_SPLIT_GROUP": lambda: bool( [](#__codelineno-0-346) int(os.getenv("VLLM_DISTRIBUTED_USE_SPLIT_GROUP", "0")) [](#__codelineno-0-347) ), [](#__codelineno-0-348) # Use dedicated multiprocess context for workers. [](#__codelineno-0-349) # Both spawn and fork work [](#__codelineno-0-350) "VLLM_WORKER_MULTIPROC_METHOD": env_with_choices( [](#__codelineno-0-351) "VLLM_WORKER_MULTIPROC_METHOD", "fork", ["spawn", "fork"] [](#__codelineno-0-352) ), [](#__codelineno-0-353) # Path to the cache for storing downloaded assets [](#__codelineno-0-354) "VLLM_ASSETS_CACHE": lambda: os.path.expanduser( [](#__codelineno-0-355) os.getenv( [](#__codelineno-0-356) "VLLM_ASSETS_CACHE", [](#__codelineno-0-357) os.path.join(get_default_cache_root(), "vllm", "assets"), [](#__codelineno-0-358) ) [](#__codelineno-0-359) ), [](#__codelineno-0-360) # If the env var is set, we will clean model file in [](#__codelineno-0-361) # this path $VLLM_ASSETS_CACHE/model_streamer/$model_name [](#__codelineno-0-362) "VLLM_ASSETS_CACHE_MODEL_CLEAN": lambda: bool( [](#__codelineno-0-363) int(os.getenv("VLLM_ASSETS_CACHE_MODEL_CLEAN", "0")) [](#__codelineno-0-364) ), [](#__codelineno-0-365) # Timeout for fetching images when serving multimodal models [](#__codelineno-0-366) # Default is 5 seconds [](#__codelineno-0-367) "VLLM_IMAGE_FETCH_TIMEOUT": lambda: int(os.getenv("VLLM_IMAGE_FETCH_TIMEOUT", "5")), [](#__codelineno-0-368) # Timeout for fetching videos when serving multimodal models [](#__codelineno-0-369) # Default is 30 seconds [](#__codelineno-0-370) "VLLM_VIDEO_FETCH_TIMEOUT": lambda: int( [](#__codelineno-0-371) os.getenv("VLLM_VIDEO_FETCH_TIMEOUT", "30") [](#__codelineno-0-372) ), [](#__codelineno-0-373) # Timeout for fetching audio when serving multimodal models [](#__codelineno-0-374) # Default is 10 seconds [](#__codelineno-0-375) "VLLM_AUDIO_FETCH_TIMEOUT": lambda: int( [](#__codelineno-0-376) os.getenv("VLLM_AUDIO_FETCH_TIMEOUT", "10") [](#__codelineno-0-377) ), [](#__codelineno-0-378) # Directory for caching media downloads (images, video, audio fetched [](#__codelineno-0-379) # from URLs during inference). Empty string disables caching. [](#__codelineno-0-380) "VLLM_MEDIA_CACHE": lambda: os.getenv("VLLM_MEDIA_CACHE", ""), [](#__codelineno-0-381) # Maximum cache size in MB. When exceeded, least-recently-used entries [](#__codelineno-0-382) # are evicted. Default is 5120 (5 GB). [](#__codelineno-0-383) "VLLM_MEDIA_CACHE_MAX_SIZE_MB": lambda: int( [](#__codelineno-0-384) os.getenv("VLLM_MEDIA_CACHE_MAX_SIZE_MB", "5120") [](#__codelineno-0-385) ), [](#__codelineno-0-386) # Time-to-live in hours for cached media files. Entries older than this [](#__codelineno-0-387) # are evicted regardless of cache size. Default is 24 hours. [](#__codelineno-0-388) "VLLM_MEDIA_CACHE_TTL_HOURS": lambda: float( [](#__codelineno-0-389) os.getenv("VLLM_MEDIA_CACHE_TTL_HOURS", "24") [](#__codelineno-0-390) ), [](#__codelineno-0-391) # Maximum number of retries for fetching media (images, audio, video) [](#__codelineno-0-392) # from URLs. Each retry quadruples the timeout. Default is 3. [](#__codelineno-0-393) "VLLM_MEDIA_FETCH_MAX_RETRIES": lambda: int( [](#__codelineno-0-394) os.getenv("VLLM_MEDIA_FETCH_MAX_RETRIES", "3") [](#__codelineno-0-395) ), [](#__codelineno-0-396) # Whether to allow HTTP redirects when fetching from media URLs. [](#__codelineno-0-397) # Default to True [](#__codelineno-0-398) "VLLM_MEDIA_URL_ALLOW_REDIRECTS": lambda: bool( [](#__codelineno-0-399) int(os.getenv("VLLM_MEDIA_URL_ALLOW_REDIRECTS", "1")) [](#__codelineno-0-400) ), [](#__codelineno-0-401) # Max number of workers for the thread pool handling [](#__codelineno-0-402) # media bytes loading. Set to 1 to disable parallel processing. [](#__codelineno-0-403) # Default is 8 [](#__codelineno-0-404) "VLLM_MEDIA_LOADING_THREAD_COUNT": lambda: int( [](#__codelineno-0-405) os.getenv("VLLM_MEDIA_LOADING_THREAD_COUNT", "8") [](#__codelineno-0-406) ), [](#__codelineno-0-407) # Maximum filesize in MB for a single audio file when processing [](#__codelineno-0-408) # speech-to-text requests. Files larger than this will be rejected. [](#__codelineno-0-409) # Default is 25 MB [](#__codelineno-0-410) "VLLM_MAX_AUDIO_CLIP_FILESIZE_MB": lambda: int( [](#__codelineno-0-411) os.getenv("VLLM_MAX_AUDIO_CLIP_FILESIZE_MB", "25") [](#__codelineno-0-412) ), [](#__codelineno-0-413) # Backend for Video IO — selects the frame-sampling algorithm. [](#__codelineno-0-414) # - "opencv": uniform sampling. [](#__codelineno-0-415) # - "opencv_dynamic": duration-aware dynamic sampling. [](#__codelineno-0-416) # - "identity": returns raw video bytes for model processor to handle. [](#__codelineno-0-417) # [](#__codelineno-0-418) # Custom backend implementations can be registered [](#__codelineno-0-419) # via `@VIDEO_LOADER_REGISTRY.register("my_custom_video_loader")` and [](#__codelineno-0-420) # imported at runtime. [](#__codelineno-0-421) # If a non-existing backend is used, an AssertionError will be thrown. [](#__codelineno-0-422) "VLLM_VIDEO_LOADER_BACKEND": lambda: os.getenv( [](#__codelineno-0-423) "VLLM_VIDEO_LOADER_BACKEND", "opencv" [](#__codelineno-0-424) ), [](#__codelineno-0-425) # Media connector implementation. [](#__codelineno-0-426) # - "http": Default connector that supports fetching media via HTTP. [](#__codelineno-0-427) # [](#__codelineno-0-428) # Custom implementations can be registered [](#__codelineno-0-429) # via `@MEDIA_CONNECTOR_REGISTRY.register("my_custom_media_connector")` and [](#__codelineno-0-430) # imported at runtime. [](#__codelineno-0-431) # If a non-existing backend is used, an AssertionError will be thrown. [](#__codelineno-0-432) "VLLM_MEDIA_CONNECTOR": lambda: os.getenv("VLLM_MEDIA_CONNECTOR", "http"), [](#__codelineno-0-433) # Hash algorithm for multimodal content hashing. [](#__codelineno-0-434) # - "blake3": Default, fast cryptographic hash (not FIPS 140-3 compliant) [](#__codelineno-0-435) # - "sha256": FIPS 140-3 compliant, widely supported [](#__codelineno-0-436) # - "sha512": FIPS 140-3 compliant, faster on 64-bit systems [](#__codelineno-0-437) # Use sha256 or sha512 for FIPS compliance in government/enterprise deployments [](#__codelineno-0-438) "VLLM_MM_HASHER_ALGORITHM": env_with_choices( [](#__codelineno-0-439) "VLLM_MM_HASHER_ALGORITHM", [](#__codelineno-0-440) "blake3", [](#__codelineno-0-441) ["blake3", "sha256", "sha512"], [](#__codelineno-0-442) case_sensitive=False, [](#__codelineno-0-443) ), [](#__codelineno-0-444) # Path to the XLA persistent cache directory. [](#__codelineno-0-445) # Only used for XLA devices such as TPUs. [](#__codelineno-0-446) "VLLM_XLA_CACHE_PATH": lambda: os.path.expanduser( [](#__codelineno-0-447) os.getenv( [](#__codelineno-0-448) "VLLM_XLA_CACHE_PATH", [](#__codelineno-0-449) os.path.join(get_default_cache_root(), "vllm", "xla_cache"), [](#__codelineno-0-450) ) [](#__codelineno-0-451) ), [](#__codelineno-0-452) # If set, assert on XLA recompilation after each execution step. [](#__codelineno-0-453) "VLLM_XLA_CHECK_RECOMPILATION": lambda: bool( [](#__codelineno-0-454) int(os.getenv("VLLM_XLA_CHECK_RECOMPILATION", "0")) [](#__codelineno-0-455) ), [](#__codelineno-0-456) # Enable SPMD mode for TPU backend. [](#__codelineno-0-457) "VLLM_XLA_USE_SPMD": lambda: bool(int(os.getenv("VLLM_XLA_USE_SPMD", "0"))), [](#__codelineno-0-458) # Maximum size (in MB) for logits tensor in sparse MLA indexer prefill chunks. [](#__codelineno-0-459) # Bounds the [M, N] float32 logits tensor to prevent CUDA OOM. [](#__codelineno-0-460) # Default: 512 MB [](#__codelineno-0-461) "VLLM_SPARSE_INDEXER_MAX_LOGITS_MB": lambda: int( [](#__codelineno-0-462) os.getenv("VLLM_SPARSE_INDEXER_MAX_LOGITS_MB", "512") [](#__codelineno-0-463) ), [](#__codelineno-0-464) # If set, the OpenAI API server will stay alive even after the underlying [](#__codelineno-0-465) # AsyncLLMEngine errors and stops serving requests [](#__codelineno-0-466) "VLLM_KEEP_ALIVE_ON_ENGINE_DEATH": lambda: bool( [](#__codelineno-0-467) int(os.getenv("VLLM_KEEP_ALIVE_ON_ENGINE_DEATH", "0")) [](#__codelineno-0-468) ), [](#__codelineno-0-469) # If the env var VLLM_ALLOW_LONG_MAX_MODEL_LEN is set, it allows [](#__codelineno-0-470) # the user to specify a max sequence length greater than [](#__codelineno-0-471) # the max length derived from the model's config.json. [](#__codelineno-0-472) # To enable this, set VLLM_ALLOW_LONG_MAX_MODEL_LEN=1. [](#__codelineno-0-473) "VLLM_ALLOW_LONG_MAX_MODEL_LEN": lambda: ( [](#__codelineno-0-474) os.environ.get("VLLM_ALLOW_LONG_MAX_MODEL_LEN", "0").strip().lower() [](#__codelineno-0-475) in ("1", "true") [](#__codelineno-0-476) ), [](#__codelineno-0-477) # If set, forces FP8 Marlin to be used for FP8 quantization regardless [](#__codelineno-0-478) # of the hardware support for FP8 compute. [](#__codelineno-0-479) "VLLM_TEST_FORCE_FP8_MARLIN": lambda: ( [](#__codelineno-0-480) os.environ.get("VLLM_TEST_FORCE_FP8_MARLIN", "0").strip().lower() [](#__codelineno-0-481) in ("1", "true") [](#__codelineno-0-482) ), [](#__codelineno-0-483) "VLLM_TEST_FORCE_LOAD_FORMAT": lambda: os.getenv( [](#__codelineno-0-484) "VLLM_TEST_FORCE_LOAD_FORMAT", "dummy" [](#__codelineno-0-485) ), [](#__codelineno-0-486) # Timeout in seconds for keeping HTTP connections alive in API server [](#__codelineno-0-487) "VLLM_HTTP_TIMEOUT_KEEP_ALIVE": lambda: int( [](#__codelineno-0-488) os.environ.get("VLLM_HTTP_TIMEOUT_KEEP_ALIVE", "5") [](#__codelineno-0-489) ), [](#__codelineno-0-490) # Maximum allowed value for the `n` sampling parameter (number of output [](#__codelineno-0-491) # sequences per request). Limits resource consumption to prevent [](#__codelineno-0-492) # denial-of-service via excessively large fan-out. Default: 16384. [](#__codelineno-0-493) "VLLM_MAX_N_SEQUENCES": lambda: int( [](#__codelineno-0-494) os.environ.get("VLLM_MAX_N_SEQUENCES", "16384") [](#__codelineno-0-495) ), [](#__codelineno-0-496) # a list of plugin names to load, separated by commas. [](#__codelineno-0-497) # if this is not set, it means all plugins will be loaded [](#__codelineno-0-498) # if this is set to an empty string, no plugins will be loaded [](#__codelineno-0-499) "VLLM_PLUGINS": lambda: ( [](#__codelineno-0-500) None [](#__codelineno-0-501) if "VLLM_PLUGINS" not in os.environ [](#__codelineno-0-502) else os.environ["VLLM_PLUGINS"].split(",") [](#__codelineno-0-503) ), [](#__codelineno-0-504) # Retain local sliding-window KV checkpoints for prefix caching. [](#__codelineno-0-505) # Unset (default) preserves the dense local checkpointing behavior. `0` [](#__codelineno-0-506) # retains only the latest completed prompt boundary. Positive values retain [](#__codelineno-0-507) # checkpoints at the specified interval boundaries (rounded up to the [](#__codelineno-0-508) # prefix-cache alignment). [](#__codelineno-0-509) # Applies to sliding-window attention for now but not yet Mamba/linear attention. [](#__codelineno-0-510) "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": lambda: ( [](#__codelineno-0-511) int(os.environ["VLLM_PREFIX_CACHE_RETENTION_INTERVAL"]) [](#__codelineno-0-512) if "VLLM_PREFIX_CACHE_RETENTION_INTERVAL" in os.environ [](#__codelineno-0-513) else None [](#__codelineno-0-514) ), [](#__codelineno-0-515) # a local directory to look in for unrecognized LoRA adapters. [](#__codelineno-0-516) # only works if plugins are enabled and [](#__codelineno-0-517) # VLLM_ALLOW_RUNTIME_LORA_UPDATING is enabled. [](#__codelineno-0-518) "VLLM_LORA_RESOLVER_CACHE_DIR": lambda: os.getenv( [](#__codelineno-0-519) "VLLM_LORA_RESOLVER_CACHE_DIR", None [](#__codelineno-0-520) ), [](#__codelineno-0-521) # A remote HF repo(s) containing one or more LoRA adapters, which [](#__codelineno-0-522) # may be downloaded and leveraged as needed. Only works if plugins [](#__codelineno-0-523) # are enabled and VLLM_ALLOW_RUNTIME_LORA_UPDATING is enabled. [](#__codelineno-0-524) # Values should be comma separated. [](#__codelineno-0-525) "VLLM_LORA_RESOLVER_HF_REPO_LIST": lambda: os.getenv( [](#__codelineno-0-526) "VLLM_LORA_RESOLVER_HF_REPO_LIST", None [](#__codelineno-0-527) ), [](#__codelineno-0-528) # If set, vLLM will use Triton implementations of AWQ. [](#__codelineno-0-529) "VLLM_USE_TRITON_AWQ": lambda: bool(int(os.getenv("VLLM_USE_TRITON_AWQ", "0"))), [](#__codelineno-0-530) # If set, allow loading or unloading lora adapters in runtime, [](#__codelineno-0-531) "VLLM_ALLOW_RUNTIME_LORA_UPDATING": lambda: ( [](#__codelineno-0-532) os.environ.get("VLLM_ALLOW_RUNTIME_LORA_UPDATING", "0").strip().lower() [](#__codelineno-0-533) in ("1", "true") [](#__codelineno-0-534) ), [](#__codelineno-0-535) # We assume drivers can report p2p status correctly. [](#__codelineno-0-536) # If the program hangs when using custom allreduce, [](#__codelineno-0-537) # potantially caused by a bug in the driver (535 series), [](#__codelineno-0-538) # if might be helpful to set VLLM_SKIP_P2P_CHECK=0 [](#__codelineno-0-539) # so that vLLM can verify if p2p is actually working. [](#__codelineno-0-540) # See https://github.com/vllm-project/vllm/blob/a9b15c606fea67a072416ea0ea115261a2756058/vllm/distributed/device_communicators/custom_all_reduce_utils.py#L101-L108 for details. # noqa [](#__codelineno-0-541) "VLLM_SKIP_P2P_CHECK": lambda: os.getenv("VLLM_SKIP_P2P_CHECK", "1") == "1", [](#__codelineno-0-542) # List of quantization kernels that should be disabled, used for testing [](#__codelineno-0-543) # and performance comparisons. Currently only affects MPLinearKernel [](#__codelineno-0-544) # selection [](#__codelineno-0-545) # (kernels: MacheteLinearKernel, MarlinLinearKernel, ExllamaLinearKernel) [](#__codelineno-0-546) "VLLM_DISABLED_KERNELS": lambda: ( [](#__codelineno-0-547) [] [](#__codelineno-0-548) if "VLLM_DISABLED_KERNELS" not in os.environ [](#__codelineno-0-549) else os.environ["VLLM_DISABLED_KERNELS"].split(",") [](#__codelineno-0-550) ), [](#__codelineno-0-551) "VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE": lambda: bool( [](#__codelineno-0-552) int(os.getenv("VLLM_ENABLE_FLA_PACKED_RECURRENT_DECODE", "1")) [](#__codelineno-0-553) ), [](#__codelineno-0-554) # Disable pynccl (using torch.distributed instead) [](#__codelineno-0-555) "VLLM_DISABLE_PYNCCL": lambda: ( [](#__codelineno-0-556) os.getenv("VLLM_DISABLE_PYNCCL", "False").lower() in ("true", "1") [](#__codelineno-0-557) ), [](#__codelineno-0-558) # Optional: enable external Oink custom ops (e.g., Blackwell RMSNorm). [](#__codelineno-0-559) # Disabled by default. [](#__codelineno-0-560) "VLLM_USE_OINK_OPS": lambda: ( [](#__codelineno-0-561) os.getenv("VLLM_USE_OINK_OPS", "False").lower() in ("true", "1") [](#__codelineno-0-562) ), [](#__codelineno-0-563) # Disable aiter ops unless specifically enabled. [](#__codelineno-0-564) # Acts as a parent switch to enable the rest of the other operations. [](#__codelineno-0-565) "VLLM_ROCM_USE_AITER": lambda: ( [](#__codelineno-0-566) os.getenv("VLLM_ROCM_USE_AITER", "False").lower() in ("true", "1") [](#__codelineno-0-567) ), [](#__codelineno-0-568) # Whether to use aiter paged attention. [](#__codelineno-0-569) # By default is disabled. [](#__codelineno-0-570) "VLLM_ROCM_USE_AITER_PAGED_ATTN": lambda: ( [](#__codelineno-0-571) os.getenv("VLLM_ROCM_USE_AITER_PAGED_ATTN", "False").lower() in ("true", "1") [](#__codelineno-0-572) ), [](#__codelineno-0-573) # use aiter linear op if aiter ops are enabled [](#__codelineno-0-574) # The following list of related ops [](#__codelineno-0-575) # - scaled_mm (per-tensor / rowwise) [](#__codelineno-0-576) # - use aiter tuned gemms for unquantized gemms [](#__codelineno-0-577) "VLLM_ROCM_USE_AITER_LINEAR": lambda: ( [](#__codelineno-0-578) os.getenv("VLLM_ROCM_USE_AITER_LINEAR", "True").lower() in ("true", "1") [](#__codelineno-0-579) ), [](#__codelineno-0-580) "VLLM_ROCM_USE_AITER_LINEAR_HIPBMM": lambda: ( [](#__codelineno-0-581) os.getenv("VLLM_ROCM_USE_AITER_LINEAR_HIPBMM", "False").lower() in ("true", "1") [](#__codelineno-0-582) ), [](#__codelineno-0-583) # Whether to use aiter moe ops. [](#__codelineno-0-584) # By default is enabled. [](#__codelineno-0-585) "VLLM_ROCM_USE_AITER_MOE": lambda: ( [](#__codelineno-0-586) os.getenv("VLLM_ROCM_USE_AITER_MOE", "True").lower() in ("true", "1") [](#__codelineno-0-587) ), [](#__codelineno-0-588) # MoE sorting dispatch policy for AITER fused MoE kernels. [](#__codelineno-0-589) # 0 = auto (default): single-pass for small batches, multi-pass [](#__codelineno-0-590) # for large batches [](#__codelineno-0-591) # 1 = always single-pass: one kernel launch, no workspace, [](#__codelineno-0-592) # may be preferred for low-concurrency decode workloads [](#__codelineno-0-593) # 2 = always multi-pass: can be faster for MoE-heavy models [](#__codelineno-0-594) # (e.g., +2-5% on Qwen3-Next, +1.5% on DeepSeek-V3 at TP4, [](#__codelineno-0-595) # see PR #39177 for benchmarks) [](#__codelineno-0-596) "VLLM_ROCM_AITER_MOE_DISPATCH_POLICY": lambda: int( [](#__codelineno-0-597) os.getenv("VLLM_ROCM_AITER_MOE_DISPATCH_POLICY", "0") [](#__codelineno-0-598) ), [](#__codelineno-0-599) # use aiter rms norm op if aiter ops are enabled. [](#__codelineno-0-600) "VLLM_ROCM_USE_AITER_RMSNORM": lambda: ( [](#__codelineno-0-601) os.getenv("VLLM_ROCM_USE_AITER_RMSNORM", "True").lower() in ("true", "1") [](#__codelineno-0-602) ), [](#__codelineno-0-603) # Whether to use aiter mla ops. [](#__codelineno-0-604) # By default is enabled. [](#__codelineno-0-605) "VLLM_ROCM_USE_AITER_MLA": lambda: ( [](#__codelineno-0-606) os.getenv("VLLM_ROCM_USE_AITER_MLA", "True").lower() in ("true", "1") [](#__codelineno-0-607) ), [](#__codelineno-0-608) # Whether to use aiter mha ops. [](#__codelineno-0-609) # By default is enabled. [](#__codelineno-0-610) "VLLM_ROCM_USE_AITER_MHA": lambda: ( [](#__codelineno-0-611) os.getenv("VLLM_ROCM_USE_AITER_MHA", "True").lower() in ("true", "1") [](#__codelineno-0-612) ), [](#__codelineno-0-613) # Whether to use aiter fp4 gemm asm. [](#__codelineno-0-614) # By default is disabled. [](#__codelineno-0-615) "VLLM_ROCM_USE_AITER_FP4_ASM_GEMM": lambda: ( [](#__codelineno-0-616) os.getenv("VLLM_ROCM_USE_AITER_FP4_ASM_GEMM", "False").lower() in ("true", "1") [](#__codelineno-0-617) ), [](#__codelineno-0-618) # Whether to use aiter rope. [](#__codelineno-0-619) # By default is disabled. [](#__codelineno-0-620) "VLLM_ROCM_USE_AITER_TRITON_ROPE": lambda: ( [](#__codelineno-0-621) os.getenv("VLLM_ROCM_USE_AITER_TRITON_ROPE", "False").lower() in ("true", "1") [](#__codelineno-0-622) ), [](#__codelineno-0-623) # Whether to use aiter triton fp8 bmm kernel [](#__codelineno-0-624) # By default is enabled. [](#__codelineno-0-625) "VLLM_ROCM_USE_AITER_FP8BMM": lambda: ( [](#__codelineno-0-626) os.getenv("VLLM_ROCM_USE_AITER_FP8BMM", "True").lower() in ("true", "1") [](#__codelineno-0-627) ), [](#__codelineno-0-628) # Whether to use aiter triton fp4 bmm kernel [](#__codelineno-0-629) # By default is enabled. [](#__codelineno-0-630) "VLLM_ROCM_USE_AITER_FP4BMM": lambda: ( [](#__codelineno-0-631) os.getenv("VLLM_ROCM_USE_AITER_FP4BMM", "True").lower() in ("true", "1") [](#__codelineno-0-632) ), [](#__codelineno-0-633) # Use AITER triton unified attention for V1 attention [](#__codelineno-0-634) "VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION": lambda: ( [](#__codelineno-0-635) os.getenv("VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION", "False").lower() [](#__codelineno-0-636) in ("true", "1") [](#__codelineno-0-637) ), [](#__codelineno-0-638) # Whether to use aiter fusion shared experts ops. [](#__codelineno-0-639) # By default is disabled. [](#__codelineno-0-640) "VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS": lambda: ( [](#__codelineno-0-641) os.getenv("VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS", "False").lower() [](#__codelineno-0-642) in ("true", "1") [](#__codelineno-0-643) ), [](#__codelineno-0-644) # Whether to use aiter triton kernels for gemm ops. [](#__codelineno-0-645) # By default is enabled. [](#__codelineno-0-646) "VLLM_ROCM_USE_AITER_TRITON_GEMM": lambda: ( [](#__codelineno-0-647) os.getenv("VLLM_ROCM_USE_AITER_TRITON_GEMM", "True").lower() in ("true", "1") [](#__codelineno-0-648) ), [](#__codelineno-0-649) # use rocm skinny gemms [](#__codelineno-0-650) "VLLM_ROCM_USE_SKINNY_GEMM": lambda: ( [](#__codelineno-0-651) os.getenv("VLLM_ROCM_USE_SKINNY_GEMM", "True").lower() in ("true", "1") [](#__codelineno-0-652) ), [](#__codelineno-0-653) # Pad the fp8 weights to 256 bytes for ROCm [](#__codelineno-0-654) "VLLM_ROCM_FP8_PADDING": lambda: bool(int(os.getenv("VLLM_ROCM_FP8_PADDING", "1"))), [](#__codelineno-0-655) # Pad the weights for the moe kernel [](#__codelineno-0-656) "VLLM_ROCM_MOE_PADDING": lambda: bool(int(os.getenv("VLLM_ROCM_MOE_PADDING", "1"))), [](#__codelineno-0-657) # Whether to use the shuffled kv cache layout [](#__codelineno-0-658) "VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT": lambda: ( [](#__codelineno-0-659) os.getenv("VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT", "False").lower() in ("true", "1") [](#__codelineno-0-660) ), [](#__codelineno-0-661) # Custom quick allreduce kernel for MI3* cards [](#__codelineno-0-662) # Choice of quantization level: FP, INT8, INT6, INT4 or NONE [](#__codelineno-0-663) # Recommended for large models to get allreduce [](#__codelineno-0-664) "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION": env_with_choices( [](#__codelineno-0-665) "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION", [](#__codelineno-0-666) "NONE", [](#__codelineno-0-667) ["FP", "INT8", "INT6", "INT4", "NONE"], [](#__codelineno-0-668) ), [](#__codelineno-0-669) # Custom quick allreduce kernel for MI3* cards [](#__codelineno-0-670) # Due to the lack of the bfloat16 asm instruction, bfloat16 [](#__codelineno-0-671) # kernels are slower than fp16, [](#__codelineno-0-672) # If environment variable is set to 1, the input is converted to fp16 [](#__codelineno-0-673) "VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16": lambda: ( [](#__codelineno-0-674) os.getenv("VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16", "True").lower() [](#__codelineno-0-675) in ("true", "1") [](#__codelineno-0-676) ), [](#__codelineno-0-677) # Custom quick allreduce kernel for MI3* cards. [](#__codelineno-0-678) # Controls the maximum allowed number of data bytes(MB) for custom quick [](#__codelineno-0-679) # allreduce communication. [](#__codelineno-0-680) # Default: 2048 MB. [](#__codelineno-0-681) # Data exceeding this size will use either custom allreduce or RCCL [](#__codelineno-0-682) # communication. [](#__codelineno-0-683) "VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB": lambda: maybe_convert_int( [](#__codelineno-0-684) os.environ.get("VLLM_ROCM_QUICK_REDUCE_MAX_SIZE_BYTES_MB", None) [](#__codelineno-0-685) ), [](#__codelineno-0-686) # Custom quick allreduce kernel for MI3* cards. [](#__codelineno-0-687) # Controls the minimum allowed number of data bytes(MB) required to use [](#__codelineno-0-688) # custom quick allreduce communication. [](#__codelineno-0-689) # If unset, use the built-in threshold table. [](#__codelineno-0-690) "VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB": lambda: maybe_convert_int( [](#__codelineno-0-691) os.environ.get("VLLM_ROCM_QUICK_REDUCE_MIN_SIZE_BYTES_MB", None) [](#__codelineno-0-692) ), [](#__codelineno-0-693) # Controls the minimum tensor size (KB, where 1 KB = 1024 bytes) required [](#__codelineno-0-694) # to use the configured QuickReduce codec. Smaller tensors use FP [](#__codelineno-0-695) # QuickReduce. This does not affect QuickReduce eligibility. [](#__codelineno-0-696) "VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB": lambda: maybe_convert_int( [](#__codelineno-0-697) os.environ.get("VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB", None) [](#__codelineno-0-698) ), [](#__codelineno-0-699) # Divisor for dynamic query scale factor calculation for FP8 KV Cache [](#__codelineno-0-700) "Q_SCALE_CONSTANT": lambda: int(os.getenv("Q_SCALE_CONSTANT", "200")), [](#__codelineno-0-701) # Divisor for dynamic key scale factor calculation for FP8 KV Cache [](#__codelineno-0-702) "K_SCALE_CONSTANT": lambda: int(os.getenv("K_SCALE_CONSTANT", "200")), [](#__codelineno-0-703) # Divisor for dynamic value scale factor calculation for FP8 KV Cache [](#__codelineno-0-704) "V_SCALE_CONSTANT": lambda: int(os.getenv("V_SCALE_CONSTANT", "100")), [](#__codelineno-0-705) # If set, enable multiprocessing in LLM for the V1 code path. [](#__codelineno-0-706) "VLLM_ENABLE_V1_MULTIPROCESSING": lambda: bool( [](#__codelineno-0-707) int(os.getenv("VLLM_ENABLE_V1_MULTIPROCESSING", "1")) [](#__codelineno-0-708) ), [](#__codelineno-0-709) "VLLM_LOG_BATCHSIZE_INTERVAL": lambda: float( [](#__codelineno-0-710) os.getenv("VLLM_LOG_BATCHSIZE_INTERVAL", "-1") [](#__codelineno-0-711) ), [](#__codelineno-0-712) "VLLM_DISABLE_COMPILE_CACHE": disable_compile_cache, [](#__codelineno-0-713) # If set to "0", disable LayerName opaque type for layer_name [](#__codelineno-0-714) # parameters in custom ops. Defaults to enabled on torch >= 2.11. [](#__codelineno-0-715) "VLLM_USE_LAYERNAME": lambda: bool(int(os.getenv("VLLM_USE_LAYERNAME", "1"))), [](#__codelineno-0-716) # If set, use the Rust frontend binary instead of the Python API server [](#__codelineno-0-717) # process(es). [](#__codelineno-0-718) "VLLM_USE_RUST_FRONTEND": lambda: bool( [](#__codelineno-0-719) int(os.getenv("VLLM_USE_RUST_FRONTEND", "0")) [](#__codelineno-0-720) ), [](#__codelineno-0-721) # Path to the Rust frontend binary. Defaults to "auto" which discovers [](#__codelineno-0-722) # the binary installed with the vllm package. Only used when [](#__codelineno-0-723) # VLLM_USE_RUST_FRONTEND=1. [](#__codelineno-0-724) "VLLM_RUST_FRONTEND_PATH": lambda: _resolve_rust_frontend_path(), [](#__codelineno-0-725) # If set, vllm will run in development mode, which will enable [](#__codelineno-0-726) # some additional endpoints for developing and debugging, [](#__codelineno-0-727) # e.g. `/reset_prefix_cache` [](#__codelineno-0-728) "VLLM_SERVER_DEV_MODE": lambda: bool(int(os.getenv("VLLM_SERVER_DEV_MODE", "0"))), [](#__codelineno-0-729) # Controls the maximum number of requests to handle in a [](#__codelineno-0-730) # single asyncio task when processing per-token outputs in the [](#__codelineno-0-731) # V1 AsyncLLM interface. It is applicable when handling a high [](#__codelineno-0-732) # concurrency of streaming requests. [](#__codelineno-0-733) # Setting this too high can result in a higher variance of [](#__codelineno-0-734) # inter-message latencies. Setting it too low can negatively impact [](#__codelineno-0-735) # TTFT and overall throughput. [](#__codelineno-0-736) "VLLM_V1_OUTPUT_PROC_CHUNK_SIZE": lambda: int( [](#__codelineno-0-737) os.getenv("VLLM_V1_OUTPUT_PROC_CHUNK_SIZE", "128") [](#__codelineno-0-738) ), [](#__codelineno-0-739) # If set, vLLM will disable the MLA attention optimizations. [](#__codelineno-0-740) "VLLM_MLA_DISABLE": lambda: bool(int(os.getenv("VLLM_MLA_DISABLE", "0"))), [](#__codelineno-0-741) # If set, vLLM will pick up the provided Flash Attention MLA [](#__codelineno-0-742) # Number of GPUs per worker in Ray, if it is set to be a fraction, [](#__codelineno-0-743) # it allows ray to schedule multiple actors on a single GPU, [](#__codelineno-0-744) # so that users can colocate other actors on the same GPUs as vLLM. [](#__codelineno-0-745) "VLLM_RAY_PER_WORKER_GPUS": lambda: float( [](#__codelineno-0-746) os.getenv("VLLM_RAY_PER_WORKER_GPUS", "1.0") [](#__codelineno-0-747) ), [](#__codelineno-0-748) # Bundle indices for Ray, if it is set, it can control precisely [](#__codelineno-0-749) # which indices are used for the Ray bundle, for every worker. [](#__codelineno-0-750) # Format: comma-separated list of integers, e.g. "0,1,2,3" [](#__codelineno-0-751) "VLLM_RAY_BUNDLE_INDICES": lambda: os.getenv("VLLM_RAY_BUNDLE_INDICES", ""), [](#__codelineno-0-752) # In some system, find_loaded_library() may not work. So we allow users to [](#__codelineno-0-753) # specify the path through environment variable VLLM_CUDART_SO_PATH. [](#__codelineno-0-754) "VLLM_CUDART_SO_PATH": lambda: os.getenv("VLLM_CUDART_SO_PATH", None), [](#__codelineno-0-755) # Rank of the process in the data parallel setting [](#__codelineno-0-756) "VLLM_DP_RANK": lambda: int(os.getenv("VLLM_DP_RANK", "0")), [](#__codelineno-0-757) # Rank of the process in the data parallel setting. [](#__codelineno-0-758) # Defaults to VLLM_DP_RANK when not set. [](#__codelineno-0-759) "VLLM_DP_RANK_LOCAL": lambda: int( [](#__codelineno-0-760) os.getenv("VLLM_DP_RANK_LOCAL", sys.modules[__name__].VLLM_DP_RANK) [](#__codelineno-0-761) ), [](#__codelineno-0-762) # World size of the data parallel setting [](#__codelineno-0-763) "VLLM_DP_SIZE": lambda: int(os.getenv("VLLM_DP_SIZE", "1")), [](#__codelineno-0-764) # IP address of the master node in the data parallel setting [](#__codelineno-0-765) "VLLM_DP_MASTER_IP": lambda: os.getenv("VLLM_DP_MASTER_IP", "127.0.0.1"), [](#__codelineno-0-766) # Port of the master node in the data parallel setting [](#__codelineno-0-767) "VLLM_DP_MASTER_PORT": lambda: int(os.getenv("VLLM_DP_MASTER_PORT", "0")), [](#__codelineno-0-768) # Randomize inputs during dummy runs when using Data Parallel [](#__codelineno-0-769) "VLLM_RANDOMIZE_DP_DUMMY_INPUTS": lambda: ( [](#__codelineno-0-770) os.environ.get("VLLM_RANDOMIZE_DP_DUMMY_INPUTS", "0") == "1" [](#__codelineno-0-771) ), [](#__codelineno-0-772) # Strategy to pack the data parallel ranks for Ray. [](#__codelineno-0-773) # Available options: [](#__codelineno-0-774) # - "fill": [](#__codelineno-0-775) # for DP master node, allocate exactly data-parallel-size-local DP ranks, [](#__codelineno-0-776) # for non-master nodes, allocate as many DP ranks as can fit; [](#__codelineno-0-777) # - "strict": [](#__codelineno-0-778) # allocate exactly data-parallel-size-local DP ranks to each picked node; [](#__codelineno-0-779) # - "span": [](#__codelineno-0-780) # Should be used only when a single DP rank requires multiple nodes. [](#__codelineno-0-781) # allocate one DP rank over as many nodes as required for set world_size; [](#__codelineno-0-782) # This environment variable is ignored if data-parallel-backend is not Ray. [](#__codelineno-0-783) "VLLM_RAY_DP_PACK_STRATEGY": lambda: os.getenv( [](#__codelineno-0-784) "VLLM_RAY_DP_PACK_STRATEGY", "strict" [](#__codelineno-0-785) ), [](#__codelineno-0-786) # Optional comma-separated list of node IPs that Ray data-parallel [](#__codelineno-0-787) # placement groups may use. When set, create_dp_placement_groups only [](#__codelineno-0-788) # considers these nodes (the DP master node is always included). [](#__codelineno-0-789) # This environment variable is ignored if data-parallel-backend is not Ray. [](#__codelineno-0-790) "VLLM_RAY_DP_PLACEMENT_NODE_IPS": lambda: os.getenv( [](#__codelineno-0-791) "VLLM_RAY_DP_PLACEMENT_NODE_IPS", "" [](#__codelineno-0-792) ), [](#__codelineno-0-793) # Comma-separated *additional* prefixes of env vars to copy from the [](#__codelineno-0-794) # driver to Ray workers. These are merged with the built-in defaults [](#__codelineno-0-795) # defined in ``vllm.ray.ray_env`` (VLLM_, etc.). Example: "MYLIB_,OTHER_" [](#__codelineno-0-796) "VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY": lambda: os.getenv( [](#__codelineno-0-797) "VLLM_RAY_EXTRA_ENV_VAR_PREFIXES_TO_COPY", "" [](#__codelineno-0-798) ), [](#__codelineno-0-799) # Comma-separated *additional* individual env var names to copy from [](#__codelineno-0-800) # the driver to Ray workers. Merged with the built-in defaults [](#__codelineno-0-801) # defined in ``vllm.ray.ray_env`` (PYTHONHASHSEED). [](#__codelineno-0-802) # Example: "MY_SECRET,MY_FLAG" [](#__codelineno-0-803) "VLLM_RAY_EXTRA_ENV_VARS_TO_COPY": lambda: os.getenv( [](#__codelineno-0-804) "VLLM_RAY_EXTRA_ENV_VARS_TO_COPY", "" [](#__codelineno-0-805) ), [](#__codelineno-0-806) # Whether to use S3 path for model loading in CI via RunAI Streamer [](#__codelineno-0-807) "VLLM_CI_USE_S3": lambda: os.environ.get("VLLM_CI_USE_S3", "0") == "1", [](#__codelineno-0-808) # Use model_redirect to redirect the model name to a local folder. [](#__codelineno-0-809) # `model_redirect` can be a json file mapping the model between [](#__codelineno-0-810) # repo_id and local folder: [](#__codelineno-0-811) # {"meta-llama/Llama-3.2-1B": "/tmp/Llama-3.2-1B"} [](#__codelineno-0-812) # or a space separated values table file: [](#__codelineno-0-813) # meta-llama/Llama-3.2-1B /tmp/Llama-3.2-1B [](#__codelineno-0-814) "VLLM_MODEL_REDIRECT_PATH": lambda: os.environ.get( [](#__codelineno-0-815) "VLLM_MODEL_REDIRECT_PATH", None [](#__codelineno-0-816) ), [](#__codelineno-0-817) # Whether to use atomicAdd reduce in gptq/awq marlin kernel. [](#__codelineno-0-818) "VLLM_MARLIN_USE_ATOMIC_ADD": lambda: ( [](#__codelineno-0-819) os.environ.get("VLLM_MARLIN_USE_ATOMIC_ADD", "0") == "1" [](#__codelineno-0-820) ), [](#__codelineno-0-821) # Whether to use marlin kernel in mxfp4 quantization method [](#__codelineno-0-822) # Deprecated: use --moe-backend marlin (MoE) or --linear-backend marlin [](#__codelineno-0-823) # (linear) instead. [](#__codelineno-0-824) "VLLM_MXFP4_USE_MARLIN": deprecated_env( [](#__codelineno-0-825) "VLLM_MXFP4_USE_MARLIN", [](#__codelineno-0-826) "v0.23", [](#__codelineno-0-827) "Use --moe-backend marlin or --linear-backend marlin.", [](#__codelineno-0-828) lambda: maybe_convert_bool(os.environ.get("VLLM_MXFP4_USE_MARLIN", None)), [](#__codelineno-0-829) ), [](#__codelineno-0-830) # The activation dtype for marlin kernel [](#__codelineno-0-831) "VLLM_MARLIN_INPUT_DTYPE": env_with_choices( [](#__codelineno-0-832) "VLLM_MARLIN_INPUT_DTYPE", None, ["int8", "fp8"] [](#__codelineno-0-833) ), [](#__codelineno-0-834) # The online quantization dtype for humming kernel [](#__codelineno-0-835) "VLLM_HUMMING_ONLINE_QUANT_CONFIG": lambda: maybe_convert_json_str_or_file( [](#__codelineno-0-836) os.environ.get("VLLM_HUMMING_ONLINE_QUANT_CONFIG", None) [](#__codelineno-0-837) ), [](#__codelineno-0-838) # The activation dtype config for humming kernel [](#__codelineno-0-839) "VLLM_HUMMING_INPUT_QUANT_CONFIG": lambda: maybe_convert_json_str_or_file( [](#__codelineno-0-840) os.environ.get("VLLM_HUMMING_INPUT_QUANT_CONFIG", None) [](#__codelineno-0-841) ), [](#__codelineno-0-842) # Whether to use fp16 accumulator mma [](#__codelineno-0-843) "VLLM_HUMMING_USE_F16_ACCUM": lambda: maybe_convert_bool( [](#__codelineno-0-844) os.environ.get("VLLM_HUMMING_USE_F16_ACCUM", "0") [](#__codelineno-0-845) ), [](#__codelineno-0-846) # Whether to use indexed gemm for humming moe [](#__codelineno-0-847) # if 1, force use indexed gemm [](#__codelineno-0-848) # if 0, force use grouped gemm [](#__codelineno-0-849) # if None, choose better gemm type automatically [](#__codelineno-0-850) "VLLM_HUMMING_MOE_GEMM_TYPE": lambda: os.environ.get( [](#__codelineno-0-851) "VLLM_HUMMING_MOE_GEMM_TYPE", None [](#__codelineno-0-852) ), [](#__codelineno-0-853) # Whether to use DeepEPLL kernels for NVFP4 quantization and dispatch method [](#__codelineno-0-854) # only supported on Blackwell GPUs and with [](#__codelineno-0-855) # https://github.com/deepseek-ai/DeepEP/pull/341 [](#__codelineno-0-856) "VLLM_DEEPEPLL_NVFP4_DISPATCH": lambda: bool( [](#__codelineno-0-857) int(os.getenv("VLLM_DEEPEPLL_NVFP4_DISPATCH", "0")) [](#__codelineno-0-858) ), [](#__codelineno-0-859) # Whether to turn on the outlines cache for V1 [](#__codelineno-0-860) # This cache is unbounded and on disk, so it's not safe to use in [](#__codelineno-0-861) # an environment with potentially malicious users. [](#__codelineno-0-862) "VLLM_V1_USE_OUTLINES_CACHE": lambda: ( [](#__codelineno-0-863) os.environ.get("VLLM_V1_USE_OUTLINES_CACHE", "0") == "1" [](#__codelineno-0-864) ), [](#__codelineno-0-865) # Gap between padding buckets for the forward pass. So we have [](#__codelineno-0-866) # 8, we will run forward pass with [16, 24, 32, ...]. [](#__codelineno-0-867) "VLLM_TPU_BUCKET_PADDING_GAP": lambda: ( [](#__codelineno-0-868) int(os.environ["VLLM_TPU_BUCKET_PADDING_GAP"]) [](#__codelineno-0-869) if "VLLM_TPU_BUCKET_PADDING_GAP" in os.environ [](#__codelineno-0-870) else 0 [](#__codelineno-0-871) ), [](#__codelineno-0-872) "VLLM_TPU_MOST_MODEL_LEN": lambda: maybe_convert_int( [](#__codelineno-0-873) os.environ.get("VLLM_TPU_MOST_MODEL_LEN", None) [](#__codelineno-0-874) ), [](#__codelineno-0-875) # Whether using Pathways [](#__codelineno-0-876) "VLLM_TPU_USING_PATHWAYS": lambda: bool( [](#__codelineno-0-877) "proxy" in os.getenv("JAX_PLATFORMS", "").lower() [](#__codelineno-0-878) ), [](#__codelineno-0-879) # Allow use of DeepGemm kernels for fused moe ops. [](#__codelineno-0-880) "VLLM_USE_DEEP_GEMM": lambda: bool(int(os.getenv("VLLM_USE_DEEP_GEMM", "1"))), [](#__codelineno-0-881) # Allow use of DeepGemm specifically for MoE fused ops (overrides only MoE). [](#__codelineno-0-882) "VLLM_MOE_USE_DEEP_GEMM": lambda: bool( [](#__codelineno-0-883) int(os.getenv("VLLM_MOE_USE_DEEP_GEMM", "1")) [](#__codelineno-0-884) ), [](#__codelineno-0-885) # Whether to use E8M0 scaling when DeepGEMM is used on Blackwell GPUs. [](#__codelineno-0-886) "VLLM_USE_DEEP_GEMM_E8M0": lambda: bool( [](#__codelineno-0-887) int(os.getenv("VLLM_USE_DEEP_GEMM_E8M0", "1")) [](#__codelineno-0-888) ), [](#__codelineno-0-889) # Whether to create TMA-aligned scale tensor when DeepGEMM is used. [](#__codelineno-0-890) "VLLM_USE_DEEP_GEMM_TMA_ALIGNED_SCALES": lambda: bool( [](#__codelineno-0-891) int(os.getenv("VLLM_USE_DEEP_GEMM_TMA_ALIGNED_SCALES", "1")) [](#__codelineno-0-892) ), [](#__codelineno-0-893) # DeepGemm JITs the kernels on-demand. The warmup attempts to make DeepGemm [](#__codelineno-0-894) # JIT all the required kernels before model execution so there is no [](#__codelineno-0-895) # JIT'ing in the hot-path. However, this warmup increases the engine [](#__codelineno-0-896) # startup time by a couple of minutes. [](#__codelineno-0-897) # Available options: [](#__codelineno-0-898) # - "skip" : Skip warmup. [](#__codelineno-0-899) # - "full" : Warmup deepgemm by running all possible gemm shapes the [](#__codelineno-0-900) # engine could encounter. [](#__codelineno-0-901) # - "relax" : Select gemm shapes to run based on some heuristics. The [](#__codelineno-0-902) # heuristic aims to have the same effect as running all possible gemm [](#__codelineno-0-903) # shapes, but provides no guarantees. [](#__codelineno-0-904) "VLLM_DEEP_GEMM_WARMUP": env_with_choices( [](#__codelineno-0-905) "VLLM_DEEP_GEMM_WARMUP", [](#__codelineno-0-906) "relax", [](#__codelineno-0-907) [ [](#__codelineno-0-908) "skip", [](#__codelineno-0-909) "full", [](#__codelineno-0-910) "relax", [](#__codelineno-0-911) ], [](#__codelineno-0-912) ), [](#__codelineno-0-913) # Whether to use fused grouped_topk used for MoE expert selection. [](#__codelineno-0-914) "VLLM_USE_FUSED_MOE_GROUPED_TOPK": lambda: bool( [](#__codelineno-0-915) int(os.getenv("VLLM_USE_FUSED_MOE_GROUPED_TOPK", "1")) [](#__codelineno-0-916) ), [](#__codelineno-0-917) # Allow use of FlashInfer FP8 block-scale GEMM for linear layers. [](#__codelineno-0-918) # This uses TensorRT-LLM kernels and requires SM90+ (Hopper). [](#__codelineno-0-919) "VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER": lambda: bool( [](#__codelineno-0-920) int(os.getenv("VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER", "1")) [](#__codelineno-0-921) ), [](#__codelineno-0-922) # Allow use of FlashInfer BF16 MoE kernels for fused moe ops. [](#__codelineno-0-923) # Deprecated: use --moe-backend to select a kernel explicitly. [](#__codelineno-0-924) "VLLM_USE_FLASHINFER_MOE_FP16": deprecated_env( [](#__codelineno-0-925) "VLLM_USE_FLASHINFER_MOE_FP16", [](#__codelineno-0-926) "v0.23", [](#__codelineno-0-927) "Use --moe-backend (e.g. flashinfer_trtllm, flashinfer_cutlass).", [](#__codelineno-0-928) lambda: bool(int(os.getenv("VLLM_USE_FLASHINFER_MOE_FP16", "0"))), [](#__codelineno-0-929) ), [](#__codelineno-0-930) # Allow use of FlashInfer FP8 MoE kernels for fused moe ops. [](#__codelineno-0-931) # Deprecated: use --moe-backend to select a kernel explicitly. [](#__codelineno-0-932) "VLLM_USE_FLASHINFER_MOE_FP8": deprecated_env( [](#__codelineno-0-933) "VLLM_USE_FLASHINFER_MOE_FP8", [](#__codelineno-0-934) "v0.23", [](#__codelineno-0-935) "Use --moe-backend (e.g. flashinfer_trtllm, flashinfer_cutlass).", [](#__codelineno-0-936) lambda: bool(int(os.getenv("VLLM_USE_FLASHINFER_MOE_FP8", "0"))), [](#__codelineno-0-937) ), [](#__codelineno-0-938) # Allow use of FlashInfer NVFP4 MoE kernels for fused moe ops. [](#__codelineno-0-939) # Deprecated: use --moe-backend to select a kernel explicitly. [](#__codelineno-0-940) "VLLM_USE_FLASHINFER_MOE_FP4": deprecated_env( [](#__codelineno-0-941) "VLLM_USE_FLASHINFER_MOE_FP4", [](#__codelineno-0-942) "v0.23", [](#__codelineno-0-943) "Use --moe-backend (e.g. flashinfer_trtllm, flashinfer_cutlass, " [](#__codelineno-0-944) "flashinfer_cutedsl).", [](#__codelineno-0-945) lambda: bool(int(os.getenv("VLLM_USE_FLASHINFER_MOE_FP4", "0"))), [](#__codelineno-0-946) ), [](#__codelineno-0-947) # Allow use of FlashInfer MxInt4 MoE kernels for fused moe ops. [](#__codelineno-0-948) "VLLM_USE_FLASHINFER_MOE_INT4": lambda: bool( [](#__codelineno-0-949) int(os.getenv("VLLM_USE_FLASHINFER_MOE_INT4", "0")) [](#__codelineno-0-950) ), [](#__codelineno-0-951) # If set to 1, use the FlashInfer [](#__codelineno-0-952) # MXFP8 (activation) x MXFP4 (weight) MoE backend. [](#__codelineno-0-953) # Deprecated: use --moe-backend flashinfer_trtllm combined with [](#__codelineno-0-954) # --quantization_config.moe.activation mxfp8. [](#__codelineno-0-955) "VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8": deprecated_env( [](#__codelineno-0-956) "VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8", [](#__codelineno-0-957) "v0.23", [](#__codelineno-0-958) "Use --moe-backend flashinfer_trtllm with " [](#__codelineno-0-959) "--quantization_config.moe.activation mxfp8.", [](#__codelineno-0-960) lambda: bool(int(os.getenv("VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8", "0"))), [](#__codelineno-0-961) ), [](#__codelineno-0-962) # If set to 1, use the FlashInfer CUTLASS backend for [](#__codelineno-0-963) # MXFP8 (activation) x MXFP4 (weight) MoE. [](#__codelineno-0-964) # Deprecated: use --moe-backend flashinfer_cutlass combined with [](#__codelineno-0-965) # --quantization_config.moe.activation mxfp8. [](#__codelineno-0-966) "VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS": deprecated_env( [](#__codelineno-0-967) "VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS", [](#__codelineno-0-968) "v0.23", [](#__codelineno-0-969) "Use --moe-backend flashinfer_cutlass with " [](#__codelineno-0-970) "--quantization_config.moe.activation mxfp8.", [](#__codelineno-0-971) lambda: bool( [](#__codelineno-0-972) int(os.getenv("VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS", "0")) [](#__codelineno-0-973) ), [](#__codelineno-0-974) ), [](#__codelineno-0-975) # If set to 1, use the FlashInfer [](#__codelineno-0-976) # BF16 (activation) x MXFP4 (weight) MoE backend. [](#__codelineno-0-977) # Deprecated: use --moe-backend to select a kernel explicitly. [](#__codelineno-0-978) "VLLM_USE_FLASHINFER_MOE_MXFP4_BF16": deprecated_env( [](#__codelineno-0-979) "VLLM_USE_FLASHINFER_MOE_MXFP4_BF16", [](#__codelineno-0-980) "v0.23", [](#__codelineno-0-981) "Use --moe-backend (e.g. flashinfer_trtllm, flashinfer_cutlass).", [](#__codelineno-0-982) lambda: bool(int(os.getenv("VLLM_USE_FLASHINFER_MOE_MXFP4_BF16", "0"))), [](#__codelineno-0-983) ), [](#__codelineno-0-984) # Control the cache sized used by the xgrammar compiler. The default [](#__codelineno-0-985) # of 512 MB should be enough for roughly 1000 JSON schemas. [](#__codelineno-0-986) # It can be changed with this variable if needed for some reason. [](#__codelineno-0-987) "VLLM_XGRAMMAR_CACHE_MB": lambda: int(os.getenv("VLLM_XGRAMMAR_CACHE_MB", "512")), [](#__codelineno-0-988) # Control the threshold for msgspec to use 'zero copy' for [](#__codelineno-0-989) # serialization/deserialization of tensors. Tensors below [](#__codelineno-0-990) # this limit will be encoded into the msgpack buffer, and [](#__codelineno-0-991) # tensors above will instead be sent via a separate message. [](#__codelineno-0-992) # While the sending side still actually copies the tensor [](#__codelineno-0-993) # in all cases, on the receiving side, tensors above this [](#__codelineno-0-994) # limit will actually be zero-copy decoded. [](#__codelineno-0-995) "VLLM_MSGPACK_ZERO_COPY_THRESHOLD": lambda: int( [](#__codelineno-0-996) os.getenv("VLLM_MSGPACK_ZERO_COPY_THRESHOLD", "256") [](#__codelineno-0-997) ), [](#__codelineno-0-998) # If set, allow insecure serialization using pickle. [](#__codelineno-0-999) # This is useful for environments where it is deemed safe to use the [](#__codelineno-0-1000) # insecure method and it is needed for some reason. [](#__codelineno-0-1001) "VLLM_ALLOW_INSECURE_SERIALIZATION": lambda: bool( [](#__codelineno-0-1002) int(os.getenv("VLLM_ALLOW_INSECURE_SERIALIZATION", "0")) [](#__codelineno-0-1003) ), [](#__codelineno-0-1004) # Temporary: skip adding random suffix to internal request IDs. May be [](#__codelineno-0-1005) # needed for KV connectors that match request IDs across instances. [](#__codelineno-0-1006) "VLLM_DISABLE_REQUEST_ID_RANDOMIZATION": lambda: bool( [](#__codelineno-0-1007) int(os.getenv("VLLM_DISABLE_REQUEST_ID_RANDOMIZATION", "0")) [](#__codelineno-0-1008) ), [](#__codelineno-0-1009) # IP address used for NIXL handshake between remote agents. [](#__codelineno-0-1010) "VLLM_NIXL_SIDE_CHANNEL_HOST": lambda: os.getenv( [](#__codelineno-0-1011) "VLLM_NIXL_SIDE_CHANNEL_HOST", "localhost" [](#__codelineno-0-1012) ), [](#__codelineno-0-1013) # Port used for NIXL handshake between remote agents. [](#__codelineno-0-1014) "VLLM_NIXL_SIDE_CHANNEL_PORT": lambda: int( [](#__codelineno-0-1015) os.getenv("VLLM_NIXL_SIDE_CHANNEL_PORT", "5600") [](#__codelineno-0-1016) ), [](#__codelineno-0-1017) # Port used for Mooncake handshake between remote agents. [](#__codelineno-0-1018) "VLLM_MOONCAKE_BOOTSTRAP_PORT": lambda: int( [](#__codelineno-0-1019) os.getenv("VLLM_MOONCAKE_BOOTSTRAP_PORT", "8998") [](#__codelineno-0-1020) ), [](#__codelineno-0-1021) # Log per-batch memory/disk tier breakdown on external GETs. [](#__codelineno-0-1022) "VLLM_MOONCAKE_STORE_TIER_LOG": lambda: ( [](#__codelineno-0-1023) os.getenv("VLLM_MOONCAKE_STORE_TIER_LOG", "False").lower() in ("true", "1") [](#__codelineno-0-1024) ), [](#__codelineno-0-1025) # Fraction of the owner's DirectIO staging buffer to fill per GET batch. [](#__codelineno-0-1026) "VLLM_MOONCAKE_DISK_STAGING_USABLE_RATIO": lambda: float( [](#__codelineno-0-1027) os.getenv("VLLM_MOONCAKE_DISK_STAGING_USABLE_RATIO", "0.9") [](#__codelineno-0-1028) ), [](#__codelineno-0-1029) # Pin this rank to a specific owner segment ("host:port"). [](#__codelineno-0-1030) "MOONCAKE_PREFERRED_SEGMENT": lambda: os.getenv("MOONCAKE_PREFERRED_SEGMENT"), [](#__codelineno-0-1031) # Override the hostname the rank registers as a Mooncake requester. [](#__codelineno-0-1032) "MOONCAKE_REQUESTER_LOCAL_HOSTNAME": lambda: os.getenv( [](#__codelineno-0-1033) "MOONCAKE_REQUESTER_LOCAL_HOSTNAME" [](#__codelineno-0-1034) ), [](#__codelineno-0-1035) # Flashinfer MoE backend for vLLM's fused Mixture-of-Experts support. [](#__codelineno-0-1036) # Both require compute capability 10.0 or above. [](#__codelineno-0-1037) # Available options: [](#__codelineno-0-1038) # - "throughput": [default] [](#__codelineno-0-1039) # Uses CUTLASS kernels optimized for high-throughput batch inference. [](#__codelineno-0-1040) # - "latency": [](#__codelineno-0-1041) # Uses TensorRT-LLM kernels optimized for low-latency inference. [](#__codelineno-0-1042) # Deprecated: pass --moe-backend flashinfer_{trtllm,cutlass,cutedsl} directly. [](#__codelineno-0-1043) "VLLM_FLASHINFER_MOE_BACKEND": deprecated_env( [](#__codelineno-0-1044) "VLLM_FLASHINFER_MOE_BACKEND", [](#__codelineno-0-1045) "v0.23", [](#__codelineno-0-1046) "Use --moe-backend flashinfer_trtllm, flashinfer_cutlass, or " [](#__codelineno-0-1047) "flashinfer_cutedsl.", [](#__codelineno-0-1048) env_with_choices( [](#__codelineno-0-1049) "VLLM_FLASHINFER_MOE_BACKEND", [](#__codelineno-0-1050) "latency", [](#__codelineno-0-1051) ["throughput", "latency", "masked_gemm"], [](#__codelineno-0-1052) ), [](#__codelineno-0-1053) ), [](#__codelineno-0-1054) # Override the directory for the FlashInfer autotune config cache. [](#__codelineno-0-1055) "VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR": lambda: os.getenv( [](#__codelineno-0-1056) "VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR", None [](#__codelineno-0-1057) ), [](#__codelineno-0-1058) # Flashinfer fused allreduce backend. [](#__codelineno-0-1059) "VLLM_FLASHINFER_ALLREDUCE_BACKEND": env_with_choices( [](#__codelineno-0-1060) "VLLM_FLASHINFER_ALLREDUCE_BACKEND", [](#__codelineno-0-1061) "auto", [](#__codelineno-0-1062) ["auto", "trtllm", "mnnvl"], [](#__codelineno-0-1063) ), [](#__codelineno-0-1064) # Control the workspace buffer size for the FlashInfer backend. [](#__codelineno-0-1065) "VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE": lambda: int( [](#__codelineno-0-1066) os.getenv("VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE", str(394 * 1024 * 1024)) [](#__codelineno-0-1067) ), [](#__codelineno-0-1068) # Control the maximum number of tokens per expert supported by the [](#__codelineno-0-1069) # NVFP4 MoE CUTLASS Kernel. This value is used to create a buffer for [](#__codelineno-0-1070) # the blockscale tensor of activations NVFP4 Quantization. [](#__codelineno-0-1071) # This is used to prevent the kernel from running out of memory. [](#__codelineno-0-1072) "VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE": lambda: int( [](#__codelineno-0-1073) os.getenv("VLLM_MAX_TOKENS_PER_EXPERT_FP4_MOE", "163840") [](#__codelineno-0-1074) ), [](#__codelineno-0-1075) # Specifies the thresholds of the communicated tensor sizes under which [](#__codelineno-0-1076) # vllm should use flashinfer fused allreduce. The variable should be a [](#__codelineno-0-1077) # JSON with the following format: [](#__codelineno-0-1078) # { : } [](#__codelineno-0-1079) # Unspecified world sizes will fall back to [](#__codelineno-0-1080) # { 2: 64, 4: 1, : 0.5 } [](#__codelineno-0-1081) "VLLM_FLASHINFER_ALLREDUCE_FUSION_THRESHOLDS_MB": lambda: json.loads( [](#__codelineno-0-1082) os.getenv("VLLM_FLASHINFER_ALLREDUCE_FUSION_THRESHOLDS_MB", "{}") [](#__codelineno-0-1083) ), [](#__codelineno-0-1084) # MoE routing strategy selector. [](#__codelineno-0-1085) # See `RoutingSimulator.get_available_strategies()` # for available [](#__codelineno-0-1086) # strategies. [](#__codelineno-0-1087) # Custom routing strategies can be registered by [](#__codelineno-0-1088) # RoutingSimulator.register_strategy() [](#__codelineno-0-1089) # Note: custom strategies may not produce correct model outputs [](#__codelineno-0-1090) "VLLM_MOE_ROUTING_SIMULATION_STRATEGY": lambda: os.environ.get( [](#__codelineno-0-1091) "VLLM_MOE_ROUTING_SIMULATION_STRATEGY", "" [](#__codelineno-0-1092) ).lower(), [](#__codelineno-0-1093) # Regex timeout for use by the vLLM tool parsing plugins. [](#__codelineno-0-1094) "VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS": lambda: int( [](#__codelineno-0-1095) os.getenv("VLLM_TOOL_PARSE_REGEX_TIMEOUT_SECONDS", "1") [](#__codelineno-0-1096) ), [](#__codelineno-0-1097) # Control the max chunk bytes (in MB) for the rpc message queue. [](#__codelineno-0-1098) # Object larger than this threshold will be broadcast to worker [](#__codelineno-0-1099) # processes via zmq. [](#__codelineno-0-1100) "VLLM_MQ_MAX_CHUNK_BYTES_MB": lambda: int( [](#__codelineno-0-1101) os.getenv("VLLM_MQ_MAX_CHUNK_BYTES_MB", "16") [](#__codelineno-0-1102) ), [](#__codelineno-0-1103) # Timeout in seconds for execute_model RPC calls in multiprocessing [](#__codelineno-0-1104) # executor (only applies when TP > 1). [](#__codelineno-0-1105) "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": lambda: int( [](#__codelineno-0-1106) os.getenv("VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS", "300") [](#__codelineno-0-1107) ), [](#__codelineno-0-1108) # KV Cache layout used throughout vllm. [](#__codelineno-0-1109) # Some common values are: [](#__codelineno-0-1110) # - NHD [](#__codelineno-0-1111) # - HND [](#__codelineno-0-1112) # Where N=num_blocks, H=num_heads and D=head_size. The default value will [](#__codelineno-0-1113) # leave the layout choice to the backend. Mind that backends may only [](#__codelineno-0-1114) # implement and support a subset of all possible layouts. [](#__codelineno-0-1115) "VLLM_KV_CACHE_LAYOUT": env_with_choices( [](#__codelineno-0-1116) "VLLM_KV_CACHE_LAYOUT", None, ["NHD", "HND"] [](#__codelineno-0-1117) ), [](#__codelineno-0-1118) # SSM conv state layout used for Mamba models. [](#__codelineno-0-1119) # - SD: (state_len, dim) — dim contiguous (default) [](#__codelineno-0-1120) # - DS: (dim, state_len) — TP-sharded dim on dim1, [](#__codelineno-0-1121) # consistent with SSM temporal state and HND KV cache layout. [](#__codelineno-0-1122) "VLLM_SSM_CONV_STATE_LAYOUT": env_with_choices( [](#__codelineno-0-1123) "VLLM_SSM_CONV_STATE_LAYOUT", None, ["SD", "DS"] [](#__codelineno-0-1124) ), [](#__codelineno-0-1125) # Enable checking whether the generated logits contain NaNs, [](#__codelineno-0-1126) # indicating corrupted output. Useful for debugging low level bugs [](#__codelineno-0-1127) # or bad hardware but it may add compute overhead. [](#__codelineno-0-1128) "VLLM_COMPUTE_NANS_IN_LOGITS": lambda: bool( [](#__codelineno-0-1129) int(os.getenv("VLLM_COMPUTE_NANS_IN_LOGITS", "0")) [](#__codelineno-0-1130) ), [](#__codelineno-0-1131) # Controls whether or not emulations are used for NVFP4 [](#__codelineno-0-1132) # generations on machines < 100 for compressed-tensors [](#__codelineno-0-1133) # models [](#__codelineno-0-1134) # Deprecated: use --linear-backend emulation instead. [](#__codelineno-0-1135) "VLLM_USE_NVFP4_CT_EMULATIONS": deprecated_env( [](#__codelineno-0-1136) "VLLM_USE_NVFP4_CT_EMULATIONS", [](#__codelineno-0-1137) "v0.23", [](#__codelineno-0-1138) "Use --linear-backend emulation.", [](#__codelineno-0-1139) lambda: bool(int(os.getenv("VLLM_USE_NVFP4_CT_EMULATIONS", "0"))), [](#__codelineno-0-1140) ), [](#__codelineno-0-1141) # Timeout (in seconds) for MooncakeConnector in PD disaggregated setup. [](#__codelineno-0-1142) "VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT": lambda: int( [](#__codelineno-0-1143) os.getenv("VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT", "480") [](#__codelineno-0-1144) ), [](#__codelineno-0-1145) # If set, it means we pre-downloaded cubin files and flashinfer will [](#__codelineno-0-1146) # read the cubin files directly. [](#__codelineno-0-1147) "VLLM_HAS_FLASHINFER_CUBIN": lambda: bool( [](#__codelineno-0-1148) int(os.getenv("VLLM_HAS_FLASHINFER_CUBIN", "0")) [](#__codelineno-0-1149) ), [](#__codelineno-0-1150) # Supported options: [](#__codelineno-0-1151) # - "flashinfer-cudnn": use flashinfer cudnn GEMM backend [](#__codelineno-0-1152) # - "flashinfer-trtllm": use flashinfer trtllm GEMM backend [](#__codelineno-0-1153) # - "flashinfer-cutlass": use flashinfer cutlass GEMM backend [](#__codelineno-0-1154) # - "marlin": use marlin GEMM backend (for GPUs without native FP4 support) [](#__codelineno-0-1155) # - "emulation": [](#__codelineno-0-1156) # use BF16/FP16 GEMM, dequantizing weights and running QDQ on activations. [](#__codelineno-0-1157) # This is only meant for research purposes to run on devices where NVFP4 [](#__codelineno-0-1158) # GEMM kernels are not available. [](#__codelineno-0-1159) # - : automatically pick an available backend [](#__codelineno-0-1160) # Deprecated: use --linear-backend instead. [](#__codelineno-0-1161) "VLLM_NVFP4_GEMM_BACKEND": deprecated_env( [](#__codelineno-0-1162) "VLLM_NVFP4_GEMM_BACKEND", [](#__codelineno-0-1163) "v0.23", [](#__codelineno-0-1164) "Use --linear-backend.", [](#__codelineno-0-1165) env_with_choices( [](#__codelineno-0-1166) "VLLM_NVFP4_GEMM_BACKEND", [](#__codelineno-0-1167) None, [](#__codelineno-0-1168) [ [](#__codelineno-0-1169) "flashinfer-b12x", [](#__codelineno-0-1170) "flashinfer-cudnn", [](#__codelineno-0-1171) "flashinfer-trtllm", [](#__codelineno-0-1172) "flashinfer-cutlass", [](#__codelineno-0-1173) "cutlass", [](#__codelineno-0-1174) "marlin", [](#__codelineno-0-1175) "emulation", [](#__codelineno-0-1176) ], [](#__codelineno-0-1177) ), [](#__codelineno-0-1178) ), [](#__codelineno-0-1179) # Controls garbage collection during CUDA graph capture. [](#__codelineno-0-1180) # If set to 0 (default), enables GC freezing to speed up capture time. [](#__codelineno-0-1181) # If set to 1, allows GC to run during capture. [](#__codelineno-0-1182) "VLLM_ENABLE_CUDAGRAPH_GC": lambda: bool( [](#__codelineno-0-1183) int(os.getenv("VLLM_ENABLE_CUDAGRAPH_GC", "0")) [](#__codelineno-0-1184) ), [](#__codelineno-0-1185) # Used to force set up loopback IP [](#__codelineno-0-1186) "VLLM_LOOPBACK_IP": lambda: os.getenv("VLLM_LOOPBACK_IP", ""), [](#__codelineno-0-1187) # Used to set the process name prefix for vLLM processes. [](#__codelineno-0-1188) # This is useful for debugging and monitoring purposes. [](#__codelineno-0-1189) # The default value is "VLLM". [](#__codelineno-0-1190) "VLLM_PROCESS_NAME_PREFIX": lambda: os.getenv("VLLM_PROCESS_NAME_PREFIX", "VLLM"), [](#__codelineno-0-1191) # Allow chunked local attention with hybrid kv cache manager. [](#__codelineno-0-1192) # Currently using the Hybrid KV cache manager with chunked local attention [](#__codelineno-0-1193) # in the Llama4 models (the only models currently using chunked local attn) [](#__codelineno-0-1194) # causes a latency regression. For this reason, we disable it by default. [](#__codelineno-0-1195) # This flag is used to allow users to enable it if they want to (to save on [](#__codelineno-0-1196) # kv-cache memory usage and enable longer contexts) [](#__codelineno-0-1197) # TODO(lucas): Remove this flag once latency regression is resolved. [](#__codelineno-0-1198) "VLLM_ALLOW_CHUNKED_LOCAL_ATTN_WITH_HYBRID_KV_CACHE": lambda: bool( [](#__codelineno-0-1199) int(os.getenv("VLLM_ALLOW_CHUNKED_LOCAL_ATTN_WITH_HYBRID_KV_CACHE", "1")) [](#__codelineno-0-1200) ), [](#__codelineno-0-1201) # Enables support for the "store" option in the OpenAI Responses API. [](#__codelineno-0-1202) # When set to 1, vLLM's OpenAI server will retain the input and output [](#__codelineno-0-1203) # messages for those requests in memory. By default, this is disabled (0), [](#__codelineno-0-1204) # and the "store" option is ignored. [](#__codelineno-0-1205) # NOTE/WARNING: [](#__codelineno-0-1206) # 1. Messages are kept in memory only (not persisted to disk) and will be [](#__codelineno-0-1207) # lost when the vLLM server shuts down. [](#__codelineno-0-1208) # 2. Enabling this option will cause a memory leak, as stored messages are [](#__codelineno-0-1209) # never removed from memory until the server terminates. [](#__codelineno-0-1210) "VLLM_ENABLE_RESPONSES_API_STORE": lambda: bool( [](#__codelineno-0-1211) int(os.getenv("VLLM_ENABLE_RESPONSES_API_STORE", "0")) [](#__codelineno-0-1212) ), [](#__codelineno-0-1213) # If set, use the fp8 mfma in rocm paged attention. [](#__codelineno-0-1214) "VLLM_ROCM_FP8_MFMA_PAGE_ATTN": lambda: bool( [](#__codelineno-0-1215) int(os.getenv("VLLM_ROCM_FP8_MFMA_PAGE_ATTN", "0")) [](#__codelineno-0-1216) ), [](#__codelineno-0-1217) # Whether to use pytorch symmetric memory for allreduce [](#__codelineno-0-1218) "VLLM_ALLREDUCE_USE_SYMM_MEM": lambda: bool( [](#__codelineno-0-1219) int(os.getenv("VLLM_ALLREDUCE_USE_SYMM_MEM", "1")) [](#__codelineno-0-1220) ), [](#__codelineno-0-1221) # Whether to use FlashInfer allreduce [](#__codelineno-0-1222) "VLLM_ALLREDUCE_USE_FLASHINFER": lambda: bool( [](#__codelineno-0-1223) int(os.getenv("VLLM_ALLREDUCE_USE_FLASHINFER", "0")) [](#__codelineno-0-1224) ), [](#__codelineno-0-1225) # Experimental: use this to enable MCP tool calling for non harmony models [](#__codelineno-0-1226) "VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT": lambda: bool( [](#__codelineno-0-1227) int(os.getenv("VLLM_USE_EXPERIMENTAL_PARSER_CONTEXT", "0")) [](#__codelineno-0-1228) ), [](#__codelineno-0-1229) # User override folder for tuned Triton-kernel configs. Shared by MoE, [](#__codelineno-0-1230) # Mamba SSU, and LoRA. Filenames are distinct so one folder can hold all. [](#__codelineno-0-1231) # Each component first checks this folder, then the configs shipped with [](#__codelineno-0-1232) # vLLM (if any). If no JSON matches, it uses a hard-coded heuristic. [](#__codelineno-0-1233) "VLLM_TUNED_CONFIG_FOLDER": lambda: os.getenv("VLLM_TUNED_CONFIG_FOLDER", None), [](#__codelineno-0-1234) # Valid values are container,code_interpreter,web_search_preview [](#__codelineno-0-1235) # ex VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS=container,code_interpreter [](#__codelineno-0-1236) # If the server_label of your mcp tool is not in this list it will [](#__codelineno-0-1237) # be completely ignored. [](#__codelineno-0-1238) "VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS": env_set_with_choices( [](#__codelineno-0-1239) "VLLM_GPT_OSS_SYSTEM_TOOL_MCP_LABELS", [](#__codelineno-0-1240) default=[], [](#__codelineno-0-1241) choices=["container", "code_interpreter", "web_search_preview"], [](#__codelineno-0-1242) ), [](#__codelineno-0-1243) # Allows harmony instructions to be injected on system messages [](#__codelineno-0-1244) "VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS": lambda: bool( [](#__codelineno-0-1245) int(os.getenv("VLLM_GPT_OSS_HARMONY_SYSTEM_INSTRUCTIONS", "0")) [](#__codelineno-0-1246) ), [](#__codelineno-0-1247) # Pin the conversation start date injected into the Harmony system [](#__codelineno-0-1248) # message. When unset the current date is used, which introduces [](#__codelineno-0-1249) # non-determinism (different tokens -> different model behaviour at [](#__codelineno-0-1250) # temperature=0). Set to an ISO date string, e.g. "2023-09-12", [](#__codelineno-0-1251) # for reproducible inference or testing. [](#__codelineno-0-1252) "VLLM_SYSTEM_START_DATE": lambda: os.getenv("VLLM_SYSTEM_START_DATE", None), [](#__codelineno-0-1253) # Enable automatic retry when tool call JSON parsing fails [](#__codelineno-0-1254) # If enabled, returns an error message to the model to retry [](#__codelineno-0-1255) # If disabled (default), raises an exception and fails the request [](#__codelineno-0-1256) "VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY": lambda: bool( [](#__codelineno-0-1257) int(os.getenv("VLLM_TOOL_JSON_ERROR_AUTOMATIC_RETRY", "0")) [](#__codelineno-0-1258) ), [](#__codelineno-0-1259) # When 1,the model structural tags will be used to enforce the model [](#__codelineno-0-1260) # output conforming to the model's tool-calling format and schema. [](#__codelineno-0-1261) # Default 0 (off). [](#__codelineno-0-1262) "VLLM_ENFORCE_STRICT_TOOL_CALLING": lambda: bool( [](#__codelineno-0-1263) int(os.getenv("VLLM_ENFORCE_STRICT_TOOL_CALLING", "0")) [](#__codelineno-0-1264) ), [](#__codelineno-0-1265) # Add optional custom scopes for profiling, disable to avoid overheads [](#__codelineno-0-1266) "VLLM_CUSTOM_SCOPES_FOR_PROFILING": lambda: bool( [](#__codelineno-0-1267) int(os.getenv("VLLM_CUSTOM_SCOPES_FOR_PROFILING", "0")) [](#__codelineno-0-1268) ), [](#__codelineno-0-1269) # Add optional nvtx scopes for profiling, disable to avoid overheads [](#__codelineno-0-1270) "VLLM_NVTX_SCOPES_FOR_PROFILING": lambda: bool( [](#__codelineno-0-1271) int(os.getenv("VLLM_NVTX_SCOPES_FOR_PROFILING", "0")) [](#__codelineno-0-1272) ), [](#__codelineno-0-1273) # Represent block hashes in KV cache events as 64-bit integers instead of [](#__codelineno-0-1274) # raw bytes. Defaults to True for backward compatibility. [](#__codelineno-0-1275) "VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES": lambda: bool( [](#__codelineno-0-1276) int(os.getenv("VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES", "1")) [](#__codelineno-0-1277) ), [](#__codelineno-0-1278) # Name of the shared memory buffer used for object storage. [](#__codelineno-0-1279) # Only effective when mm_config.mm_processor_cache_type == "shm". [](#__codelineno-0-1280) # Automatically generates a unique UUID-based name per process tree [](#__codelineno-0-1281) # if not explicitly set. [](#__codelineno-0-1282) "VLLM_OBJECT_STORAGE_SHM_BUFFER_NAME": get_env_or_set_default( [](#__codelineno-0-1283) "VLLM_OBJECT_STORAGE_SHM_BUFFER_NAME", [](#__codelineno-0-1284) lambda: f"VLLM_OBJECT_STORAGE_SHM_BUFFER_{uuid.uuid4().hex}", [](#__codelineno-0-1285) ), [](#__codelineno-0-1286) # The size in MB of the buffers (NVL and RDMA) used by DeepEP [](#__codelineno-0-1287) "VLLM_DEEPEP_BUFFER_SIZE_MB": lambda: int( [](#__codelineno-0-1288) os.getenv("VLLM_DEEPEP_BUFFER_SIZE_MB", "1024") [](#__codelineno-0-1289) ), [](#__codelineno-0-1290) # Force DeepEP to use intranode kernel for inter-node communication in [](#__codelineno-0-1291) # high throughput mode. This is useful archive higher prefill throughput [](#__codelineno-0-1292) # on system supports multi-node nvlink (e.g GB200). [](#__codelineno-0-1293) "VLLM_DEEPEP_HIGH_THROUGHPUT_FORCE_INTRA_NODE": lambda: bool( [](#__codelineno-0-1294) int(os.getenv("VLLM_DEEPEP_HIGH_THROUGHPUT_FORCE_INTRA_NODE", "0")) [](#__codelineno-0-1295) ), [](#__codelineno-0-1296) # Allow DeepEP to use MNNVL (multi-node nvlink) for internode_ll kernel, [](#__codelineno-0-1297) # turn this for better latency on GB200 like system [](#__codelineno-0-1298) "VLLM_DEEPEP_LOW_LATENCY_USE_MNNVL": lambda: bool( [](#__codelineno-0-1299) int(os.getenv("VLLM_DEEPEP_LOW_LATENCY_USE_MNNVL", "0")) [](#__codelineno-0-1300) ), [](#__codelineno-0-1301) # The number of SMs/CUs to allocate for communication kernels when [](#__codelineno-0-1302) # running DBO; the rest will be allocated to compute. [](#__codelineno-0-1303) # Default: 20 on CUDA (SMs), 64 on ROCm (CUs). [](#__codelineno-0-1304) "VLLM_DBO_COMM_SMS": lambda: int( [](#__codelineno-0-1305) os.getenv( [](#__codelineno-0-1306) "VLLM_DBO_COMM_SMS", [](#__codelineno-0-1307) "64" [](#__codelineno-0-1308) if hasattr(__import__("torch").version, "hip") [](#__codelineno-0-1309) and __import__("torch").version.hip is not None [](#__codelineno-0-1310) else "20", [](#__codelineno-0-1311) ) [](#__codelineno-0-1312) ), [](#__codelineno-0-1313) # Enable max_autotune & coordinate_descent_tuning in inductor_config [](#__codelineno-0-1314) # to compile static shapes passed from compile_sizes in compilation_config [](#__codelineno-0-1315) # If set to 1, enable max_autotune; By default, this is enabled (1) [](#__codelineno-0-1316) "VLLM_ENABLE_INDUCTOR_MAX_AUTOTUNE": lambda: bool( [](#__codelineno-0-1317) int(os.getenv("VLLM_ENABLE_INDUCTOR_MAX_AUTOTUNE", "1")) [](#__codelineno-0-1318) ), [](#__codelineno-0-1319) # If set to 1, enable coordinate_descent_tuning; [](#__codelineno-0-1320) # By default, this is enabled (1) [](#__codelineno-0-1321) "VLLM_ENABLE_INDUCTOR_COORDINATE_DESCENT_TUNING": lambda: bool( [](#__codelineno-0-1322) int(os.getenv("VLLM_ENABLE_INDUCTOR_COORDINATE_DESCENT_TUNING", "1")) [](#__codelineno-0-1323) ), [](#__codelineno-0-1324) # Flag to enable NCCL symmetric memory allocation and registration [](#__codelineno-0-1325) "VLLM_USE_NCCL_SYMM_MEM": lambda: bool( [](#__codelineno-0-1326) int(os.getenv("VLLM_USE_NCCL_SYMM_MEM", "0")) [](#__codelineno-0-1327) ), [](#__codelineno-0-1328) # NCCL header path [](#__codelineno-0-1329) "VLLM_NCCL_INCLUDE_PATH": lambda: os.environ.get("VLLM_NCCL_INCLUDE_PATH", None), [](#__codelineno-0-1330) # Flag to enable FBGemm kernels on model execution [](#__codelineno-0-1331) # Deprecated: use --linear-backend fbgemm instead. [](#__codelineno-0-1332) "VLLM_USE_FBGEMM": deprecated_env( [](#__codelineno-0-1333) "VLLM_USE_FBGEMM", [](#__codelineno-0-1334) "v0.23", [](#__codelineno-0-1335) "Use --linear-backend fbgemm.", [](#__codelineno-0-1336) lambda: bool(int(os.getenv("VLLM_USE_FBGEMM", "0"))), [](#__codelineno-0-1337) ), [](#__codelineno-0-1338) # GC debug config [](#__codelineno-0-1339) # - VLLM_GC_DEBUG=0: disable GC debugger [](#__codelineno-0-1340) # - VLLM_GC_DEBUG=1: enable GC debugger with gc.collect elpased times [](#__codelineno-0-1341) # - VLLM_GC_DEBUG='{"top_objects":5}': enable GC debugger with [](#__codelineno-0-1342) # top 5 collected objects [](#__codelineno-0-1343) "VLLM_GC_DEBUG": lambda: os.getenv("VLLM_GC_DEBUG", ""), [](#__codelineno-0-1344) # Debug workspace allocations. [](#__codelineno-0-1345) # logging of workspace resize operations. [](#__codelineno-0-1346) "VLLM_DEBUG_WORKSPACE": lambda: bool(int(os.getenv("VLLM_DEBUG_WORKSPACE", "0"))), [](#__codelineno-0-1347) # Disables parallel execution of shared_experts via separate cuda stream [](#__codelineno-0-1348) "VLLM_DISABLE_SHARED_EXPERTS_STREAM": lambda: bool( [](#__codelineno-0-1349) int(os.getenv("VLLM_DISABLE_SHARED_EXPERTS_STREAM", "0")) [](#__codelineno-0-1350) ), [](#__codelineno-0-1351) # Limits when we run shared_experts in a separate stream. [](#__codelineno-0-1352) # We found out that for large batch sizes, the separate stream [](#__codelineno-0-1353) # execution is not beneficial (most likely because of the input clone) [](#__codelineno-0-1354) # TODO(alexm-redhat): Tune to be more dynamic based on GPU type [](#__codelineno-0-1355) "VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD": lambda: int( [](#__codelineno-0-1356) int(os.getenv("VLLM_SHARED_EXPERTS_STREAM_TOKEN_THRESHOLD", 256)) [](#__codelineno-0-1357) ), [](#__codelineno-0-1358) # Token-count cutoff for multi-stream overlap of the attention input [](#__codelineno-0-1359) # GEMM with auxiliary GEMMs (e.g. fused_wqa_wkv overlapped with indexer [](#__codelineno-0-1360) # weights / kv-score projections in DeepSeek-V4). At or below this many [](#__codelineno-0-1361) # tokens the FP8 main GEMM has idle SMs to share with the bf16 aux GEMMs [](#__codelineno-0-1362) # and overlap is a 5-45% win; above it the FP8 GEMM saturates the device [](#__codelineno-0-1363) # and the cross-stream sync becomes pure overhead. Set to 0 to disable [](#__codelineno-0-1364) # the multi-stream path entirely. See #PR 41526 for the empirical result [](#__codelineno-0-1365) # for the default value of 1024 tokens. [](#__codelineno-0-1366) "VLLM_MULTI_STREAM_GEMM_TOKEN_THRESHOLD": lambda: int( [](#__codelineno-0-1367) os.getenv("VLLM_MULTI_STREAM_GEMM_TOKEN_THRESHOLD", "1024") [](#__codelineno-0-1368) ), [](#__codelineno-0-1369) # Format for saving torch.compile cache artifacts [](#__codelineno-0-1370) # - "binary": saves as binary file [](#__codelineno-0-1371) # Safe for multiple vllm serve processes accessing the same torch compile cache. [](#__codelineno-0-1372) # - "unpacked": saves as directory structure (for inspection/debugging) [](#__codelineno-0-1373) # NOT multiprocess safe - race conditions may occur with multiple processes. [](#__codelineno-0-1374) # Allows viewing and setting breakpoints in Inductor's code output files. [](#__codelineno-0-1375) "VLLM_COMPILE_CACHE_SAVE_FORMAT": env_with_choices( [](#__codelineno-0-1376) "VLLM_COMPILE_CACHE_SAVE_FORMAT", "binary", ["binary", "unpacked"] [](#__codelineno-0-1377) ), [](#__codelineno-0-1378) # Flag to control the v2 model runner. If unset, use config defaults. [](#__codelineno-0-1379) "VLLM_USE_V2_MODEL_RUNNER": lambda: maybe_convert_bool( [](#__codelineno-0-1380) os.getenv("VLLM_USE_V2_MODEL_RUNNER", None) [](#__codelineno-0-1381) ), [](#__codelineno-0-1382) # Log model inspection after loading. [](#__codelineno-0-1383) # If enabled, logs a transformers-style hierarchical view of the model [](#__codelineno-0-1384) # with quantization methods and attention backends. [](#__codelineno-0-1385) "VLLM_LOG_MODEL_INSPECTION": lambda: bool( [](#__codelineno-0-1386) int(os.getenv("VLLM_LOG_MODEL_INSPECTION", "0")) [](#__codelineno-0-1387) ), [](#__codelineno-0-1388) # Debug logging for --enable-mfu-metrics [](#__codelineno-0-1389) "VLLM_DEBUG_MFU_METRICS": lambda: bool( [](#__codelineno-0-1390) int(os.getenv("VLLM_DEBUG_MFU_METRICS", "0")) [](#__codelineno-0-1391) ), [](#__codelineno-0-1392) # Disable using pytorch's pin memory for CPU offloading. [](#__codelineno-0-1393) "VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY": lambda: bool( [](#__codelineno-0-1394) int(os.getenv("VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY", "0")) [](#__codelineno-0-1395) ), [](#__codelineno-0-1396) # Disable using UVA (Unified Virtual Addressing) for CPU offloading. [](#__codelineno-0-1397) "VLLM_WEIGHT_OFFLOADING_DISABLE_UVA": lambda: bool( [](#__codelineno-0-1398) int(os.getenv("VLLM_WEIGHT_OFFLOADING_DISABLE_UVA", "0")) [](#__codelineno-0-1399) ), [](#__codelineno-0-1400) # Disable logging of vLLM logo at server startup time. [](#__codelineno-0-1401) "VLLM_DISABLE_LOG_LOGO": lambda: bool(int(os.getenv("VLLM_DISABLE_LOG_LOGO", "0"))), [](#__codelineno-0-1402) # Disable PDL for LoRA, as enabling PDL with LoRA on SM100 causes [](#__codelineno-0-1403) # Triton compilation to fail. [](#__codelineno-0-1404) "VLLM_LORA_DISABLE_PDL": lambda: bool(int(os.getenv("VLLM_LORA_DISABLE_PDL", "0"))), [](#__codelineno-0-1405) # Enable CUDA compatibility mode for datacenter GPUs with older [](#__codelineno-0-1406) # driver versions than the CUDA toolkit major version of vLLM. [](#__codelineno-0-1407) "VLLM_ENABLE_CUDA_COMPATIBILITY": lambda: ( [](#__codelineno-0-1408) os.environ.get("VLLM_ENABLE_CUDA_COMPATIBILITY", "0").strip().lower() [](#__codelineno-0-1409) in ("1", "true") [](#__codelineno-0-1410) ), [](#__codelineno-0-1411) # Path to the CUDA compatibility libraries when CUDA compatibility is enabled. [](#__codelineno-0-1412) "VLLM_CUDA_COMPATIBILITY_PATH": lambda: os.environ.get( [](#__codelineno-0-1413) "VLLM_CUDA_COMPATIBILITY_PATH", None [](#__codelineno-0-1414) ), [](#__codelineno-0-1415) # Skip model name validation in OpenAI API requests. [](#__codelineno-0-1416) # When set to 1, any model name will be accepted in the 'model' field [](#__codelineno-0-1417) # of API requests. This is useful for proxy/gateway scenarios where [](#__codelineno-0-1418) # the actual model is served but different names may be used in requests. [](#__codelineno-0-1419) "VLLM_SKIP_MODEL_NAME_VALIDATION": lambda: ( [](#__codelineno-0-1420) os.getenv("VLLM_SKIP_MODEL_NAME_VALIDATION", "0").strip().lower() [](#__codelineno-0-1421) in ("1", "true") [](#__codelineno-0-1422) ), [](#__codelineno-0-1423) # Whether it is a scale up launch engine for elastic EP, [](#__codelineno-0-1424) # Should only be set by EngineCoreClient. [](#__codelineno-0-1425) "VLLM_ELASTIC_EP_SCALE_UP_LAUNCH": lambda: bool( [](#__codelineno-0-1426) int(os.getenv("VLLM_ELASTIC_EP_SCALE_UP_LAUNCH", "0")) [](#__codelineno-0-1427) ), [](#__codelineno-0-1428) # Whether to wait for all requests to drain before sending the [](#__codelineno-0-1429) # scaling command in elastic EP. [](#__codelineno-0-1430) "VLLM_ELASTIC_EP_DRAIN_REQUESTS": lambda: bool( [](#__codelineno-0-1431) int(os.getenv("VLLM_ELASTIC_EP_DRAIN_REQUESTS", "0")) [](#__codelineno-0-1432) ), [](#__codelineno-0-1433) # If set to 1, enable CUDA graph memory estimation during memory profiling. [](#__codelineno-0-1434) # This profiles CUDA graph memory usage to provide more accurate KV cache [](#__codelineno-0-1435) # memory allocation. Enabled by default as of v0.21.0 [](#__codelineno-0-1436) "VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS": lambda: bool( [](#__codelineno-0-1437) int(os.getenv("VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS", "1")) [](#__codelineno-0-1438) ), [](#__codelineno-0-1439) # NIXL EP environment variables [](#__codelineno-0-1440) "VLLM_NIXL_EP_MAX_NUM_RANKS": lambda: int( [](#__codelineno-0-1441) os.getenv("VLLM_NIXL_EP_MAX_NUM_RANKS", "32") [](#__codelineno-0-1442) ), [](#__codelineno-0-1443) # Whether enable XPU graph on Intel GPU [](#__codelineno-0-1444) "VLLM_XPU_ENABLE_XPU_GRAPH": lambda: bool( [](#__codelineno-0-1445) int(os.getenv("VLLM_XPU_ENABLE_XPU_GRAPH", "0")) [](#__codelineno-0-1446) ), [](#__codelineno-0-1447) # whether use xpu specific sample kernel [](#__codelineno-0-1448) "VLLM_XPU_USE_SAMPLER_KERNEL": lambda: bool( [](#__codelineno-0-1449) int(os.getenv("VLLM_XPU_USE_SAMPLER_KERNEL", "1")) [](#__codelineno-0-1450) ), [](#__codelineno-0-1451) # Enable simple KV offload. [](#__codelineno-0-1452) "VLLM_USE_SIMPLE_KV_OFFLOAD": lambda: bool( [](#__codelineno-0-1453) int(os.getenv("VLLM_USE_SIMPLE_KV_OFFLOAD", "0")) [](#__codelineno-0-1454) ), [](#__codelineno-0-1455) # Whether to enable dual cuda streams for LoRA computation [](#__codelineno-0-1456) # (used by both BaseLinearLayerWithLoRA and FusedMoEWithLoRA to [](#__codelineno-0-1457) # overlap the base layer compute with the LoRA fast path). [](#__codelineno-0-1458) "VLLM_LORA_ENABLE_DUAL_STREAM": lambda: bool( [](#__codelineno-0-1459) int(os.getenv("VLLM_LORA_ENABLE_DUAL_STREAM", "0")) [](#__codelineno-0-1460) ), [](#__codelineno-0-1461) # If set to 1, use Python spinloop extension to poll in a more efficient [](#__codelineno-0-1462) # way when using the mp backend. [](#__codelineno-0-1463) "VLLM_USE_SPINLOOP_EXT": lambda: bool(int(os.getenv("VLLM_USE_SPINLOOP_EXT", "0"))), [](#__codelineno-0-1464) # Comma-separated GPU_BDF=NIC_BDF pairs for RDMA NIC selection. [](#__codelineno-0-1465) # Must be set together with VLLM_NIC_SELECTION_VARS. [](#__codelineno-0-1466) "VLLM_GPU_NIC_PCIE_MAPPING": lambda: os.getenv("VLLM_GPU_NIC_PCIE_MAPPING", ""), [](#__codelineno-0-1467) # Comma-separated list of env vars to set from the GPU-NIC mapping. [](#__codelineno-0-1468) # Each entry is VAR_NAME or VAR_NAME: (suffix appended to [](#__codelineno-0-1469) # RDMA device name). Must be set together with VLLM_GPU_NIC_PCIE_MAPPING. [](#__codelineno-0-1470) "VLLM_NIC_SELECTION_VARS": lambda: os.getenv("VLLM_NIC_SELECTION_VARS", ""), [](#__codelineno-0-1471)}``` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/configuration/model_resolution.md "Edit this page") vLLM loads HuggingFace-compatible models by inspecting the `architectures` field in `config.json` of the model repository and finding the corresponding implementation that is registered to vLLM. Nevertheless, our model resolution may fail for the following reasons: - The `config.json` of the model repository lacks the `architectures` field. - Unofficial repositories refer to a model using alternative names which are not recorded in vLLM. - The same architecture name is used for multiple models, creating ambiguity as to which model should be loaded. To fix this, explicitly specify the model architecture by passing `config.json` overrides to the `hf_overrides` option. For example: `[](#__codelineno-0-1)from vllm import LLM [](#__codelineno-0-2)[](#__codelineno-0-3)llm = LLM( [](#__codelineno-0-4) model="cerebras/Cerebras-GPT-1.3B", [](#__codelineno-0-5) hf_overrides={"architectures": ["GPT2LMHeadModel"]}, # GPT-2 [](#__codelineno-0-6))` Our [list of supported models](https://docs.vllm.ai/en/models/supported_models/) shows the model architectures that are recognized by vLLM. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/configuration/optimization.md "Edit this page") This guide covers optimization strategies and performance tuning for vLLM V1. Tip Running out of memory? Consult [this guide](https://docs.vllm.ai/en/latest/conserving_memory/) on how to conserve memory. ## Optimization Levels[¶](#optimization-levels "Permanent link") vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow users to trade off startup time for performance: - `-O0`: No optimizations. Fastest startup time, but lowest performance. - `-O1`: Fast optimization. Simple compilation and fast fusions, and PIECEWISE cudagraphs. - `-O2`: Default optimization. Additional compilation ranges, additional fusions, FULL\_AND\_PIECEWISE cudagraphs. - `-O3`: Aggressive optimization. Currently equal to `-O2`, but may include additional time-consuming or experimental optimizations in the future. For more information, see the [optimization level documentation](https://docs.vllm.ai/en/design/optimization_levels/). ## Preemption[¶](#preemption "Permanent link") Due to the autoregressive nature of transformer architecture, there are times when KV cache space is insufficient to handle all batched requests. In such cases, vLLM can preempt requests to free up KV cache space for other requests. Preempted requests are recomputed when sufficient KV cache space becomes available again. When this occurs, you may see the following warning: `[](#__codelineno-0-1)WARNING 05-09 00:49:33 scheduler.py:1057 Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space. This can affect the end-to-end performance. Increase gpu_memory_utilization or tensor_parallel_size to provide more KV cache memory. total_cumulative_preemption_cnt=1` While this mechanism ensures system robustness, preemption and recomputation can adversely affect end-to-end latency. If you frequently encounter preemptions, consider the following actions: - Increase `gpu_memory_utilization`. vLLM pre-allocates GPU cache using this percentage of memory. By increasing utilization, you can provide more KV cache space. - Decrease `max_num_seqs` or `max_num_batched_tokens`. This reduces the number of concurrent requests in a batch, thereby requiring less KV cache space. - Increase `tensor_parallel_size`. This shards model weights across GPUs, allowing each GPU to have more memory available for KV cache. However, increasing this value may cause excessive synchronization overhead. - Increase `pipeline_parallel_size`. This distributes model layers across GPUs, reducing the memory needed for model weights on each GPU, indirectly leaving more memory available for KV cache. However, increasing this value may cause latency penalties. You can monitor the number of preemption requests through Prometheus metrics exposed by vLLM. Additionally, you can log the cumulative number of preemption requests by setting `disable_log_stats=False`. In vLLM V1, the default preemption mode is `RECOMPUTE` rather than `SWAP`, as recomputation has lower overhead in the V1 architecture. ## Chunked Prefill[¶](#chunked-prefill "Permanent link") Chunked prefill allows vLLM to process large prefills in smaller chunks and batch them together with decode requests. This feature helps improve both throughput and latency by better balancing compute-bound (prefill) and memory-bound (decode) operations. In V1, **chunked prefill is enabled by default whenever possible**. With chunked prefill enabled, the scheduling policy prioritizes decode requests. It batches all pending decode requests before scheduling any prefill operations. When there are available tokens in the `max_num_batched_tokens` budget, it schedules pending prefills. If a pending prefill request cannot fit into `max_num_batched_tokens`, it automatically chunks it. This policy has two benefits: - It improves inter-token latency (ITL) and generation decode because decode requests are prioritized. - It helps achieve better GPU utilization by locating compute-bound (prefill) and memory-bound (decode) requests to the same batch. ### Performance Tuning with Chunked Prefill[¶](#performance-tuning-with-chunked-prefill "Permanent link") You can tune the performance by adjusting `max_num_batched_tokens`: - Smaller values (e.g., 2048) achieve better ITL because there are fewer prefills slowing down decodes. - Higher values achieve better time to first token (TTFT) as you can process more prefill tokens in a batch. - For optimal throughput, we recommend setting `max_num_batched_tokens > 8192` especially for smaller models on large GPUs. - If `max_num_batched_tokens` is the same as `max_model_len`, that's almost the equivalent to the V0 default scheduling policy (except that it still prioritizes decodes). Warning When chunked prefill is disabled, `max_num_batched_tokens` must be greater than `max_model_len`. In that case, if `max_num_batched_tokens < max_model_len`, vLLM may crash at server start‑up. `[](#__codelineno-1-1)from vllm import LLM [](#__codelineno-1-2)[](#__codelineno-1-3)# Set max_num_batched_tokens to tune performance [](#__codelineno-1-4)llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", max_num_batched_tokens=16384)` See related papers for more details ([https://arxiv.org/pdf/2401.08671](https://arxiv.org/pdf/2401.08671) or [https://arxiv.org/pdf/2308.16369](https://arxiv.org/pdf/2308.16369)). ## Parallelism Strategies[¶](#parallelism-strategies "Permanent link") vLLM supports multiple parallelism strategies that can be combined to optimize performance across different hardware configurations. ### Tensor Parallelism (TP)[¶](#tensor-parallelism-tp "Permanent link") Tensor parallelism shards model parameters across multiple GPUs within each model layer. This is the most common strategy for large model inference within a single node. **When to use:** - When the model is too large to fit on a single GPU - When you need to reduce memory pressure per GPU to allow more KV cache space for higher throughput `[](#__codelineno-2-1)from vllm import LLM [](#__codelineno-2-2)[](#__codelineno-2-3)# Split model across 4 GPUs [](#__codelineno-2-4)llm = LLM(model="meta-llama/Llama-3.3-70B-Instruct", tensor_parallel_size=4)` For models that are too large to fit on a single GPU (like 70B parameter models), tensor parallelism is essential. ### Pipeline Parallelism (PP)[¶](#pipeline-parallelism-pp "Permanent link") Pipeline parallelism distributes model layers across multiple GPUs. Each GPU processes different parts of the model in sequence. **When to use:** - When you've already maxed out efficient tensor parallelism but need to distribute the model further, or across nodes - For very deep and narrow models where layer distribution is more efficient than tensor sharding Pipeline parallelism can be combined with tensor parallelism for very large models: `[](#__codelineno-3-1)from vllm import LLM [](#__codelineno-3-2)[](#__codelineno-3-3)# Combine pipeline and tensor parallelism [](#__codelineno-3-4)llm = LLM( [](#__codelineno-3-5) model="meta-llama/Llama-3.3-70B-Instruct, [](#__codelineno-3-6) tensor_parallel_size=4, [](#__codelineno-3-7) pipeline_parallel_size=2, [](#__codelineno-3-8))` ### Expert Parallelism (EP)[¶](#expert-parallelism-ep "Permanent link") Expert parallelism is a specialized form of parallelism for Mixture of Experts (MoE) models, where different expert networks are distributed across GPUs. **When to use:** - Specifically for MoE models (like DeepSeekV3, Qwen3MoE, Llama-4) - When you want to balance the expert computation load across GPUs Expert parallelism is enabled by setting `enable_expert_parallel=True`, which will use expert parallelism instead of tensor parallelism for MoE layers. It will use the same degree of parallelism as what you have set for tensor parallelism. ### Data Parallelism (DP)[¶](#data-parallelism-dp "Permanent link") Data parallelism replicates the entire model across multiple GPU sets and processes different batches of requests in parallel. **When to use:** - When you have enough GPUs to replicate the entire model - When you need to scale throughput rather than model size - In multi-user environments where isolation between request batches is beneficial Data parallelism can be combined with the other parallelism strategies and is set by `data_parallel_size=N`. Note that MoE layers will be sharded according to the product of the tensor parallel size and data parallel size. ### NUMA Binding for Multi-Socket GPU Nodes[¶](#numa-binding-for-multi-socket-gpu-nodes "Permanent link") On multi-socket GPU servers, GPU worker processes can lose performance if their CPU execution and memory allocation drift away from the NUMA node nearest to the GPU. vLLM can pin each worker with `numactl` before the Python subprocess starts, so the interpreter, imports, and early allocator state are created with the desired NUMA policy from the beginning. Use `--numa-bind` to enable the feature. By default, vLLM auto-detects the GPU-to-NUMA mapping and uses `--cpunodebind= --membind=` for each worker. When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will switch to `--physcpubind= --membind=`. These `--numa-bind*` options only apply to GPU execution processes. They do not configure the CPU backend's separate thread-affinity controls. Automatic GPU-to-NUMA detection is currently implemented for CUDA/NVML-based as well as ROCM-based platforms; other GPU backends must provide explicit binding lists if they use these options. `--numa-bind-nodes` takes one non-negative NUMA node index per visible GPU, in the same order as the GPU indices. `--numa-bind-cpus` takes one `numactl` CPU list per visible GPU, in the same order as the GPU indices. Each CPU list must use `numactl --physcpubind` syntax such as `0-3`, `0,2,4-7`, or `16-31,48-63`. `[](#__codelineno-4-1)# Auto-detect NUMA nodes for visible GPUs [](#__codelineno-4-2)vllm serve meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-4-3) --tensor-parallel-size 4 \ [](#__codelineno-4-4) --numa-bind [](#__codelineno-4-5)[](#__codelineno-4-6)# Explicit NUMA-node mapping [](#__codelineno-4-7)vllm serve meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-4-8) --tensor-parallel-size 4 \ [](#__codelineno-4-9) --numa-bind \ [](#__codelineno-4-10) --numa-bind-nodes 0 0 1 1 [](#__codelineno-4-11)[](#__codelineno-4-12)# Explicit CPU pinning, useful for PCT or other high-frequency core layouts [](#__codelineno-4-13)vllm serve meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-4-14) --tensor-parallel-size 4 \ [](#__codelineno-4-15) --numa-bind \ [](#__codelineno-4-16) --numa-bind-nodes 0 0 1 1 \ [](#__codelineno-4-17) --numa-bind-cpus 0-3 4-7 48-51 52-55` Notes: - CLI usage forces multiprocessing to use the `spawn` method automatically. If you enable NUMA binding through the Python API, also set `VLLM_WORKER_MULTIPROC_METHOD=spawn`. - Automatic detection relies on NVML and NUMA support from the host. If it cannot determine the mapping reliably, pass `--numa-bind-nodes` explicitly. - Explicit `--numa-bind-nodes` and `--numa-bind-cpus` values must be valid `numactl` inputs. vLLM does a small amount of validation, but the effective binding semantics are still determined by `numactl`. - The current implementation binds GPU execution processes such as [`EngineCore`](https://docs.vllm.ai/en/api/vllm/v1/engine/core/#vllm.v1.engine.core.EngineCore " EngineCore") and multiprocessing workers. It does not apply NUMA binding to frontend API server processes or the DP coordinator. - In containerized environments, NUMA policy syscalls may require extra permissions, such as `--cap-add SYS_NICE` when running via `docker run`. ### CPU Backend Thread Affinity[¶](#cpu-backend-thread-affinity "Permanent link") The CPU backend uses a different mechanism from `--numa-bind`. CPU execution is configured through CPU-specific environment variables such as `VLLM_CPU_OMP_THREADS_BIND`, `VLLM_CPU_NUM_OF_RESERVED_CPU`, and `CPU_VISIBLE_MEMORY_NODES`, rather than the GPU-oriented `--numa-bind*` CLI options. By default, `VLLM_CPU_OMP_THREADS_BIND=auto` derives OpenMP placement from the available CPU and NUMA topology for each CPU worker. To override the automatic policy, set `VLLM_CPU_OMP_THREADS_BIND` explicitly using the CPU list format documented for the CPU backend, or use `nobind` to disable this behavior. For the current CPU backend setup and tuning guidance, see: - [Related runtime environment variables](https://docs.vllm.ai/en/getting_started/installation/cpu/#related-runtime-environment-variables) - [How to decide `VLLM_CPU_OMP_THREADS_BIND`](https://docs.vllm.ai/en/getting_started/installation/cpu/#how-to-decide-vllm_cpu_omp_threads_bind) The GPU-only `--numa-bind`, `--numa-bind-nodes`, and `--numa-bind-cpus` options do not configure CPU worker affinity. ### Batch-level DP for Multi-Modal Encoders[¶](#batch-level-dp-for-multi-modal-encoders "Permanent link") By default, TP is used to shard the weights of multi-modal encoders just like for language decoders, in order to reduce the memory and compute load on each GPU. However, since the size of multi-modal encoders is very small compared to language decoders, there is relatively little gain from TP. On the other hand, TP incurs significant communication overhead because of all-reduce being performed after every layer. Given this, it may be advantageous to instead shard the batched input data using TP, essentially performing batch-level DP. This has been shown to improve the throughput and TTFT by around 10% for `tensor_parallel_size=8`. For vision encoders that use hardware-unoptimized Conv3D operations, batch-level DP can provide another 40% improvement compared to regular TP. Nevertheless, since the weights of the multi-modal encoder are replicated across each TP rank, there will be a minor increase in memory consumption and may cause OOM if you can barely fit the model already. You can enable batch-level DP by setting `mm_encoder_tp_mode="data"`, for example: `[](#__codelineno-5-1)from vllm import LLM [](#__codelineno-5-2)[](#__codelineno-5-3)llm = LLM( [](#__codelineno-5-4) model="Qwen/Qwen2.5-VL-72B-Instruct", [](#__codelineno-5-5) tensor_parallel_size=4, [](#__codelineno-5-6) # When mm_encoder_tp_mode="data", [](#__codelineno-5-7) # the vision encoder uses TP=4 (not DP=1) to shard the input data, [](#__codelineno-5-8) # so the TP size becomes the effective DP size. [](#__codelineno-5-9) # Note that this is independent of the DP size for language decoder which is used in expert parallel setting. [](#__codelineno-5-10) mm_encoder_tp_mode="data", [](#__codelineno-5-11) # The language decoder uses TP=4 to shard the weights regardless [](#__codelineno-5-12) # of the setting of mm_encoder_tp_mode [](#__codelineno-5-13))` Important Batch-level DP is not to be confused with API request-level DP (which is instead controlled by `data_parallel_size`). Batch-level DP needs to be implemented on a per-model basis, and enabled by setting `supports_encoder_tp_data = True` in the model class. Regardless, you need to set `mm_encoder_tp_mode="data"` in engine arguments to use this feature. Known supported models (with corresponding benchmarks): - dots\_ocr ( [Pull Request #25466](https://github.com/vllm-project/vllm/pull/25466)) - GLM-4.1V or above ( [Pull Request #23168](https://github.com/vllm-project/vllm/pull/23168)) - InternVL ( [Pull Request #23909](https://github.com/vllm-project/vllm/pull/23909)) - Kimi-VL ( [Pull Request #23817](https://github.com/vllm-project/vllm/pull/23817)) - Llama4 ( [Pull Request #18368](https://github.com/vllm-project/vllm/pull/18368)) - MiniCPM-V-2.5 or above ( [Pull Request #23327](https://github.com/vllm-project/vllm/pull/23327), [Pull Request #23948](https://github.com/vllm-project/vllm/pull/23948)) - Qwen2-VL or above ( [Pull Request #22742](https://github.com/vllm-project/vllm/pull/22742), [Pull Request #24955](https://github.com/vllm-project/vllm/pull/24955), [Pull Request #25445](https://github.com/vllm-project/vllm/pull/25445)) - Step3 ( [Pull Request #22697](https://github.com/vllm-project/vllm/pull/22697)) ## Input Processing[¶](#input-processing "Permanent link") ### fastokens Backend[¶](#fastokens-backend "Permanent link") By default vLLM uses the standard Hugging Face `tokenizers` library to power the fast tokenizer. For BPE tokenizers (Qwen, Llama, DeepSeek, GPT-OSS, etc.) you can switch to the [fastokens](https://github.com/crusoecloud/fastokens) Rust backend, a drop-in replacement that's substantially faster on encode/decode and on streaming detokenization. Enable it by setting `VLLM_USE_FASTOKENS=1`: `[](#__codelineno-6-1)VLLM_USE_FASTOKENS=1 vllm serve Qwen/Qwen3-8B` Equivalent in the offline API: `[](#__codelineno-7-1)import os [](#__codelineno-7-2)os.environ["VLLM_USE_FASTOKENS"] = "1" [](#__codelineno-7-3)[](#__codelineno-7-4)from vllm import LLM [](#__codelineno-7-5)llm = LLM(model="Qwen/Qwen3-8B")` The `fastokens` Python package (>= 0.2.0) must be installed; if it isn't, vLLM raises a clear `ImportError` at tokenizer load. The override applies to any `--tokenizer-mode` that ends up loading an HF fast tokenizer (`hf`, `deepseek_v32`, `deepseek_v4`, `qwen_vl`, …). Models that don't use the HF fast tokenizer (`mistral`, `grok2`, `kimi_audio`) ignore the flag. Tokenizer-bound workloads — long shared prefixes, bursty short prompts, batch detokenization — see the largest wins. If your bottleneck is GPU prefill/decode, the tokenizer change is unlikely to be visible end-to-end. ### Parallel Processing[¶](#parallel-processing "Permanent link") You can run input processing in parallel via [API server scale-out](https://docs.vllm.ai/en/serving/data_parallel_deployment/#internal-load-balancing). This is useful when input processing (which is run inside the API server) becomes a bottleneck compared to model execution (which is run inside engine core) and you have excess CPU capacity. `[](#__codelineno-8-1)# Run 4 API processes and 1 engine core process [](#__codelineno-8-2)vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4 [](#__codelineno-8-3)[](#__codelineno-8-4)# Run 4 API processes and 2 engine core processes [](#__codelineno-8-5)vllm serve Qwen/Qwen2.5-VL-3B-Instruct --api-server-count 4 -dp 2` Note API server scale-out is only available for online inference. Warning By default, 8 CPU threads are used in each API server to load media items (e.g. images) from request data. If you apply API server scale-out, consider adjusting `VLLM_MEDIA_LOADING_THREAD_COUNT` to avoid CPU resource exhaustion. Note API server scale-out disables [multi-modal IPC caching](#ipc-caching) because it requires a one-to-one correspondence between API and engine core processes. This does not impact [multi-modal processor caching](#processor-caching). ## Multi-Modal Caching[¶](#multi-modal-caching "Permanent link") Multi-modal caching avoids repeated transfer or processing of the same multi-modal data, which commonly occurs in multi-turn conversations. ### Processor Caching[¶](#processor-caching "Permanent link") Multi-modal processor caching is automatically enabled to avoid repeatedly processing the same multi-modal inputs in [`BaseMultiModalProcessor`](https://docs.vllm.ai/en/api/vllm/multimodal/processing/processor/#vllm.multimodal.processing.processor.BaseMultiModalProcessor " BaseMultiModalProcessor"). ### IPC Caching[¶](#ipc-caching "Permanent link") Multi-modal IPC caching is automatically enabled when there is a one-to-one correspondence between API (`P0`) and engine core (`P1`) processes, to avoid repeatedly transferring the same multi-modal inputs between them. #### Key-Replicated Cache[¶](#key-replicated-cache "Permanent link") By default, IPC caching uses a **key-replicated cache**, where cache keys exist in both the API (`P0`) and engine core (`P1`) processes, but the actual cache data resides only in `P1`. #### Shared Memory Cache[¶](#shared-memory-cache "Permanent link") When multiple worker processes are involved (e.g., when TP > 1), a **shared-memory cache** is more efficient. This can be enabled by setting `mm_processor_cache_type="shm"`. In this mode, cache keys are stored on `P0`, while the cache data itself lives in shared memory accessible by all processes. ### Configuration[¶](#configuration "Permanent link") You can adjust the size of the cache by setting the value of `mm_processor_cache_gb` (default 4 GiB). If you do not benefit much from the cache, you can disable both IPC and processor caching completely via `mm_processor_cache_gb=0`. Examples: `[](#__codelineno-9-1)# Use a larger cache [](#__codelineno-9-2)llm = LLM( [](#__codelineno-9-3) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-9-4) mm_processor_cache_gb=8, [](#__codelineno-9-5)) [](#__codelineno-9-6)[](#__codelineno-9-7)# Use a shared-memory based IPC cache [](#__codelineno-9-8)llm = LLM( [](#__codelineno-9-9) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-9-10) tensor_parallel_size=2, [](#__codelineno-9-11) mm_processor_cache_type="shm", [](#__codelineno-9-12) mm_processor_cache_gb=8, [](#__codelineno-9-13)) [](#__codelineno-9-14)[](#__codelineno-9-15)# Disable the cache [](#__codelineno-9-16)llm = LLM( [](#__codelineno-9-17) model="Qwen/Qwen2.5-VL-3B-Instruct", [](#__codelineno-9-18) mm_processor_cache_gb=0, [](#__codelineno-9-19))` ### Cache Placement[¶](#cache-placement "Permanent link") Based on the configuration, the content of the multi-modal caches on `P0` and `P1` are as follows: mm\_processor\_cache\_type Cache Type `P0` Cache `P1` Engine Cache `P1` Worker Cache Max. Memory lru Processor Caching K + V N/A N/A `mm_processor_cache_gb * data_parallel_size` lru Key-Replicated Caching K K + V N/A `mm_processor_cache_gb * api_server_count` shm Shared Memory Caching K N/A V `mm_processor_cache_gb * api_server_count` N/A Disabled N/A N/A N/A `0` K: Stores the hashes of multi-modal items V: Stores the processed tensor data of multi-modal items ## CPU Resources for GPU Deployments[¶](#cpu-resources-for-gpu-deployments "Permanent link") vLLM V1 uses a multi-process architecture (see [V1 Process Architecture](https://docs.vllm.ai/en/design/arch_overview/#v1-process-architecture)) where each process requires CPU resources. Underprovisioning CPU cores is a common source of performance degradation, especially in virtualized environments. ### Minimum CPU Requirements[¶](#minimum-cpu-requirements "Permanent link") For a deployment with `N` GPUs, there are at minimum: - **1 API server process** -- handles HTTP requests, tokenization, and input processing - **1 engine core process** -- runs the scheduler and coordinates GPU workers - **N GPU worker processes** -- one per GPU, executes model forward passes This means there are always at least **`2 + N` processes** competing for CPU time. Warning Using fewer physical CPU cores than processes will cause contention and significantly degrade throughput and latency. The engine core process runs a busy loop and is particularly sensitive to CPU starvation. The minimum is `2 + N` physical cores (1 for the API server, 1 for the engine core, and 1 per GPU worker). In practice, allocating more cores improves performance because the OS, PyTorch background threads, and other system processes also need CPU time. Important Please note we are referring to **physical CPU cores** here. If your system has hyperthreading enabled, then 1 vCPU = 1 hyperthread = 1/2 physical CPU core, so you need `2 x (2 + N)` minimum vCPUs. ### Data Parallel and Multi-API Server Deployments[¶](#data-parallel-and-multi-api-server-deployments "Permanent link") When using data parallelism or multiple API servers, the CPU requirements increase: `[](#__codelineno-10-1)Minimum physical cores = A + DP + N + (1 if DP > 1 else 0)` where `A` is the API server count (defaults to `DP`), `DP` is the data parallel size, and `N` is the total number of GPUs. For example, with `DP=4, TP=2` on 8 GPUs: `[](#__codelineno-11-1)4 API servers + 4 engine cores + 8 GPU workers + 1 DP coordinator = 17 processes` ### Performance Impact[¶](#performance-impact "Permanent link") CPU underprovisioning particularly impacts: - **Input processing throughput** -- tokenization, chat template rendering, and multi-modal data loading all run on CPU - **Scheduling latency** -- the engine core scheduler runs on CPU and directly affects how quickly new tokens are dispatched to the GPU workers - **Output processing** -- detokenization, networking, and especially streaming token responses use CPU cycles If you observe that GPU utilization is lower than expected, CPU contention may be the bottleneck. Increasing the number of available CPU cores and even the clock speed can significantly improve end-to-end performance. ## Attention Backend Selection[¶](#attention-backend-selection "Permanent link") vLLM supports multiple attention backends optimized for different hardware and use cases. The backend is automatically selected based on your GPU architecture, model type, and configuration, but you can also manually specify one for optimal performance. For detailed information on available backends, their feature support, and how to configure them, see the [Attention Backend Feature Support](https://docs.vllm.ai/en/design/attention_backends/) documentation. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/configuration/serve_args.md "Edit this page") The `vllm serve` command is used to launch the OpenAI-compatible server. ## CLI Arguments[¶](#cli-arguments "Permanent link") The `vllm serve` command is used to launch the OpenAI-compatible server. To see the available options, take a look at the [CLI Reference](https://docs.vllm.ai/en/cli/)! ## Configuration file[¶](#configuration-file "Permanent link") You can load CLI arguments via a [YAML](https://yaml.org/) config file. The argument names must be the long form of those outlined [above](https://docs.vllm.ai/en/latest/configuration/). For example: `[](#__codelineno-0-1)# config.yaml [](#__codelineno-0-2)[](#__codelineno-0-3)model: meta-llama/Llama-3.1-8B-Instruct [](#__codelineno-0-4)host: "127.0.0.1" [](#__codelineno-0-5)port: 6379 [](#__codelineno-0-6)uvicorn-log-level: "info"` To use the above config file: `[](#__codelineno-1-1)vllm serve --config config.yaml` Note In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence. The order of priorities is `command line > config file values > defaults`. e.g. `vllm serve SOME_MODEL --config config.yaml`, SOME\_MODEL takes precedence over `model` in config file. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/README.md "Edit this page") Thank you for your interest in contributing to vLLM! Our community is open to everyone and welcomes all kinds of contributions, no matter how small or large. There are several ways you can contribute to the project: - Identify and report any issues or bugs. - Request or add support for a new model. - Suggest or implement new features. - Improve documentation or contribute a how-to guide. We also believe in the power of community support; thus, answering queries, offering PR reviews, and assisting others are also highly regarded and beneficial contributions. Finally, one of the most impactful ways to support us is by raising awareness about vLLM. Talk about it in your blog posts and highlight how it's driving your incredible projects. Express your support on social media if you're using vLLM, or simply offer your appreciation by starring our repository! ## Job Board[¶](#job-board "Permanent link") Unsure on where to start? Check out the following links for tasks to work on: - [Good first issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22) - [Selected onboarding tasks](https://github.com/orgs/vllm-project/projects/6) - [New model requests](https://github.com/vllm-project/vllm/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22new-model%22) - [Models with multi-modal capabilities](https://github.com/orgs/vllm-project/projects/10) ## License[¶](#license "Permanent link") See [LICENSE](https://github.com/vllm-project/vllm/blob/main/LICENSE). ## Developing[¶](#developing "Permanent link") The first step of contributing to vLLM is to clone the GitHub repository: `[](#__codelineno-0-1)git clone https://github.com/vllm-project/vllm.git [](#__codelineno-0-2)cd vllm` Then, configure your Python virtual environment. It's recommended to use [uv](https://docs.astral.sh/uv/), a very fast Python environment manager, to create and manage Python environments. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install `uv`. After installing `uv`, you can create a new Python environment using the following commands: `[](#__codelineno-1-1)uv venv --python 3.12 --seed --managed-python [](#__codelineno-1-2)source .venv/bin/activate` If you are only developing vLLM's Python code, install vLLM using: `[](#__codelineno-2-1)VLLM_USE_PRECOMPILED=1 uv pip install -e .` To rebuild only the Rust frontend binary: `[](#__codelineno-3-1)./build_rust.sh # release build [](#__codelineno-3-2)./build_rust.sh --debug # faster build for development` If you are developing vLLM's Python and CUDA/C++ code, install Pytorch first: `[](#__codelineno-4-1)uv pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu129` Then install the necessary build dependencies from `requirements/build/cuda.txt`, skipping `torch` as it was installed in the previous step: `[](#__codelineno-5-1)grep -v '^torch==' requirements/build/cuda.txt | uv pip install -r -` Finally install vLLM using: `[](#__codelineno-6-1)uv pip install -e . --no-build-isolation` For more details about installing from source and installing for other hardware, check out the [installation instructions](https://docs.vllm.ai/en/getting_started/installation/) for your hardware and head to the "Build wheel from source" section. For an optimized workflow when iterating on C++/CUDA kernels, see the [Incremental Compilation Workflow](https://docs.vllm.ai/en/latest/incremental_build/) for recommendations. Tip vLLM is compatible with Python versions 3.10 to 3.13. However, vLLM's default [Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile) ships with Python 3.12 and tests in CI (except `mypy`) are run with Python 3.12. Therefore, we recommend developing with Python 3.12 to minimise the chance of your local environment clashing with our CI environment. ### Linting[¶](#linting "Permanent link") vLLM uses `pre-commit` to lint and format the codebase. See [https://pre-commit.com/#usage](https://pre-commit.com/#usage) if `pre-commit` is new to you. Setting up `pre-commit` is as easy as: `[](#__codelineno-7-1)uv pip install pre-commit>=4.5.1 [](#__codelineno-7-2)pre-commit install` vLLM's `pre-commit` hooks will now run automatically every time you commit. Tips You can manually run the `pre-commit` hooks using: `[](#__codelineno-8-1)pre-commit run # runs on staged files [](#__codelineno-8-2)pre-commit run -a # runs on all files (short for --all-files)` * * * Some `pre-commit` hooks only run in CI. If you need to, you can run them locally with: `[](#__codelineno-9-1)pre-commit run --hook-stage manual mypy-3.11` ### Documentation[¶](#documentation "Permanent link") MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file, [mkdocs.yaml](https://github.com/vllm-project/vllm/blob/main/mkdocs.yaml). Get started with: `[](#__codelineno-10-1)uv pip install -r requirements/docs.txt` Tip Ensure that your Python version is compatible with the plugins (e.g., `mkdocs-awesome-nav` requires Python 3.10+) MkDocs comes with a built-in dev-server that lets you preview your documentation as you work on it. From the root of the repository, run: `[](#__codelineno-11-1)mkdocs serve # with API ref (~10 minutes) [](#__codelineno-11-2)API_AUTONAV_EXCLUDE=vllm mkdocs serve # API ref off (~15 seconds)` Once you see `Serving on http://127.0.0.1:8000/` in the logs, the live preview is ready! Open [http://127.0.0.1:8000/](http://127.0.0.1:8000/) in your browser to see it. For additional features and advanced configurations, refer to the: - [MkDocs documentation](https://www.mkdocs.org/) - [Material for MkDocs documentation](https://squidfunk.github.io/mkdocs-material/) (the MkDocs theme we use) ### Testing[¶](#testing "Permanent link") vLLM uses `pytest` to test the codebase. `[](#__codelineno-12-1)# Install the test dependencies used in CI (CUDA only) [](#__codelineno-12-2)uv pip install -r requirements/common.txt -r requirements/dev.txt --torch-backend=auto [](#__codelineno-12-3)[](#__codelineno-12-4)# Install some common test dependencies (hardware agnostic) [](#__codelineno-12-5)uv pip install pytest pytest-asyncio [](#__codelineno-12-6)[](#__codelineno-12-7)# Run all tests [](#__codelineno-12-8)pytest tests/ [](#__codelineno-12-9)[](#__codelineno-12-10)# Run tests for a single test file with detailed output [](#__codelineno-12-11)pytest -s -v tests/test_logger.py` Install python3-dev if Python.h is missing If any of the above commands fails with `Python.h: No such file or directory`, install `python3-dev` with `sudo apt install python3-dev`. Warnings Currently, the repository is not fully checked by `mypy`. * * * Currently, not all unit tests pass when run on CPU platforms. If you don't have access to a GPU platform to run unit tests locally, rely on the continuous integration system to run the tests for now. ## Issues[¶](#issues "Permanent link") If you encounter a bug or have a feature request, please [search existing issues](https://github.com/vllm-project/vllm/issues?q=is%3Aissue) first to see if it has already been reported. If not, please [file a new issue](https://github.com/vllm-project/vllm/issues/new/choose), providing as much relevant information as possible. Important If you discover a security vulnerability, please follow the instructions [here](https://github.com/vllm-project/vllm/blob/main/SECURITY.md). ## Pull Requests & Code Reviews[¶](#pull-requests-code-reviews "Permanent link") Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process. ### DCO and Signed-off-by[¶](#dco-and-signed-off-by "Permanent link") When contributing changes to this project, you must agree to the [DCO](https://github.com/vllm-project/vllm/blob/main/DCO). Commits must include a `Signed-off-by:` header which certifies agreement with the terms of the DCO. Using `-s` with `git commit` will automatically add this header. Tip You can enable automatic sign-off via your IDE: - **PyCharm**: Click on the `Show Commit Options` icon to the right of the `Commit and Push...` button in the `Commit` window. It will bring up a `git` window where you can modify the `Author` and enable `Sign-off commit`. - **VSCode**: Open the [Settings editor](https://code.visualstudio.com/docs/configure/settings) and enable the `Git: Always Sign Off` (`git.alwaysSignOff`) field. ### AI Assisted Contributions[¶](#ai-assisted-contributions "Permanent link") Before making an AI assisted contribution, you must: 1. **Be involved**: Do not submit "pure agent" PRs. The human submitter is responsible for reviewing all changed lines, validating behavior end-to-end, and running relevant tests. 2. **Ensure significance**: Avoid one-off "busywork" PRs (single typo, isolated style cleanup, one mutable default fix, etc.). Bundle mechanical cleanups into a clear, systematic scope. When AI tools provide non-trivial assistance in generating or modifying code, you must: 1. **Review thoroughly**: You remain responsible for all code you submit. Review and understand AI-generated code with the same care as code you write manually. 2. **Disclose in PR**: Always mention when a pull request includes AI-generated code. Add a note in the PR description. 3. **Mark commits**: Add attribution using commit trailers such as `Co-authored-by:` (other projects use `Assisted-by:` or `Generated-by:`). For example: `[](#__codelineno-13-1)Your commit message here [](#__codelineno-13-2)[](#__codelineno-13-3)Co-authored-by: GitHub Copilot [](#__codelineno-13-4)Co-authored-by: Claude [](#__codelineno-13-5)Co-authored-by: gemini-code-assist [](#__codelineno-13-6)Signed-off-by: Your Name <[[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection)>` AI-assisted code must meet all quality standards: proper testing, documentation, adherence to style guides, and thorough review. Attribution helps reviewers evaluate contributions in context and maintains legal clarity for the project. ### PR Title and Classification[¶](#pr-title-and-classification "Permanent link") Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following: - `[Bugfix]` for bug fixes. - `[CI/Build]` for build or continuous integration improvements. - `[Doc]` for documentation fixes and improvements. - `[Model]` for adding a new model or improving an existing model. Model name should appear in the title. - `[Frontend]` For changes on the vLLM frontend (e.g., OpenAI API server, [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class, etc.) - `[Kernel]` for changes affecting CUDA kernels or other compute kernels. - `[Core]` for changes in the core vLLM logic (e.g., [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine"), `AsyncLLMEngine`, `Scheduler`, etc.) - `[Hardware][Vendor]` for hardware-specific changes. Vendor name should appear in the prefix (e.g., `[Hardware][AMD]`). - `[Misc]` for PRs that do not fit the above categories. Please use this sparingly. Note If the PR spans more than one category, please include all relevant prefixes. ### Code Quality[¶](#code-quality "Permanent link") The PR needs to meet the following code quality standards: - We adhere to [Google Python style guide](https://google.github.io/styleguide/pyguide.html) and [Google C++ style guide](https://google.github.io/styleguide/cppguide.html). - Pass all linter checks. - The code needs to be well-documented to ensure future contributors can easily understand the code. - Include sufficient tests to ensure the project stays correct and robust. This includes both unit tests and integration tests. - Please add documentation to `docs/` if the PR modifies the user-facing behaviors of vLLM. It helps vLLM users understand and utilize the new features or changes. ### Adding or Changing Kernels[¶](#adding-or-changing-kernels "Permanent link") When actively developing or modifying kernels, using the [Incremental Compilation Workflow](https://docs.vllm.ai/en/latest/incremental_build/) is highly recommended for faster build times. Each custom kernel needs a schema and one or more implementations to be registered with PyTorch. - Make sure custom ops are registered following PyTorch guidelines: [Custom C++ and CUDA Operators](https://pytorch.org/tutorials/advanced/cpp_custom_ops.html#cpp-custom-ops-tutorial) and [The Custom Operators Manual](https://docs.google.com/document/d/1_W62p8WJOQQUzPsJYa7s701JXt0qf2OfLub2sbkHOaU). - Custom operations that return `Tensors` require meta-functions. Meta-functions should be implemented and registered in Python so that dynamic dims can be handled automatically. See above documents for a description of meta-functions. - Use [torch.library.opcheck()](https://pytorch.org/docs/stable/library.html#torch.library.opcheck) to test the function registration and meta-function for any registered ops. See `tests/kernels` for examples. - When changing the C++ signature of an existing op, the schema must be updated to reflect the changes. - If a new custom type is needed, see the following document: [Custom Class Support in PT2](https://docs.google.com/document/d/18fBMPuOJ0fY5ZQ6YyrHUppw9FA332CpNtgB6SOIgyuA). ### Notes for Large Changes[¶](#notes-for-large-changes "Permanent link") Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with `rfc-required` and might not go through the PR. ### What to Expect for the Reviews[¶](#what-to-expect-for-the-reviews "Permanent link") The goal of the vLLM team is to be a _transparent reviewing machine_. We would like to make the review process transparent and efficient and make sure no contributor feels confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process: - After the PR is submitted, the PR will be assigned to a reviewer. Every reviewer will pick up the PRs based on their expertise and availability. - After the PR is assigned, the reviewer will provide status updates every 2-3 days. If the PR is not reviewed within 7 days, please feel free to ping the reviewer or the vLLM team. - After the review, the reviewer will put an `action-required` label on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR. - Please respond to all comments within a reasonable time frame. If a comment isn't clear or you disagree with a suggestion, feel free to ask for clarification or discuss the suggestion. - Note that not all CI checks will be executed due to limited computational resources. The reviewer will add `ready` label to the PR when the PR is ready to merge or a full CI run is needed. ### Escalating Stalled Contributions[¶](#escalating-stalled-contributions "Permanent link") If you have an important contribution that has not yet received maintainer attention, please email us at: [\[email protected\]](https://docs.vllm.ai/cdn-cgi/l/email-protection#4232306f3027342b27356f3027333727313602342e2e2f6c232b) Using a verifiable company or university email, include: - your production or research use case - the problem you encountered - how your contribution addresses it ## Thank You[¶](#thank-you "Permanent link") Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. All of your contributions help make vLLM a great tool and community for everyone! --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/deprecation_policy.md "Edit this page") This document outlines the official policy and process for deprecating features in the vLLM project. ## Overview[¶](#overview "Permanent link") vLLM uses a structured "deprecation pipeline" to guide the lifecycle of deprecated features. This policy ensures that users are given clear and sufficient notice when a feature is deprecated and that deprecations proceed in a consistent and predictable manner. We aim to strike a balance between continued innovation and respecting users’ reliance on existing functionality. Deprecations are tied to our **minor (Y) releases** following semantic versioning (X.Y.Z), where: - **X** is a major version (rare) - **Y** is a minor version (used for significant changes, including deprecations/removals) - **Z** is a patch version (used for fixes and safer enhancements) Features that fall under this policy include (at a minimum) the following: - CLI flags - Environment variables - Configuration files - APIs in the OpenAI-compatible API server - Public Python APIs for the `vllm` library ## Deprecation Pipeline[¶](#deprecation-pipeline "Permanent link") The deprecation process consists of several clearly defined stages that span multiple Y releases: ### 1\. Deprecated (Still On By Default)[¶](#1-deprecated-still-on-by-default "Permanent link") - **Action**: Feature is marked as deprecated. - **Timeline**: A removal version is explicitly stated in the deprecation warning (e.g., "This will be removed in v0.10.0"). - **Communication**: Deprecation is noted in the following, as applicable: - Help strings - Log output - API responses - `/metrics` output (for metrics features) - User-facing documentation - Release notes - GitHub Issue (RFC) for feedback - Documentation and use of the `@typing_extensions.deprecated` decorator for Python APIs ### 2\. Deprecated (Off By Default)[¶](#2-deprecated-off-by-default "Permanent link") - **Action**: Feature is disabled by default, but can still be re-enabled via a CLI flag or environment variable. Feature throws an error when used without re-enabling. - **Purpose**: Allows users who missed earlier warnings a temporary escape hatch while signaling imminent removal. Ensures any remaining usage is clearly surfaced and blocks silent breakage before full removal. ### 3\. Removed[¶](#3-removed "Permanent link") - **Action**: Feature is completely removed from the codebase. - **Note**: Only features that have passed through the previous deprecation stages will be removed. ## Example Timeline[¶](#example-timeline "Permanent link") Assume a feature is deprecated in `v0.9.0`. Release Status `v0.9.0` Feature is deprecated with clear removal version listed. `v0.10.0` Feature is now off by default, throws an error when used, and can be re-enabled for legacy use. `v0.11.0` Feature is removed. ## Important Guidelines[¶](#important-guidelines "Permanent link") - **No Removals in Patch Releases**: Removing deprecated features in patch (`.Z`) releases is disallowed to avoid surprising users. - **Grace Period for Existing Deprecations**: Any feature deprecated **before this policy** will have its grace period start **now**, not retroactively. - **Documentation is Critical**: Ensure every stage of the pipeline is documented clearly for users. ## Final Notes[¶](#final-notes "Permanent link") This policy is a living document and may evolve as the needs of the project and its users change. Community feedback is welcome and encouraged as we refine the process. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/editing-agent-instructions.md "Edit this page") > Read this before modifying `AGENTS.md` or any guide it links to. ## Token Budget Mindset[¶](#token-budget-mindset "Permanent link") `AGENTS.md` loads on every agent request; domain guides load on entry to a relevant area. Keep `AGENTS.md` under **200 lines** and each domain guide under **300 lines**. When a file exceeds its budget, split or prune — do not compress prose to fit. ## When NOT to Add Content[¶](#when-not-to-add-content "Permanent link") Before writing a new rule, ask whether it is actually needed: - **Agents already do it.** Test with a prompt first. If the agent behaves correctly without the rule, don't add it. - **One-off incident.** Prefer a code-level fix (lint rule, CI check, test assertion) over a new doc rule. - **Hardcoded paths.** File paths change; use "search for X" patterns instead. - **Upstream docs.** Don't reproduce pytest, ruff, or other tool docs — link to them. - **Contradicts an existing rule.** Search all linked guides before adding. If two rules conflict, consolidate into one. - **Already covered elsewhere.** Search `AGENTS.md` and every linked guide for overlapping guidance. If any of the above apply, **do not add the content**. ## Where Content Belongs[¶](#where-content-belongs "Permanent link") The goal is a lean `AGENTS.md` plus rich domain guides that teach agents what they can't learn from the code alone. Scope File Project-wide invariants (contribution policy, env setup, test/lint commands, commit conventions) `AGENTS.md` Area-specific knowledge (model patterns, format details, deprecation timelines) Domain guide **Rules of thumb:** - If it only matters for one area, put it in a domain guide. - If it matters for all areas, consider `AGENTS.md` — but first verify agents don't already do it. - Create a new domain guide when you have 5 or more non-obvious instructions sharing a coherent scope. ## What Makes a Good Domain Guide[¶](#what-makes-a-good-domain-guide "Permanent link") Add what agents can't infer from the code or public docs: project-specific conventions that differ from standard patterns, correct approaches that require cross-file context, and fixes for repeated mistakes. Each entry should be short, specific, and actionable — e.g., which files to touch, what order to change them in, and which tests to run. ## Keeping Docs Lean[¶](#keeping-docs-lean "Permanent link") - Every addition should trigger review of surrounding content for stale or redundant items. - Prefer examples over explanations — a 3-line snippet beats a paragraph of prose. - Merge related bullets into one principle instead of listing variants. - Use `search for X` instead of hardcoded file paths. - PR references are fine in domain guides for traceability, but avoid them in `AGENTS.md`. ## Anti-Patterns[¶](#anti-patterns "Permanent link") Pattern Problem Reactive accumulation Adding a rule per incident without pruning leads to bloat Copy-paste between guides Duplicated content drifts apart; keep in one place, link from the other Imperative walls Long DO NOT lists that agents skim past; consolidate into principles Config snapshots Show the command to get the value, not the value itself ## Change Checklist[¶](#change-checklist "Permanent link") Before submitting changes to any agent instruction file: - \[ \] **Non-obvious?** Would an agent do the wrong thing without this rule? - \[ \] **No conflicts?** Searched all linked guides for contradictions? - \[ \] **Right file?** Project-wide goes in `AGENTS.md`, area-specific in a domain guide? - \[ \] **Offset the addition?** Removed or consolidated something to compensate? - \[ \] **Under budget?** `AGENTS.md` < 200 lines, domain guides < 300 lines? - \[ \] **No hardcoded paths?** Uses "search for X" where paths may change? - \[ \] **Tested?** Verified that an agent actually follows the new instruction? --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/incremental_build.md "Edit this page") When working on vLLM's C++/CUDA kernels located in the `csrc/` directory, recompiling the entire project with `uv pip install -e .` for every change can be time-consuming. An incremental compilation workflow using CMake allows for faster iteration by only recompiling the necessary components after an initial setup. This guide details how to set up and use such a workflow, which complements your editable Python installation. ## Prerequisites[¶](#prerequisites "Permanent link") Before setting up the incremental build: 1. **vLLM Editable Install:** Ensure you have vLLM installed from source in an editable mode. Using pre-compiled wheels for the initial editable setup can be faster, as the CMake workflow will handle subsequent kernel recompilations. `[](#__codelineno-0-1)uv venv --python 3.12 --seed [](#__codelineno-0-2)source .venv/bin/activate [](#__codelineno-0-3)VLLM_USE_PRECOMPILED=1 uv pip install -U -e . --torch-backend=auto` 2. **CUDA Toolkit:** Verify that the NVIDIA CUDA Toolkit is correctly installed and `nvcc` is accessible in your `PATH`. CMake relies on `nvcc` to compile CUDA code. You can typically find `nvcc` in `$CUDA_HOME/bin/nvcc` or by running `which nvcc`. If you encounter issues, refer to the [official CUDA Toolkit installation guides](https://developer.nvidia.com/cuda-toolkit-archive) and vLLM's main [GPU installation documentation](https://docs.vllm.ai/en/getting_started/installation/gpu/#troubleshooting) for troubleshooting. The `CMAKE_CUDA_COMPILER` variable in your `CMakeUserPresets.json` should also point to your `nvcc` binary. 3. **Build Tools:** It is highly recommended to install `ccache` for fast rebuilds by caching compilation results (e.g., `sudo apt install ccache` or `conda install ccache`). Also, ensure the core build dependencies like `cmake` and `ninja` are installed. These are installable through `requirements/build/cuda.txt` or your system's package manager. `[](#__codelineno-1-1)uv pip install -r requirements/build/cuda.txt --torch-backend=auto` ## Setting up the CMake Build Environment[¶](#setting-up-the-cmake-build-environment "Permanent link") The incremental build process is managed through CMake. You can configure your build settings using a `CMakeUserPresets.json` file at the root of the vLLM repository. ### Generate `CMakeUserPresets.json` using the helper script[¶](#generate-cmakeuserpresetsjson-using-the-helper-script "Permanent link") To simplify the setup, vLLM provides a helper script that attempts to auto-detect your system's configuration (like CUDA path, Python environment, and CPU cores) and generates the `CMakeUserPresets.json` file for you. **Run the script:** Navigate to the root of your vLLM clone and execute the following command: `[](#__codelineno-2-1)python tools/generate_cmake_presets.py` The script will prompt you if it cannot automatically determine certain paths (e.g., `nvcc` or a specific Python executable for your vLLM development environment). Follow the on-screen prompts. If an existing `CMakeUserPresets.json` is found, the script will ask for confirmation before overwriting it. **Force overwrite existing file:** To automatically overwrite an existing `CMakeUserPresets.json` without prompting, use the `--force-overwrite` flag: `[](#__codelineno-3-1)python tools/generate_cmake_presets.py --force-overwrite` This is particularly useful in automated scripts or CI/CD environments where interactive prompts are not desired. After running the script, a `CMakeUserPresets.json` file will be created in the root of your vLLM repository. ### Example `CMakeUserPresets.json`[¶](#example-cmakeuserpresetsjson "Permanent link") Below is an example of what the generated `CMakeUserPresets.json` might look like. The script will tailor these values based on your system and any input you provide. `[](#__codelineno-4-1){ [](#__codelineno-4-2) "version": 6, [](#__codelineno-4-3) "cmakeMinimumRequired": { [](#__codelineno-4-4) "major": 3, [](#__codelineno-4-5) "minor": 26, [](#__codelineno-4-6) "patch": 1 [](#__codelineno-4-7) }, [](#__codelineno-4-8) "configurePresets": [ [](#__codelineno-4-9) { [](#__codelineno-4-10) "name": "release", [](#__codelineno-4-11) "generator": "Ninja", [](#__codelineno-4-12) "binaryDir": "${sourceDir}/cmake-build-release", [](#__codelineno-4-13) "cacheVariables": { [](#__codelineno-4-14) "CMAKE_CUDA_COMPILER": "/usr/local/cuda/bin/nvcc", [](#__codelineno-4-15) "CMAKE_C_COMPILER_LAUNCHER": "ccache", [](#__codelineno-4-16) "CMAKE_CXX_COMPILER_LAUNCHER": "ccache", [](#__codelineno-4-17) "CMAKE_CUDA_COMPILER_LAUNCHER": "ccache", [](#__codelineno-4-18) "CMAKE_BUILD_TYPE": "Release", [](#__codelineno-4-19) "VLLM_PYTHON_EXECUTABLE": "/home/user/venvs/vllm/bin/python", [](#__codelineno-4-20) "CMAKE_INSTALL_PREFIX": "${sourceDir}", [](#__codelineno-4-21) "CMAKE_CUDA_FLAGS": "", [](#__codelineno-4-22) "NVCC_THREADS": "4", [](#__codelineno-4-23) "CMAKE_JOB_POOLS": "compile=32" [](#__codelineno-4-24) } [](#__codelineno-4-25) } [](#__codelineno-4-26) ], [](#__codelineno-4-27) "buildPresets": [ [](#__codelineno-4-28) { [](#__codelineno-4-29) "name": "release", [](#__codelineno-4-30) "configurePreset": "release", [](#__codelineno-4-31) "jobs": 32 [](#__codelineno-4-32) } [](#__codelineno-4-33) ] [](#__codelineno-4-34)}` **What do the various configurations mean?** - `CMAKE_CUDA_COMPILER`: Path to your `nvcc` binary. The script attempts to find this automatically. - `CMAKE_C_COMPILER_LAUNCHER`, `CMAKE_CXX_COMPILER_LAUNCHER`, `CMAKE_CUDA_COMPILER_LAUNCHER`: Setting these to `ccache` (or `sccache`) significantly speeds up rebuilds by caching compilation results. Ensure `ccache` is installed (e.g., `sudo apt install ccache` or `conda install ccache`). The script sets these by default. - `VLLM_PYTHON_EXECUTABLE`: Path to the Python executable in your vLLM development environment. The script will prompt for this, defaulting to the current Python environment if suitable. - `CMAKE_INSTALL_PREFIX: "${sourceDir}"`: Specifies that the compiled components should be installed back into your vLLM source directory. This is crucial for the editable install, as it makes the newly built kernels immediately available to your Python environment. - `CMAKE_JOB_POOLS` and `jobs` in build presets: Control the parallelism of the build. The script sets these based on the number of CPU cores detected on your system. - `binaryDir`: Specifies where the build artifacts will be stored (e.g., `cmake-build-release`). ## Building and Installing with CMake[¶](#building-and-installing-with-cmake "Permanent link") Once your `CMakeUserPresets.json` is configured: 1. **Initialize the CMake build environment:** This step configures the build system according to your chosen preset (e.g., `release`) and creates the build directory at `binaryDir` `[](#__codelineno-5-1)cmake --preset release` 2. **Build and install the vLLM components:** This command compiles the code and installs the resulting binaries into your vLLM source directory, making them available to your editable Python installation. `[](#__codelineno-6-1)cmake --build --preset release --target install` 3. **Make changes and repeat!** Now you start using your editable install of vLLM, testing and making changes as needed. If you need to build again to update based on changes, simply run the CMake command again to build only the affected files. `[](#__codelineno-7-1)cmake --build --preset release --target install` ## Verifying the Build[¶](#verifying-the-build "Permanent link") After a successful build, you will find a populated build directory (e.g., `cmake-build-release/` if you used the `release` preset and the example configuration). `[](#__codelineno-8-1)> ls cmake-build-release/ [](#__codelineno-8-2)bin cmake_install.cmake _deps machete_generation.log [](#__codelineno-8-3)build.ninja CPackConfig.cmake detect_cuda_compute_capabilities.cu marlin_generation.log [](#__codelineno-8-4)_C.abi3.so CPackSourceConfig.cmake detect_cuda_version.cc _moe_C.abi3.so [](#__codelineno-8-5)CMakeCache.txt ctest _flashmla_C.abi3.so moe_marlin_generation.log [](#__codelineno-8-6)CMakeFiles cumem_allocator.abi3.so install_local_manifest.txt vllm-flash-attn` The `cmake --build ... --target install` command copies the compiled shared libraries (like `_C.abi3.so`, `_moe_C.abi3.so`, etc.) into the appropriate `vllm` package directory within your source tree. This updates your editable installation with the newly compiled kernels. ## Additional Tips[¶](#additional-tips "Permanent link") - **Adjust Parallelism:** Fine-tune the `CMAKE_JOB_POOLS` in `configurePresets` and `jobs` in `buildPresets` in your `CMakeUserPresets.json`. Too many jobs can overload systems with limited RAM or CPU cores, leading to slower builds or system instability. Too few won't fully utilize available resources. - **Clean Builds When Necessary:** If you encounter persistent or strange build errors, especially after significant changes or switching branches, consider removing the CMake build directory (e.g., `rm -rf cmake-build-release`) and re-running the `cmake --preset` and `cmake --build` commands. - **Specific Target Builds:** For even faster iterations when working on a specific module, you can sometimes build a specific target instead of the full `install` target, though `install` ensures all necessary components are updated in your Python environment. Refer to CMake documentation for more advanced target management. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/profiling.md "Edit this page") Warning Profiling is only intended for vLLM developers and maintainers to understand the proportion of time spent in different parts of the codebase. **vLLM end-users should never turn on profiling** as it will significantly slow down the inference. Choosing a profiler - Use **Nsight Systems** for low-overhead, performance-critical profiling. - Use **PyTorch Profiler** for medium-overhead profiling with richer debugging information (e.g., stack traces, memory, shapes). Note that enabling these features adds overhead and is not recommended for benchmarking. ## Profile with PyTorch Profiler[¶](#profile-with-pytorch-profiler "Permanent link") We support tracing vLLM workers using different profilers. You can enable profiling by setting the `--profiler-config` flag when launching the server. Note The `--profiler-config` flag is available in vLLM v0.13.0 and later. If you are using an earlier version, please upgrade to use this feature. To use the `torch.profiler` module, set the `profiler` entry to `'torch'` and `torch_profiler_dir` to the directory where you want to save the traces. Additionally, you can control the profiling content by specifying the following additional arguments in the config: - `torch_profiler_record_shapes` to enable recording Tensor Shapes, off by default - `torch_profiler_with_memory` to record memory, off by default - `torch_profiler_with_stack` to enable recording stack information, on by default - `torch_profiler_with_flops` to enable recording FLOPs, off by default - `torch_profiler_use_gzip` to control gzip-compressing profiling files, on by default - `torch_profiler_dump_cuda_time_total` to control dumping and printing the aggregated CUDA self time table, on by default When using `vllm bench serve`, you can enable profiling by passing the `--profile` flag. Traces can be visualized using [https://ui.perfetto.dev/](https://ui.perfetto.dev/). Tip You can directly call bench module without installing vLLM using `python -m vllm.entrypoints.cli.main bench`. Tip Only send a few requests through vLLM when profiling, as the traces can get quite large. Also, no need to untar the traces, they can be viewed directly. Tip To stop the profiler - it flushes out all the profile trace files to the directory. This takes time, for example for about 100 requests worth of data for a llama 70b, it takes about 10 minutes to flush out on a H100. The engine client waits for this flush to complete without timing out, so simply allow the stop call to run to completion. ### Example commands and usage[¶](#example-commands-and-usage "Permanent link") #### Offline Inference[¶](#offline-inference "Permanent link") Refer to [examples/features/profiling/simple\_profiling\_offline.py](https://github.com/vllm-project/vllm/blob/main/examples/features/profiling/simple_profiling_offline.py) for an example. #### OpenAI Server[¶](#openai-server "Permanent link") `[](#__codelineno-0-1)vllm serve meta-llama/Llama-3.1-8B-Instruct --profiler-config '{"profiler": "torch", "torch_profiler_dir": "./vllm_profile"}'` vllm bench command: `[](#__codelineno-1-1)vllm bench serve \ [](#__codelineno-1-2) --backend vllm \ [](#__codelineno-1-3) --model meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-1-4) --dataset-name sharegpt \ [](#__codelineno-1-5) --dataset-path sharegpt.json \ [](#__codelineno-1-6) --profile \ [](#__codelineno-1-7) --num-prompts 2` Or use http request: `[](#__codelineno-2-1)# We need first call /start_profile api to start profile. [](#__codelineno-2-2)$ curl -X POST http://localhost:8000/start_profile [](#__codelineno-2-3)[](#__codelineno-2-4)# Call model generate. [](#__codelineno-2-5)curl -X POST http://localhost:8000/v1/chat/completions \ [](#__codelineno-2-6) -H "Content-Type: application/json" \ [](#__codelineno-2-7) -d '{ [](#__codelineno-2-8) "model": "meta-llama/Llama-3.1-8B-Instruct", [](#__codelineno-2-9) "messages": [ [](#__codelineno-2-10) { [](#__codelineno-2-11) "role": "user", [](#__codelineno-2-12) "content": "San Francisco is a" [](#__codelineno-2-13) } [](#__codelineno-2-14) ] [](#__codelineno-2-15) }' [](#__codelineno-2-16)[](#__codelineno-2-17)# After need call /stop_profile api to stop profile. [](#__codelineno-2-18)$ curl -X POST http://localhost:8000/stop_profile` ## Profile with NVIDIA Nsight Systems[¶](#profile-with-nvidia-nsight-systems "Permanent link") Nsight systems is an advanced tool that exposes more profiling details, such as register and shared memory usage, annotated code regions and low-level CUDA APIs and events. [Install nsight-systems](https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html) using your package manager. The following block is an example for Ubuntu. `[](#__codelineno-3-1)apt update [](#__codelineno-3-2)apt install -y --no-install-recommends gnupg [](#__codelineno-3-3)echo "deb http://developer.download.nvidia.com/devtools/repos/ubuntu$(source /etc/lsb-release; echo "$DISTRIB_RELEASE" | tr -d .)/$(dpkg --print-architecture) /" | tee /etc/apt/sources.list.d/nvidia-devtools.list [](#__codelineno-3-4)apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub [](#__codelineno-3-5)apt update [](#__codelineno-3-6)apt install nsight-systems-cli` Tip When profiling with `nsys`, it is advisable to set the environment variable `VLLM_WORKER_MULTIPROC_METHOD=spawn`. The default is to use the `fork` method instead of `spawn`. More information on the topic can be found in the [Nsight Systems release notes](https://docs.nvidia.com/nsight-systems/ReleaseNotes/index.html#general-issues). The Nsight Systems profiler can be launched with `nsys profile ...`, with a few recommended flags for vLLM: `--trace-fork-before-exec=true --cuda-graph-trace=node`. ### Example commands and usage[¶](#example-commands-and-usage_1 "Permanent link") #### Offline Inference[¶](#offline-inference_1 "Permanent link") For basic usage, you can just append the profiling command before any existing script you would run for offline inference. The following is an example using the `vllm bench latency` script: `[](#__codelineno-4-1)nsys profile \ [](#__codelineno-4-2) --trace-fork-before-exec=true \ [](#__codelineno-4-3) --cuda-graph-trace=node \ [](#__codelineno-4-4)vllm bench latency \ [](#__codelineno-4-5) --model meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-4-6) --num-iters-warmup 5 \ [](#__codelineno-4-7) --num-iters 1 \ [](#__codelineno-4-8) --batch-size 16 \ [](#__codelineno-4-9) --input-len 512 \ [](#__codelineno-4-10) --output-len 8` #### OpenAI Server[¶](#openai-server_1 "Permanent link") To profile the server, you will want to prepend your `vllm serve` command with `nsys profile` just like for offline inference, but you will need to specify a few other arguments to enable dynamic capture similarly to the Torch Profiler: `[](#__codelineno-5-1)# server [](#__codelineno-5-2)nsys profile \ [](#__codelineno-5-3) --trace-fork-before-exec=true \ [](#__codelineno-5-4) --cuda-graph-trace=node \ [](#__codelineno-5-5) --capture-range=cudaProfilerApi \ [](#__codelineno-5-6) --capture-range-end repeat \ [](#__codelineno-5-7) vllm serve meta-llama/Llama-3.1-8B-Instruct --profiler-config.profiler cuda [](#__codelineno-5-8)[](#__codelineno-5-9)# client [](#__codelineno-5-10)vllm bench serve \ [](#__codelineno-5-11) --backend vllm \ [](#__codelineno-5-12) --model meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-5-13) --dataset-name sharegpt \ [](#__codelineno-5-14) --dataset-path sharegpt.json \ [](#__codelineno-5-15) --profile \ [](#__codelineno-5-16) --num-prompts 2` With `--profile`, vLLM will capture a profile for each run of `vllm bench serve`. Once the server is killed, the profiles will all be saved. #### Analysis[¶](#analysis "Permanent link") You can view these profiles either as summaries in the CLI, using `nsys stats [profile-file]`, or in the GUI by installing Nsight [locally following the directions here](https://developer.nvidia.com/nsight-systems/get-started). CLI example `[](#__codelineno-6-1)nsys stats report1.nsys-rep [](#__codelineno-6-2)... [](#__codelineno-6-3)** CUDA GPU Kernel Summary (cuda_gpu_kern_sum): [](#__codelineno-6-4)[](#__codelineno-6-5)Time (%) Total Time (ns) Instances Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name [](#__codelineno-6-6)-------- --------------- --------- ----------- ----------- -------- --------- ----------- ---------------------------------------------------------------------------------------------------- [](#__codelineno-6-7) 46.3 10,327,352,338 17,505 589,965.9 144,383.0 27,040 3,126,460 944,263.8 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize128x128x64_warpgroupsize1x1x1_execute_segment_k_of… [](#__codelineno-6-8) 14.8 3,305,114,764 5,152 641,520.7 293,408.0 287,296 2,822,716 867,124.9 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize256x128x64_warpgroupsize2x1x1_execute_segment_k_of… [](#__codelineno-6-9) 12.1 2,692,284,876 14,280 188,535.4 83,904.0 19,328 2,862,237 497,999.9 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize64x128x64_warpgroupsize1x1x1_execute_segment_k_off… [](#__codelineno-6-10) 9.5 2,116,600,578 33,920 62,399.8 21,504.0 15,326 2,532,285 290,954.1 sm90_xmma_gemm_bf16bf16_bf16f32_f32_tn_n_tilesize64x64x64_warpgroupsize1x1x1_execute_segment_k_off_… [](#__codelineno-6-11) 5.0 1,119,749,165 18,912 59,208.4 9,056.0 6,784 2,578,366 271,581.7 void vllm::act_and_mul_kernel, (bool)1>(T1 *, cons… [](#__codelineno-6-12) 4.1 916,662,515 21,312 43,011.6 19,776.0 8,928 2,586,205 199,790.1 void cutlass::device_kernel(int)0&&vllm::_typeConvert::exists, void>::type vllm::fused_add_rms_norm_kern… [](#__codelineno-6-14) 1.9 418,362,605 18,912 22,121.5 3,871.0 3,328 2,523,870 175,248.2 void vllm::rotary_embedding_kernel(const long *, T1 *, T1 *, const T1 *, in… [](#__codelineno-6-15) 0.7 167,083,069 18,880 8,849.7 2,240.0 1,471 2,499,996 101,436.1 void vllm::reshape_and_cache_flash_kernel<__nv_bfloat16, __nv_bfloat16, (vllm::Fp8KVCacheDataType)0… [](#__codelineno-6-16)...` GUI example: [![Screenshot 2025-03-05 at 11 48 42 AM](https://github.com/user-attachments/assets/c7cff1ae-6d6f-477d-a342-bd13c4fc424c)](https://github.com/user-attachments/assets/c7cff1ae-6d6f-477d-a342-bd13c4fc424c) ## Continuous Profiling[¶](#continuous-profiling "Permanent link") There is a [GitHub CI workflow](https://github.com/pytorch/pytorch-integration-testing/actions/workflows/vllm-profiling.yml) in the PyTorch infrastructure repository that provides continuous profiling for different models on vLLM. This automated profiling helps track performance characteristics over time and across different model configurations. ### How It Works[¶](#how-it-works "Permanent link") The workflow currently runs weekly profiling sessions for selected models, generating detailed performance traces that can be analyzed using different tools to identify performance regressions or optimization opportunities. But, it can be triggered manually as well, using the Github Action tool. ### Adding New Models[¶](#adding-new-models "Permanent link") To extend the continuous profiling to additional models, you can modify the [profiling-tests.json](https://github.com/pytorch/pytorch-integration-testing/blob/main/vllm-profiling/cuda/profiling-tests.json) configuration file in the PyTorch integration testing repository. Simply add your model specifications to this file to include them in the automated profiling runs. ### Viewing Profiling Results[¶](#viewing-profiling-results "Permanent link") The profiling traces generated by the continuous profiling workflow are publicly available on the [vLLM Performance Dashboard](https://hud.pytorch.org/benchmark/llms?repoName=vllm-project%2Fvllm). Look for the **Profiling traces** table to access and download the traces for different models and runs. ## Profiling vLLM Python Code[¶](#profiling-vllm-python-code "Permanent link") The Python standard library includes [cProfile](https://docs.python.org/3/library/profile.html) for profiling Python code. ### Example usage - function call[¶](#example-usage-function-call "Permanent link") If a filename is specified, the profile will be saved to that file. If no filename is specified, profile data can be printed to stdout. `[](#__codelineno-7-1)import cProfile [](#__codelineno-7-2) [](#__codelineno-7-3)[](#__codelineno-7-4)def expensive_function(): [](#__codelineno-7-5) # some expensive code [](#__codelineno-7-6) pass [](#__codelineno-7-7) [](#__codelineno-7-8)[](#__codelineno-7-9)profiler = cProfile.Profile() [](#__codelineno-7-10)profiler.runcall(expensive_function) [](#__codelineno-7-11)profiler.dump_stats("expensive_function.prof")` ### Example usage - context manager style[¶](#example-usage-context-manager-style "Permanent link") `[](#__codelineno-8-1)import cProfile [](#__codelineno-8-2) [](#__codelineno-8-3)[](#__codelineno-8-4)def another_function(): [](#__codelineno-8-5) # more expensive code [](#__codelineno-8-6) pass [](#__codelineno-8-7) [](#__codelineno-8-8)[](#__codelineno-8-9)profiler = cProfile.Profile() [](#__codelineno-8-10)profiler.enable() [](#__codelineno-8-11)try: [](#__codelineno-8-12) another_function() [](#__codelineno-8-13)finally: [](#__codelineno-8-14) profiler.disable() [](#__codelineno-8-15) profiler.dump_stats("another_function.prof")` ### Analyzing Profile Results[¶](#analyzing-profile-results "Permanent link") There are multiple tools available that can help analyze the profile results. One example is [snakeviz](https://jiffyclub.github.io/snakeviz/). `[](#__codelineno-9-1)pip install snakeviz [](#__codelineno-9-2)snakeviz expensive_function.prof` ### Analyzing Garbage Collection Costs[¶](#analyzing-garbage-collection-costs "Permanent link") Leverage VLLM\_GC\_DEBUG environment variable to debug GC costs. - VLLM\_GC\_DEBUG=1: enable GC debugger with gc.collect elapsed times - VLLM\_GC\_DEBUG='{"top\_objects":5}': enable GC debugger to log top 5 collected objects for each gc.collect --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/vulnerability_management.md "Edit this page") ## Reporting Vulnerabilities[¶](#reporting-vulnerabilities "Permanent link") As mentioned in the [security policy](https://github.com/vllm-project/vllm/tree/main/SECURITY.md), security vulnerabilities may be reported privately to the project via [GitHub](https://github.com/vllm-project/vllm/security/advisories/new). ## Vulnerability Management Team[¶](#vulnerability-management-team "Permanent link") Once a vulnerability has been reported to the project, the Vulnerability Management Team (VMT) is responsible for managing the vulnerability. The VMT is responsible for: - Triaging the vulnerability. - Coordinating with reporters and project maintainers on vulnerability analysis and resolution. - Drafting of security advisories for confirmed vulnerabilities, as appropriate. - Coordination with project maintainers on a coordinated release of the fix and security advisory. ### Security Advisories[¶](#security-advisories "Permanent link") Advisories are published via GitHub through the same system used to report vulnerabilities. More information on the process can be found in the [GitHub documentation](https://docs.github.com/en/code-security/security-advisories/working-with-repository-security-advisories/about-repository-security-advisories). ### Team Members[¶](#team-members "Permanent link") We prefer to keep all vulnerability-related communication on the security report on GitHub. However, if you need to contact the VMT directly for an urgent issue, you may contact the following individuals: - Simon Mo - [\[email protected\]](https://docs.vllm.ai/cdn-cgi/l/email-protection#ec9f85818382c28183ac848995c28f8381) - Russell Bryant - [\[email protected\]](https://docs.vllm.ai/cdn-cgi/l/email-protection#80f2e2f2f9e1eef4c0f2e5e4e8e1f4aee3efed) - Juan Pérez de Algaba - [\[email protected\]](https://docs.vllm.ai/cdn-cgi/l/email-protection#1973697c6b7c637d7c596b7c7d71786d377a7674) - Huzaifa Sidhpurwala - [\[email protected\]](https://docs.vllm.ai/cdn-cgi/l/email-protection#cca4b9b6ada5aaadbf8cbea9a8a4adb8e2afa3a1) ## Slack Discussion[¶](#slack-discussion "Permanent link") You may use the `#security` channel in the [vLLM Slack](https://slack.vllm.ai/) to discuss security-related topics. However, please do not disclose any vulnerabilities in this channel. If you need to report a vulnerability, please use the GitHub security advisory system or contact a VMT member privately. ## Vulnerability Disclosure[¶](#vulnerability-disclosure "Permanent link") The process for disclosing vulnerabilities is the following: - The VMT will work with the project maintainers to develop a fix for the vulnerability. - The VMT will coordinate with the reporter and project maintainers to prepare a security advisory that adequately describes the vulnerability and its impact. - The VMT will coordinate with the project maintainers to publish a fix and release an update that includes that fix. - The VMT will publish the security advisory on GitHub. Release notes will be updated to include a reference to the security advisory. The VMT and project maintainers will work to minimize the amount of time in between disclosing any public information about the vulnerability and making a release and advisory available. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/ci/failures.md "Edit this page") What should I do when a CI job fails on my PR, but I don't think my PR caused the failure? - Check the dashboard of current CI test failures: 👉 [CI Failures Dashboard](https://github.com/orgs/vllm-project/projects/20) - If your failure **is already listed**, it's likely unrelated to your PR. Help fixing it is always welcome! - Leave comments with links to additional instances of the failure. - React with a 👍 to signal how many are affected. - If your failure **is not listed**, you should **file an issue**. ## Filing a CI Test Failure Issue[¶](#filing-a-ci-test-failure-issue "Permanent link") - **File a bug report:** 👉 [New CI Failure Report](https://github.com/vllm-project/vllm/issues/new?template=450-ci-failure.yml) - **Use this title format:** `[](#__codelineno-0-1)[CI Failure]: failing-test-job - regex/matching/failing:test` - **For the environment field:** `[](#__codelineno-1-1)Still failing on main as of commit abcdef123` - **In the description, include failing tests:** `[](#__codelineno-2-1)FAILED failing/test.py:failing_test1 - Failure description [](#__codelineno-2-2)FAILED failing/test.py:failing_test2 - Failure description [](#__codelineno-2-3)https://github.com/orgs/vllm-project/projects/20 [](#__codelineno-2-4)https://github.com/vllm-project/vllm/issues/new?template=400-bug-report.yml [](#__codelineno-2-5)FAILED failing/test.py:failing_test3 - Failure description` - **Attach logs** (collapsible section example): Logs: `[](#__codelineno-3-1)ERROR 05-20 03:26:38 [dump_input.py:68] Dumping input data [](#__codelineno-3-2)--- Logging error --- [](#__codelineno-3-3)Traceback (most recent call last): [](#__codelineno-3-4) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 203, in execute_model [](#__codelineno-3-5) return self.model_executor.execute_model(scheduler_output) [](#__codelineno-3-6)... [](#__codelineno-3-7)FAILED failing/test.py:failing_test1 - Failure description [](#__codelineno-3-8)FAILED failing/test.py:failing_test2 - Failure description [](#__codelineno-3-9)FAILED failing/test.py:failing_test3 - Failure description` ## Logs Wrangling[¶](#logs-wrangling "Permanent link") Download a job's log (no Buildkite login required): [.buildkite/scripts/ci-fetch-log.sh](https://github.com/vllm-project/vllm/blob/main/.buildkite/scripts/ci-fetch-log.sh) `[](#__codelineno-4-1)# Find the failing job. Each row's URL is .../builds/#: [](#__codelineno-4-2)gh pr checks --repo vllm-project/vllm [](#__codelineno-4-3)[](#__codelineno-4-4)# Download + strip timestamps/ANSI in one step: [](#__codelineno-4-5).buildkite/scripts/ci-fetch-log.sh "https://buildkite.com/vllm/ci/builds/#"` To clean an already-downloaded log: [.buildkite/scripts/ci-clean-log.sh](https://github.com/vllm-project/vllm/blob/main/.buildkite/scripts/ci-clean-log.sh) `[](#__codelineno-5-1)./ci-clean-log.sh ci.log` Use a tool [wl-clipboard](https://github.com/bugaevc/wl-clipboard) for quick copy-pasting: `[](#__codelineno-6-1)tail -525 ci_build.log | wl-copy` ## Investigating a CI Test Failure[¶](#investigating-a-ci-test-failure "Permanent link") 1. Go to 👉 [Buildkite main branch](https://buildkite.com/vllm/ci/builds?branch=main) 2. Bisect to find the first build that shows the issue. 3. Add your findings to the GitHub issue. 4. If you find a strong candidate PR, mention it in the issue and ping contributors. ## Reproducing a Failure[¶](#reproducing-a-failure "Permanent link") CI test failures may be flaky. Use a bash loop to run repeatedly: [.buildkite/scripts/rerun-test.sh](https://github.com/vllm-project/vllm/blob/main/.buildkite/scripts/rerun-test.sh) `[](#__codelineno-7-1)./rerun-test.sh tests/v1/engine/test_engine_core_client.py::test_kv_cache_events[True-tcp]` ## Submitting a PR[¶](#submitting-a-pr "Permanent link") If you submit a PR to fix a CI failure: - Link the PR to the issue: Add `Closes #12345` to the PR description. - Add the `ci-failure` label: This helps track it in the [CI Failures GitHub Project](https://github.com/orgs/vllm-project/projects/20). ## Other Resources[¶](#other-resources "Permanent link") - 🔍 [Test Reliability on `main`](https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests?branch=main&order=ASC&sort_by=reliability) - 🧪 [Latest Buildkite CI Runs](https://buildkite.com/vllm/ci/builds?branch=main) ## Daily Triage[¶](#daily-triage "Permanent link") Use [Buildkite analytics (2-day view)](https://buildkite.com/organizations/vllm/analytics/suites/ci-1/tests?branch=main&period=2days) to: - Identify recent test failures **on `main`**. - Exclude legitimate test failures on PRs. - (Optional) Ignore tests with 0% reliability. Compare to the [CI Failures Dashboard](https://github.com/orgs/vllm-project/projects/20). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/ci/nightly_builds.md "Edit this page") vLLM maintains a per-commit wheel repository (commonly referred to as "nightly") at `https://wheels.vllm.ai` that provides pre-built wheels for every commit on the `main` branch since `v0.5.3`. This document explains how the nightly wheel index mechanism works. ## Build and Upload Process on CI[¶](#build-and-upload-process-on-ci "Permanent link") ### Wheel Building[¶](#wheel-building "Permanent link") Wheels are built in the `Release` pipeline (`.buildkite/release-pipeline.yaml`) after a PR is merged into the main branch, with multiple variants: - **Backend variants**: `cpu` and `cuXXX` (e.g., `cu129`, `cu130`). - **Architecture variants**: `x86_64` and `aarch64`. Each build step: 1. Builds the wheel in a Docker container. 2. Renames the wheel filename to use the correct manylinux tag (currently `manylinux_2_31`) for PEP 600 compliance. 3. Uploads the wheel to S3 bucket `vllm-wheels` under `/{commit_hash}/`. ### Index Generation[¶](#index-generation "Permanent link") After uploading each wheel, the `.buildkite/scripts/upload-wheels.sh` script: 1. **Lists all existing wheels** in the commit directory from S3 2. **Generates indices** using `.buildkite/scripts/generate-nightly-index.py`: - Parses wheel filenames to extract metadata (version, variant, platform tags). - Creates HTML index files (`index.html`) for PyPI compatibility. - Generates machine-readable `metadata.json` files. 3. **Uploads indices** to multiple locations (overriding existing ones): - `/{commit_hash}/` - Always uploaded for commit-specific access. - `/nightly/` - Only for commits on `main` branch (not PRs). - `/{version}/` - Only for release wheels (no `dev` in its version). Handling Concurrent Builds The index generation script can handle multiple variants being built concurrently by always listing all wheels in the commit directory before generating indices, avoiding race conditions. ## Directory Structure[¶](#directory-structure "Permanent link") The S3 bucket structure follows this pattern: `[](#__codelineno-0-1)s3://vllm-wheels/ [](#__codelineno-0-2)├── {commit_hash}/ # Commit-specific wheels and indices [](#__codelineno-0-3)│ ├── vllm-*.whl # All wheel files [](#__codelineno-0-4)│ ├── index.html # Project list (default variant) [](#__codelineno-0-5)│ ├── vllm/ [](#__codelineno-0-6)│ │ ├── index.html # Package index (default variant) [](#__codelineno-0-7)│ │ └── metadata.json # Metadata (default variant) [](#__codelineno-0-8)│ ├── cu129/ # Variant subdirectory [](#__codelineno-0-9)│ │ ├── index.html # Project list (cu129 variant) [](#__codelineno-0-10)│ │ └── vllm/ [](#__codelineno-0-11)│ │ ├── index.html # Package index (cu129 variant) [](#__codelineno-0-12)│ │ └── metadata.json # Metadata (cu129 variant) [](#__codelineno-0-13)│ ├── cu130/ # Variant subdirectory [](#__codelineno-0-14)│ ├── cpu/ # Variant subdirectory [](#__codelineno-0-15)│ └── .../ # More variant subdirectories [](#__codelineno-0-16)├── nightly/ # Latest main branch wheels (mirror of latest commit) [](#__codelineno-0-17)└── {version}/ # Release version indices (e.g., 0.11.2)` All built wheels are stored in `/{commit_hash}/`, while different indices are generated and reference them. This avoids duplication of wheel files. For example, you can specify the following URLs to use different indices: - `https://wheels.vllm.ai/nightly/cu130` for the latest main branch wheels built with CUDA 13.0. - `https://wheels.vllm.ai/{commit_hash}` for wheels built at a specific commit (default variant). - `https://wheels.vllm.ai/0.12.0/cpu` for 0.12.0 release wheels built for CPU variant. Please note that not all variants are present on every commit. The available variants are subject to change over time, e.g., changing cu130 to cu131. ### Variant Organization[¶](#variant-organization "Permanent link") Indices are organized by variant: - **Default variant**: Wheels without variant suffix (i.e., built with the current `VLLM_MAIN_CUDA_VERSION`) are placed in the root. - **Variant subdirectories**: Wheels with variant suffixes (e.g., `+cu130`, `.cpu`) are organized in subdirectories. - **Alias to default**: The default variant can have an alias (e.g., `cu129` for now) for consistency and convenience. The variant is extracted from the wheel filename (as described in the [file name convention](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-name-convention)): - The variant is encoded in the local version identifier (e.g. `+cu129` or `dev+g.cu130`). - Examples: - `vllm-0.11.2.dev278+gdbc3d9991-cp38-abi3-manylinux1_x86_64.whl` → default variant - `vllm-0.10.2rc2+cu129-cp38-abi3-manylinux2014_aarch64.whl` → `cu129` variant - `vllm-0.11.1rc8.dev14+gaa384b3c0.cu130-cp38-abi3-manylinux1_x86_64.whl` → `cu130` variant ## Index Generation Details[¶](#index-generation-details "Permanent link") The `generate-nightly-index.py` script performs the following: 1. **Parses wheel filenames** using regex to extract: - Package name - Version (with variant extracted) - Python tag, ABI tag, platform tag - Build tag (if present) 2. **Groups wheels by variant**, then by package name: - Currently only `vllm` is built, but the structure supports multiple packages in the future. 3. **Generates HTML indices** (compliant with the [Simple repository API](https://packaging.python.org/en/latest/specifications/simple-repository-api/#simple-repository-api)): - Top-level `index.html`: Lists all packages and variant subdirectories - Package-level `index.html`: Lists all wheel files for that package - Uses relative paths to wheel files for portability 4. **Generates metadata.json**: - Machine-readable JSON containing all wheel metadata - Includes `path` field with URL-encoded relative path to wheel file - Used by `setup.py` to locate compatible pre-compiled wheels during Python-only builds ### Special Handling for AWS Services[¶](#special-handling-for-aws-services "Permanent link") The wheels and indices are directly stored on AWS S3, and we use AWS CloudFront as a CDN in front of the S3 bucket. Since S3 does not provide proper directory listing, to support PyPI-compatible simple repository API behavior, we deploy a CloudFront Function that: - redirects any URL that does not end with `/` and does not look like a file (i.e., does not contain a dot `.` in the last path segment) to the same URL with a trailing `/` - appends `/index.html` to any URL that ends with `/` For example, the following requests would be handled as: - `/nightly` -> `/nightly/index.html` - `/nightly/cu130/` -> `/nightly/cu130/index.html` - `/nightly/index.html` or `/nightly/vllm.whl` -> unchanged AWS S3 Filename Escaping S3 will automatically escape filenames upon upload according to its [naming rule](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). The direct impact on vllm is that `+` in filenames will be converted to `%2B`. We take special care in the index generation script to escape filenames properly when generating the HTML indices and JSON metadata, to ensure the URLs are correct and can be directly used. ## Usage of precompiled wheels in `setup.py`[¶](#precompiled-wheels-usage "Permanent link") When installing vLLM with `VLLM_USE_PRECOMPILED=1`, the `setup.py` script: 1. **Determines wheel location** via `precompiled_wheel_utils.determine_wheel_url()`: - Env var `VLLM_PRECOMPILED_WHEEL_LOCATION` (user-specified URL/path) always takes precedence and skips all other steps. - Determines the variant from `VLLM_MAIN_CUDA_VERSION` (can be overridden with env var `VLLM_PRECOMPILED_WHEEL_VARIANT`); the default variant will also be tried as a fallback. - Determines the _base commit_ (explained later) of this branch (can be overridden with env var `VLLM_PRECOMPILED_WHEEL_COMMIT`). 2. **Fetches metadata** from `https://wheels.vllm.ai/{commit}/vllm/metadata.json` (for the default variant) or `https://wheels.vllm.ai/{commit}/{variant}/vllm/metadata.json` (for a specific variant). 3. **Selects compatible wheel** based on: - Package name (`vllm`) - Platform tag (architecture match) 4. **Downloads and extracts** precompiled artifacts from the wheel: - Native extension modules (`.so` files) - The `vllm-rs` Rust frontend binary - Flash Attention Python modules and Triton/FlashMLA Python files 5. **Patches package\_data** to include extracted files in the installation What is the base commit? The base commit is determined by finding the merge-base between the current branch and upstream `main`, ensuring compatibility between source code and precompiled binaries. _Note: it's users' responsibility to ensure there is no native code (e.g., C++ or CUDA) changes before using precompiled wheels._ ## Implementation Files[¶](#implementation-files "Permanent link") Key files involved in the nightly wheel mechanism: - **`.buildkite/release-pipeline.yaml`**: CI pipeline that builds wheels - **`.buildkite/scripts/upload-wheels.sh`**: Script that uploads wheels and generates indices - **`.buildkite/scripts/generate-nightly-index.py`**: Python script that generates PyPI-compatible indices - **`setup.py`**: Contains `precompiled_wheel_utils` class for fetching and using precompiled wheels --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/ci/update_pytorch_version.md "Edit this page") vLLM's current policy is to always use the latest PyTorch stable release in CI/CD. It is standard practice to submit a PR to update the PyTorch version as early as possible when a new [PyTorch stable release](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-cadence) becomes available. This process is non-trivial due to the gap between PyTorch releases. Using [Pull Request #16859](https://github.com/vllm-project/vllm/pull/16859) as an example, this document outlines common steps to achieve this update along with a list of potential issues and how to address them. ## Test PyTorch release candidates (RCs)[¶](#test-pytorch-release-candidates-rcs "Permanent link") Updating PyTorch in vLLM after the official release is not ideal because any issues discovered at that point can only be resolved by waiting for the next release or by implementing hacky workarounds in vLLM. The better solution is to test vLLM with PyTorch release candidates (RC) to ensure compatibility before each release. PyTorch release candidates can be downloaded from [PyTorch test index](https://download.pytorch.org/whl/test). For example, `torch2.7.0+cu12.8` RC can be installed using the following command: `[](#__codelineno-0-1)uv pip install torch torchvision torchaudio \ [](#__codelineno-0-2) --index-url https://download.pytorch.org/whl/test/cu128` When the final RC is ready for testing, it will be announced to the community on the [PyTorch dev-discuss forum](https://dev-discuss.pytorch.org/c/release-announcements). After this announcement, we can begin testing vLLM integration by drafting a pull request following this 3-step process: 1. Update [requirements files](https://github.com/vllm-project/vllm/tree/main/requirements) to point to the new releases for `torch`, `torchvision`, and `torchaudio`. 2. Use the following option to get the final release candidates' wheels. Some common platforms are `cpu`, `cu128`, and `rocm6.2.4`. `[](#__codelineno-1-1)--extra-index-url https://download.pytorch.org/whl/test/` 3. Since vLLM uses `uv`, ensure the following index strategy is applied: - Via environment variable: `[](#__codelineno-2-1)export UV_INDEX_STRATEGY=unsafe-best-match` - Or via CLI flag: `[](#__codelineno-3-1)--index-strategy unsafe-best-match` If failures are found in the pull request, raise them as issues on vLLM and cc the PyTorch release team to initiate discussion on how to address them. ## Update CUDA version[¶](#update-cuda-version "Permanent link") The PyTorch release matrix includes both stable and experimental [CUDA versions](https://github.com/pytorch/pytorch/blob/main/RELEASE.md#release-compatibility-matrix). Due to limitations, only the latest stable CUDA version (for example, torch `2.7.1+cu126`) is uploaded to PyPI. However, vLLM may require a different CUDA version, such as 12.8 for Blackwell support. This complicates the process as we cannot use the out-of-the-box `pip install torch torchvision torchaudio` command. The solution is to use `--extra-index-url` in vLLM's Dockerfiles. - Important indexes at the moment include: Platform `--extra-index-url` CUDA 12.8 [https://download.pytorch.org/whl/cu128](https://download.pytorch.org/whl/cu128) CPU [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu) ROCm 6.2 [https://download.pytorch.org/whl/rocm6.2.4](https://download.pytorch.org/whl/rocm6.2.4) ROCm 6.3 [https://download.pytorch.org/whl/rocm6.3](https://download.pytorch.org/whl/rocm6.3) XPU [https://download.pytorch.org/whl/xpu](https://download.pytorch.org/whl/xpu) - Update the below files to match the CUDA version from step 1. This makes sure that the release vLLM wheel is tested on CI. - `.buildkite/release-pipeline.yaml` - `.buildkite/scripts/upload-wheels.sh` ## Manually running vLLM builds on BuildKiteCI[¶](#manually-running-vllm-builds-on-buildkiteci "Permanent link") When building vLLM with a new PyTorch/CUDA version, the vLLM sccache S3 bucket will not have any cached artifacts, which can cause CI build jobs to exceed 5 hours. Furthermore, vLLM's fastcheck pipeline operates in read-only mode and does not populate the cache, making it ineffective for cache warm-up purposes. To address this, manually trigger a build on Buildkite to accomplish two objectives: 1. Run the complete test suite against the PyTorch RC build by setting the environment variables: `RUN_ALL=1` and `NIGHTLY=1` 2. Populate the vLLM sccache S3 bucket with compiled artifacts, enabling faster subsequent builds [![Buildkite new build popup](https://github.com/user-attachments/assets/3b07f71b-bb18-4ca3-aeaf-da0fe79d315f)](https://github.com/user-attachments/assets/3b07f71b-bb18-4ca3-aeaf-da0fe79d315f) ## Update all the different vLLM platforms[¶](#update-all-the-different-vllm-platforms "Permanent link") Rather than attempting to update all vLLM platforms in a single pull request, it's more manageable to handle some platforms separately. The separation of requirements and Dockerfiles for different platforms in vLLM CI/CD allows us to selectively choose which platforms to update. For instance, updating XPU requires the corresponding release from [Intel Extension for PyTorch](https://github.com/intel/intel-extension-for-pytorch) by Intel. While [Pull Request #16859](https://github.com/vllm-project/vllm/pull/16859) updated vLLM to PyTorch 2.7.0 on CPU, CUDA, and ROCm, [Pull Request #17444](https://github.com/vllm-project/vllm/pull/17444) completed the update for XPU. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/dockerfile/dockerfile.md "Edit this page") We provide a [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile) to construct the image for running an OpenAI compatible server with vLLM. More information about deploying with Docker can be found [here](https://docs.vllm.ai/en/deployment/docker/). Below is a visual representation of the multi-stage Dockerfile. The build graph contains the following nodes: - All build stages - The default build target (highlighted in grey) - External images (with dashed borders) The edges of the build graph represent: - `FROM ...` dependencies (with a solid line and a full arrow head) - `COPY --from=...` dependencies (with a dashed line and an empty arrow head) - `RUN --mount=(.\*)from=...` dependencies (with a dotted line and an empty diamond arrow head) > [![query](https://docs.vllm.ai/en/assets/contributing/dockerfile-stages-dependency.png)](https://docs.vllm.ai/en/assets/contributing/dockerfile-stages-dependency.png) > > Made using: [https://github.com/patrickhoefler/dockerfilegraph](https://github.com/patrickhoefler/dockerfilegraph) > > Commands to regenerate the build graph (make sure to run it **from the \`root\` directory of the vLLM repository** where the dockerfile is present): > > `[](#__codelineno-0-1)dockerfilegraph \ [](#__codelineno-0-2) -o png \ [](#__codelineno-0-3) --legend \ [](#__codelineno-0-4) --dpi 200 \ [](#__codelineno-0-5) --max-label-length 50 \ [](#__codelineno-0-6) --filename docker/Dockerfile` > > or in case you want to run it directly with the docker image: > > `[](#__codelineno-1-1)docker run \ [](#__codelineno-1-2) --rm \ [](#__codelineno-1-3) --user "$(id -u):$(id -g)" \ [](#__codelineno-1-4) --workdir /workspace \ [](#__codelineno-1-5) --volume "$(pwd)":/workspace \ [](#__codelineno-1-6) ghcr.io/patrickhoefler/dockerfilegraph:alpine \ [](#__codelineno-1-7) --output png \ [](#__codelineno-1-8) --dpi 200 \ [](#__codelineno-1-9) --max-label-length 50 \ [](#__codelineno-1-10) --filename docker/Dockerfile \ [](#__codelineno-1-11) --legend` > > (To run it for a different file, you can pass in a different argument to the flag `--filename`.) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/model/README.md "Edit this page") Important Many decoder language models can now be automatically loaded using the [Transformers modeling backend](https://docs.vllm.ai/en/models/supported_models/#transformers) without having to implement them in vLLM. See if `vllm serve ` works first! vLLM models are specialized [PyTorch](https://pytorch.org/) models that take advantage of various [features](https://docs.vllm.ai/en/features/#compatibility-matrix) to optimize their performance. The complexity of integrating a model into vLLM depends heavily on the model's architecture. The process is considerably straightforward if the model shares a similar architecture with an existing model in vLLM. However, this can be more complex for models that include new operators (e.g., a new attention mechanism). Read through these pages for a step-by-step guide: - [Basic Model](https://docs.vllm.ai/en/latest/contributing/basic/) - [Registering a Model](https://docs.vllm.ai/en/latest/contributing/registration/) - [Unit Testing](https://docs.vllm.ai/en/latest/contributing/tests/) - [Multi-Modal Support](https://docs.vllm.ai/en/latest/contributing/multimodal/) - [Speech-to-Text Support](https://docs.vllm.ai/en/latest/contributing/transcription/) Tip If you are encountering issues while integrating your model into vLLM, feel free to open a [GitHub issue](https://github.com/vllm-project/vllm/issues) or ask on our [developer slack](https://slack.vllm.ai/). We will be happy to help you out! --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/model/registration.md "Edit this page") vLLM relies on a model registry to determine how to run each model. A list of pre-registered architectures can be found [here](https://docs.vllm.ai/en/models/supported_models/). If your model is not on this list, you must register it to vLLM. This page provides detailed instructions on how to do so. ## Built-in models[¶](#built-in-models "Permanent link") To add a model directly to the vLLM library, start by forking our [GitHub repository](https://github.com/vllm-project/vllm) and then [build it from source](https://docs.vllm.ai/en/getting_started/installation/gpu/#build-wheel-from-source). This gives you the ability to modify the codebase and test your model. After you have implemented your model (see [tutorial](https://docs.vllm.ai/en/latest/contributing/basic/)), put it into the [vllm/model\_executor/models](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) directory. Then, add your model class to `_VLLM_MODELS` in [vllm/model\_executor/models/registry.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/registry.py) so that it is automatically registered upon importing vLLM. Finally, update our [list of supported models](https://docs.vllm.ai/en/models/supported_models/) to promote your model! Important The list of models in each section should be maintained in alphabetical order. ## Out-of-tree models[¶](#out-of-tree-models "Permanent link") You can load an external model [using a plugin](https://docs.vllm.ai/en/design/plugin_system/) without modifying the vLLM codebase. To register the model, use the following code: `[](#__codelineno-0-1)# The entrypoint of your plugin [](#__codelineno-0-2)def register(): [](#__codelineno-0-3) from vllm import ModelRegistry [](#__codelineno-0-4) from your_code import YourModelForCausalLM [](#__codelineno-0-5) [](#__codelineno-0-6) ModelRegistry.register_model("YourModelForCausalLM", YourModelForCausalLM)` If your model imports modules that initialize CUDA, consider lazy-importing it to avoid errors like `RuntimeError: Cannot re-initialize CUDA in forked subprocess`: `[](#__codelineno-1-1)# The entrypoint of your plugin [](#__codelineno-1-2)def register(): [](#__codelineno-1-3) from vllm import ModelRegistry [](#__codelineno-1-4) [](#__codelineno-1-5) ModelRegistry.register_model( [](#__codelineno-1-6) "YourModelForCausalLM", [](#__codelineno-1-7) "your_code:YourModelForCausalLM", [](#__codelineno-1-8) )` Important If your model is a multimodal model, ensure the model class implements the [SupportsMultiModal](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal " SupportsMultiModal") interface. Read more about that [here](https://docs.vllm.ai/en/latest/contributing/multimodal/). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/model/transcription.md "Edit this page") This document walks you through the steps to add support for speech-to-text (ASR) models to vLLM’s transcription and translation APIs by implementing [SupportsTranscription](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription " SupportsTranscription"). Please refer to the [supported models](https://docs.vllm.ai/en/models/supported_models/#transcription) for further guidance. ## Update the base vLLM model[¶](#update-the-base-vllm-model "Permanent link") It is assumed you have already implemented your model in vLLM according to the basic model guide. Extend your model with the [SupportsTranscription](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription " SupportsTranscription") interface and implement the following class attributes and methods. ### `supported_languages` and `supports_transcription_only`[¶](#supported_languages-and-supports_transcription_only "Permanent link") Declare supported languages and capabilities: - The `supported_languages` mapping is validated at init time. - Set `supports_transcription_only=True` if the model should not serve text generation (eg Whisper). supported\_languages and supports\_transcription\_only `[](#__codelineno-0-1)from typing import ClassVar, Mapping, Literal [](#__codelineno-0-2)import numpy as np [](#__codelineno-0-3)import torch [](#__codelineno-0-4)from torch import nn [](#__codelineno-0-5)[](#__codelineno-0-6)from vllm.config import ModelConfig, SpeechToTextConfig [](#__codelineno-0-7)from vllm.inputs import PromptType [](#__codelineno-0-8)from vllm.model_executor.models.interfaces import SupportsTranscription [](#__codelineno-0-9)[](#__codelineno-0-10)class YourASRModel(nn.Module, SupportsTranscription): [](#__codelineno-0-11) # Map of ISO 639-1 language codes to language names [](#__codelineno-0-12) supported_languages: ClassVar[Mapping[str, str]] = { [](#__codelineno-0-13) "en": "English", [](#__codelineno-0-14) "it": "Italian", [](#__codelineno-0-15) # ... add more as needed [](#__codelineno-0-16) } [](#__codelineno-0-17) [](#__codelineno-0-18) # If your model only supports audio-conditioned generation [](#__codelineno-0-19) # (no text-only generation), enable this flag. [](#__codelineno-0-20) supports_transcription_only: ClassVar[bool] = True` Provide an ASR configuration via [get\_speech\_to\_text\_config](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription.get_speech_to_text_config " get_speech_to_text_config(model_config, task_type) classmethod "). This is for controlling general behavior of the API when serving your model: get\_speech\_to\_text\_config() `[](#__codelineno-1-1)class YourASRModel(nn.Module, SupportsTranscription): [](#__codelineno-1-2) ... [](#__codelineno-1-3) [](#__codelineno-1-4) @classmethod [](#__codelineno-1-5) def get_speech_to_text_config( [](#__codelineno-1-6) cls, [](#__codelineno-1-7) model_config: ModelConfig, [](#__codelineno-1-8) task_type: Literal["transcribe", "translate"], [](#__codelineno-1-9) ) -> SpeechToTextConfig: [](#__codelineno-1-10) return SpeechToTextConfig( [](#__codelineno-1-11) sample_rate=16_000, [](#__codelineno-1-12) max_audio_clip_s=30, [](#__codelineno-1-13) # Set to None to disable server-side chunking if your [](#__codelineno-1-14) # model/processor handles it already [](#__codelineno-1-15) min_energy_split_window_size=None, [](#__codelineno-1-16) )` See [Audio preprocessing and chunking](#audio-preprocessing-and-chunking) for what each field controls. Implement the prompt construction via [get\_generation\_prompt](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription.get_generation_prompt " get_generation_prompt(stt_params) classmethod "). The server builds a [SpeechToTextParams](https://docs.vllm.ai/en/api/vllm/config/speech_to_text/#vllm.config.speech_to_text.SpeechToTextParams " SpeechToTextParams dataclass ") object that bundles the resampled waveform, task parameters, and request-specific options. Your model receives this single object and returns a valid [PromptType](https://docs.vllm.ai/en/api/vllm/inputs/llm/#vllm.inputs.llm.PromptType " PromptType = DecoderOnlyPrompt | EncoderDecoderPrompt module-attribute "). There are two common patterns: #### Multimodal LLM with audio embeddings (e.g., Voxtral, Gemma3n)[¶](#multimodal-llm-with-audio-embeddings-eg-voxtral-gemma3n "Permanent link") Return a dict containing `multi_modal_data` with the audio, and either a `prompt` string or `prompt_token_ids`: get\_generation\_prompt() `[](#__codelineno-2-1)from vllm.config.speech_to_text import SpeechToTextParams [](#__codelineno-2-2)[](#__codelineno-2-3)class YourASRModel(nn.Module, SupportsTranscription): [](#__codelineno-2-4) ... [](#__codelineno-2-5) [](#__codelineno-2-6) @classmethod [](#__codelineno-2-7) def get_generation_prompt( [](#__codelineno-2-8) cls, [](#__codelineno-2-9) stt_params: SpeechToTextParams, [](#__codelineno-2-10) ) -> PromptType: [](#__codelineno-2-11) audio = stt_params.audio [](#__codelineno-2-12) stt_config = stt_params.stt_config [](#__codelineno-2-13) task_type = stt_params.task_type [](#__codelineno-2-14) [](#__codelineno-2-15) task_word = "Transcribe" if task_type == "transcribe" else "Translate" [](#__codelineno-2-16) prompt = ( [](#__codelineno-2-17) "user\n" [](#__codelineno-2-18) f"{task_word} this audio: " [](#__codelineno-2-19) "\nmodel\n" [](#__codelineno-2-20) ) [](#__codelineno-2-21) [](#__codelineno-2-22) return { [](#__codelineno-2-23) "multi_modal_data": {"audio": (audio, stt_config.sample_rate)}, [](#__codelineno-2-24) "prompt": prompt, [](#__codelineno-2-25) }` For further clarification on multi modal inputs, please refer to [Multi-Modal Inputs](https://docs.vllm.ai/en/features/multimodal_inputs/). #### Encoder–decoder audio-only (e.g., Whisper)[¶](#encoderdecoder-audio-only-eg-whisper "Permanent link") Return a dict with separate `encoder_prompt` and `decoder_prompt` entries: get\_generation\_prompt() `[](#__codelineno-3-1)from vllm.config.speech_to_text import SpeechToTextParams [](#__codelineno-3-2)[](#__codelineno-3-3)class YourASRModel(nn.Module, SupportsTranscription): [](#__codelineno-3-4) ... [](#__codelineno-3-5) [](#__codelineno-3-6) @classmethod [](#__codelineno-3-7) def get_generation_prompt( [](#__codelineno-3-8) cls, [](#__codelineno-3-9) stt_params: SpeechToTextParams, [](#__codelineno-3-10) ) -> PromptType: [](#__codelineno-3-11) audio = stt_params.audio [](#__codelineno-3-12) stt_config = stt_params.stt_config [](#__codelineno-3-13) language = stt_params.language [](#__codelineno-3-14) task_type = stt_params.task_type [](#__codelineno-3-15) request_prompt = stt_params.request_prompt [](#__codelineno-3-16) [](#__codelineno-3-17) if language is None: [](#__codelineno-3-18) raise ValueError("Language must be specified") [](#__codelineno-3-19) [](#__codelineno-3-20) prompt = { [](#__codelineno-3-21) "encoder_prompt": { [](#__codelineno-3-22) "prompt": "", [](#__codelineno-3-23) "multi_modal_data": { [](#__codelineno-3-24) "audio": (audio, stt_config.sample_rate), [](#__codelineno-3-25) }, [](#__codelineno-3-26) }, [](#__codelineno-3-27) "decoder_prompt": ( [](#__codelineno-3-28) (f"<|prev|>{request_prompt}" if request_prompt else "") [](#__codelineno-3-29) + f"<|startoftranscript|><|{language}|>" [](#__codelineno-3-30) + f"<|{task_type}|><|notimestamps|>" [](#__codelineno-3-31) ), [](#__codelineno-3-32) } [](#__codelineno-3-33) return cast(PromptType, prompt)` ### `validate_language` (optional)[¶](#validate_language-optional "Permanent link") Language validation via [validate\_language](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription.validate_language " validate_language(language) classmethod ") If your model requires a language and you want a default, override this method (see Whisper): validate\_language() ``[](#__codelineno-4-1)@classmethod [](#__codelineno-4-2)def validate_language(cls, language: str | None) -> str | None: [](#__codelineno-4-3) if language is None: [](#__codelineno-4-4) logger.warning( [](#__codelineno-4-5) "Defaulting to language='en'. If you wish to transcribe " [](#__codelineno-4-6) "audio in a different language, pass the `language` field " [](#__codelineno-4-7) "in the TranscriptionRequest." [](#__codelineno-4-8) ) [](#__codelineno-4-9) language = "en" [](#__codelineno-4-10) return super().validate_language(language)`` ### `get_num_audio_tokens` (optional)[¶](#get_num_audio_tokens-optional "Permanent link") Token accounting for streaming via [get\_num\_audio\_tokens](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription.get_num_audio_tokens " get_num_audio_tokens(audio_duration_s, stt_config, model_config) classmethod ") Provide a fast duration→token estimate to improve streaming usage statistics: get\_num\_audio\_tokens() `[](#__codelineno-5-1)class YourASRModel(nn.Module, SupportsTranscription): [](#__codelineno-5-2) ... [](#__codelineno-5-3) [](#__codelineno-5-4) @classmethod [](#__codelineno-5-5) def get_num_audio_tokens( [](#__codelineno-5-6) cls, [](#__codelineno-5-7) audio_duration_s: float, [](#__codelineno-5-8) stt_config: SpeechToTextConfig, [](#__codelineno-5-9) model_config: ModelConfig, [](#__codelineno-5-10) ) -> int | None: [](#__codelineno-5-11) # Return None if unknown; otherwise return an estimate. [](#__codelineno-5-12) return int(audio_duration_s * stt_config.sample_rate // 320) # example` ## Audio preprocessing and chunking[¶](#audio-preprocessing-and-chunking "Permanent link") The API server takes care of basic audio I/O and optional chunking before building prompts: - Resampling: Input audio is resampled to `SpeechToTextConfig.sample_rate` using [`AudioResampler`](https://docs.vllm.ai/en/api/vllm/multimodal/audio/#vllm.multimodal.audio.AudioResampler " AudioResampler"). - Chunking: If `SpeechToTextConfig.allow_audio_chunking` is True and the duration exceeds `max_audio_clip_s`, the server splits the audio into overlapping chunks and generates a prompt per chunk. Overlap is controlled by `overlap_chunk_second`. - Energy-aware splitting: When `min_energy_split_window_size` is set, the server finds low-energy regions to minimize cutting within words. Relevant server logic: \_preprocess\_speech\_to\_text() `[](#__codelineno-6-1)# vllm/entrypoints/openai/speech_to_text.py [](#__codelineno-6-2)async def _preprocess_speech_to_text(...): [](#__codelineno-6-3) language = self.model_cls.validate_language(request.language) [](#__codelineno-6-4) ... [](#__codelineno-6-5) y, sr = load_audio(bytes_, sr=self.asr_config.sample_rate) [](#__codelineno-6-6) duration = get_audio_duration(y=y, sr=sr) [](#__codelineno-6-7) do_split_audio = (self.asr_config.allow_audio_chunking [](#__codelineno-6-8) and duration > self.asr_config.max_audio_clip_s) [](#__codelineno-6-9) chunks = [y] if not do_split_audio else self._split_audio(y, int(sr)) [](#__codelineno-6-10) prompts = [] [](#__codelineno-6-11) for chunk in chunks: [](#__codelineno-6-12) stt_params = request.build_stt_params( [](#__codelineno-6-13) audio=chunk, [](#__codelineno-6-14) stt_config=self.asr_config, [](#__codelineno-6-15) model_config=self.model_config, [](#__codelineno-6-16) task_type=self.task_type, [](#__codelineno-6-17) ) [](#__codelineno-6-18) prompt = self.model_cls.get_generation_prompt(stt_params) [](#__codelineno-6-19) prompts.append(prompt) [](#__codelineno-6-20) return prompts, duration` ## Exposing tasks automatically[¶](#exposing-tasks-automatically "Permanent link") vLLM automatically advertises transcription support if your model implements the interface: `[](#__codelineno-7-1)if supports_transcription(model): [](#__codelineno-7-2) if model.supports_transcription_only: [](#__codelineno-7-3) return ["transcription"] [](#__codelineno-7-4) supported_tasks.append("transcription")` When enabled, the server initializes the transcription and translation handlers: `[](#__codelineno-8-1)state.openai_serving_transcription = OpenAIServingTranscription(...) if "transcription" in supported_tasks else None [](#__codelineno-8-2)state.openai_serving_translation = OpenAIServingTranslation(...) if "transcription" in supported_tasks else None` No extra registration is required beyond having your model class available via the model registry and implementing [`SupportsTranscription`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription " SupportsTranscription"). ## Examples in-tree[¶](#examples-in-tree "Permanent link") - Whisper encoder–decoder (audio-only): [vllm/model\_executor/models/whisper.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/whisper.py) - Voxtral decoder-only (audio embeddings + LLM): [vllm/model\_executor/models/voxtral.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/voxtral.py). Make sure to have installed `mistral-common[audio]`. - Gemma3n decoder-only with fixed instruction prompt: [vllm/model\_executor/models/gemma3n\_mm.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/gemma3n_mm.py) - Qwen3-Omni multimodal with audio embeddings: [vllm/model\_executor/models/qwen3\_omni\_moe\_thinker.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen3_omni_moe_thinker.py) ## Test with the API[¶](#test-with-the-api "Permanent link") Once your model implements [`SupportsTranscription`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsTranscription " SupportsTranscription"), you can test the endpoints (API mimics OpenAI): - Transcription (ASR): `[](#__codelineno-9-1)curl -s -X POST \ [](#__codelineno-9-2) -H "Authorization: Bearer $VLLM_API_KEY" \ [](#__codelineno-9-3) -H "Content-Type: multipart/form-data" \ [](#__codelineno-9-4) -F "file=@/path/to/audio.wav" \ [](#__codelineno-9-5) -F "model=$MODEL_ID" \ [](#__codelineno-9-6) http://localhost:8000/v1/audio/transcriptions` - Translation (source → English unless otherwise supported): `[](#__codelineno-10-1)curl -s -X POST \ [](#__codelineno-10-2) -H "Authorization: Bearer $VLLM_API_KEY" \ [](#__codelineno-10-3) -H "Content-Type: multipart/form-data" \ [](#__codelineno-10-4) -F "file=@/path/to/audio.wav" \ [](#__codelineno-10-5) -F "model=$MODEL_ID" \ [](#__codelineno-10-6) http://localhost:8000/v1/audio/translations` Or check out more examples in [examples/speech\_to\_text](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text). Note - If your model handles chunking internally (e.g., via its processor or encoder), set `min_energy_split_window_size=None` in the returned [`SpeechToTextConfig`](https://docs.vllm.ai/en/api/vllm/config/speech_to_text/#vllm.config.speech_to_text.SpeechToTextConfig " SpeechToTextConfig") to disable server-side chunking. - Implementing `get_num_audio_tokens` improves accuracy of streaming usage metrics (`prompt_tokens`) without an extra forward pass. - For multilingual behavior, keep `supported_languages` aligned with actual model capabilities. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/model/basic.md "Edit this page") This guide walks you through the steps to implement a basic vLLM model. ## 1\. Bring your model code[¶](#1-bring-your-model-code "Permanent link") First, clone the PyTorch model code from the source repository. For instance, vLLM's [OPT model](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/opt.py) was adapted from HuggingFace's [modeling\_opt.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/opt/modeling_opt.py) file. Warning Make sure to review and adhere to the original code's copyright and licensing terms! ## 2\. Make your code compatible with vLLM[¶](#2-make-your-code-compatible-with-vllm "Permanent link") To ensure compatibility with vLLM, your model must meet the following requirements: ### Initialization Code[¶](#initialization-code "Permanent link") All vLLM modules within the model must include a `prefix` argument in their constructor. This `prefix` is typically the full name of the module in the model's state dictionary and is crucial for: - Runtime support: vLLM's attention operators are registered in a model's state by their full names. Each attention operator must have a unique prefix as its layer name to avoid conflicts. - Non-uniform quantization support: A quantized checkpoint can selectively quantize certain layers while keeping others in full precision. By providing the `prefix` during initialization, vLLM can match the current layer's `prefix` with the quantization configuration to determine if the layer should be initialized in quantized mode. The initialization code should look like this: Code `[](#__codelineno-0-1)from torch import nn [](#__codelineno-0-2)from vllm.config import VllmConfig [](#__codelineno-0-3)from vllm.model_executor.layers.attention import Attention [](#__codelineno-0-4)[](#__codelineno-0-5)class MyAttention(nn.Module): [](#__codelineno-0-6) def __init__(self, vllm_config: VllmConfig, prefix: str): [](#__codelineno-0-7) super().__init__() [](#__codelineno-0-8) self.attn = Attention(prefix=f"{prefix}.attn") [](#__codelineno-0-9)[](#__codelineno-0-10)class MyDecoderLayer(nn.Module): [](#__codelineno-0-11) def __init__(self, vllm_config: VllmConfig, prefix: str): [](#__codelineno-0-12) super().__init__() [](#__codelineno-0-13) self.self_attn = MyAttention(prefix=f"{prefix}.self_attn") [](#__codelineno-0-14)[](#__codelineno-0-15)class MyModel(nn.Module): [](#__codelineno-0-16) def __init__(self, vllm_config: VllmConfig, prefix: str): [](#__codelineno-0-17) super().__init__() [](#__codelineno-0-18) self.layers = nn.ModuleList( [](#__codelineno-0-19) [MyDecoderLayer(vllm_config, prefix=f"{prefix}.layers.{i}") for i in range(vllm_config.model_config.hf_config.num_hidden_layers)] [](#__codelineno-0-20) ) [](#__codelineno-0-21)[](#__codelineno-0-22)class MyModelForCausalLM(nn.Module): [](#__codelineno-0-23) def __init__(self, vllm_config: VllmConfig, prefix: str = ""): [](#__codelineno-0-24) super().__init__() [](#__codelineno-0-25) self.model = MyModel(vllm_config, prefix=f"{prefix}.model")` ### Computation Code[¶](#computation-code "Permanent link") - Add a `embed_input_ids` method inside `MyModel` module that returns the text embeddings given `input_ids`. This is equivalent to directly calling the text embedding layer, but provides a unified interface in case `MyModel` is used within a composite multimodal model. `[](#__codelineno-1-1)class MyModel(nn.Module): [](#__codelineno-1-2) ... [](#__codelineno-1-3) [](#__codelineno-1-4) def embed_input_ids(self, input_ids: torch.Tensor) -> torch.Tensor: [](#__codelineno-1-5) ...` - Rewrite the [forward](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.forward) method of your model to remove any unnecessary code, such as training-specific code. Modify the input parameters to treat `input_ids` and `positions` as flattened tensors with a single batch size dimension, without a max-sequence length dimension. `[](#__codelineno-2-1)def forward( [](#__codelineno-2-2) self, [](#__codelineno-2-3) input_ids: torch.Tensor | None, [](#__codelineno-2-4) positions: torch.Tensor, [](#__codelineno-2-5) intermediate_tensors: IntermediateTensors | None = None, [](#__codelineno-2-6) inputs_embeds: torch.Tensor | None = None, [](#__codelineno-2-7)) -> torch.Tensor: [](#__codelineno-2-8) ...` Note Currently, vLLM supports the basic multi-head attention mechanism and its variant with rotary positional embeddings. If your model employs a different attention mechanism, you will need to implement a new attention layer in vLLM. For reference, check out our [Llama implementation](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llama.py). vLLM already supports a large number of models. It is recommended to find a model similar to yours and adapt it to your model's architecture. Check out [vllm/model\_executor/models](https://github.com/vllm-project/vllm/tree/main/vllm/model_executor/models) for more examples. ## 3\. (Optional) Implement tensor parallelism and quantization support[¶](#3-optional-implement-tensor-parallelism-and-quantization-support "Permanent link") If your model is too large to fit into a single GPU, you can use tensor parallelism to manage it. To do this, substitute your model's linear and embedding layers with their tensor-parallel versions. For the embedding layer, you can simply replace [torch.nn.Embedding](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html#torch.nn.Embedding) with [`VocabParallelEmbedding`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/vocab_parallel_embedding/#vllm.model_executor.layers.vocab_parallel_embedding.VocabParallelEmbedding " VocabParallelEmbedding"). For the output LM head, you can use [`ParallelLMHead`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/vocab_parallel_embedding/#vllm.model_executor.layers.vocab_parallel_embedding.ParallelLMHead " ParallelLMHead"). When it comes to the linear layers, we provide the following options to parallelize them: - [`ReplicatedLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.ReplicatedLinear " ReplicatedLinear"): Replicates the inputs and weights across multiple GPUs. No memory saving. - [`RowParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.RowParallelLinear " RowParallelLinear"): The input tensor is partitioned along the hidden dimension. The weight matrix is partitioned along the rows (input dimension). An _all-reduce_ operation is performed after the matrix multiplication to reduce the results. Typically used for the second FFN layer and the output linear transformation of the attention layer. - [`ColumnParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.ColumnParallelLinear " ColumnParallelLinear"): The input tensor is replicated. The weight matrix is partitioned along the columns (output dimension). The result is partitioned along the column dimension. Typically used for the first FFN layer and the separated QKV transformation of the attention layer in the original Transformer. - [`MergedColumnParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.MergedColumnParallelLinear " MergedColumnParallelLinear"): Column-parallel linear that merges multiple [`ColumnParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.ColumnParallelLinear " ColumnParallelLinear") operators. Typically used for the first FFN layer with weighted activation functions (e.g., SiLU). This class handles the sharded weight loading logic of multiple weight matrices. - [`QKVParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.QKVParallelLinear " QKVParallelLinear"): Parallel linear layer for the query, key, and value projections of the multi-head and grouped-query attention mechanisms. When number of key/value heads are less than the world size, this class replicates the key/value heads properly. This class handles the weight loading and replication of the weight matrices. Note that all the linear layers above take `linear_method` as an input. vLLM will set this parameter according to different quantization schemes to support weight quantization. ## 4\. Implement the weight loading logic[¶](#4-implement-the-weight-loading-logic "Permanent link") You now need to implement the `load_weights` method in your `*ForCausalLM` class. This method should load the weights from the HuggingFace's checkpoint file and assign them to the corresponding layers in your model. Specifically, for [`MergedColumnParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.MergedColumnParallelLinear " MergedColumnParallelLinear") and [`QKVParallelLinear`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/linear/#vllm.model_executor.layers.linear.QKVParallelLinear " QKVParallelLinear") layers, if the original model has separated weight matrices, you need to load the different parts separately. ## 5\. Register your model[¶](#5-register-your-model "Permanent link") See [this page](https://docs.vllm.ai/en/latest/contributing/registration/) for instructions on how to register your new model to be used by vLLM. ## Frequently Asked Questions[¶](#frequently-asked-questions "Permanent link") ### How to support models with interleaving sliding windows?[¶](#how-to-support-models-with-interleaving-sliding-windows "Permanent link") To support a model with interleaving sliding windows, we need to take care of the following details: - Make sure the model's `config.json` contains `layer_types`. - In the modeling code, parse the correct sliding window value for every layer, and pass it to the attention layer's `per_layer_sliding_window` argument. For reference, check [this line](https://github.com/vllm-project/vllm/blob/996357e4808ca5eab97d4c97c7d25b3073f46aab/vllm/model_executor/models/llama.py#L171). With these two steps, interleaved sliding windows should work with the model. ### How to support models that use Mamba?[¶](#how-to-support-models-that-use-mamba "Permanent link") We consider 3 different scenarios: 1. Models that use Mamba layers (either Mamba-1 or Mamba-2) but do not use attention layers. 2. Models that combine Mamba layers (either Mamba-1 or Mamba-2) together with attention layers. 3. Models that combine Mamba-like mechanisms (e.g., Linear Attention, ShortConv) together with attention layers. For case (1), we recommend looking at the implementation of [`MambaForCausalLM`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/mamba.py) (for Mamba-1) or [`Mamba2ForCausalLM`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/mamba2.py) (for Mamba-2) as a reference. The model should inherit protocol [`IsAttentionFree`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.IsAttentionFree " IsAttentionFree") and also implement class methods `get_mamba_state_dtype_from_config` and `get_mamba_state_shape_from_config` to calculate the state shapes and data types from the config. For the mamba layers themselves, please use the [`MambaMixer`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/mamba/mamba_mixer.py) (for Mamba-1) or [`MambaMixer2`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/mamba/mamba_mixer2.py) (for Mamba-2) classes. The model should also be added to the `MODELS_CONFIG_MAP` dictionary in [vllm/model\_executor/models/config.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/config.py) to ensure that the runtime defaults are optimized. For case (2), we recommend using as a reference the implementation of [`JambaForCausalLM`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/jamba.py) (for an example of a model that uses Mamba-1 and attention together) or [`BambaForCausalLM`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/bamba.py) (for an example of a model that uses Mamba-2 and attention together). These models should follow the same instructions as case (1), but they should inherit protocol [`IsHybrid`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.IsHybrid " IsHybrid") (instead of [`IsAttentionFree`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.IsAttentionFree " IsAttentionFree")) and it is _not_ necessary to add them to the `MODELS_CONFIG_MAP` (their runtime defaults will be inferred from the protocol). For case (3), we recommend looking at the implementation of [`MiniMaxText01ForCausalLM`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/minimax_text_01.py) or [`Lfm2ForCausalLM`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/lfm2.py) as a reference, which use custom "mamba-like" layers `MiniMaxText01LinearAttention` and `ShortConv` respectively. Please follow the same guidelines as case (2) for implementing these models. We use "mamba-like" to refer to layers that possess a state that is updated in-place, rather than being appended-to (like KV cache for attention). For implementing new custom mamba-like layers, one should inherit from [`MambaBase`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/mamba/abstract/#vllm.model_executor.layers.mamba.abstract.MambaBase " MambaBase") and implement the methods `get_state_dtype`, `get_state_shape` to calculate the data types and state shapes at runtime, as well as `mamba_type` and `get_attn_backend`. It is also necessary to implement the "attention meta-data" class which handles the meta-data that is common across all layers. Please see [`LinearAttentionMetadata`](https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/linear_attn.py) or [`ShortConvAttentionMetadata`](https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/short_conv_attn.py) for examples of this. It is also worth noting that we should update [`MambaAttentionBackendEnum`](https://docs.vllm.ai/en/api/vllm/v1/attention/backends/registry/#vllm.v1.attention.backends.registry.MambaAttentionBackendEnum " MambaAttentionBackendEnum") in [`registry.py`](https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/registry.py) when adding a new mamba backend. Finally, if one wants to support torch compile and CUDA graphs, it necessary to wrap the call to the mamba-like layer inside a custom op and register it. Please see the calls to `direct_register_custom_op` in [vllm/model\_executor/models/minimax\_text\_01.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/minimax_text_01.py) or [vllm/model\_executor/layers/mamba/short\_conv.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/mamba/short_conv.py) for examples of this. The new custom op should then be added to the list `_attention_ops` in [vllm/config/compilation.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/compilation.py) to ensure that piecewise CUDA graphs works as intended. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/docker.md "Edit this page") ## Pre-built images[¶](#pre-built-images "Permanent link") NVIDIA CUDAAMD ROCmIntel XPUApple Silicon vLLM offers an official Docker image for deployment. The image can be used to run OpenAI compatible server and is available on Docker Hub as [vllm/vllm-openai](https://hub.docker.com/r/vllm/vllm-openai/tags). `[](#__codelineno-0-1)docker run --runtime nvidia --gpus all \ [](#__codelineno-0-2) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-0-3) --env "HF_TOKEN=$HF_TOKEN" \ [](#__codelineno-0-4) -p 8000:8000 \ [](#__codelineno-0-5) --ipc=host \ [](#__codelineno-0-6) vllm/vllm-openai:latest \ [](#__codelineno-0-7) --model Qwen/Qwen3-0.6B` This image can also be used with other container engines such as [Podman](https://podman.io/). `[](#__codelineno-1-1)podman run --device nvidia.com/gpu=all \ [](#__codelineno-1-2)-v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-1-3)--env "HF_TOKEN=$HF_TOKEN" \ [](#__codelineno-1-4)-p 8000:8000 \ [](#__codelineno-1-5)--ipc=host \ [](#__codelineno-1-6)docker.io/vllm/vllm-openai:latest \ [](#__codelineno-1-7)--model Qwen/Qwen3-0.6B` You can add any other [engine-args](https://docs.vllm.ai/en/latest/configuration/engine_args/) you need after the image tag (`vllm/vllm-openai:latest`). Note You can either use the `ipc=host` flag or `--shm-size` flag to allow the container to access the host's shared memory. vLLM uses PyTorch, which uses shared memory to share data between processes under the hood, particularly for tensor parallel inference. Note Optional dependencies are not included in order to avoid licensing issues (e.g. [Issue #8030](https://github.com/vllm-project/vllm/issues/8030)). If you need to use those dependencies (having accepted the license terms), create a custom Dockerfile on top of the base image with an extra layer that installs them: ``[](#__codelineno-2-1)FROM vllm/vllm-openai:v0.11.0 [](#__codelineno-2-2)[](#__codelineno-2-3)# e.g. install the `audio` optional dependencies [](#__codelineno-2-4)# NOTE: Make sure the version of vLLM matches the base image! [](#__codelineno-2-5)RUN uv pip install --system vllm[audio]==0.11.0`` Tip Some new models may only be available on the main branch of [HF Transformers](https://github.com/huggingface/transformers). To use the development version of `transformers`, create a custom Dockerfile on top of the base image with an extra layer that installs their code from source: `[](#__codelineno-3-1)FROM vllm/vllm-openai:latest [](#__codelineno-3-2)[](#__codelineno-3-3)RUN uv pip install --system git+https://github.com/huggingface/transformers.git` #### Running on Systems with Older CUDA Drivers[¶](#running-on-systems-with-older-cuda-drivers "Permanent link") vLLM's Docker image comes with [CUDA compatibility libraries](https://docs.nvidia.com/deploy/cuda-compatibility/index.html) pre-installed. This allows you to run vLLM on systems with NVIDIA drivers that are older than the CUDA Toolkit version used in the image, but only supports select professional and datacenter NVIDIA GPUs. To enable this feature, set the `VLLM_ENABLE_CUDA_COMPATIBILITY` environment variable to `1` or `true` when running the container: `[](#__codelineno-4-1)docker run --runtime nvidia --gpus all \ [](#__codelineno-4-2) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-4-3) -p 8000:8000 \ [](#__codelineno-4-4) --env "HF_TOKEN=" \ [](#__codelineno-4-5) --env "VLLM_ENABLE_CUDA_COMPATIBILITY=1" \ [](#__codelineno-4-6) vllm/vllm-openai ` This will automatically configure `LD_LIBRARY_PATH` to point to the compatibility libraries before loading PyTorch and other dependencies. vLLM offers official Docker images for deployment. The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags). - `vllm/vllm-openai-rocm:latest` — stable release - `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes `[](#__codelineno-5-1)docker run --rm \ [](#__codelineno-5-2) --group-add=video \ [](#__codelineno-5-3) --cap-add=SYS_PTRACE \ [](#__codelineno-5-4) --security-opt seccomp=unconfined \ [](#__codelineno-5-5) --device /dev/kfd \ [](#__codelineno-5-6) --device /dev/dri \ [](#__codelineno-5-7) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-5-8) --env "HF_TOKEN=$HF_TOKEN" \ [](#__codelineno-5-9) -p 8000:8000 \ [](#__codelineno-5-10) --ipc=host \ [](#__codelineno-5-11) vllm/vllm-openai-rocm: \ [](#__codelineno-5-12) --model Qwen/Qwen3-0.6B` To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint. Commands `[](#__codelineno-6-1)docker run --rm -it \ [](#__codelineno-6-2) --group-add=video \ [](#__codelineno-6-3) --cap-add=SYS_PTRACE \ [](#__codelineno-6-4) --security-opt seccomp=unconfined \ [](#__codelineno-6-5) --device /dev/kfd \ [](#__codelineno-6-6) --device /dev/dri \ [](#__codelineno-6-7) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-6-8) --env "HF_TOKEN=$HF_TOKEN" \ [](#__codelineno-6-9) --network=host \ [](#__codelineno-6-10) --ipc=host \ [](#__codelineno-6-11) --entrypoint /bin/bash \ [](#__codelineno-6-12) vllm/vllm-openai-rocm:` #### Use AMD's Docker Images (Deprecated)[¶](#use-amds-docker-images-deprecated "Permanent link") Deprecated AMD's Docker images (`rocm/vllm` and `rocm/vllm-dev`) are deprecated in favor of the official vLLM Docker images above (`vllm/vllm-openai-rocm`). Please migrate to the official images. Prior to January 20th, 2026 when the official docker images became available on [upstream vLLM docker hub](https://hub.docker.com/v2/repositories/vllm/vllm-openai-rocm/tags/), the [AMD Infinity hub for vLLM](https://hub.docker.com/r/rocm/vllm/tags) offered a prebuilt, optimized docker image designed for validating inference performance on the AMD Instinct MI300X™ accelerator. AMD also offered nightly prebuilt docker image from [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev), which has vLLM and all its dependencies installed. The entrypoint of this docker image is `/bin/bash` (different from the vLLM's Official Docker Image). Currently, we release prebuilt XPU images at docker [hub](https://hub.docker.com/r/intel/vllm/tags) based on vLLM released version. For more information, please refer release [note](https://github.com/intel/ai-containers/blob/main/vllm). ## Run as a non-root user[¶](#run-as-a-non-root-user "Permanent link") The CUDA `vllm/vllm-openai` image runs as root by default for backward compatibility. It is also prepared to run as the built-in `vllm` user (UID 2000, GID 0): `[](#__codelineno-7-1)docker run --rm --gpus all \ [](#__codelineno-7-2) --user 2000:0 \ [](#__codelineno-7-3) -p 8000:8000 \ [](#__codelineno-7-4) vllm/vllm-openai:latest \ [](#__codelineno-7-5) meta-llama/Llama-3.1-8B-Instruct` When mounting model or cache volumes for a non-root container, mount writable paths under `/home/vllm` instead of `/root`. For example, mount the Hugging Face cache at `/home/vllm/.cache/huggingface` and make the mounted directory writable by group 0. `[](#__codelineno-8-1)docker run --rm --gpus all \ [](#__codelineno-8-2) --user 2000:0 \ [](#__codelineno-8-3) -v ~/.cache/huggingface:/home/vllm/.cache/huggingface \ [](#__codelineno-8-4) -p 8000:8000 \ [](#__codelineno-8-5) vllm/vllm-openai:latest \ [](#__codelineno-8-6) meta-llama/Llama-3.1-8B-Instruct` To build an image that defaults to the non-root `vllm` user, use the opt-in `vllm-openai-nonroot` target: `[](#__codelineno-9-1)docker build --target vllm-openai-nonroot \ [](#__codelineno-9-2) -t vllm-openai-nonroot:local \ [](#__codelineno-9-3) -f docker/Dockerfile . [](#__codelineno-9-4)[](#__codelineno-9-5)docker run --rm --gpus all \ [](#__codelineno-9-6) -p 8000:8000 \ [](#__codelineno-9-7) vllm-openai-nonroot:local \ [](#__codelineno-9-8) meta-llama/Llama-3.1-8B-Instruct` The `vllm-openai-nonroot` target also supports OpenShift-style arbitrary UIDs when the runtime UID is a member of group 0. In Kubernetes manifests, set the container security context accordingly and keep mounted cache/model paths writable by group 0: `[](#__codelineno-10-1)securityContext: [](#__codelineno-10-2) runAsNonRoot: true [](#__codelineno-10-3) runAsUser: 1000540000 [](#__codelineno-10-4) runAsGroup: 0 [](#__codelineno-10-5) fsGroup: 0` Runtime UIDs outside group 0 are not part of the documented support matrix because they may be unable to write to `/home/vllm` or `/opt/uv/cache`. ## Build image from source[¶](#build-image-from-source "Permanent link") NVIDIA CUDAAMD ROCmIntel XPUApple Silicon You can build and run vLLM from source via the provided [docker/Dockerfile](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile). To build vLLM: `[](#__codelineno-11-1)# optionally specifies: --build-arg max_jobs=8 --build-arg nvcc_threads=2 [](#__codelineno-11-2)DOCKER_BUILDKIT=1 docker build . \ [](#__codelineno-11-3) --target vllm-openai \ [](#__codelineno-11-4) --tag vllm/vllm-openai \ [](#__codelineno-11-5) --file docker/Dockerfile` Note By default vLLM will build for all GPU types for widest distribution. If you are just building for the current GPU type the machine is running on, you can add the argument `--build-arg torch_cuda_arch_list=""` for vLLM to find the current GPU type and build for that. If you are using Podman instead of Docker, you might need to disable SELinux labeling by adding `--security-opt label=disable` when running `podman build` command to avoid certain [existing issues](https://github.com/containers/buildah/discussions/4184). Note If you have not changed any C++ or CUDA kernel code, you can use precompiled wheels to significantly reduce Docker build time. - **Enable the feature** by adding the build argument: `--build-arg VLLM_USE_PRECOMPILED="1"`. - **How it works**: By default, vLLM automatically finds the correct wheels from our [Nightly Builds](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/) by using the merge-base commit with the upstream `main` branch. - **Override commit**: To use wheels from a specific commit, provide the `--build-arg VLLM_PRECOMPILED_WHEEL_COMMIT=` argument. For a detailed explanation, refer to the documentation on 'Set up using Python-only build (without compilation)' part in [Build wheel from source](https://docs.vllm.ai/en/latest/contributing/ci/nightly_builds/#precompiled-wheels-usage), these args are similar. #### Building vLLM's Docker Image from Source for Arm64/aarch64[¶](#building-vllms-docker-image-from-source-for-arm64aarch64 "Permanent link") A docker container can be built for aarch64 systems such as the Nvidia Grace-Hopper and Grace-Blackwell. Using the flag `--platform "linux/arm64"` will build for arm64. Note Multiple modules must be compiled, so this process can take a while. Recommend using `--build-arg max_jobs=` & `--build-arg nvcc_threads=` flags to speed up build process. However, ensure your `max_jobs` is substantially larger than `nvcc_threads` to get the most benefits. Keep an eye on memory usage with parallel jobs as it can be substantial (see example below). Command `[](#__codelineno-12-1)# Example of building on Nvidia GH200 server. (Memory usage: ~15GB, Build time: ~1475s / ~25 min, Image size: 6.93GB) [](#__codelineno-12-2)DOCKER_BUILDKIT=1 docker build . \ [](#__codelineno-12-3)--file docker/Dockerfile \ [](#__codelineno-12-4)--target vllm-openai \ [](#__codelineno-12-5)--platform "linux/arm64" \ [](#__codelineno-12-6)-t vllm/vllm-gh200-openai:latest \ [](#__codelineno-12-7)--build-arg max_jobs=66 \ [](#__codelineno-12-8)--build-arg nvcc_threads=2 \ [](#__codelineno-12-9)--build-arg torch_cuda_arch_list="9.0 10.0+PTX" \ [](#__codelineno-12-10)--build-arg RUN_WHEEL_CHECK=false` For (G)B300, we recommend using CUDA 13, as shown in the following command. Command `[](#__codelineno-13-1)DOCKER_BUILDKIT=1 docker build \ [](#__codelineno-13-2)--build-arg CUDA_VERSION=13.0.2 \ [](#__codelineno-13-3)--build-arg BUILD_BASE_IMAGE=nvidia/cuda:13.0.2-devel-ubuntu22.04 \ [](#__codelineno-13-4)--build-arg max_jobs=256 \ [](#__codelineno-13-5)--build-arg nvcc_threads=2 \ [](#__codelineno-13-6)--build-arg RUN_WHEEL_CHECK=false \ [](#__codelineno-13-7)--build-arg torch_cuda_arch_list='9.0 10.0+PTX' \ [](#__codelineno-13-8)--platform "linux/arm64" \ [](#__codelineno-13-9)--tag vllm/vllm-gb300-openai:latest \ [](#__codelineno-13-10)--target vllm-openai \ [](#__codelineno-13-11)-f docker/Dockerfile \ [](#__codelineno-13-12).` Note If you are building the `linux/arm64` image on a non-ARM host (e.g., an x86\_64 machine), you need to ensure your system is set up for cross-compilation using QEMU. This allows your host machine to emulate ARM64 execution. Run the following command on your host machine to register QEMU user static handlers: `[](#__codelineno-14-1)docker run --rm --privileged multiarch/qemu-user-static --reset -p yes` After setting up QEMU, you can use the `--platform "linux/arm64"` flag in your `docker build` command. #### Use the custom-built vLLM Docker image\*\*[¶](#use-the-custom-built-vllm-docker-image "Permanent link") To run vLLM with the custom-built Docker image: `[](#__codelineno-15-1)docker run --runtime nvidia --gpus all \ [](#__codelineno-15-2) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-15-3) -p 8000:8000 \ [](#__codelineno-15-4) --env "HF_TOKEN=" \ [](#__codelineno-15-5) vllm/vllm-openai ` The argument `vllm/vllm-openai` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command). Note **For version 0.4.1 and 0.4.2 only** - the vLLM docker images under these versions are supposed to be run under the root user since a library under the root user's home directory, i.e. `/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` is required to be loaded during runtime. If you are running the container under a different user, you may need to first change the permissions of the library (and all the parent directories) to allow the user to access it, then run vLLM with environment variable `VLLM_NCCL_SO_PATH=/root/.config/vllm/nccl/cu12/libnccl.so.2.18.1` . You can build and run vLLM from source via the provided [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm). (Optional) Build an image with ROCm software stack Build a docker image from [docker/Dockerfile.rocm\_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM. **This step is optional as this rocm\_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** If you choose to build this rocm\_base image yourself, the steps are as follows. It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon: `[](#__codelineno-16-1){ [](#__codelineno-16-2) "features": { [](#__codelineno-16-3) "buildkit": true [](#__codelineno-16-4) } [](#__codelineno-16-5)}` To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default: `[](#__codelineno-17-1)DOCKER_BUILDKIT=1 docker build \ [](#__codelineno-17-2) -f docker/Dockerfile.rocm_base \ [](#__codelineno-17-3) -t rocm/vllm-dev:base .` First, build a docker image from [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) and launch a docker container from the image. It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration /etc/docker/daemon.json as follows and restart the daemon: `[](#__codelineno-18-1){ [](#__codelineno-18-2) "features": { [](#__codelineno-18-3) "buildkit": true [](#__codelineno-18-4) } [](#__codelineno-18-5)}` [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches. It provides flexibility to customize the build of docker image using the following arguments: - `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm\_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) - `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image Their values can be passed in when running `docker build` with `--build-arg` options. To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint): `[](#__codelineno-19-1)DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .` To run vLLM with the custom-built Docker image: `[](#__codelineno-20-1)docker run --rm \ [](#__codelineno-20-2) --group-add=video \ [](#__codelineno-20-3) --cap-add=SYS_PTRACE \ [](#__codelineno-20-4) --security-opt seccomp=unconfined \ [](#__codelineno-20-5) --device /dev/kfd \ [](#__codelineno-20-6) --device /dev/dri \ [](#__codelineno-20-7) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-20-8) --env "HF_TOKEN=$HF_TOKEN" \ [](#__codelineno-20-9) -p 8000:8000 \ [](#__codelineno-20-10) --ipc=host \ [](#__codelineno-20-11) vllm/vllm-openai-rocm ` The argument `vllm/vllm-openai-rocm` specifies the image to run, and should be replaced with the name of the custom-built image (the `-t` tag from the build command). To use the docker image as base for development, you can launch it in interactive session through overriding the entrypoint. Commands `[](#__codelineno-21-1)docker run --rm -it \ [](#__codelineno-21-2) --group-add=video \ [](#__codelineno-21-3) --cap-add=SYS_PTRACE \ [](#__codelineno-21-4) --security-opt seccomp=unconfined \ [](#__codelineno-21-5) --device /dev/kfd \ [](#__codelineno-21-6) --device /dev/dri \ [](#__codelineno-21-7) -v ~/.cache/huggingface:/root/.cache/huggingface \ [](#__codelineno-21-8) --env "HF_TOKEN=$HF_TOKEN" \ [](#__codelineno-21-9) --network=host \ [](#__codelineno-21-10) --ipc=host \ [](#__codelineno-21-11) --entrypoint bash \ [](#__codelineno-21-12) vllm/vllm-openai-rocm` `[](#__codelineno-22-1)docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g . [](#__codelineno-22-2)docker run -it \ [](#__codelineno-22-3) --rm \ [](#__codelineno-22-4) --network=host \ [](#__codelineno-22-5) --device /dev/dri:/dev/dri \ [](#__codelineno-22-6) -v /dev/dri/by-path:/dev/dri/by-path \ [](#__codelineno-22-7) --ipc=host \ [](#__codelineno-22-8) --privileged \ [](#__codelineno-22-9) vllm-xpu-env` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/nginx.md "Edit this page") This document shows how to launch multiple vLLM serving containers and use Nginx to act as a load balancer between the servers. ## Build Nginx Container[¶](#build-nginx-container "Permanent link") This guide assumes that you have just cloned the vLLM project and you're currently in the vllm root directory. `` [](#__codelineno-0-1)export vllm_root=`pwd` `` Create a file named `Dockerfile.nginx`: `[](#__codelineno-1-1)FROM nginx:latest [](#__codelineno-1-2)RUN rm /etc/nginx/conf.d/default.conf [](#__codelineno-1-3)EXPOSE 80 [](#__codelineno-1-4)CMD ["nginx", "-g", "daemon off;"]` Build the container: `[](#__codelineno-2-1)docker build . -f Dockerfile.nginx --tag nginx-lb` ## Create Simple Nginx Config file[¶](#create-simple-nginx-config-file "Permanent link") Create a file named `nginx_conf/nginx.conf`. Note that you can add as many servers as you'd like. In the below example we'll start with two. To add more, add another `server vllmN:8000 max_fails=3 fail_timeout=10000s;` entry to `upstream backend`. Config `[](#__codelineno-3-1)upstream backend { [](#__codelineno-3-2) least_conn; [](#__codelineno-3-3) server vllm0:8000 max_fails=3 fail_timeout=10000s; [](#__codelineno-3-4) server vllm1:8000 max_fails=3 fail_timeout=10000s; [](#__codelineno-3-5)} [](#__codelineno-3-6)server { [](#__codelineno-3-7) listen 80; [](#__codelineno-3-8) location / { [](#__codelineno-3-9) proxy_pass http://backend; [](#__codelineno-3-10) proxy_set_header Host $host; [](#__codelineno-3-11) proxy_set_header X-Real-IP $remote_addr; [](#__codelineno-3-12) proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; [](#__codelineno-3-13) proxy_set_header X-Forwarded-Proto $scheme; [](#__codelineno-3-14) } [](#__codelineno-3-15)}` ## Build vLLM Container[¶](#build-vllm-container "Permanent link") `[](#__codelineno-4-1)cd $vllm_root [](#__codelineno-4-2)docker build -f docker/Dockerfile . --tag vllm` If you are behind proxy, you can pass the proxy settings to the docker build command as shown below: `[](#__codelineno-5-1)cd $vllm_root [](#__codelineno-5-2)docker build \ [](#__codelineno-5-3) -f docker/Dockerfile . \ [](#__codelineno-5-4) --tag vllm \ [](#__codelineno-5-5) --build-arg http_proxy=$http_proxy \ [](#__codelineno-5-6) --build-arg https_proxy=$https_proxy` ## Create Docker Network[¶](#create-docker-network "Permanent link") `[](#__codelineno-6-1)docker network create vllm_nginx` ## Launch vLLM Containers[¶](#launch-vllm-containers "Permanent link") Notes: - If you have your HuggingFace models cached somewhere else, update `hf_cache_dir` below. - If you don't have an existing HuggingFace cache you will want to start `vllm0` and wait for the model to complete downloading and the server to be ready. This will ensure that `vllm1` can leverage the model you just downloaded and it won't have to be downloaded again. - The below example assumes GPU backend used. If you are using CPU backend, remove `--gpus device=ID`, add `VLLM_CPU_KVCACHE_SPACE` and `VLLM_CPU_OMP_THREADS_BIND` environment variables to the docker run command. - Adjust the model name that you want to use in your vLLM servers if you don't want to use `Llama-2-7b-chat-hf`. Commands `[](#__codelineno-7-1)mkdir -p ~/.cache/huggingface/hub/ [](#__codelineno-7-2)hf_cache_dir=~/.cache/huggingface/ [](#__codelineno-7-3)docker run \ [](#__codelineno-7-4) -itd \ [](#__codelineno-7-5) --ipc host \ [](#__codelineno-7-6) --network vllm_nginx \ [](#__codelineno-7-7) --gpus device=0 \ [](#__codelineno-7-8) --shm-size=10.24gb \ [](#__codelineno-7-9) -v $hf_cache_dir:/root/.cache/huggingface/ \ [](#__codelineno-7-10) -p 8081:8000 \ [](#__codelineno-7-11) --name vllm0 vllm \ [](#__codelineno-7-12) --model meta-llama/Llama-2-7b-chat-hf [](#__codelineno-7-13)docker run \ [](#__codelineno-7-14) -itd \ [](#__codelineno-7-15) --ipc host \ [](#__codelineno-7-16) --network vllm_nginx \ [](#__codelineno-7-17) --gpus device=1 \ [](#__codelineno-7-18) --shm-size=10.24gb \ [](#__codelineno-7-19) -v $hf_cache_dir:/root/.cache/huggingface/ \ [](#__codelineno-7-20) -p 8082:8000 \ [](#__codelineno-7-21) --name vllm1 vllm \ [](#__codelineno-7-22) --model meta-llama/Llama-2-7b-chat-hf` Note If you are behind proxy, you can pass the proxy settings to the docker run command via `-e http_proxy=$http_proxy -e https_proxy=$https_proxy`. ## Launch Nginx[¶](#launch-nginx "Permanent link") `[](#__codelineno-8-1)docker run \ [](#__codelineno-8-2) -itd \ [](#__codelineno-8-3) -p 8000:80 \ [](#__codelineno-8-4) --network vllm_nginx \ [](#__codelineno-8-5) -v ./nginx_conf/:/etc/nginx/conf.d/ \ [](#__codelineno-8-6) --name nginx-lb nginx-lb:latest` ## Verify That vLLM Servers Are Ready[¶](#verify-that-vllm-servers-are-ready "Permanent link") `[](#__codelineno-9-1)docker logs vllm0 | grep Uvicorn [](#__codelineno-9-2)docker logs vllm1 | grep Uvicorn` Both outputs should look like this: `[](#__codelineno-10-1)INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/model/multimodal.md "Edit this page") This document walks you through the steps to extend a basic model so that it accepts [multi-modal inputs](https://docs.vllm.ai/en/features/multimodal_inputs/). ## 1\. Update the base vLLM model[¶](#1-update-the-base-vllm-model "Permanent link") It is assumed that you have already implemented the model in vLLM according to [these steps](https://docs.vllm.ai/en/latest/contributing/basic/). Further update the model as follows: - Implement [get\_placeholder\_str](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal.get_placeholder_str " get_placeholder_str(modality, i) classmethod ") to define the placeholder string which is used to represent the multi-modal item in the text prompt. This should be consistent with the chat template of the model. Code `[](#__codelineno-0-1)class YourModelForImage2Seq(nn.Module): [](#__codelineno-0-2) ... [](#__codelineno-0-3) [](#__codelineno-0-4) @classmethod [](#__codelineno-0-5) def get_placeholder_str(cls, modality: str, i: int) -> str | None: [](#__codelineno-0-6) if modality.startswith("image"): [](#__codelineno-0-7) return "" [](#__codelineno-0-8) [](#__codelineno-0-9) raise ValueError("Only image modality is supported")` - Inside `__init__` method, initialize the language components of the model inside [\_mark\_language\_model](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal._mark_language_model " _mark_language_model(vllm_config, *, targets=None)"), and the multimodal components of the model inside [\_mark\_tower\_model](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal._mark_tower_model " _mark_tower_model(vllm_config, modalities, *, targets=None)"), e.g.: `[](#__codelineno-1-1) def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None: [](#__codelineno-1-2) super().__init__() [](#__codelineno-1-3) [](#__codelineno-1-4) config = vllm_config.model_config.hf_config [](#__codelineno-1-5) [](#__codelineno-1-6) with self._mark_tower_model(vllm_config, "image"): [](#__codelineno-1-7) self.vision_encoder = ... [](#__codelineno-1-8) self.multi_modal_projector = ... [](#__codelineno-1-9) [](#__codelineno-1-10) with self._mark_language_model(vllm_config): [](#__codelineno-1-11) self.language_model = init_vllm_registered_model( [](#__codelineno-1-12) vllm_config=vllm_config, [](#__codelineno-1-13) hf_config=config.text_config, [](#__codelineno-1-14) prefix=maybe_prefix(prefix, "language_model"), [](#__codelineno-1-15) )` - Remove the embedding part from the [forward](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.forward) method: - Move the multi-modal embedding to [embed\_multimodal](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal " embed_multimodal(**kwargs)"). - The text embedding and embedding merge are handled automatically by a default implementation of [embed\_input\_ids](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal.embed_input_ids " embed_input_ids(input_ids, multimodal_embeddings=None, *, is_multimodal=None)"). It does not need to be overridden in most cases. `[](#__codelineno-2-1) def forward( [](#__codelineno-2-2) self, [](#__codelineno-2-3) input_ids: torch.Tensor | None, [](#__codelineno-2-4)- pixel_values: torch.Tensor, [](#__codelineno-2-5) positions: torch.Tensor, [](#__codelineno-2-6) intermediate_tensors: IntermediateTensors | None = None, [](#__codelineno-2-7) inputs_embeds: torch.Tensor | None = None, [](#__codelineno-2-8) ) -> torch.Tensor: [](#__codelineno-2-9)- if inputs_embeds is None: [](#__codelineno-2-10)- inputs_embeds = self.get_input_embeddings()(input_ids) [](#__codelineno-2-11)- [](#__codelineno-2-12)- if pixel_values is not None: [](#__codelineno-2-13)- image_features = self.get_image_features( [](#__codelineno-2-14)- pixel_values=pixel_values, [](#__codelineno-2-15)- ) [](#__codelineno-2-16)- special_image_mask = self.get_placeholder_mask( [](#__codelineno-2-17)- input_ids, [](#__codelineno-2-18)- inputs_embeds=inputs_embeds, [](#__codelineno-2-19)- image_features=image_features, [](#__codelineno-2-20)- ) [](#__codelineno-2-21)- inputs_embeds = inputs_embeds.masked_scatter( [](#__codelineno-2-22)- special_image_mask, [](#__codelineno-2-23)- image_features, [](#__codelineno-2-24)- ) [](#__codelineno-2-25) [](#__codelineno-2-26) hidden_states = self.language_model( [](#__codelineno-2-27) input_ids, [](#__codelineno-2-28) positions, [](#__codelineno-2-29) intermediate_tensors, [](#__codelineno-2-30) inputs_embeds=inputs_embeds, [](#__codelineno-2-31) ) [](#__codelineno-2-32) ... [](#__codelineno-2-33)[](#__codelineno-2-34)+ def embed_multimodal( [](#__codelineno-2-35)+ self, [](#__codelineno-2-36)+ pixel_values: torch.Tensor, [](#__codelineno-2-37)+ ) -> MultiModalEmbeddings | None: [](#__codelineno-2-38)+ return self.get_image_features( [](#__codelineno-2-39)+ pixel_values=pixel_values, [](#__codelineno-2-40)+ )` Below we provide a boilerplate of a typical implementation pattern of [embed\_multimodal](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal.embed_multimodal " embed_multimodal(**kwargs)"), but feel free to adjust it to your own needs. `[](#__codelineno-3-1)def _process_image_input(self, image_input: YourModelImageInputs) -> torch.Tensor: [](#__codelineno-3-2) image_features = self.vision_encoder(image_input) [](#__codelineno-3-3) return self.multi_modal_projector(image_features) [](#__codelineno-3-4)[](#__codelineno-3-5)def embed_multimodal( [](#__codelineno-3-6) self, [](#__codelineno-3-7) **kwargs: object, [](#__codelineno-3-8)) -> MultiModalEmbeddings | None: [](#__codelineno-3-9) # Validate the multimodal input keyword arguments [](#__codelineno-3-10) image_input = self._parse_and_validate_image_input(**kwargs) [](#__codelineno-3-11) if image_input is None: [](#__codelineno-3-12) return None [](#__codelineno-3-13) [](#__codelineno-3-14) # Run multimodal inputs through encoder and projector [](#__codelineno-3-15) vision_embeddings = self._process_image_input(image_input) [](#__codelineno-3-16) return vision_embeddings` Important The returned `multimodal_embeddings` must be either a **3D [torch.Tensor](https://pytorch.org/docs/stable/tensors.html#torch.Tensor)** of shape `(num_items, feature_size, hidden_size)`, or a **list / tuple of 2D [torch.Tensor](https://pytorch.org/docs/stable/tensors.html#torch.Tensor)'s** of shape `(feature_size, hidden_size)`, so that `multimodal_embeddings[i]` retrieves the embeddings generated from the `i`\-th multimodal data item (e.g, image) of the request. Note By default, vLLM merges the multimodal embeddings into text embeddings depending on the information of their locations defined in [PlaceholderRange](https://docs.vllm.ai/en/api/vllm/multimodal/inputs/#vllm.multimodal.inputs.PlaceholderRange " PlaceholderRange dataclass ") from input processing. This logic can be found at [embed\_input\_ids](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal.embed_input_ids " embed_input_ids(input_ids, multimodal_embeddings=None, *, is_multimodal=None)"). You may override this method if additional logic is required for your model when merging embeddings. - Once the above steps are done, update the model class with the [SupportsMultiModal](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsMultiModal " SupportsMultiModal") interface. `[](#__codelineno-4-1)+ from vllm.model_executor.models.interfaces import SupportsMultiModal [](#__codelineno-4-2)[](#__codelineno-4-3)- class YourModelForImage2Seq(nn.Module): [](#__codelineno-4-4)+ class YourModelForImage2Seq(nn.Module, SupportsMultiModal):` ## 2\. Specify processing information[¶](#2-specify-processing-information "Permanent link") Next, create a subclass of [BaseProcessingInfo](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseProcessingInfo " BaseProcessingInfo") to provide basic information related to HF processing. ### Maximum number of input items[¶](#maximum-number-of-input-items "Permanent link") You need to override the abstract method [get\_supported\_mm\_limits](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseProcessingInfo.get_supported_mm_limits " get_supported_mm_limits() abstractmethod ") to return the maximum number of input items for each modality supported by the model. For example, if the model supports any number of images but only one video per prompt: `[](#__codelineno-5-1)def get_supported_mm_limits(self) -> Mapping[str, int | None]: [](#__codelineno-5-2) return {"image": None, "video": 1}` ## 3\. Specify dummy inputs[¶](#3-specify-dummy-inputs "Permanent link") Then, inherit [BaseDummyInputsBuilder](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseDummyInputsBuilder " BaseDummyInputsBuilder") to construct dummy inputs for HF processing. The processed outputs are also used for memory profiling. Override the abstract methods [get\_dummy\_text](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text " get_dummy_text(mm_counts) abstractmethod ") and [get\_dummy\_mm\_data](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_mm_data " get_dummy_mm_data(seq_len, mm_counts, mm_options) abstractmethod ") to construct dummy inputs. These dummy inputs should result in the worst-case memory usage of the model so that vLLM can reserve the correct amount of memory for it. Assuming that the memory usage increases with the number of tokens, the dummy inputs can be constructed to maximize the number of output embeddings, which is the same number as placeholder feature tokens. Basic example: LLaVANo input placeholders: Fuyu Looking at the code of HF's `LlavaForConditionalGeneration`: Code `[](#__codelineno-6-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L530-L544 [](#__codelineno-6-2)n_image_tokens = (input_ids == self.config.image_token_index).sum().item() [](#__codelineno-6-3)n_image_features = image_features.shape[0] * image_features.shape[1] [](#__codelineno-6-4)[](#__codelineno-6-5)if n_image_tokens != n_image_features: [](#__codelineno-6-6) raise ValueError( [](#__codelineno-6-7) f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}" [](#__codelineno-6-8) ) [](#__codelineno-6-9)special_image_mask = ( [](#__codelineno-6-10) (input_ids == self.config.image_token_index) [](#__codelineno-6-11) .unsqueeze(-1) [](#__codelineno-6-12) .expand_as(inputs_embeds) [](#__codelineno-6-13) .to(inputs_embeds.device) [](#__codelineno-6-14)) [](#__codelineno-6-15)image_features = image_features.to(inputs_embeds.device, inputs_embeds.dtype) [](#__codelineno-6-16)inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features)` The number of placeholder feature tokens per image is `image_features.shape[1]`. `image_features` is calculated inside the `get_image_features` method: Code `[](#__codelineno-7-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/modeling_llava.py#L290-L300 [](#__codelineno-7-2)image_outputs = self.vision_tower(pixel_values, output_hidden_states=True) [](#__codelineno-7-3)[](#__codelineno-7-4)selected_image_feature = image_outputs.hidden_states[vision_feature_layer] [](#__codelineno-7-5)if vision_feature_select_strategy == "default": [](#__codelineno-7-6) selected_image_feature = selected_image_feature[:, 1:] [](#__codelineno-7-7)elif vision_feature_select_strategy == "full": [](#__codelineno-7-8) selected_image_feature = selected_image_feature [](#__codelineno-7-9)else: [](#__codelineno-7-10) raise ValueError(f"Unexpected select feature strategy: {self.config.vision_feature_select_strategy}") [](#__codelineno-7-11)image_features = self.multi_modal_projector(selected_image_feature) [](#__codelineno-7-12)return image_features` We can infer that `image_features.shape[1]` is based on `image_outputs.hidden_states.shape[1]` from the vision tower (`CLIPVisionModel` for the [`llava-hf/llava-1.5-7b-hf`](https://huggingface.co/llava-hf/llava-1.5-7b-hf) model). Moreover, we only need the sequence length (the second dimension of the tensor) to get `image_features.shape[1]`. The sequence length is determined by the initial hidden states in `CLIPVisionTransformer` since the attention mechanism doesn't change the sequence length of the output hidden states. `[](#__codelineno-8-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L1094-L1102 [](#__codelineno-8-2)hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding) [](#__codelineno-8-3)hidden_states = self.pre_layrnorm(hidden_states) [](#__codelineno-8-4)[](#__codelineno-8-5)encoder_outputs = self.encoder( [](#__codelineno-8-6) inputs_embeds=hidden_states, [](#__codelineno-8-7) output_attentions=output_attentions, [](#__codelineno-8-8) output_hidden_states=output_hidden_states, [](#__codelineno-8-9) return_dict=return_dict, [](#__codelineno-8-10))` To find the sequence length, we turn to the code of `CLIPVisionEmbeddings`: Code `[](#__codelineno-9-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L247-L257 [](#__codelineno-9-2)target_dtype = self.patch_embedding.weight.dtype [](#__codelineno-9-3)patch_embeds = self.patch_embedding(pixel_values.to(dtype=target_dtype)) # shape = [*, width, grid, grid] [](#__codelineno-9-4)patch_embeds = patch_embeds.flatten(2).transpose(1, 2) [](#__codelineno-9-5)[](#__codelineno-9-6)class_embeds = self.class_embedding.expand(batch_size, 1, -1) [](#__codelineno-9-7)embeddings = torch.cat([class_embeds, patch_embeds], dim=1) [](#__codelineno-9-8)if interpolate_pos_encoding: [](#__codelineno-9-9) embeddings = embeddings + self.interpolate_pos_encoding(embeddings, height, width) [](#__codelineno-9-10)else: [](#__codelineno-9-11) embeddings = embeddings + self.position_embedding(self.position_ids) [](#__codelineno-9-12)return embeddings` We can infer that `embeddings.shape[1] == self.num_positions`, where `[](#__codelineno-10-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/modeling_clip.py#L195-L196 [](#__codelineno-10-2)self.num_patches = (self.image_size // self.patch_size) ** 2 [](#__codelineno-10-3)self.num_positions = self.num_patches + 1` Overall, the number of placeholder feature tokens for an image can be calculated as: Code `[](#__codelineno-11-1)def get_num_image_tokens( [](#__codelineno-11-2) self, [](#__codelineno-11-3) *, [](#__codelineno-11-4) image_width: int, [](#__codelineno-11-5) image_height: int, [](#__codelineno-11-6)) -> int: [](#__codelineno-11-7) hf_config = self.get_hf_config() [](#__codelineno-11-8) hf_processor = self.get_hf_processor() [](#__codelineno-11-9) [](#__codelineno-11-10) image_size = hf_config.vision_config.image_size [](#__codelineno-11-11) patch_size = hf_config.vision_config.patch_size [](#__codelineno-11-12) [](#__codelineno-11-13) num_image_tokens = (image_size // patch_size) ** 2 + 1 [](#__codelineno-11-14) if hf_processor.vision_feature_select_strategy == "default": [](#__codelineno-11-15) num_image_tokens -= 1 [](#__codelineno-11-16) [](#__codelineno-11-17) return num_image_tokens` Notice that the number of image tokens doesn't depend on the image width and height. We can simply use a dummy `image_size` to calculate the multimodal profiling data: Code ``[](#__codelineno-12-1)# NOTE: In actuality, this is usually implemented as part of the [](#__codelineno-12-2)# model's subclass of [`BaseProcessingInfo`][vllm.multimodal.processing.context.BaseProcessingInfo], but we show it as is [](#__codelineno-12-3)# here for simplicity. [](#__codelineno-12-4)def get_image_size_with_most_features(self) -> ImageSize: [](#__codelineno-12-5) hf_config = self.get_hf_config() [](#__codelineno-12-6) width = height = hf_config.image_size [](#__codelineno-12-7) return ImageSize(width=width, height=height) [](#__codelineno-12-8)[](#__codelineno-12-9)def get_dummy_mm_data( [](#__codelineno-12-10) self, [](#__codelineno-12-11) seq_len: int, [](#__codelineno-12-12) mm_counts: Mapping[str, int], [](#__codelineno-12-13) mm_options: Mapping[str, BaseDummyOptions], [](#__codelineno-12-14)) -> MultiModalDataDict: [](#__codelineno-12-15) num_images = mm_counts.get("image", 0) [](#__codelineno-12-16) [](#__codelineno-12-17) target_width, target_height = \ [](#__codelineno-12-18) self.info.get_image_size_with_most_features() [](#__codelineno-12-19) [](#__codelineno-12-20) image_overrides = mm_options.get("image") [](#__codelineno-12-21) [](#__codelineno-12-22) return { [](#__codelineno-12-23) "image": self._get_dummy_images( [](#__codelineno-12-24) width=target_width, [](#__codelineno-12-25) height=target_height, [](#__codelineno-12-26) num_images=num_images, [](#__codelineno-12-27) overrides=image_overrides, [](#__codelineno-12-28) ) [](#__codelineno-12-29) }`` For the text, we simply expand the multimodal image token from the model config to match the desired number of images. `[](#__codelineno-13-1)def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str: [](#__codelineno-13-2) num_images = mm_counts.get("image", 0) [](#__codelineno-13-3) [](#__codelineno-13-4) processor = self.info.get_hf_processor() [](#__codelineno-13-5) image_token = processor.image_token [](#__codelineno-13-6) [](#__codelineno-13-7) return image_token * num_images` Looking at the code of HF's `FuyuForCausalLM`: Code `[](#__codelineno-14-1)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/modeling_fuyu.py#L311-L322 [](#__codelineno-14-2)if image_patches is not None and past_key_values is None: [](#__codelineno-14-3) patch_embeddings = [ [](#__codelineno-14-4) self.vision_embed_tokens(patch.to(self.vision_embed_tokens.weight.dtype)) [](#__codelineno-14-5) .squeeze(0) [](#__codelineno-14-6) .to(inputs_embeds.device) [](#__codelineno-14-7) for patch in image_patches [](#__codelineno-14-8) ] [](#__codelineno-14-9) inputs_embeds = self.gather_continuous_embeddings( [](#__codelineno-14-10) word_embeddings=inputs_embeds, [](#__codelineno-14-11) continuous_embeddings=patch_embeddings, [](#__codelineno-14-12) image_patch_input_indices=image_patches_indices, [](#__codelineno-14-13) )` The number of placeholder feature tokens for the `i`th item in the batch is `patch_embeddings[i].shape[0]`, which is the same as `image_patches[i].shape[0]`, i.e. `num_total_patches`. Unlike LLaVA, Fuyu does not define the number of patches inside the modeling file. Where can we get more information? Considering that the model input comes from the output of `FuyuProcessor`, let's **look at the preprocessing files**. The image outputs are obtained by calling `FuyuImageProcessor.preprocess` and then `FuyuImageProcessor.preprocess_with_tokenizer_info` inside `FuyuProcessor`. In `FuyuImageProcessor.preprocess`, the images are resized and padded to the target `FuyuImageProcessor.size`, returning the dimensions after resizing (but before padding) as metadata. Code `[](#__codelineno-15-1)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L541-L544 [](#__codelineno-15-2)image_encoding = self.image_processor.preprocess(images, **output_kwargs["images_kwargs"]) [](#__codelineno-15-3)batch_images = image_encoding["images"] [](#__codelineno-15-4)image_unpadded_heights = image_encoding["image_unpadded_heights"] [](#__codelineno-15-5)image_unpadded_widths = image_encoding["image_unpadded_widths"] [](#__codelineno-15-6)[](#__codelineno-15-7)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L480-L [](#__codelineno-15-8)if do_resize: [](#__codelineno-15-9) batch_images = [ [](#__codelineno-15-10) [self.resize(image, size=size, input_data_format=input_data_format) for image in images] [](#__codelineno-15-11) for images in batch_images [](#__codelineno-15-12) ] [](#__codelineno-15-13)[](#__codelineno-15-14)image_sizes = [get_image_size(images[0], channel_dim=input_data_format) for images in batch_images] [](#__codelineno-15-15)image_unpadded_heights = [[image_size[0]] for image_size in image_sizes] [](#__codelineno-15-16)image_unpadded_widths = [[image_size[1]] for image_size in image_sizes] [](#__codelineno-15-17)[](#__codelineno-15-18)if do_pad: [](#__codelineno-15-19) batch_images = [ [](#__codelineno-15-20) [ [](#__codelineno-15-21) self.pad_image( [](#__codelineno-15-22) image, [](#__codelineno-15-23) size=size, [](#__codelineno-15-24) mode=padding_mode, [](#__codelineno-15-25) constant_values=padding_value, [](#__codelineno-15-26) input_data_format=input_data_format, [](#__codelineno-15-27) ) [](#__codelineno-15-28) for image in images [](#__codelineno-15-29) ] [](#__codelineno-15-30) for images in batch_images [](#__codelineno-15-31) ]` In `FuyuImageProcessor.preprocess_with_tokenizer_info`, the images are split into patches based on this metadata: Code `[](#__codelineno-16-1)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L425 [](#__codelineno-16-2)model_image_input = self.image_processor.preprocess_with_tokenizer_info( [](#__codelineno-16-3) image_input=tensor_batch_images, [](#__codelineno-16-4) image_present=image_present, [](#__codelineno-16-5) image_unpadded_h=image_unpadded_heights, [](#__codelineno-16-6) image_unpadded_w=image_unpadded_widths, [](#__codelineno-16-7) image_placeholder_id=image_placeholder_id, [](#__codelineno-16-8) image_newline_id=image_newline_id, [](#__codelineno-16-9) variable_sized=True, [](#__codelineno-16-10)) [](#__codelineno-16-11)[](#__codelineno-16-12)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L638-L658 [](#__codelineno-16-13)image_height, image_width = image.shape[1], image.shape[2] [](#__codelineno-16-14)if variable_sized: # variable_sized=True [](#__codelineno-16-15) new_h = min( [](#__codelineno-16-16) image_height, [](#__codelineno-16-17) math.ceil(image_unpadded_h[batch_index, subseq_index] / patch_height) * patch_height, [](#__codelineno-16-18) ) [](#__codelineno-16-19) new_w = min( [](#__codelineno-16-20) image_width, [](#__codelineno-16-21) math.ceil(image_unpadded_w[batch_index, subseq_index] / patch_width) * patch_width, [](#__codelineno-16-22) ) [](#__codelineno-16-23) image = image[:, :new_h, :new_w] [](#__codelineno-16-24) image_height, image_width = new_h, new_w [](#__codelineno-16-25)[](#__codelineno-16-26)num_patches = self.get_num_patches(image_height=image_height, image_width=image_width) [](#__codelineno-16-27)tensor_of_image_ids = torch.full( [](#__codelineno-16-28) [num_patches], image_placeholder_id, dtype=torch.int32, device=image_input.device [](#__codelineno-16-29)) [](#__codelineno-16-30)patches = self.patchify_image(image=image.unsqueeze(0)).squeeze(0) [](#__codelineno-16-31)assert num_patches == patches.shape[0]` The number of patches is in turn defined by `FuyuImageProcessor.get_num_patches`: Code `[](#__codelineno-17-1)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L552-L562 [](#__codelineno-17-2)patch_size = patch_size if patch_size is not None else self.patch_size [](#__codelineno-17-3)patch_height, patch_width = self.patch_size["height"], self.patch_size["width"] [](#__codelineno-17-4)[](#__codelineno-17-5)if image_height % patch_height != 0: [](#__codelineno-17-6) raise ValueError(f"{image_height=} must be divisible by {patch_height}") [](#__codelineno-17-7)if image_width % patch_width != 0: [](#__codelineno-17-8) raise ValueError(f"{image_width=} must be divisible by {patch_width}") [](#__codelineno-17-9)[](#__codelineno-17-10)num_patches_per_dim_h = image_height // patch_height [](#__codelineno-17-11)num_patches_per_dim_w = image_width // patch_width [](#__codelineno-17-12)num_patches = num_patches_per_dim_h * num_patches_per_dim_w` These image patches correspond to placeholder tokens (`|SPEAKER|`). So, we just need to maximize the number of image patches. Since input images are first resized to fit within `image_processor.size`, we can maximize the number of image patches by inputting an image with size equal to `image_processor.size`. `[](#__codelineno-18-1)def get_image_size_with_most_features(self) -> ImageSize: [](#__codelineno-18-2) image_processor = self.get_image_processor() [](#__codelineno-18-3) return ImageSize( [](#__codelineno-18-4) width=image_processor.size["width"], [](#__codelineno-18-5) height=image_processor.size["height"], [](#__codelineno-18-6) )` Fuyu does not expect image placeholders in the inputs to HF processor, so the dummy prompt text is empty regardless of the number of images. `[](#__codelineno-19-1)def get_dummy_text(self, mm_counts: Mapping[str, int]) -> str: [](#__codelineno-19-2) return ""` For the multimodal image profiling data, the logic is very similar to LLaVA: Code `[](#__codelineno-20-1)def get_dummy_mm_data( [](#__codelineno-20-2) self, [](#__codelineno-20-3) seq_len: int, [](#__codelineno-20-4) mm_counts: Mapping[str, int], [](#__codelineno-20-5) mm_options: Mapping[str, BaseDummyOptions], [](#__codelineno-20-6)) -> MultiModalDataDict: [](#__codelineno-20-7) target_width, target_height = \ [](#__codelineno-20-8) self.info.get_image_size_with_most_features() [](#__codelineno-20-9) num_images = mm_counts.get("image", 0) [](#__codelineno-20-10) [](#__codelineno-20-11) image_overrides = mm_options.get("image") [](#__codelineno-20-12) [](#__codelineno-20-13) return { [](#__codelineno-20-14) "image": self._get_dummy_images( [](#__codelineno-20-15) width=target_width, [](#__codelineno-20-16) height=target_height, [](#__codelineno-20-17) num_images=num_images, [](#__codelineno-20-18) overrides=image_overrides, [](#__codelineno-20-19) ) [](#__codelineno-20-20) }` ## 4\. Specify processing details[¶](#4-specify-processing-details "Permanent link") Afterwards, create a subclass of [BaseMultiModalProcessor](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor " BaseMultiModalProcessor") to fill in the missing details about HF processing. ### Multi-modal fields[¶](#multi-modal-fields "Permanent link") Override [\_get\_mm\_fields\_config](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config " _get_mm_fields_config(hf_inputs, hf_processor_mm_kwargs) abstractmethod ") to return a schema of the tensors outputted by the HF processor that are related to the input multi-modal items. Basic example: LLaVAWith postprocessing: Fuyu The output of `CLIPImageProcessor` is a simple tensor with shape `(num_images, num_channels, image_height, image_width)`: `[](#__codelineno-21-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/clip/image_processing_clip.py#L339-L345 [](#__codelineno-21-2)images = [ [](#__codelineno-21-3) to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format) [](#__codelineno-21-4) for image in all_images [](#__codelineno-21-5)] [](#__codelineno-21-6)[](#__codelineno-21-7)data = {"pixel_values": images} [](#__codelineno-21-8)return BatchFeature(data=data, tensor_type=return_tensors)` So, we override [\_get\_mm\_fields\_config](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config " _get_mm_fields_config(hf_inputs, hf_processor_mm_kwargs) abstractmethod ") as follows: `[](#__codelineno-22-1)def _get_mm_fields_config( [](#__codelineno-22-2) self, [](#__codelineno-22-3) hf_inputs: BatchFeature, [](#__codelineno-22-4) hf_processor_mm_kwargs: Mapping[str, object], [](#__codelineno-22-5)) -> Mapping[str, MultiModalFieldConfig]: [](#__codelineno-22-6) return dict( [](#__codelineno-22-7) pixel_values=MultiModalFieldConfig.batched("image"), [](#__codelineno-22-8) )` Note Our [actual code](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/llava.py) additionally supports pre-computed image embeddings, which can be passed to be model via the `image_embeds` argument. The `image_patches` output of `FuyuImageProcessor.preprocess_with_tokenizer_info` concatenates the patches from each image belonging to an item in the batch: `[](#__codelineno-23-1)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/image_processing_fuyu.py#L673-L679 [](#__codelineno-23-2) image_input_ids.append(tensor_of_image_ids) [](#__codelineno-23-3) image_patches.append(patches) [](#__codelineno-23-4) else: [](#__codelineno-23-5) image_input_ids.append(torch.tensor([], dtype=torch.int32, device=image_input.device)) [](#__codelineno-23-6)[](#__codelineno-23-7)batch_image_input_ids.append(image_input_ids) [](#__codelineno-23-8)batch_image_patches.append(image_patches)` The shape of `image_patches` outputted by `FuyuImageProcessor` is therefore `(1, num_images, num_patches, patch_width * patch_height * num_channels)`. In order to support the use of [MultiModalFieldConfig.batched](https://docs.vllm.ai/en/api/vllm/multimodal/inputs/#vllm.multimodal.inputs.MultiModalFieldConfig.batched " batched(modality, *, keep_on_cpu=False) staticmethod ") like in LLaVA, we remove the extra batch dimension by overriding [BaseMultiModalProcessor.\_call\_hf\_processor](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor " _call_hf_processor(prompt, mm_data, mm_kwargs, tok_kwargs)"): Code `[](#__codelineno-24-1)def _call_hf_processor( [](#__codelineno-24-2) self, [](#__codelineno-24-3) prompt: str, [](#__codelineno-24-4) mm_data: Mapping[str, object], [](#__codelineno-24-5) mm_kwargs: Mapping[str, object], [](#__codelineno-24-6) tok_kwargs: Mapping[str, object], [](#__codelineno-24-7)) -> BatchFeature: [](#__codelineno-24-8) processed_outputs = super()._call_hf_processor( [](#__codelineno-24-9) prompt=prompt, [](#__codelineno-24-10) mm_data=mm_data, [](#__codelineno-24-11) mm_kwargs=mm_kwargs, [](#__codelineno-24-12) tok_kwargs=tok_kwargs, [](#__codelineno-24-13) ) [](#__codelineno-24-14) [](#__codelineno-24-15) image_patches = processed_outputs.get("image_patches") [](#__codelineno-24-16) if image_patches is not None: [](#__codelineno-24-17) images = mm_data["images"] [](#__codelineno-24-18) assert isinstance(images, list) [](#__codelineno-24-19) [](#__codelineno-24-20) # Original output: (1, num_images, Pn, Px * Py * C) [](#__codelineno-24-21) # New output: (num_images, Pn, Px * Py * C) [](#__codelineno-24-22) assert (isinstance(image_patches, list) [](#__codelineno-24-23) and len(image_patches) == 1) [](#__codelineno-24-24) assert (isinstance(image_patches[0], torch.Tensor) [](#__codelineno-24-25) and len(image_patches[0]) == len(images)) [](#__codelineno-24-26) [](#__codelineno-24-27) processed_outputs["image_patches"] = image_patches[0] [](#__codelineno-24-28) [](#__codelineno-24-29) return processed_outputs` Note Our [actual code](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/fuyu.py) has special handling for text-only inputs to prevent unnecessary warnings from HF processor. Note The `_call_hf_processor` method specifies both `mm_kwargs` and `tok_kwargs` for processing. `mm_kwargs` is used to both initialize and call the huggingface processor, whereas `tok_kwargs` is only used to call the huggingface processor. This lets us override [\_get\_mm\_fields\_config](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_mm_fields_config " _get_mm_fields_config(hf_inputs, hf_processor_mm_kwargs) abstractmethod ") as follows: `[](#__codelineno-25-1)def _get_mm_fields_config( [](#__codelineno-25-2) self, [](#__codelineno-25-3) hf_inputs: BatchFeature, [](#__codelineno-25-4) hf_processor_mm_kwargs: Mapping[str, object], [](#__codelineno-25-5)) -> Mapping[str, MultiModalFieldConfig]: [](#__codelineno-25-6) return dict(image_patches=MultiModalFieldConfig.batched("image"))` ### Prompt updates[¶](#prompt-updates "Permanent link") Override [\_get\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates " _get_prompt_updates(mm_items, hf_processor_mm_kwargs, out_mm_kwargs) abstractmethod ") to return a list of [PromptUpdate](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.PromptUpdate " PromptUpdate dataclass ") instances. Each [PromptUpdate](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.PromptUpdate " PromptUpdate dataclass ") instance specifies an update operation (e.g.: insertion, replacement) performed by the HF processor. Basic example: LLaVAHandling additional tokens: Fuyu Looking at HF's `LlavaProcessor`: `[](#__codelineno-26-1)# https://github.com/huggingface/transformers/blob/v4.47.1/src/transformers/models/llava/processing_llava.py#L167-L170 [](#__codelineno-26-2)prompt_strings = [] [](#__codelineno-26-3)for sample in text: [](#__codelineno-26-4) sample = sample.replace(self.image_token, self.image_token * num_image_tokens) [](#__codelineno-26-5) prompt_strings.append(sample)` It simply repeats each input `image_token` a number of times equal to the number of placeholder feature tokens (`num_image_tokens`). Based on this, we override [\_get\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates " _get_prompt_updates(mm_items, hf_processor_mm_kwargs, out_mm_kwargs) abstractmethod ") as follows: Code `[](#__codelineno-27-1)def _get_prompt_updates( [](#__codelineno-27-2) self, [](#__codelineno-27-3) mm_items: MultiModalDataItems, [](#__codelineno-27-4) hf_processor_mm_kwargs: Mapping[str, object], [](#__codelineno-27-5) out_mm_kwargs: MultiModalKwargsItems, [](#__codelineno-27-6)) -> Sequence[PromptUpdate]: [](#__codelineno-27-7) hf_config = self.info.get_hf_config() [](#__codelineno-27-8) image_token_id = hf_config.image_token_index [](#__codelineno-27-9) [](#__codelineno-27-10) def get_replacement(item_idx: int): [](#__codelineno-27-11) images = mm_items.get_items("image", ImageProcessorItems) [](#__codelineno-27-12) [](#__codelineno-27-13) image_size = images.get_image_size(item_idx) [](#__codelineno-27-14) num_image_tokens = self.info.get_num_image_tokens( [](#__codelineno-27-15) image_width=image_size.width, [](#__codelineno-27-16) image_height=image_size.height, [](#__codelineno-27-17) ) [](#__codelineno-27-18) [](#__codelineno-27-19) return [image_token_id] * num_image_tokens [](#__codelineno-27-20) [](#__codelineno-27-21) return [ [](#__codelineno-27-22) PromptReplacement( [](#__codelineno-27-23) modality="image", [](#__codelineno-27-24) target=[image_token_id], [](#__codelineno-27-25) replacement=get_replacement, [](#__codelineno-27-26) ), [](#__codelineno-27-27) ]` Recall the layout of feature tokens from Step 2: `[](#__codelineno-28-1)|SPEAKER||SPEAKER|...|SPEAKER||NEWLINE| [](#__codelineno-28-2)|SPEAKER||SPEAKER|...|SPEAKER||NEWLINE| [](#__codelineno-28-3)... [](#__codelineno-28-4)|SPEAKER||SPEAKER|...|SPEAKER||NEWLINE|` We define a helper function to return `ncols` and `nrows` directly: Code `[](#__codelineno-29-1)def get_image_feature_grid_size( [](#__codelineno-29-2) self, [](#__codelineno-29-3) *, [](#__codelineno-29-4) image_width: int, [](#__codelineno-29-5) image_height: int, [](#__codelineno-29-6)) -> tuple[int, int]: [](#__codelineno-29-7) image_processor = self.get_image_processor() [](#__codelineno-29-8) target_width = image_processor.size["width"] [](#__codelineno-29-9) target_height = image_processor.size["height"] [](#__codelineno-29-10) patch_width = image_processor.patch_size["width"] [](#__codelineno-29-11) patch_height = image_processor.patch_size["height"] [](#__codelineno-29-12) [](#__codelineno-29-13) if not (image_width <= target_width and image_height <= target_height): [](#__codelineno-29-14) height_scale_factor = target_height / image_height [](#__codelineno-29-15) width_scale_factor = target_width / image_width [](#__codelineno-29-16) optimal_scale_factor = min(height_scale_factor, width_scale_factor) [](#__codelineno-29-17) [](#__codelineno-29-18) image_height = int(image_height * optimal_scale_factor) [](#__codelineno-29-19) image_width = int(image_width * optimal_scale_factor) [](#__codelineno-29-20) [](#__codelineno-29-21) ncols = math.ceil(image_width / patch_width) [](#__codelineno-29-22) nrows = math.ceil(image_height / patch_height) [](#__codelineno-29-23) return ncols, nrows` Based on this, we can initially define our replacement tokens as: Code ``[](#__codelineno-30-1)def get_replacement(item_idx: int): [](#__codelineno-30-2) images = mm_items.get_items("image", ImageProcessorItems) [](#__codelineno-30-3) image_size = images.get_image_size(item_idx) [](#__codelineno-30-4) [](#__codelineno-30-5) ncols, nrows = self.info.get_image_feature_grid_size( [](#__codelineno-30-6) image_width=image_size.width, [](#__codelineno-30-7) image_height=image_size.height, [](#__codelineno-30-8) ) [](#__codelineno-30-9) [](#__codelineno-30-10) # `_IMAGE_TOKEN_ID` corresponds to `|SPEAKER|` [](#__codelineno-30-11) # `_NEWLINE_TOKEN_ID` corresponds to `|NEWLINE|` [](#__codelineno-30-12) return ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows`` However, this is not entirely correct. After `FuyuImageProcessor.preprocess_with_tokenizer_info` is called, a BOS token (``) is also added to the prompt: Code `[](#__codelineno-31-1)# https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/fuyu/processing_fuyu.py#L417-L435 [](#__codelineno-31-2)model_image_input = self.image_processor.preprocess_with_tokenizer_info( [](#__codelineno-31-3) image_input=tensor_batch_images, [](#__codelineno-31-4) image_present=image_present, [](#__codelineno-31-5) image_unpadded_h=image_unpadded_heights, [](#__codelineno-31-6) image_unpadded_w=image_unpadded_widths, [](#__codelineno-31-7) image_placeholder_id=image_placeholder_id, [](#__codelineno-31-8) image_newline_id=image_newline_id, [](#__codelineno-31-9) variable_sized=True, [](#__codelineno-31-10)) [](#__codelineno-31-11)prompt_tokens, prompts_length = _tokenize_prompts_with_image_and_batch( [](#__codelineno-31-12) tokenizer=self.tokenizer, [](#__codelineno-31-13) prompts=prompts, [](#__codelineno-31-14) scale_factors=scale_factors, [](#__codelineno-31-15) max_tokens_to_generate=self.max_tokens_to_generate, [](#__codelineno-31-16) max_position_embeddings=self.max_position_embeddings, [](#__codelineno-31-17) add_BOS=True, [](#__codelineno-31-18) add_beginning_of_answer_token=True, [](#__codelineno-31-19))` To assign the vision embeddings to only the image tokens, instead of a string you can return an instance of [PromptUpdateDetails](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.PromptUpdateDetails " PromptUpdateDetails dataclass "): Code ``[](#__codelineno-32-1)hf_config = self.info.get_hf_config() [](#__codelineno-32-2)bos_token_id = hf_config.bos_token_id # `` [](#__codelineno-32-3)assert isinstance(bos_token_id, int) [](#__codelineno-32-4)[](#__codelineno-32-5)def get_replacement_fuyu(item_idx: int): [](#__codelineno-32-6) images = mm_items.get_items("image", ImageProcessorItems) [](#__codelineno-32-7) image_size = images.get_image_size(item_idx) [](#__codelineno-32-8) [](#__codelineno-32-9) ncols, nrows = self.info.get_image_feature_grid_size( [](#__codelineno-32-10) image_width=image_size.width, [](#__codelineno-32-11) image_height=image_size.height, [](#__codelineno-32-12) ) [](#__codelineno-32-13) image_tokens = ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows [](#__codelineno-32-14) [](#__codelineno-32-15) return PromptUpdateDetails.select_token_id( [](#__codelineno-32-16) image_tokens + [bos_token_id], [](#__codelineno-32-17) embed_token_id=_IMAGE_TOKEN_ID, [](#__codelineno-32-18) )`` Finally, noticing that the HF processor removes the `|ENDOFTEXT|` token from the tokenized prompt, we can search for it to conduct the replacement at the start of the string: Code `[](#__codelineno-33-1)def _get_prompt_updates( [](#__codelineno-33-2) self, [](#__codelineno-33-3) mm_items: MultiModalDataItems, [](#__codelineno-33-4) hf_processor_mm_kwargs: Mapping[str, object], [](#__codelineno-33-5) out_mm_kwargs: MultiModalKwargsItems, [](#__codelineno-33-6)) -> Sequence[PromptUpdate]: [](#__codelineno-33-7) hf_config = self.info.get_hf_config() [](#__codelineno-33-8) bos_token_id = hf_config.bos_token_id [](#__codelineno-33-9) assert isinstance(bos_token_id, int) [](#__codelineno-33-10) [](#__codelineno-33-11) tokenizer = self.info.get_tokenizer() [](#__codelineno-33-12) eot_token_id = tokenizer.bos_token_id [](#__codelineno-33-13) assert isinstance(eot_token_id, int) [](#__codelineno-33-14) [](#__codelineno-33-15) def get_replacement_fuyu(item_idx: int): [](#__codelineno-33-16) images = mm_items.get_items("image", ImageProcessorItems) [](#__codelineno-33-17) image_size = images.get_image_size(item_idx) [](#__codelineno-33-18) [](#__codelineno-33-19) ncols, nrows = self.info.get_image_feature_grid_size( [](#__codelineno-33-20) image_width=image_size.width, [](#__codelineno-33-21) image_height=image_size.height, [](#__codelineno-33-22) ) [](#__codelineno-33-23) image_tokens = ([_IMAGE_TOKEN_ID] * ncols + [_NEWLINE_TOKEN_ID]) * nrows [](#__codelineno-33-24) [](#__codelineno-33-25) return PromptUpdateDetails.select_token_id( [](#__codelineno-33-26) image_tokens + [bos_token_id], [](#__codelineno-33-27) embed_token_id=_IMAGE_TOKEN_ID, [](#__codelineno-33-28) ) [](#__codelineno-33-29) [](#__codelineno-33-30) return [ [](#__codelineno-33-31) PromptReplacement( [](#__codelineno-33-32) modality="image", [](#__codelineno-33-33) target=[eot_token_id], [](#__codelineno-33-34) replacement=get_replacement_fuyu, [](#__codelineno-33-35) ) [](#__codelineno-33-36) ]` After you have defined [BaseProcessingInfo](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseProcessingInfo " BaseProcessingInfo") (Step 2), [BaseDummyInputsBuilder](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseDummyInputsBuilder " BaseDummyInputsBuilder") (Step 3), and [BaseMultiModalProcessor](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor " BaseMultiModalProcessor") (Step 4), decorate the model class with [MULTIMODAL\_REGISTRY.register\_processor](https://docs.vllm.ai/en/api/vllm/multimodal/registry/#vllm.multimodal.registry.MultiModalRegistry.register_processor " register_processor(processor, *, info, dummy_inputs)") to register them to the multi-modal registry: `[](#__codelineno-34-1) from vllm.model_executor.models.interfaces import SupportsMultiModal [](#__codelineno-34-2)+ from vllm.multimodal import MULTIMODAL_REGISTRY [](#__codelineno-34-3)[](#__codelineno-34-4)+ @MULTIMODAL_REGISTRY.register_processor( [](#__codelineno-34-5)+ YourMultiModalProcessor, [](#__codelineno-34-6)+ info=YourProcessingInfo, [](#__codelineno-34-7)+ dummy_inputs=YourDummyInputsBuilder, [](#__codelineno-34-8)+ ) [](#__codelineno-34-9) class YourModelForImage2Seq(nn.Module, SupportsMultiModal):` ## Notes[¶](#notes "Permanent link") ### Inserting feature tokens without replacement[¶](#inserting-feature-tokens-without-replacement "Permanent link") Some HF processors directly insert feature tokens without replacing anything in the original prompt. In that case, you can use [PromptInsertion](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.PromptInsertion " PromptInsertion dataclass ") instead of [PromptReplacement](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.PromptReplacement " PromptReplacement dataclass ") inside [\_get\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates " _get_prompt_updates(mm_items, hf_processor_mm_kwargs, out_mm_kwargs) abstractmethod "). Examples: - BLIP-2 (insert at start of prompt): [vllm/model\_executor/models/blip2.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/blip2.py) - Molmo (insert after `<|endoftext|>` token): [vllm/model\_executor/models/molmo.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/molmo.py) ### Handling prompt updates unrelated to multi-modal data[¶](#handling-prompt-updates-unrelated-to-multi-modal-data "Permanent link") [\_get\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates " _get_prompt_updates(mm_items, hf_processor_mm_kwargs, out_mm_kwargs) abstractmethod ") assumes that each application of prompt update corresponds to one multi-modal item. If the HF processor performs additional processing regardless of how many multi-modal items there are, you should override [\_apply\_hf\_processor\_tokens\_only](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_tokens_only " _apply_hf_processor_tokens_only(prompt_tokens)") so that the processed token inputs are consistent with the result of applying the HF processor on text inputs. This is because token inputs bypass the HF processor according to [our design](https://docs.vllm.ai/en/design/mm_processing/). Examples: - Chameleon (appends `sep_token`): [vllm/model\_executor/models/chameleon.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/chameleon.py) - Fuyu (appends `boa_token`): [vllm/model\_executor/models/fuyu.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/fuyu.py) - Molmo (applies chat template which is not defined elsewhere): [vllm/model\_executor/models/molmo.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/molmo.py) ### Custom HF processor[¶](#custom-hf-processor "Permanent link") Some models don't define an HF processor class on HF Hub. In that case, you can define a custom HF processor that has the same call signature as HF processors and pass it to [\_call\_hf\_processor](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._call_hf_processor " _call_hf_processor(prompt, mm_data, mm_kwargs, tok_kwargs)"). Examples: - DeepSeek-VL2: [vllm/model\_executor/models/deepseek\_vl2.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/deepseek_vl2.py) - InternVL: [vllm/model\_executor/models/internvl.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/internvl.py) - Qwen-VL: [vllm/model\_executor/models/qwen\_vl.py](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/models/qwen_vl.py) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/k8s.md "Edit this page") Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine learning models. This guide walks you through deploying vLLM using native Kubernetes. - [Deployment with CPUs](#deployment-with-cpus) - [Deployment with GPUs](#deployment-with-gpus) - [Serving with gRPC](#serving-with-grpc) - [Troubleshooting](#troubleshooting) - [Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"](#startup-probe-or-readiness-probe-failure-container-log-contains-keyboardinterrupt-terminated) - [Conclusion](#conclusion) Alternatively, you can deploy vLLM to Kubernetes using any of the following: - [Helm](https://docs.vllm.ai/en/latest/frameworks/helm/) - [NVIDIA Dynamo](https://docs.vllm.ai/en/latest/integrations/dynamo/) - [InftyAI/llmaz](https://docs.vllm.ai/en/latest/integrations/llmaz/) - [llm-d](https://docs.vllm.ai/en/latest/integrations/llm-d/) - [KAITO](https://docs.vllm.ai/en/latest/integrations/kaito/) - [KServe](https://docs.vllm.ai/en/latest/integrations/kserve/) - [Kthena](https://docs.vllm.ai/en/latest/integrations/kthena/) - [KubeRay](https://docs.vllm.ai/en/latest/integrations/kuberay/) - [kubernetes-sigs/lws](https://docs.vllm.ai/en/latest/frameworks/lws/) - [meta-llama/llama-stack](https://docs.vllm.ai/en/latest/integrations/llamastack/) - [substratusai/kubeai](https://docs.vllm.ai/en/latest/integrations/kubeai/) - [vllm-project/AIBrix](https://docs.vllm.ai/en/latest/integrations/aibrix/) - [vllm-project/production-stack](https://docs.vllm.ai/en/latest/integrations/production-stack/) ## Deployment with CPUs[¶](#deployment-with-cpus "Permanent link") Note The use of CPUs here is for demonstration and testing purposes only and its performance will not be on par with GPUs. First, create a Kubernetes PVC and Secret for downloading and storing Hugging Face model: Config `[](#__codelineno-0-1)cat <`: `[](#__codelineno-8-1)kubectl apply -f deployment.yaml [](#__codelineno-8-2)kubectl apply -f service.yaml` To test the deployment, run the following `curl` command: `[](#__codelineno-9-1)curl http://mistral-7b.default.svc.cluster.local/v1/completions \ [](#__codelineno-9-2) -H "Content-Type: application/json" \ [](#__codelineno-9-3) -d '{ [](#__codelineno-9-4) "model": "mistralai/Mistral-7B-Instruct-v0.3", [](#__codelineno-9-5) "prompt": "San Francisco is a", [](#__codelineno-9-6) "max_tokens": 7, [](#__codelineno-9-7) "temperature": 0 [](#__codelineno-9-8) }'` If the service is correctly deployed, you should receive a response from the vLLM model. ## Serving with gRPC[¶](#serving-with-grpc "Permanent link") vLLM can serve models over gRPC instead of HTTP by passing the `--grpc` flag. This requires the optional gRPC dependencies: `[](#__codelineno-10-1)pip install vllm[grpc]` When using `--grpc`, the server exposes the standard [gRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md) (`grpc.health.v1.Health`), which integrates with Kubernetes [native gRPC probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) (available since Kubernetes 1.24). To deploy with gRPC, change the `vllm serve` command to include `--grpc` and replace `httpGet` probes with `grpc` probes: `[](#__codelineno-11-1)containers: [](#__codelineno-11-2)- name: mistral-7b [](#__codelineno-11-3) image: vllm/vllm-openai:latest [](#__codelineno-11-4) command: ["/bin/sh", "-c"] [](#__codelineno-11-5) args: [ [](#__codelineno-11-6) "pip install vllm[grpc] && vllm serve mistralai/Mistral-7B-Instruct-v0.3 --grpc --port 50051 --trust-remote-code" [](#__codelineno-11-7) ] [](#__codelineno-11-8) ports: [](#__codelineno-11-9) - containerPort: 50051 [](#__codelineno-11-10) livenessProbe: [](#__codelineno-11-11) grpc: [](#__codelineno-11-12) port: 50051 [](#__codelineno-11-13) initialDelaySeconds: 120 [](#__codelineno-11-14) periodSeconds: 10 [](#__codelineno-11-15) readinessProbe: [](#__codelineno-11-16) grpc: [](#__codelineno-11-17) port: 50051 [](#__codelineno-11-18) initialDelaySeconds: 120 [](#__codelineno-11-19) periodSeconds: 5` Note The gRPC health service checks the engine status on every probe. If the engine is unhealthy or the server is shutting down, the probe returns `NOT_SERVING`. You can also verify the health service manually with `grpcurl`: `[](#__codelineno-12-1)grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check` ## Troubleshooting[¶](#troubleshooting "Permanent link") ### Startup Probe or Readiness Probe Failure, container log contains "KeyboardInterrupt: terminated"[¶](#startup-probe-or-readiness-probe-failure-container-log-contains-keyboardinterrupt-terminated "Permanent link") If the startup or readiness probe failureThreshold is too low for the time needed to start up the server, Kubernetes scheduler will kill the container. A couple of indications that this has happened: 1. container log contains "KeyboardInterrupt: terminated" 2. `kubectl get events` shows message `Container $NAME failed startup probe, will be restarted` To mitigate, increase the failureThreshold to allow more time for the model server to start serving. You can identify an ideal failureThreshold by removing the probes from the manifest and measuring how much time it takes for the model server to show it's ready to serve. ## Conclusion[¶](#conclusion "Permanent link") Deploying vLLM with Kubernetes allows for efficient scaling and management of ML models leveraging GPU resources. By following the steps outlined above, you should be able to set up and test a vLLM deployment within your Kubernetes cluster. If you encounter any issues or have suggestions, please feel free to contribute to the documentation. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/anyscale.md "Edit this page") [Anyscale](https://www.anyscale.com/) is a managed, multi-cloud platform developed by the creators of Ray. Anyscale automates the entire lifecycle of Ray clusters in your AWS, GCP, or Azure account, delivering the flexibility of open-source Ray without the operational overhead of maintaining Kubernetes control planes, configuring autoscalers, managing observability stacks, or manually managing head and worker nodes with helper scripts like [examples/ray\_serving/run\_cluster.sh](https://github.com/vllm-project/vllm/blob/main/examples/ray_serving/run_cluster.sh). When serving large language models with vLLM, Anyscale can rapidly provision [production-ready HTTPS endpoints](https://docs.anyscale.com/examples/deploy-ray-serve-llms) or [fault-tolerant batch inference jobs](https://docs.anyscale.com/examples/ray-data-llm). ## Production-ready vLLM on Anyscale quickstarts[¶](#production-ready-vllm-on-anyscale-quickstarts "Permanent link") - [Offline batch inference](https://console.anyscale.com/template-preview/llm_batch_inference?utm_source=vllm_docs) - [Deploy vLLM services](https://console.anyscale.com/template-preview/llm_serving?utm_source=vllm_docs) - [Curate a dataset](https://console.anyscale.com/template-preview/audio-dataset-curation-llm-judge?utm_source=vllm_docs) - [Finetune an LLM](https://console.anyscale.com/template-preview/entity-recognition-with-llms?utm_source=vllm_docs) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/contributing/model/tests.md "Edit this page") This page explains how to write unit tests to verify the implementation of your model. ## Required Tests[¶](#required-tests "Permanent link") These tests are necessary to get your PR merged into vLLM library. Without them, the CI for your PR will fail. ### Model loading[¶](#model-loading "Permanent link") Include an example HuggingFace repository for your model in [tests/models/registry.py](https://github.com/vllm-project/vllm/blob/main/tests/models/registry.py). This enables a unit test that loads dummy weights to ensure that the model can be initialized in vLLM. Important The list of models in each section should be maintained in alphabetical order. Tip If your model requires a development version of HF Transformers, you can set `min_transformers_version` to skip the test in CI until the model is released. ## Optional Tests[¶](#optional-tests "Permanent link") These tests are optional to get your PR merged into vLLM library. Passing these tests provides more confidence that your implementation is correct, and helps avoid future regressions. ### Model correctness[¶](#model-correctness "Permanent link") These tests compare the model outputs of vLLM against [HF Transformers](https://github.com/huggingface/transformers). You can add new tests under the subdirectories of [tests/models](https://github.com/vllm-project/vllm/tree/main/tests/models). #### Generative models[¶](#generative-models "Permanent link") For [generative models](https://docs.vllm.ai/en/models/generative_models/), there are two levels of correctness tests, as defined in [tests/models/utils.py](https://github.com/vllm-project/vllm/blob/main/tests/models/utils.py): - Exact correctness (`check_outputs_equal`): The text outputted by vLLM should exactly match the text outputted by HF. - Logprobs similarity (`check_logprobs_close`): The logprobs outputted by vLLM should be in the top-k logprobs outputted by HF, and vice versa. #### Pooling models[¶](#pooling-models "Permanent link") For [pooling models](https://docs.vllm.ai/en/models/pooling_models/), we simply check the cosine similarity, as defined in [tests/models/utils.py](https://github.com/vllm-project/vllm/blob/main/tests/models/utils.py). ### Multi-modal processing[¶](#multi-modal-processing "Permanent link") #### Common tests[¶](#common-tests "Permanent link") Adding your model to [tests/models/multimodal/processing/test\_common.py](https://github.com/vllm-project/vllm/blob/main/tests/models/multimodal/processing/test_common.py) verifies that the following input combinations result in the same outputs: - Text + multi-modal data - Tokens + multi-modal data - Text + cached multi-modal data - Tokens + cached multi-modal data #### Model-specific tests[¶](#model-specific-tests "Permanent link") You can add a new file under [tests/models/multimodal/processing](https://github.com/vllm-project/vllm/tree/main/tests/models/multimodal/processing) to run tests that only apply to your model. For example, if the HF processor for your model accepts user-specified keyword arguments, you can verify that the keyword arguments are being applied correctly, such as in [tests/models/multimodal/processing/test\_phi3v.py](https://github.com/vllm-project/vllm/blob/main/tests/models/multimodal/processing/test_phi3v.py). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/anything-llm.md "Edit this page") [AnythingLLM](https://github.com/Mintplex-Labs/anything-llm) is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting. It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM environment: `[](#__codelineno-0-1)pip install vllm` ## Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with a supported chat-completion model, for example: `[](#__codelineno-1-1)vllm serve Qwen/Qwen1.5-32B-Chat-AWQ --max-model-len 4096` 2. Download and install [AnythingLLM Desktop](https://anythingllm.com/desktop). 3. Configure the AI provider: - At the bottom, click the 🔧 wrench icon -> **Open settings** -> **AI Providers** -> **LLM**. - Enter the following values: - LLM Provider: Generic OpenAI - Base URL: `http://{vllm server host}:{vllm server port}/v1` - Chat Model Name: `Qwen/Qwen1.5-32B-Chat-AWQ` [![set AI providers](https://docs.vllm.ai/en/assets/deployment/anything-llm-provider.png)](https://docs.vllm.ai/en/assets/deployment/anything-llm-provider.png) 4. Create a workspace: 1. At the bottom, click the ↺ back icon and back to workspaces. 2. Create a workspace (e.g., `vllm`) and start chatting. [![create a workspace](https://docs.vllm.ai/en/assets/deployment/anything-llm-chat-without-doc.png)](https://docs.vllm.ai/en/assets/deployment/anything-llm-chat-without-doc.png) 5. Add a document. 1. Click the 📎 attachment icon. 2. Upload a document. 3. Select and move the document into your workspace. 4. Save and embed it. [![add a document](https://docs.vllm.ai/en/assets/deployment/anything-llm-upload-doc.png)](https://docs.vllm.ai/en/assets/deployment/anything-llm-upload-doc.png) 6. Chat using your document as context. [![chat with your context](https://docs.vllm.ai/en/assets/deployment/anything-llm-chat-with-doc.png)](https://docs.vllm.ai/en/assets/deployment/anything-llm-chat-with-doc.png) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/autogen.md "Edit this page") [AutoGen](https://github.com/microsoft/autogen) is a framework for creating multi-agent AI applications that can act autonomously or work alongside humans. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM and [AutoGen](https://microsoft.github.io/autogen/0.2/docs/installation/) environment: `[](#__codelineno-0-1)pip install vllm [](#__codelineno-0-2)[](#__codelineno-0-3)# Install AgentChat and OpenAI client from Extensions [](#__codelineno-0-4)# AutoGen requires Python 3.10 or later. [](#__codelineno-0-5)pip install -U "autogen-agentchat" "autogen-ext[openai]"` ## Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-1-1)vllm serve mistralai/Mistral-7B-Instruct-v0.2` 2. Call it with AutoGen: Code `[](#__codelineno-2-1)import asyncio [](#__codelineno-2-2)from autogen_core.models import UserMessage [](#__codelineno-2-3)from autogen_ext.models.openai import OpenAIChatCompletionClient [](#__codelineno-2-4)from autogen_core.models import ModelFamily [](#__codelineno-2-5) [](#__codelineno-2-6)[](#__codelineno-2-7)async def main() -> None: [](#__codelineno-2-8) # Create a model client [](#__codelineno-2-9) model_client = OpenAIChatCompletionClient( [](#__codelineno-2-10) model="mistralai/Mistral-7B-Instruct-v0.2", [](#__codelineno-2-11) base_url="http://{your-vllm-host-ip}:{your-vllm-host-port}/v1", [](#__codelineno-2-12) api_key="EMPTY", [](#__codelineno-2-13) model_info={ [](#__codelineno-2-14) "vision": False, [](#__codelineno-2-15) "function_calling": False, [](#__codelineno-2-16) "json_output": False, [](#__codelineno-2-17) "family": ModelFamily.MISTRAL, [](#__codelineno-2-18) "structured_output": True, [](#__codelineno-2-19) }, [](#__codelineno-2-20) ) [](#__codelineno-2-21) [](#__codelineno-2-22) messages = [UserMessage(content="Write a very short story about a dragon.", source="user")] [](#__codelineno-2-23) [](#__codelineno-2-24) # Create a stream. [](#__codelineno-2-25) stream = model_client.create_stream(messages=messages) [](#__codelineno-2-26) [](#__codelineno-2-27) # Iterate over the stream and print the responses. [](#__codelineno-2-28) print("Streamed responses:") [](#__codelineno-2-29) async for response in stream: [](#__codelineno-2-30) if isinstance(response, str): [](#__codelineno-2-31) # A partial response is a string. [](#__codelineno-2-32) print(response, flush=True, end="") [](#__codelineno-2-33) else: [](#__codelineno-2-34) # The last response is a CreateResult object with the complete message. [](#__codelineno-2-35) print("\n\n------------\n") [](#__codelineno-2-36) print("The complete response:", flush=True) [](#__codelineno-2-37) print(response.content, flush=True) [](#__codelineno-2-38) [](#__codelineno-2-39) # Close the client when done. [](#__codelineno-2-40) await model_client.close() [](#__codelineno-2-41) [](#__codelineno-2-42)[](#__codelineno-2-43)asyncio.run(main())` For details, see the tutorial: - [Using vLLM in AutoGen](https://microsoft.github.io/autogen/0.2/docs/topics/non-openai-models/local-vllm/) - [OpenAI-compatible API examples](https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.models.openai.html#autogen_ext.models.openai.OpenAIChatCompletionClient) --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Deployment](https://docs.vllm.ai/en/latest/docker/) 4. [Frameworks](https://docs.vllm.ai/en/latest/deployment/anyscale/) [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/bentoml.md "Edit this page") [BentoML](https://github.com/bentoml/BentoML) allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. You can serve the model locally or containerize it as an OCI-compliant image and deploy it on Kubernetes. For details, see the tutorial [vLLM inference in the BentoML documentation](https://docs.bentoml.com/en/latest/use-cases/large-language-models/vllm.html). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/cerebrium.md "Edit this page") [![vLLM_plus_cerebrium](https://i.ibb.co/hHcScTT/Screenshot-2024-06-13-at-10-14-54.png)](https://i.ibb.co/hHcScTT/Screenshot-2024-06-13-at-10-14-54.png) vLLM can be run on a cloud based GPU machine with [Cerebrium](https://www.cerebrium.ai/), a serverless AI infrastructure platform that makes it easier for companies to build and deploy AI based applications. To install the Cerebrium client, run: `[](#__codelineno-0-1)pip install cerebrium [](#__codelineno-0-2)cerebrium login` Next, create your Cerebrium project, run: `[](#__codelineno-1-1)cerebrium init vllm-project` Next, to install the required packages, add the following to your cerebrium.toml: `[](#__codelineno-2-1)[cerebrium.deployment] [](#__codelineno-2-2)docker_base_image_url = "nvidia/cuda:12.1.1-runtime-ubuntu22.04" [](#__codelineno-2-3)[](#__codelineno-2-4)[cerebrium.dependencies.pip] [](#__codelineno-2-5)vllm = "latest"` Next, let us add our code to handle inference for the LLM of your choice (`mistralai/Mistral-7B-Instruct-v0.1` for this example), add the following code to your `main.py`: Code `[](#__codelineno-3-1)from vllm import LLM, SamplingParams [](#__codelineno-3-2)[](#__codelineno-3-3)llm = LLM(model="mistralai/Mistral-7B-Instruct-v0.1") [](#__codelineno-3-4)[](#__codelineno-3-5)def run(prompts: list[str], temperature: float = 0.8, top_p: float = 0.95): [](#__codelineno-3-6) [](#__codelineno-3-7) sampling_params = SamplingParams(temperature=temperature, top_p=top_p) [](#__codelineno-3-8) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-3-9) [](#__codelineno-3-10) # Print the outputs. [](#__codelineno-3-11) results = [] [](#__codelineno-3-12) for output in outputs: [](#__codelineno-3-13) prompt = output.prompt [](#__codelineno-3-14) generated_text = output.outputs[0].text [](#__codelineno-3-15) results.append({"prompt": prompt, "generated_text": generated_text}) [](#__codelineno-3-16) [](#__codelineno-3-17) return {"results": results}` Then, run the following code to deploy it to the cloud: `[](#__codelineno-4-1)cerebrium deploy` If successful, you should be returned a CURL command that you can call inference against. Just remember to end the url with the function name you are calling (in our case `/run`) Command `[](#__codelineno-5-1)curl -X POST https://api.cortex.cerebrium.ai/v4/p-xxxxxx/vllm/run \ [](#__codelineno-5-2)-H 'Content-Type: application/json' \ [](#__codelineno-5-3)-H 'Authorization: ' \ [](#__codelineno-5-4)--data '{ [](#__codelineno-5-5)"prompts": [ [](#__codelineno-5-6) "Hello, my name is", [](#__codelineno-5-7) "The president of the United States is", [](#__codelineno-5-8) "The capital of France is", [](#__codelineno-5-9) "The future of AI is" [](#__codelineno-5-10)] [](#__codelineno-5-11)}'` You should get a response like: Response `[](#__codelineno-6-1){ [](#__codelineno-6-2) "run_id": "52911756-3066-9ae8-bcc9-d9129d1bd262", [](#__codelineno-6-3) "result": { [](#__codelineno-6-4) "result": [ [](#__codelineno-6-5) { [](#__codelineno-6-6) "prompt": "Hello, my name is", [](#__codelineno-6-7) "generated_text": " Sarah, and I'm a teacher. I teach elementary school students. One of" [](#__codelineno-6-8) }, [](#__codelineno-6-9) { [](#__codelineno-6-10) "prompt": "The president of the United States is", [](#__codelineno-6-11) "generated_text": " elected every four years. This is a democratic system.\n\n5. What" [](#__codelineno-6-12) }, [](#__codelineno-6-13) { [](#__codelineno-6-14) "prompt": "The capital of France is", [](#__codelineno-6-15) "generated_text": " Paris.\n" [](#__codelineno-6-16) }, [](#__codelineno-6-17) { [](#__codelineno-6-18) "prompt": "The future of AI is", [](#__codelineno-6-19) "generated_text": " bright, but it's important to approach it with a balanced and nuanced perspective." [](#__codelineno-6-20) } [](#__codelineno-6-21) ] [](#__codelineno-6-22) }, [](#__codelineno-6-23) "run_time_ms": 152.53663063049316 [](#__codelineno-6-24)}` You now have an autoscaling endpoint where you only pay for the compute you use! --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/chatbox.md "Edit this page") [Chatbox](https://github.com/chatboxai/chatbox) is a desktop client for LLMs, available on Windows, Mac, Linux. It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM environment: `[](#__codelineno-0-1)pip install vllm` ## Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-1-1)vllm serve qwen/Qwen1.5-0.5B-Chat` 2. Download and install [Chatbox desktop](https://chatboxai.app/en#download). 3. On the bottom left of settings, Add Custom Provider - API Mode: `OpenAI API Compatible` - Name: vllm - API Host: `http://{vllm server host}:{vllm server port}/v1` - API Path: `/chat/completions` - Model: `qwen/Qwen1.5-0.5B-Chat` [![Chatbox settings screen](https://docs.vllm.ai/en/assets/deployment/chatbox-settings.png)](https://docs.vllm.ai/en/assets/deployment/chatbox-settings.png) 4. Go to `Just chat`, and start to chat: [![Chatbot chat screen](https://docs.vllm.ai/en/assets/deployment/chatbox-chat.png)](https://docs.vllm.ai/en/assets/deployment/chatbox-chat.png) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/dify.md "Edit this page") [Dify](https://github.com/langgenius/dify) is an open-source LLM app development platform. Its intuitive interface combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features, and more, allowing you to quickly move from prototype to production. It supports vLLM as a model provider to efficiently serve large language models. This guide walks you through deploying Dify using a vLLM backend. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM environment: `[](#__codelineno-0-1)pip install vllm` And install [Docker](https://docs.docker.com/engine/install/) and [Docker Compose](https://docs.docker.com/compose/install/). ## Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-1-1)vllm serve Qwen/Qwen1.5-7B-Chat` 2. Start the Dify server with docker compose ([details](https://github.com/langgenius/dify?tab=readme-ov-file#quick-start)): `[](#__codelineno-2-1)git clone https://github.com/langgenius/dify.git [](#__codelineno-2-2)cd dify [](#__codelineno-2-3)cd docker [](#__codelineno-2-4)cp .env.example .env [](#__codelineno-2-5)docker compose up -d` 3. Open the browser to access `http://localhost/install`, config the basic login information and login. 4. In the top-right user menu (under the profile icon), go to Settings, then click `Model Provider`, and locate the `vLLM` provider to install it. 5. Fill in the model provider details as follows: - **Model Type**: [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") - **Model Name**: `Qwen/Qwen1.5-7B-Chat` - **API Endpoint URL**: `http://{vllm_server_host}:{vllm_server_port}/v1` - **Model Name for API Endpoint**: `Qwen/Qwen1.5-7B-Chat` - **Completion Mode**: `Completion` [![Dify settings screen](https://docs.vllm.ai/en/assets/deployment/dify-settings.png)](https://docs.vllm.ai/en/assets/deployment/dify-settings.png) 6. To create a test chatbot, go to `Studio → Chatbot → Create from Blank`, then select Chatbot as the type: [![Dify create chatbot screen](https://docs.vllm.ai/en/assets/deployment/dify-create-chatbot.png)](https://docs.vllm.ai/en/assets/deployment/dify-create-chatbot.png) 7. Click the chatbot you just created to open the chat interface and start interacting with the model: [![Dify chat screen](https://docs.vllm.ai/en/assets/deployment/dify-chat.png)](https://docs.vllm.ai/en/assets/deployment/dify-chat.png) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/dstack.md "Edit this page") [![vLLM_plus_dstack](https://i.ibb.co/71kx6hW/vllm-dstack.png)](https://i.ibb.co/71kx6hW/vllm-dstack.png) vLLM can be run on a cloud based GPU machine with [dstack](https://dstack.ai/), an open-source framework for running LLMs on any cloud. This tutorial assumes that you have already configured credentials, gateway, and GPU quotas on your cloud environment. To install dstack client, run: `[](#__codelineno-0-1)pip install dstack[all] [](#__codelineno-0-2)dstack server` Next, to configure your dstack project, run: `[](#__codelineno-1-1)mkdir -p vllm-dstack [](#__codelineno-1-2)cd vllm-dstack [](#__codelineno-1-3)dstack init` Next, to provision a VM instance with LLM of your choice (`NousResearch/Llama-2-7b-chat-hf` for this example), create the following `serve.dstack.yml` file for the dstack `Service`: Config `[](#__codelineno-2-1)type: service [](#__codelineno-2-2)[](#__codelineno-2-3)python: "3.11" [](#__codelineno-2-4)env: [](#__codelineno-2-5) - MODEL=NousResearch/Llama-2-7b-chat-hf [](#__codelineno-2-6)port: 8000 [](#__codelineno-2-7)resources: [](#__codelineno-2-8) gpu: 24GB [](#__codelineno-2-9)commands: [](#__codelineno-2-10) - pip install vllm [](#__codelineno-2-11) - vllm serve $MODEL --port 8000 [](#__codelineno-2-12)model: [](#__codelineno-2-13) format: openai [](#__codelineno-2-14) type: chat [](#__codelineno-2-15) name: NousResearch/Llama-2-7b-chat-hf` Then, run the following CLI for provisioning: Command `[](#__codelineno-3-1)$ dstack run . -f serve.dstack.yml [](#__codelineno-3-2)[](#__codelineno-3-3)⠸ Getting run plan... [](#__codelineno-3-4)Configuration serve.dstack.yml [](#__codelineno-3-5)Project deep-diver-main [](#__codelineno-3-6)User deep-diver [](#__codelineno-3-7)Min resources 2..xCPU, 8GB.., 1xGPU (24GB) [](#__codelineno-3-8)Max price - [](#__codelineno-3-9)Max duration - [](#__codelineno-3-10)Spot policy auto [](#__codelineno-3-11)Retry policy no [](#__codelineno-3-12)[](#__codelineno-3-13)# BACKEND REGION INSTANCE RESOURCES SPOT PRICE [](#__codelineno-3-14)1 gcp us-central1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804 [](#__codelineno-3-15)2 gcp us-east1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804 [](#__codelineno-3-16)3 gcp us-west1 g2-standard-4 4xCPU, 16GB, 1xL4 (24GB), 100GB (disk) yes $0.223804 [](#__codelineno-3-17) ... [](#__codelineno-3-18)Shown 3 of 193 offers, $5.876 max [](#__codelineno-3-19)[](#__codelineno-3-20)Continue? [y/n]: y [](#__codelineno-3-21)⠙ Submitting run... [](#__codelineno-3-22)⠏ Launching spicy-treefrog-1 (pulling) [](#__codelineno-3-23)spicy-treefrog-1 provisioning completed (running) [](#__codelineno-3-24)Service is published at ...` After the provisioning, you can interact with the model by using the OpenAI SDK: Code `[](#__codelineno-4-1)from openai import OpenAI [](#__codelineno-4-2)[](#__codelineno-4-3)client = OpenAI( [](#__codelineno-4-4) base_url="https://gateway.", [](#__codelineno-4-5) api_key="", [](#__codelineno-4-6)) [](#__codelineno-4-7)[](#__codelineno-4-8)completion = client.chat.completions.create( [](#__codelineno-4-9) model="NousResearch/Llama-2-7b-chat-hf", [](#__codelineno-4-10) messages=[ [](#__codelineno-4-11) { [](#__codelineno-4-12) "role": "user", [](#__codelineno-4-13) "content": "Compose a poem that explains the concept of recursion in programming.", [](#__codelineno-4-14) } [](#__codelineno-4-15) ], [](#__codelineno-4-16)) [](#__codelineno-4-17)[](#__codelineno-4-18)print(completion.choices[0].message.content)` Note dstack automatically handles authentication on the gateway using dstack's tokens. Meanwhile, if you don't want to configure a gateway, you can provision dstack `Task` instead of `Service`. The `Task` is for development purpose only. If you want to know more about hands-on materials how to serve vLLM using dstack, check out [this repository](https://github.com/dstackai/dstack-examples/tree/main/deployment/vllm) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/haystack.md "Edit this page") [Haystack](https://github.com/deepset-ai/haystack) is an end-to-end LLM framework that allows you to build applications powered by LLMs, Transformer models, vector search and more. Whether you want to perform retrieval-augmented generation (RAG), document search, question answering or answer generation, Haystack can orchestrate state-of-the-art embedding models and LLMs into pipelines to build end-to-end NLP applications and solve your use case. It allows you to deploy a large language model (LLM) server with vLLM as the backend, which exposes OpenAI-compatible endpoints. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM and Haystack environment: `[](#__codelineno-0-1)pip install vllm haystack-ai` ## Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-1-1)vllm serve mistralai/Mistral-7B-Instruct-v0.1` 2. Use the `OpenAIGenerator` and `OpenAIChatGenerator` components in Haystack to query the vLLM server. Code `[](#__codelineno-2-1)from haystack.components.generators.chat import OpenAIChatGenerator [](#__codelineno-2-2)from haystack.dataclasses import ChatMessage [](#__codelineno-2-3)from haystack.utils import Secret [](#__codelineno-2-4)[](#__codelineno-2-5)generator = OpenAIChatGenerator( [](#__codelineno-2-6) # for compatibility with the OpenAI API, a placeholder api_key is needed [](#__codelineno-2-7) api_key=Secret.from_token("VLLM-PLACEHOLDER-API-KEY"), [](#__codelineno-2-8) model="mistralai/Mistral-7B-Instruct-v0.1", [](#__codelineno-2-9) api_base_url="http://{your-vLLM-host-ip}:{your-vLLM-host-port}/v1", [](#__codelineno-2-10) generation_kwargs={"max_tokens": 512}, [](#__codelineno-2-11)) [](#__codelineno-2-12)[](#__codelineno-2-13)response = generator.run( [](#__codelineno-2-14) messages=[ChatMessage.from_user("Hi. Can you help me plan my next trip to Italy?")] [](#__codelineno-2-15)) [](#__codelineno-2-16)[](#__codelineno-2-17)print("-"*30) [](#__codelineno-2-18)print(response) [](#__codelineno-2-19)print("-"*30)` `[](#__codelineno-3-1)------------------------------ [](#__codelineno-3-2){'replies': [ChatMessage(_role=, _content=[TextContent(text=' Of course! Where in Italy would you like to go and what type of trip are you looking to plan?')], _name=None, _meta={'model': 'mistralai/Mistral-7B-Instruct-v0.1', 'index': 0, 'finish_reason': 'stop', 'usage': {'completion_tokens': 23, 'prompt_tokens': 21, 'total_tokens': 44, 'completion_tokens_details': None, 'prompt_tokens_details': None}})]} [](#__codelineno-3-3)------------------------------` For details, see the tutorial [Using vLLM in Haystack](https://github.com/deepset-ai/haystack-integrations/blob/main/integrations/vllm.md). --- # page autoscaling object {"enabled":false,"maxReplicas":100,"minReplicas":1,"targetCPUUtilizationPercentage":80} Autoscaling configuration autoscaling.enabled bool false Enable autoscaling autoscaling.maxReplicas int 100 Maximum replicas autoscaling.minReplicas int 1 Minimum replicas autoscaling.targetCPUUtilizationPercentage int 80 Target CPU utilization for autoscaling configs object {} Configmap containerPort int 8000 Container port customObjects list \[\] Custom Objects configuration deploymentStrategy object {} Deployment strategy configuration externalConfigs list \[\] External configuration extraContainers list \[\] Additional containers configuration extraInit object {"modelDownload":{"enabled":true},"initContainers":\[\],"pvcStorage":"1Gi"} Additional configuration for init containers extraInit.modelDownload object {"enabled":true} Model download functionality configuration extraInit.modelDownload.enabled bool true Enable automatic model download job and wait container extraInit.modelDownload.image object {"repository":"amazon/aws-cli","tag":"2.6.4","pullPolicy":"IfNotPresent"} Image for model download operations extraInit.modelDownload.waitContainer object {} Wait container configuration (command, args, env) extraInit.modelDownload.downloadJob object {} Download job configuration (command, args, env) extraInit.initContainers list \[\] Custom init containers (appended after model download if enabled) extraInit.pvcStorage string "1Gi" Storage size for the PVC extraInit.s3modelpath string "relative\_s3\_model\_path/opt-125m" (Optional) Path of the model on S3 extraInit.awsEc2MetadataDisabled bool true (Optional) Disable AWS EC2 metadata service extraPorts list \[\] Additional ports configuration gpuModels list \["TYPE\_GPU\_USED"\] Type of gpu used image object {"command":\["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"\],"repository":"vllm/vllm-openai","tag":"latest"} Image configuration image.command list \["vllm","serve","/data/","--served-model-name","opt-125m","--host","0.0.0.0","--port","8000"\] Container launch command image.repository string "vllm/vllm-openai" Image repository image.tag string "latest" Image tag livenessProbe object {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":15,"periodSeconds":10} Liveness probe configuration livenessProbe.failureThreshold int 3 Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not alive livenessProbe.httpGet object {"path":"/health","port":8000} Configuration of the kubelet http request on the server livenessProbe.httpGet.path string "/health" Path to access on the HTTP server livenessProbe.httpGet.port int 8000 Name or number of the port to access on the container, on which the server is listening livenessProbe.initialDelaySeconds int 15 Number of seconds after the container has started before liveness probe is initiated livenessProbe.periodSeconds int 10 How often (in seconds) to perform the liveness probe maxUnavailablePodDisruptionBudget string "" Disruption Budget Configuration readinessProbe object {"failureThreshold":3,"httpGet":{"path":"/health","port":8000},"initialDelaySeconds":5,"periodSeconds":5} Readiness probe configuration readinessProbe.failureThreshold int 3 Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not ready readinessProbe.httpGet object {"path":"/health","port":8000} Configuration of the kubelet http request on the server readinessProbe.httpGet.path string "/health" Path to access on the HTTP server readinessProbe.httpGet.port int 8000 Name or number of the port to access on the container, on which the server is listening readinessProbe.initialDelaySeconds int 5 Number of seconds after the container has started before readiness probe is initiated readinessProbe.periodSeconds int 5 How often (in seconds) to perform the readiness probe replicaCount int 1 Number of replicas resources object {"limits":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1},"requests":{"cpu":4,"memory":"16Gi","nvidia.com/gpu":1}} Resource configuration resources.limits."nvidia.com/gpu" int 1 Number of GPUs used resources.limits.cpu int 4 Number of CPUs resources.limits.memory string "16Gi" CPU memory configuration resources.requests."nvidia.com/gpu" int 1 Number of GPUs used resources.requests.cpu int 4 Number of CPUs resources.requests.memory string "16Gi" CPU memory configuration secrets object {} Secrets configuration serviceName string "" Service name servicePort int 80 Service port labels.environment string test Environment name --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/hf_inference_endpoints.md "Edit this page") ## Overview[¶](#overview "Permanent link") Models compatible with vLLM can be deployed on Hugging Face Inference Endpoints, either starting from the [Hugging Face Hub](https://huggingface.co/) or directly from the [Inference Endpoints](https://endpoints.huggingface.co/) interface. This allows you to serve models in a fully managed environment with GPU acceleration, auto-scaling, and monitoring, without managing the infrastructure manually. For advanced details on vLLM integration and deployment options, see [Advanced Deployment Details](#advanced-deployment-details). ## Deployment Methods[¶](#deployment-methods "Permanent link") - [**Method 1: Deploy from the Catalog.**](#method-1-deploy-from-the-catalog) One-click deploy models from the Hugging Face Hub with ready-made optimized configurations. - [**Method 2: Guided Deployment (Transformers Models).**](#method-2-guided-deployment-transformers-models) Instantly deploy models tagged with `transformers` from the Hub UI using the **Deploy** button. - [**Method 3: Manual Deployment (Advanced Models).**](#method-3-manual-deployment-advanced-models) For models that either use custom code with the `transformers` tag, or don’t run with standard `transformers` but are supported by vLLM. This method requires manual configuration. ### Method 1: Deploy from the Catalog[¶](#method-1-deploy-from-the-catalog "Permanent link") This is the easiest way to get started with vLLM on Hugging Face Inference Endpoints. You can browse a catalog of models with verified and optimized deployment configuration at [Inference Endpoints](https://endpoints.huggingface.co/catalog) to maximize performance. 1. Go to [Endpoints Catalog](https://endpoints.huggingface.co/catalog) and in the **Inference Server** options, select `vLLM`.This will display the current list of models with optimized preconfigured options. [![Endpoints Catalog](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-catalog.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-catalog.png) 2. Select the desired model and click **Create Endpoint**. [![Create Endpoint](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-create-endpoint.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-create-endpoint.png) 3. Once the deployment is ready, you can use the endpoint. Update the `DEPLOYMENT_URL` with the URL provided in the console, remembering to append `/v1` as required. `[](#__codelineno-0-1)# pip install openai [](#__codelineno-0-2)from openai import OpenAI [](#__codelineno-0-3)import os [](#__codelineno-0-4)[](#__codelineno-0-5)client = OpenAI( [](#__codelineno-0-6) base_url=DEPLOYMENT_URL, [](#__codelineno-0-7) api_key=os.environ["HF_TOKEN"], # https://huggingface.co/settings/tokens [](#__codelineno-0-8)) [](#__codelineno-0-9)[](#__codelineno-0-10)chat_completion = client.chat.completions.create( [](#__codelineno-0-11) model="HuggingFaceTB/SmolLM3-3B", [](#__codelineno-0-12) messages=[ [](#__codelineno-0-13) { [](#__codelineno-0-14) "role": "user", [](#__codelineno-0-15) "content": [ [](#__codelineno-0-16) { [](#__codelineno-0-17) "type": "text", [](#__codelineno-0-18) "text": "Give me a brief explanation of gravity in simple terms.", [](#__codelineno-0-19) } [](#__codelineno-0-20) ], [](#__codelineno-0-21) } [](#__codelineno-0-22) ], [](#__codelineno-0-23) stream=True, [](#__codelineno-0-24)) [](#__codelineno-0-25)[](#__codelineno-0-26)for message in chat_completion: [](#__codelineno-0-27) print(message.choices[0].delta.content, end="")` Note The catalog provides models optimized for vLLM, including GPU settings and inference engine configurations. You can monitor the endpoint and update the **container or its configuration** from the Inference Endpoints UI. ### Method 2: Guided Deployment (Transformers Models)[¶](#method-2-guided-deployment-transformers-models "Permanent link") This method applies to models with the [`transformers` library tag](https://huggingface.co/models?library=transformers) in their metadata. It allows you to deploy a model directly from the Hub UI without manual configuration. 1. Navigate to a model on [Hugging Face Hub](https://huggingface.co/models). For this example we will use the [`ibm-granite/granite-docling-258M`](https://huggingface.co/ibm-granite/granite-docling-258M) model. You can verify that the model is compatible by checking the front matter in the [README](https://huggingface.co/ibm-granite/granite-docling-258M/blob/main/README.md), where the library is tagged as `library: transformers`. 2. Locate the **Deploy** button. The button appears for models tagged with `transformers` at the top right of the [model card](https://huggingface.co/ibm-granite/granite-docling-258M). [![Locate deploy button](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-locate-deploy-button.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-locate-deploy-button.png) 3. Click the **Deploy** button > **HF Inference Endpoints**. You will be taken to the Inference Endpoints interface to configure the deployment. [![Click deploy button](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-click-deploy-button.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-click-deploy-button.png) 4. Select the Hardware (we choose AWS>GPU>T4 for the example) and Container Configuration. Choose `vLLM` as the container type and finalize the deployment pressing **Create Endpoint**. [![Select Hardware](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-select-hardware.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-select-hardware.png) 5. Use the deployed endpoint. Update the `DEPLOYMENT_URL` with the URL provided in the console (remember to add `/v1` needed). You can then use your endpoint programmatically or via the SDK. `[](#__codelineno-1-1)# pip install openai [](#__codelineno-1-2)from openai import OpenAI [](#__codelineno-1-3)import os [](#__codelineno-1-4)[](#__codelineno-1-5)client = OpenAI( [](#__codelineno-1-6) base_url=DEPLOYMENT_URL, [](#__codelineno-1-7) api_key=os.environ["HF_TOKEN"], # https://huggingface.co/settings/tokens [](#__codelineno-1-8)) [](#__codelineno-1-9)[](#__codelineno-1-10)chat_completion = client.chat.completions.create( [](#__codelineno-1-11) model="ibm-granite/granite-docling-258M", [](#__codelineno-1-12) messages=[ [](#__codelineno-1-13) { [](#__codelineno-1-14) "role": "user", [](#__codelineno-1-15) "content": [ [](#__codelineno-1-16) { [](#__codelineno-1-17) "type": "image_url", [](#__codelineno-1-18) "image_url": { [](#__codelineno-1-19) "url": "https://huggingface.co/ibm-granite/granite-docling-258M/resolve/main/assets/new_arxiv.png", [](#__codelineno-1-20) }, [](#__codelineno-1-21) }, [](#__codelineno-1-22) { [](#__codelineno-1-23) "type": "text", [](#__codelineno-1-24) "text": "Convert this page to docling.", [](#__codelineno-1-25) }, [](#__codelineno-1-26) ] [](#__codelineno-1-27) } [](#__codelineno-1-28) ], [](#__codelineno-1-29) stream=True, [](#__codelineno-1-30)) [](#__codelineno-1-31)[](#__codelineno-1-32)for message in chat_completion: [](#__codelineno-1-33) print(message.choices[0].delta.content, end="")` Note This method uses best-guess defaults. You may need to adjust the configuration to fit your specific requirements. ### Method 3: Manual Deployment (Advanced Models)[¶](#method-3-manual-deployment-advanced-models "Permanent link") Some models require manual deployment because they: - Use custom code with the `transformers` tag - Don't run with standard `transformers` but are supported by `vLLM` These models cannot be deployed using the **Deploy** button on the model card. In this guide, we demonstrate manual deployment using the [`rednote-hilab/dots.ocr`](https://huggingface.co/rednote-hilab/dots.ocr) model, an OCR model integrated with vLLM (see vLLM [PR](https://github.com/vllm-project/vllm/pull/24645)). 1. Start a new deployment. Go to [Inference Endpoints](https://endpoints.huggingface.co/) and click `New`. [![New Endpoint](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-new-endpoint.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-new-endpoint.png) 2. Search the model in the Hub. In the dialog, switch to **Hub** and search for the desired model. [![Select model](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-select-model.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-select-model.png) 3. Choosing infrastructure. On the configuration page, select the cloud provider and hardware from the available options. For this demo, we choose AWS and L4 GPU. Adjust according to your hardware needs. [![Choose Infra](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-choose-infra.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-choose-infra.png) 4. Configure the container. Scroll to the **Container Configuration** and select `vLLM` as the container type. [![Configure Container](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-configure-container.png)](https://docs.vllm.ai/en/assets/deployment/hf-inference-endpoints-configure-container.png) 5. Create the endpoint. Click **Create Endpoint** to deploy the model. Once the endpoint is ready, you can use it with the OpenAI Completion API, cURL, or other SDKs. Remember to append `/v1` to the deployment URL if needed. Note You can adjust the **container settings** (Container URI, Container Arguments) from the Inference Endpoints UI and press **Update Endpoint**. This redeploys the endpoint with the updated container configuration. Changes to the model itself require creating a new endpoint or redeploying with a different model. For example, for this demo, you may need to update the Container URI to the nightly image (`vllm/vllm-openai:nightly`) and add the `--trust-remote-code` flag in the container arguments. ## Advanced Deployment Details[¶](#advanced-deployment-details "Permanent link") With the [Transformers modeling backend integration](https://blog.vllm.ai/2025/04/11/transformers-backend.html), vLLM now offers Day 0 support for any model compatible with `transformers`. This means you can deploy such models immediately, leveraging vLLM’s optimized inference without additional backend modifications. Hugging Face Inference Endpoints provides a fully managed environment for serving models via vLLM. You can deploy models without configuring servers, installing dependencies, or managing clusters. Endpoints also support deployment across multiple cloud providers (AWS, Azure, GCP) without the need for separate accounts. The platform integrates seamlessly with the Hugging Face Hub, allowing you to deploy any vLLM- or `transformers`\-compatible model, track usage, and update the inference engine directly. The vLLM engine comes preconfigured, enabling optimized inference and easy switching between models or engines without modifying your code. This setup simplifies production deployment: endpoints are ready in minutes, include monitoring and logging, and let you focus on serving models rather than maintaining infrastructure. ## Next Steps[¶](#next-steps "Permanent link") - Explore the [Inference Endpoints](https://endpoints.huggingface.co/catalog) model catalog - Read the Inference Endpoints [documentation](https://huggingface.co/docs/inference-endpoints/en/index) - Learn about [Inference Endpoints engines](https://huggingface.co/docs/inference-endpoints/en/engines/vllm) - Understand the [Transformers modeling backend integration](https://blog.vllm.ai/2025/04/11/transformers-backend.html) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/litellm.md "Edit this page") [LiteLLM](https://github.com/BerriAI/litellm) call all LLM APIs using the OpenAI format \[Bedrock, Huggingface, VertexAI, TogetherAI, Azure, OpenAI, Groq etc.\] LiteLLM manages: - Translate inputs to provider's `completion`, `embedding`, and `image_generation` endpoints - [Consistent output](https://docs.litellm.ai/docs/completion/output), text responses will always be available at `['choices'][0]['message']['content']` - Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - [Router](https://docs.litellm.ai/docs/routing) - Set Budgets & Rate limits per project, api key, model [LiteLLM Proxy Server (LLM Gateway)](https://docs.litellm.ai/docs/simple_proxy) And LiteLLM supports all models on VLLM. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM and litellm environment: `[](#__codelineno-0-1)pip install vllm litellm` ## Deploy[¶](#deploy "Permanent link") ### Chat completion[¶](#chat-completion "Permanent link") 1. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-1-1)vllm serve qwen/Qwen1.5-0.5B-Chat` 2. Call it with litellm: Code `[](#__codelineno-2-1)import litellm [](#__codelineno-2-2)[](#__codelineno-2-3)messages = [{"content": "Hello, how are you?", "role": "user"}] [](#__codelineno-2-4)[](#__codelineno-2-5)# hosted_vllm is prefix key word and necessary [](#__codelineno-2-6)response = litellm.completion( [](#__codelineno-2-7) model="hosted_vllm/qwen/Qwen1.5-0.5B-Chat", # pass the vllm model name [](#__codelineno-2-8) messages=messages, [](#__codelineno-2-9) api_base="http://{your-vllm-server-host}:{your-vllm-server-port}/v1", [](#__codelineno-2-10) temperature=0.2, [](#__codelineno-2-11) max_tokens=80, [](#__codelineno-2-12)) [](#__codelineno-2-13)[](#__codelineno-2-14)print(response)` ### Embeddings[¶](#embeddings "Permanent link") 1. Start the vLLM server with the supported embedding model, e.g. `[](#__codelineno-3-1)vllm serve BAAI/bge-base-en-v1.5` 2. Call it with litellm: `[](#__codelineno-4-1)from litellm import embedding [](#__codelineno-4-2)import os [](#__codelineno-4-3)[](#__codelineno-4-4)os.environ["HOSTED_VLLM_API_BASE"] = "http://{your-vllm-server-host}:{your-vllm-server-port}/v1" [](#__codelineno-4-5)[](#__codelineno-4-6)# hosted_vllm is prefix key word and necessary [](#__codelineno-4-7)# pass the vllm model name [](#__codelineno-4-8)embedding = embedding(model="hosted_vllm/BAAI/bge-base-en-v1.5", input=["Hello world"]) [](#__codelineno-4-9)[](#__codelineno-4-10)print(embedding)` For details, see the tutorial [Using vLLM in LiteLLM](https://docs.litellm.ai/docs/providers/vllm). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/lobe-chat.md "Edit this page") [Lobe Chat](https://github.com/lobehub/lobe-chat) is an open-source, modern-design ChatGPT/LLMs UI/Framework. Supports speech-synthesis, multi-modal, and extensible (function call) plugin system. One-click FREE deployment of your private OpenAI ChatGPT/Claude/Gemini/Groq/Ollama chat application. It supports vLLM as an AI model provider to efficiently serve large language models. For details, see the tutorial [Using vLLM in LobeChat](https://lobehub.com/docs/usage/providers/vllm). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/lws.md "Edit this page") LeaderWorkerSet (LWS) is a Kubernetes API that aims to address common deployment patterns of AI/ML inference workloads. A major use case is for multi-host/multi-node distributed inference. vLLM can be deployed with [LWS](https://github.com/kubernetes-sigs/lws) on Kubernetes for distributed model serving. ## Prerequisites[¶](#prerequisites "Permanent link") - At least two Kubernetes nodes, each with 8 GPUs, are required. - Install LWS by following the instructions found [here](https://lws.sigs.k8s.io/docs/installation/). ## Deploy and Serve[¶](#deploy-and-serve "Permanent link") Deploy the following yaml file `lws.yaml` Yaml `[](#__codelineno-0-1)apiVersion: leaderworkerset.x-k8s.io/v1 [](#__codelineno-0-2)kind: LeaderWorkerSet [](#__codelineno-0-3)metadata: [](#__codelineno-0-4) name: vllm [](#__codelineno-0-5)spec: [](#__codelineno-0-6) replicas: 1 [](#__codelineno-0-7) leaderWorkerTemplate: [](#__codelineno-0-8) size: 2 [](#__codelineno-0-9) restartPolicy: RecreateGroupOnPodRestart [](#__codelineno-0-10) leaderTemplate: [](#__codelineno-0-11) metadata: [](#__codelineno-0-12) labels: [](#__codelineno-0-13) role: leader [](#__codelineno-0-14) spec: [](#__codelineno-0-15) containers: [](#__codelineno-0-16) - name: vllm-leader [](#__codelineno-0-17) image: docker.io/vllm/vllm-openai:latest [](#__codelineno-0-18) env: [](#__codelineno-0-19) - name: HF_TOKEN [](#__codelineno-0-20) value: [](#__codelineno-0-21) command: [](#__codelineno-0-22) - sh [](#__codelineno-0-23) - -c [](#__codelineno-0-24) - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE); [](#__codelineno-0-25) vllm serve meta-llama/Meta-Llama-3.1-405B-Instruct --port 8080 --tensor-parallel-size 8 --pipeline_parallel_size 2" [](#__codelineno-0-26) resources: [](#__codelineno-0-27) limits: [](#__codelineno-0-28) nvidia.com/gpu: "8" [](#__codelineno-0-29) memory: 1124Gi [](#__codelineno-0-30) ephemeral-storage: 800Gi [](#__codelineno-0-31) requests: [](#__codelineno-0-32) ephemeral-storage: 800Gi [](#__codelineno-0-33) cpu: 125 [](#__codelineno-0-34) ports: [](#__codelineno-0-35) - containerPort: 8080 [](#__codelineno-0-36) readinessProbe: [](#__codelineno-0-37) tcpSocket: [](#__codelineno-0-38) port: 8080 [](#__codelineno-0-39) initialDelaySeconds: 15 [](#__codelineno-0-40) periodSeconds: 10 [](#__codelineno-0-41) volumeMounts: [](#__codelineno-0-42) - mountPath: /dev/shm [](#__codelineno-0-43) name: dshm [](#__codelineno-0-44) volumes: [](#__codelineno-0-45) - name: dshm [](#__codelineno-0-46) emptyDir: [](#__codelineno-0-47) medium: Memory [](#__codelineno-0-48) sizeLimit: 15Gi [](#__codelineno-0-49) workerTemplate: [](#__codelineno-0-50) spec: [](#__codelineno-0-51) containers: [](#__codelineno-0-52) - name: vllm-worker [](#__codelineno-0-53) image: docker.io/vllm/vllm-openai:latest [](#__codelineno-0-54) command: [](#__codelineno-0-55) - sh [](#__codelineno-0-56) - -c [](#__codelineno-0-57) - "bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS)" [](#__codelineno-0-58) resources: [](#__codelineno-0-59) limits: [](#__codelineno-0-60) nvidia.com/gpu: "8" [](#__codelineno-0-61) memory: 1124Gi [](#__codelineno-0-62) ephemeral-storage: 800Gi [](#__codelineno-0-63) requests: [](#__codelineno-0-64) ephemeral-storage: 800Gi [](#__codelineno-0-65) cpu: 125 [](#__codelineno-0-66) env: [](#__codelineno-0-67) - name: HF_TOKEN [](#__codelineno-0-68) value: [](#__codelineno-0-69) volumeMounts: [](#__codelineno-0-70) - mountPath: /dev/shm [](#__codelineno-0-71) name: dshm [](#__codelineno-0-72) volumes: [](#__codelineno-0-73) - name: dshm [](#__codelineno-0-74) emptyDir: [](#__codelineno-0-75) medium: Memory [](#__codelineno-0-76) sizeLimit: 15Gi [](#__codelineno-0-77)--- [](#__codelineno-0-78)apiVersion: v1 [](#__codelineno-0-79)kind: Service [](#__codelineno-0-80)metadata: [](#__codelineno-0-81) name: vllm-leader [](#__codelineno-0-82)spec: [](#__codelineno-0-83) ports: [](#__codelineno-0-84) - name: http [](#__codelineno-0-85) port: 8080 [](#__codelineno-0-86) protocol: TCP [](#__codelineno-0-87) targetPort: 8080 [](#__codelineno-0-88) selector: [](#__codelineno-0-89) leaderworkerset.sigs.k8s.io/name: vllm [](#__codelineno-0-90) role: leader [](#__codelineno-0-91) type: ClusterIP` `[](#__codelineno-1-1)kubectl apply -f lws.yaml` Verify the status of the pods: `[](#__codelineno-2-1)kubectl get pods` Should get an output similar to this: `[](#__codelineno-3-1)NAME READY STATUS RESTARTS AGE [](#__codelineno-3-2)vllm-0 1/1 Running 0 2s [](#__codelineno-3-3)vllm-0-1 1/1 Running 0 2s` Verify that the distributed tensor-parallel inference works: `[](#__codelineno-4-1)kubectl logs vllm-0 |grep -i "Loading model weights took"` Should get something similar to this: `[](#__codelineno-5-1)INFO 05-08 03:20:24 model_runner.py:173] Loading model weights took 0.1189 GB [](#__codelineno-5-2)(RayWorkerWrapper pid=169, ip=10.20.0.197) INFO 05-08 03:20:28 model_runner.py:173] Loading model weights took 0.1189 GB` ## Access ClusterIP service[¶](#access-clusterip-service "Permanent link") `[](#__codelineno-6-1)# Listen on port 8080 locally, forwarding to the targetPort of the service's port 8080 in a pod selected by the service [](#__codelineno-6-2)kubectl port-forward svc/vllm-leader 8080:8080` The output should be similar to the following: `[](#__codelineno-7-1)Forwarding from 127.0.0.1:8080 -> 8080 [](#__codelineno-7-2)Forwarding from [::1]:8080 -> 8080` ## Serve the model[¶](#serve-the-model "Permanent link") Open another terminal and send a request `[](#__codelineno-8-1)curl http://localhost:8080/v1/completions \ [](#__codelineno-8-2)-H "Content-Type: application/json" \ [](#__codelineno-8-3)-d '{ [](#__codelineno-8-4) "model": "meta-llama/Meta-Llama-3.1-405B-Instruct", [](#__codelineno-8-5) "prompt": "San Francisco is a", [](#__codelineno-8-6) "max_tokens": 7, [](#__codelineno-8-7) "temperature": 0 [](#__codelineno-8-8)}'` The output should be similar to the following Output `[](#__codelineno-9-1){ [](#__codelineno-9-2) "id": "cmpl-1bb34faba88b43f9862cfbfb2200949d", [](#__codelineno-9-3) "object": "text_completion", [](#__codelineno-9-4) "created": 1715138766, [](#__codelineno-9-5) "model": "meta-llama/Meta-Llama-3.1-405B-Instruct", [](#__codelineno-9-6) "choices": [ [](#__codelineno-9-7) { [](#__codelineno-9-8) "index": 0, [](#__codelineno-9-9) "text": " top destination for foodies, with", [](#__codelineno-9-10) "logprobs": null, [](#__codelineno-9-11) "finish_reason": "length", [](#__codelineno-9-12) "stop_reason": null [](#__codelineno-9-13) } [](#__codelineno-9-14) ], [](#__codelineno-9-15) "usage": { [](#__codelineno-9-16) "prompt_tokens": 5, [](#__codelineno-9-17) "total_tokens": 12, [](#__codelineno-9-18) "completion_tokens": 7 [](#__codelineno-9-19) } [](#__codelineno-9-20)}` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/modal.md "Edit this page") vLLM can be run on cloud GPUs with [Modal](https://modal.com/), a serverless computing platform designed for fast auto-scaling. For details on how to deploy vLLM on Modal, see [this tutorial in the Modal documentation](https://modal.com/docs/examples/vllm_inference). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/open-webui.md "Edit this page") [Open WebUI](https://github.com/open-webui/open-webui) is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in RAG capabilities, making it a powerful AI deployment solution. To get started with Open WebUI using vLLM, follow these steps: 1. Install the [Docker](https://docs.docker.com/engine/install/). 2. Start the vLLM server with a supported chat completion model: `[](#__codelineno-0-1)vllm serve Qwen/Qwen3-0.6B-Chat` Note When starting the vLLM server, be sure to specify the host and port using the `--host` and `--port` flags. For example: `[](#__codelineno-1-1)vllm serve --host 0.0.0.0 --port 8000` 3. Start the Open WebUI Docker container: `[](#__codelineno-2-1)docker run -d \ [](#__codelineno-2-2) --name open-webui \ [](#__codelineno-2-3) -p 3000:8080 \ [](#__codelineno-2-4) -v open-webui:/app/backend/data \ [](#__codelineno-2-5) -e OPENAI_API_BASE_URL=http://0.0.0.0:8000/v1 \ [](#__codelineno-2-6) --restart always \ [](#__codelineno-2-7) ghcr.io/open-webui/open-webui:main` 4. Open it in the browser: [http://open-webui-host:3000/](http://open-webui-host:3000/) At the top of the page, you should see the model `Qwen/Qwen3-0.6B-Chat`. [![Web portal of model Qwen/Qwen3-0.6B-Chat](https://docs.vllm.ai/en/assets/deployment/open_webui.png)](https://docs.vllm.ai/en/assets/deployment/open_webui.png) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/retrieval_augmented_generation.md "Edit this page") [Retrieval-augmented generation (RAG)](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) is a technique that enables generative artificial intelligence (Gen AI) models to retrieve and incorporate new information. It modifies interactions with a large language model (LLM) so that the model responds to user queries with reference to a specified set of documents, using this information to supplement information from its pre-existing training data. This allows LLMs to use domain-specific and/or updated information. Use cases include providing chatbot access to internal company data or generating responses based on authoritative sources. Here are the integrations: - vLLM + [langchain](https://github.com/langchain-ai/langchain) + [milvus](https://github.com/milvus-io/milvus) - vLLM + [llamaindex](https://github.com/run-llama/llama_index) + [milvus](https://github.com/milvus-io/milvus) ## vLLM + langchain[¶](#vllm-langchain "Permanent link") ### Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM and langchain environment: `[](#__codelineno-0-1)pip install -U vllm \ [](#__codelineno-0-2) langchain_milvus langchain_openai \ [](#__codelineno-0-3) langchain_community beautifulsoup4 \ [](#__codelineno-0-4) langchain-text-splitters` ### Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with the supported embedding model, e.g. `[](#__codelineno-1-1)# Start embedding service (port 8000) [](#__codelineno-1-2)vllm serve ssmits/Qwen2-7B-Instruct-embed-base` 2. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-2-1)# Start chat service (port 8001) [](#__codelineno-2-2)vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001` 3. Use the script: [examples/applications/rag/retrieval\_augmented\_generation\_with\_langchain.py](https://github.com/vllm-project/vllm/blob/main/examples/applications/rag/retrieval_augmented_generation_with_langchain.py) 4. Run the script `[](#__codelineno-3-1)python retrieval_augmented_generation_with_langchain.py` ## vLLM + llamaindex[¶](#vllm-llamaindex "Permanent link") ### Prerequisites[¶](#prerequisites_1 "Permanent link") Set up the vLLM and llamaindex environment: `[](#__codelineno-4-1)pip install vllm \ [](#__codelineno-4-2) llama-index llama-index-readers-web \ [](#__codelineno-4-3) llama-index-llms-openai-like \ [](#__codelineno-4-4) llama-index-embeddings-openai-like \ [](#__codelineno-4-5) llama-index-vector-stores-milvus \` ### Deploy[¶](#deploy_1 "Permanent link") 1. Start the vLLM server with the supported embedding model, e.g. `[](#__codelineno-5-1)# Start embedding service (port 8000) [](#__codelineno-5-2)vllm serve ssmits/Qwen2-7B-Instruct-embed-base` 2. Start the vLLM server with the supported chat completion model, e.g. `[](#__codelineno-6-1)# Start chat service (port 8001) [](#__codelineno-6-2)vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001` 3. Use the script: [examples/applications/rag/retrieval\_augmented\_generation\_with\_llamaindex.py](https://github.com/vllm-project/vllm/blob/main/examples/applications/rag/retrieval_augmented_generation_with_llamaindex.py) 4. Run the script: `[](#__codelineno-7-1)python retrieval_augmented_generation_with_llamaindex.py` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/runpod.md "Edit this page") vLLM can be deployed on [RunPod](https://www.runpod.io/), a cloud GPU platform that provides on-demand and serverless GPU instances for AI inference workloads. ## Prerequisites[¶](#prerequisites "Permanent link") - A RunPod account with GPU pod access - A GPU pod running a CUDA-compatible template (e.g., `runpod/pytorch`) ## Starting the Server[¶](#starting-the-server "Permanent link") SSH into your RunPod pod and launch the vLLM OpenAI-compatible server: `[](#__codelineno-0-1)vllm serve \ [](#__codelineno-0-2) --host 0.0.0.0 \ [](#__codelineno-0-3) --port 8000` Note Use `--host 0.0.0.0` to bind to all interfaces so the server is reachable from outside the container. ## Exposing Port 8000[¶](#exposing-port-8000 "Permanent link") RunPod exposes HTTP services through its proxy. To make port 8000 accessible: 1. In the RunPod dashboard, navigate to your pod settings. 2. Add `8000` to the list of exposed HTTP ports. 3. After the pod restarts, RunPod provides a public URL in the format: `[](#__codelineno-1-1)https://-8000.proxy.runpod.net` ## Troubleshooting 502 Bad Gateway[¶](#troubleshooting-502-bad-gateway "Permanent link") A `502 Bad Gateway` error from the RunPod proxy typically means the server is not yet listening. Common causes: - **Model still loading** — Large models take time to download and load into GPU memory. Check the pod logs for progress. - **Wrong host binding** — Ensure you passed `--host 0.0.0.0`. Binding to `127.0.0.1` (the default) makes the server unreachable from the proxy. - **Port mismatch** — Verify the `--port` value matches the port exposed in the RunPod dashboard. - **Out of GPU memory** — The model may be too large for the allocated GPU. Check logs for CUDA OOM errors and consider using a larger instance or adding `--tensor-parallel-size` for multi-GPU pods. ## Verifying the Deployment[¶](#verifying-the-deployment "Permanent link") Once the server is running, test it with a curl request: Command `[](#__codelineno-2-1)curl https://-8000.proxy.runpod.net/v1/chat/completions \ [](#__codelineno-2-2) -H "Content-Type: application/json" \ [](#__codelineno-2-3) -d '{ [](#__codelineno-2-4) "model": "", [](#__codelineno-2-5) "messages": [ [](#__codelineno-2-6) {"role": "user", "content": "Hello, how are you?"} [](#__codelineno-2-7) ], [](#__codelineno-2-8) "max_tokens": 50 [](#__codelineno-2-9) }'` Response `[](#__codelineno-3-1){ [](#__codelineno-3-2) "id": "chat-abc123", [](#__codelineno-3-3) "object": "chat.completion", [](#__codelineno-3-4) "choices": [ [](#__codelineno-3-5) { [](#__codelineno-3-6) "message": { [](#__codelineno-3-7) "role": "assistant", [](#__codelineno-3-8) "content": "I'm doing well, thank you for asking! How can I help you today?" [](#__codelineno-3-9) }, [](#__codelineno-3-10) "index": 0, [](#__codelineno-3-11) "finish_reason": "stop" [](#__codelineno-3-12) } [](#__codelineno-3-13) ] [](#__codelineno-3-14)}` You can also check the server health endpoint: `[](#__codelineno-4-1)curl https://-8000.proxy.runpod.net/health` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/skypilot.md "Edit this page") [![vLLM](https://imgur.com/yxtzPEu.png)](https://imgur.com/yxtzPEu.png) vLLM can be **run and scaled to multiple service replicas on clouds and Kubernetes** with [SkyPilot](https://github.com/skypilot-org/skypilot), an open-source framework for running LLMs on any cloud. More examples for various open models, such as Llama-3, Mixtral, etc., can be found in [SkyPilot AI gallery](https://skypilot.readthedocs.io/en/latest/gallery/index.html). ## Prerequisites[¶](#prerequisites "Permanent link") - Go to the [HuggingFace model page](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and request access to the model `meta-llama/Meta-Llama-3-8B-Instruct`. - Check that you have installed SkyPilot ([docs](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html)). - Check that `sky check` shows clouds or Kubernetes are enabled. `[](#__codelineno-0-1)pip install skypilot-nightly [](#__codelineno-0-2)sky check` ## Run on a single instance[¶](#run-on-a-single-instance "Permanent link") See the vLLM SkyPilot YAML for serving, [serving.yaml](https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve.yaml). Yaml ``[](#__codelineno-1-1)resources: [](#__codelineno-1-2) accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model. [](#__codelineno-1-3) use_spot: True [](#__codelineno-1-4) disk_size: 512 # Ensure model checkpoints can fit. [](#__codelineno-1-5) disk_tier: best [](#__codelineno-1-6) ports: 8081 # Expose to internet traffic. [](#__codelineno-1-7)[](#__codelineno-1-8)envs: [](#__codelineno-1-9) PYTHONUNBUFFERED: 1 [](#__codelineno-1-10) MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct [](#__codelineno-1-11) HF_TOKEN: # Change to your own huggingface token, or use --env to pass. [](#__codelineno-1-12)[](#__codelineno-1-13)setup: | [](#__codelineno-1-14) conda create -n vllm python=3.10 -y [](#__codelineno-1-15) conda activate vllm [](#__codelineno-1-16) [](#__codelineno-1-17) pip install vllm==0.4.0.post1 [](#__codelineno-1-18) # Install Gradio for web UI. [](#__codelineno-1-19) pip install gradio openai [](#__codelineno-1-20) pip install flash-attn==2.5.7 [](#__codelineno-1-21)[](#__codelineno-1-22)run: | [](#__codelineno-1-23) conda activate vllm [](#__codelineno-1-24) echo 'Starting vllm api server...' [](#__codelineno-1-25) vllm serve $MODEL_NAME \ [](#__codelineno-1-26) --port 8081 \ [](#__codelineno-1-27) --trust-remote-code \ [](#__codelineno-1-28) --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \ [](#__codelineno-1-29) 2>&1 | tee api_server.log & [](#__codelineno-1-30) [](#__codelineno-1-31) echo 'Waiting for vllm api server to start...' [](#__codelineno-1-32) while ! `cat api_server.log | grep -q 'Uvicorn running on'`; do sleep 1; done [](#__codelineno-1-33) [](#__codelineno-1-34) echo 'Starting gradio server...' [](#__codelineno-1-35) git clone https://github.com/vllm-project/vllm.git || true [](#__codelineno-1-36) python vllm/examples/applications/chatbot/gradio_openai_chatbot_webserver.py \ [](#__codelineno-1-37) -m $MODEL_NAME \ [](#__codelineno-1-38) --port 8811 \ [](#__codelineno-1-39) --model-url http://localhost:8081/v1 \ [](#__codelineno-1-40) --stop-token-ids 128009,128001`` Start the serving the Llama-3 8B model on any of the candidate GPUs listed (L4, A10g, ...): `[](#__codelineno-2-1)HF_TOKEN="your-huggingface-token" sky launch serving.yaml --env HF_TOKEN` Check the output of the command. There will be a shareable gradio link (like the last line of the following). Open it in your browser to use the LLaMA model to do the text completion. `[](#__codelineno-3-1)(task, pid=7431) Running on public URL: https://.gradio.live` **Optional**: Serve the 70B model instead of the default 8B and use more GPU: `[](#__codelineno-4-1)HF_TOKEN="your-huggingface-token" \ [](#__codelineno-4-2) sky launch serving.yaml \ [](#__codelineno-4-3) --gpus A100:8 \ [](#__codelineno-4-4) --env HF_TOKEN \ [](#__codelineno-4-5) --env MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct` ## Scale up to multiple replicas[¶](#scale-up-to-multiple-replicas "Permanent link") SkyPilot can scale up the service to multiple service replicas with built-in autoscaling, load-balancing and fault-tolerance. You can do it by adding a services section to the YAML file. Yaml `[](#__codelineno-5-1)service: [](#__codelineno-5-2) replicas: 2 [](#__codelineno-5-3) # An actual request for readiness probe. [](#__codelineno-5-4) readiness_probe: [](#__codelineno-5-5) path: /v1/chat/completions [](#__codelineno-5-6) post_data: [](#__codelineno-5-7) model: $MODEL_NAME [](#__codelineno-5-8) messages: [](#__codelineno-5-9) - role: user [](#__codelineno-5-10) content: Hello! What is your name? [](#__codelineno-5-11) max_completion_tokens: 1` Yaml `[](#__codelineno-6-1)service: [](#__codelineno-6-2) replicas: 2 [](#__codelineno-6-3) # An actual request for readiness probe. [](#__codelineno-6-4) readiness_probe: [](#__codelineno-6-5) path: /v1/chat/completions [](#__codelineno-6-6) post_data: [](#__codelineno-6-7) model: $MODEL_NAME [](#__codelineno-6-8) messages: [](#__codelineno-6-9) - role: user [](#__codelineno-6-10) content: Hello! What is your name? [](#__codelineno-6-11) max_completion_tokens: 1 [](#__codelineno-6-12)[](#__codelineno-6-13)resources: [](#__codelineno-6-14) accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model. [](#__codelineno-6-15) use_spot: True [](#__codelineno-6-16) disk_size: 512 # Ensure model checkpoints can fit. [](#__codelineno-6-17) disk_tier: best [](#__codelineno-6-18) ports: 8081 # Expose to internet traffic. [](#__codelineno-6-19)[](#__codelineno-6-20)envs: [](#__codelineno-6-21) PYTHONUNBUFFERED: 1 [](#__codelineno-6-22) MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct [](#__codelineno-6-23) HF_TOKEN: # Change to your own huggingface token, or use --env to pass. [](#__codelineno-6-24)[](#__codelineno-6-25)setup: | [](#__codelineno-6-26) conda create -n vllm python=3.10 -y [](#__codelineno-6-27) conda activate vllm [](#__codelineno-6-28) [](#__codelineno-6-29) pip install vllm==0.4.0.post1 [](#__codelineno-6-30) # Install Gradio for web UI. [](#__codelineno-6-31) pip install gradio openai [](#__codelineno-6-32) pip install flash-attn==2.5.7 [](#__codelineno-6-33)[](#__codelineno-6-34)run: | [](#__codelineno-6-35) conda activate vllm [](#__codelineno-6-36) echo 'Starting vllm api server...' [](#__codelineno-6-37) vllm serve $MODEL_NAME \ [](#__codelineno-6-38) --port 8081 \ [](#__codelineno-6-39) --trust-remote-code \ [](#__codelineno-6-40) --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \ [](#__codelineno-6-41) 2>&1 | tee api_server.log` Start the serving the Llama-3 8B model on multiple replicas: `[](#__codelineno-7-1)HF_TOKEN="your-huggingface-token" \ [](#__codelineno-7-2) sky serve up -n vllm serving.yaml \ [](#__codelineno-7-3) --env HF_TOKEN` Wait until the service is ready: `[](#__codelineno-8-1)watch -n10 sky serve status vllm` Example outputs: `[](#__codelineno-9-1)Services [](#__codelineno-9-2)NAME VERSION UPTIME STATUS REPLICAS ENDPOINT [](#__codelineno-9-3)vllm 1 35s READY 2/2 xx.yy.zz.100:30001 [](#__codelineno-9-4)[](#__codelineno-9-5)Service Replicas [](#__codelineno-9-6)SERVICE_NAME ID VERSION IP LAUNCHED RESOURCES STATUS REGION [](#__codelineno-9-7)vllm 1 1 xx.yy.zz.121 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4 [](#__codelineno-9-8)vllm 2 1 xx.yy.zz.245 18 mins ago 1x GCP([Spot]{'L4': 1}) READY us-east4` After the service is READY, you can find a single endpoint for the service and access the service with the endpoint: Commands `[](#__codelineno-10-1)ENDPOINT=$(sky serve status --endpoint 8081 vllm) [](#__codelineno-10-2)curl -L http://$ENDPOINT/v1/chat/completions \ [](#__codelineno-10-3) -H "Content-Type: application/json" \ [](#__codelineno-10-4) -d '{ [](#__codelineno-10-5) "model": "meta-llama/Meta-Llama-3-8B-Instruct", [](#__codelineno-10-6) "messages": [ [](#__codelineno-10-7) { [](#__codelineno-10-8) "role": "system", [](#__codelineno-10-9) "content": "You are a helpful assistant." [](#__codelineno-10-10) }, [](#__codelineno-10-11) { [](#__codelineno-10-12) "role": "user", [](#__codelineno-10-13) "content": "Who are you?" [](#__codelineno-10-14) } [](#__codelineno-10-15) ], [](#__codelineno-10-16) "stop_token_ids": [128009, 128001] [](#__codelineno-10-17) }'` To enable autoscaling, you could replace the `replicas` with the following configs in `service`: `[](#__codelineno-11-1)service: [](#__codelineno-11-2) replica_policy: [](#__codelineno-11-3) min_replicas: 2 [](#__codelineno-11-4) max_replicas: 4 [](#__codelineno-11-5) target_qps_per_replica: 2` This will scale the service up to when the QPS exceeds 2 for each replica. Yaml `[](#__codelineno-12-1)service: [](#__codelineno-12-2) replica_policy: [](#__codelineno-12-3) min_replicas: 2 [](#__codelineno-12-4) max_replicas: 4 [](#__codelineno-12-5) target_qps_per_replica: 2 [](#__codelineno-12-6) # An actual request for readiness probe. [](#__codelineno-12-7) readiness_probe: [](#__codelineno-12-8) path: /v1/chat/completions [](#__codelineno-12-9) post_data: [](#__codelineno-12-10) model: $MODEL_NAME [](#__codelineno-12-11) messages: [](#__codelineno-12-12) - role: user [](#__codelineno-12-13) content: Hello! What is your name? [](#__codelineno-12-14) max_completion_tokens: 1 [](#__codelineno-12-15)[](#__codelineno-12-16)resources: [](#__codelineno-12-17) accelerators: {L4, A10g, A10, L40, A40, A100, A100-80GB} # We can use cheaper accelerators for 8B model. [](#__codelineno-12-18) use_spot: True [](#__codelineno-12-19) disk_size: 512 # Ensure model checkpoints can fit. [](#__codelineno-12-20) disk_tier: best [](#__codelineno-12-21) ports: 8081 # Expose to internet traffic. [](#__codelineno-12-22)[](#__codelineno-12-23)envs: [](#__codelineno-12-24) PYTHONUNBUFFERED: 1 [](#__codelineno-12-25) MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct [](#__codelineno-12-26) HF_TOKEN: # Change to your own huggingface token, or use --env to pass. [](#__codelineno-12-27)[](#__codelineno-12-28)setup: | [](#__codelineno-12-29) conda create -n vllm python=3.10 -y [](#__codelineno-12-30) conda activate vllm [](#__codelineno-12-31) [](#__codelineno-12-32) pip install vllm==0.4.0.post1 [](#__codelineno-12-33) # Install Gradio for web UI. [](#__codelineno-12-34) pip install gradio openai [](#__codelineno-12-35) pip install flash-attn==2.5.7 [](#__codelineno-12-36)[](#__codelineno-12-37)run: | [](#__codelineno-12-38) conda activate vllm [](#__codelineno-12-39) echo 'Starting vllm api server...' [](#__codelineno-12-40) vllm serve $MODEL_NAME \ [](#__codelineno-12-41) --port 8081 \ [](#__codelineno-12-42) --trust-remote-code \ [](#__codelineno-12-43) --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \ [](#__codelineno-12-44) 2>&1 | tee api_server.log` To update the service with the new config: `[](#__codelineno-13-1)HF_TOKEN="your-huggingface-token" sky serve update vllm serving.yaml --env HF_TOKEN` To stop the service: `[](#__codelineno-14-1)sky serve down vllm` ### **Optional**: Connect a GUI to the endpoint[¶](#optional-connect-a-gui-to-the-endpoint "Permanent link") It is also possible to access the Llama-3 service with a separate GUI frontend, so the user requests send to the GUI will be load-balanced across replicas. Yaml `[](#__codelineno-15-1)envs: [](#__codelineno-15-2) MODEL_NAME: meta-llama/Meta-Llama-3-8B-Instruct [](#__codelineno-15-3) ENDPOINT: x.x.x.x:3031 # Address of the API server running vllm. [](#__codelineno-15-4)[](#__codelineno-15-5)resources: [](#__codelineno-15-6) cpus: 2 [](#__codelineno-15-7)[](#__codelineno-15-8)setup: | [](#__codelineno-15-9) conda create -n vllm python=3.10 -y [](#__codelineno-15-10) conda activate vllm [](#__codelineno-15-11) [](#__codelineno-15-12) # Install Gradio for web UI. [](#__codelineno-15-13) pip install gradio openai [](#__codelineno-15-14)[](#__codelineno-15-15)run: | [](#__codelineno-15-16) conda activate vllm [](#__codelineno-15-17) export PATH=$PATH:/sbin [](#__codelineno-15-18) [](#__codelineno-15-19) echo 'Starting gradio server...' [](#__codelineno-15-20) git clone https://github.com/vllm-project/vllm.git || true [](#__codelineno-15-21) python vllm/examples/applications/api_client/gradio_openai_chatbot_webserver.py \ [](#__codelineno-15-22) -m $MODEL_NAME \ [](#__codelineno-15-23) --port 8811 \ [](#__codelineno-15-24) --model-url http://$ENDPOINT/v1 \ [](#__codelineno-15-25) --stop-token-ids 128009,128001 | tee ~/gradio.log` 1. Start the chat web UI: `[](#__codelineno-16-1)sky launch \ [](#__codelineno-16-2) -c gui ./gui.yaml \ [](#__codelineno-16-3) --env ENDPOINT=$(sky serve status --endpoint vllm)` 2. Then, we can access the GUI at the returned gradio link: `[](#__codelineno-17-1)| INFO | stdout | Running on public URL: https://6141e84201ce0bb4ed.gradio.live` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/streamlit.md "Edit this page") [Streamlit](https://github.com/streamlit/streamlit) lets you transform Python scripts into interactive web apps in minutes, instead of weeks. Build dashboards, generate reports, or create chat apps. It can be quickly integrated with vLLM as a backend API server, enabling powerful LLM inference via API calls. ## Prerequisites[¶](#prerequisites "Permanent link") Set up the vLLM environment by installing all required packages: `[](#__codelineno-0-1)pip install vllm streamlit openai` ## Deploy[¶](#deploy "Permanent link") 1. Start the vLLM server with a supported chat completion model, e.g. `[](#__codelineno-1-1)vllm serve Qwen/Qwen1.5-0.5B-Chat` 2. Use the script: [examples/applications/chatbot/streamlit\_openai\_chatbot\_webserver.py](https://github.com/vllm-project/vllm/blob/main/examples/applications/chatbot/streamlit_openai_chatbot_webserver.py) 3. Start the streamlit web UI and start to chat: `[](#__codelineno-2-1)streamlit run streamlit_openai_chatbot_webserver.py [](#__codelineno-2-2)[](#__codelineno-2-3)# or specify the VLLM_API_BASE or VLLM_API_KEY [](#__codelineno-2-4)VLLM_API_BASE="http://vllm-server-host:vllm-server-port/v1" \ [](#__codelineno-2-5) streamlit run streamlit_openai_chatbot_webserver.py [](#__codelineno-2-6)[](#__codelineno-2-7)# start with debug mode to view more details [](#__codelineno-2-8)streamlit run streamlit_openai_chatbot_webserver.py --logger.level=debug` [![Chat with vLLM assistant in Streamlit](https://docs.vllm.ai/en/assets/deployment/streamlit-chat.png)](https://docs.vllm.ai/en/assets/deployment/streamlit-chat.png) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/frameworks/triton.md "Edit this page") The [Triton Inference Server](https://github.com/triton-inference-server) hosts a tutorial demonstrating how to quickly deploy a simple [facebook/opt-125m](https://huggingface.co/facebook/opt-125m) model using vLLM. Please see [Deploying a vLLM model in Triton](https://github.com/triton-inference-server/tutorials/blob/main/Quick_Deploy/vLLM/README.md#deploying-a-vllm-model-in-triton) for more details. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/aibrix.md "Edit this page") [AIBrix](https://github.com/vllm-project/aibrix) is a cloud-native control plane that integrates with vLLM to simplify Kubernetes deployment, scaling, routing, and LoRA adapter management for large language model inference. For installation and usage instructions, please refer to the [AIBrix documentation](https://aibrix.readthedocs.io/). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/dynamo.md "Edit this page") [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo) is an open-source framework for distributed LLM inference that can run vLLM on Kubernetes with flexible serving architectures (e.g. aggregated/disaggregated, optional router/planner). For Kubernetes deployment instructions and examples (including vLLM), see the [Deploying Dynamo on Kubernetes](https://github.com/ai-dynamo/dynamo/blob/main/docs/kubernetes/README.md) guide. Background reading: InfoQ news coverage — [NVIDIA Dynamo simplifies Kubernetes deployment for LLM inference](https://www.infoq.com/news/2025/12/nvidia-dynamo-kubernetes/). --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Deployment](https://docs.vllm.ai/en/latest/docker/) 4. [Integrations](https://docs.vllm.ai/en/latest/deployment/aibrix/) [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/kaito.md "Edit this page") [KAITO](https://kaito-project.github.io/kaito/docs/) is a Kubernetes operator that supports deploying and serving LLMs with vLLM. It offers managing large models via container images with built-in OpenAI-compatible inference, auto-provisioning GPU nodes and curated model presets. Please refer to [quick start](https://kaito-project.github.io/kaito/docs/quick-start) for more details. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/kserve.md "Edit this page") vLLM can be deployed with [KServe](https://github.com/kserve/kserve) on Kubernetes for highly scalable distributed model serving. You can use vLLM with KServe's [Hugging Face serving runtime](https://kserve.github.io/website/docs/model-serving/generative-inference/overview) or via [`LLMInferenceService` that uses llm-d](https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/kthena.md "Edit this page") [**Kthena**](https://github.com/volcano-sh/kthena) is a Kubernetes-native LLM inference platform that transforms how organizations deploy and manage Large Language Models in production. Built with declarative model lifecycle management and intelligent request routing, it provides high performance and enterprise-grade scalability for LLM inference workloads. This guide shows how to deploy a production-grade, **multi-node vLLM** service on Kubernetes. We’ll: - Install the required components (Kthena + Volcano). - Deploy a multi-node vLLM model via Kthena’s `ModelServing` CR. - Validate the deployment. * * * ## 1\. Prerequisites[¶](#1-prerequisites "Permanent link") You need: - A Kubernetes cluster with **GPU nodes**. - `kubectl` access with cluster-admin or equivalent permissions. - **Volcano** installed for gang scheduling. - **Kthena** installed with the `ModelServing` CRD available. - A valid **Hugging Face token** if loading models from Hugging Face Hub. ### 1.1 Install Volcano[¶](#11-install-volcano "Permanent link") `[](#__codelineno-0-1)helm repo add volcano-sh https://volcano-sh.github.io/helm-charts [](#__codelineno-0-2)helm repo update [](#__codelineno-0-3)helm install volcano volcano-sh/volcano -n volcano-system --create-namespace` This provides the gang-scheduling and network topology features used by Kthena. ### 1.2 Install Kthena[¶](#12-install-kthena "Permanent link") `[](#__codelineno-1-1)helm install kthena oci://ghcr.io/volcano-sh/charts/kthena --version v0.1.0 --namespace kthena-system --create-namespace` - The `kthena-system` namespace is created. - Kthena controllers and CRDs, including `ModelServing`, are installed and healthy. Validate: `[](#__codelineno-2-1)kubectl get crd | grep modelserving` You should see: `[](#__codelineno-3-1)modelservings.workload.serving.volcano.sh ...` * * * ## 2\. The Multi-Node vLLM `ModelServing` Example[¶](#2-the-multi-node-vllm-modelserving-example "Permanent link") Kthena provides an example manifest to deploy a **multi-node vLLM cluster running Llama**. Conceptually this is equivalent to the vLLM production stack Helm deployment, but expressed with `ModelServing`. A simplified version of the example (`llama-multinode`) looks like: - `spec.replicas: 1` – one `ServingGroup` (one logical model deployment). - `roles`: - `entryTemplate` – defines **leader** pods that run: - vLLM’s **multi-node cluster bootstrap script** (Ray cluster). - vLLM **OpenAI-compatible API server**. - `workerTemplate` – defines **worker** pods that join the leader’s Ray cluster. Key points from the example YAML: - **Image**: `vllm/vllm-openai:latest` (matches upstream vLLM images). - **Command** (leader): `[](#__codelineno-4-1)command: [](#__codelineno-4-2) - sh [](#__codelineno-4-3) - -c [](#__codelineno-4-4) - > [](#__codelineno-4-5) bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh leader --ray_cluster_size=2; [](#__codelineno-4-6) vllm serve meta-llama/Llama-3.1-405B-Instruct [](#__codelineno-4-7) --port 8080 [](#__codelineno-4-8) --tensor-parallel-size 8 [](#__codelineno-4-9) --pipeline-parallel-size 2` - **Command** (worker): `[](#__codelineno-5-1)command: [](#__codelineno-5-2) - sh [](#__codelineno-5-3) - -c [](#__codelineno-5-4) - > [](#__codelineno-5-5) bash /vllm-workspace/examples/ray_serving/multi-node-serving.sh worker --ray_address=$(ENTRY_ADDRESS)` * * * ## 3\. Deploying Multi-Node llama vLLM via Kthena[¶](#3-deploying-multi-node-llama-vllm-via-kthena "Permanent link") ### 3.1 Prepare the Manifest[¶](#31-prepare-the-manifest "Permanent link") **Recommended**: use a Secret instead of a raw env var: `[](#__codelineno-6-1)kubectl create secret generic hf-token \ [](#__codelineno-6-2) -n default \ [](#__codelineno-6-3) --from-literal=HUGGING_FACE_HUB_TOKEN=''` ### 3.2 Apply the `ModelServing`[¶](#32-apply-the-modelserving "Permanent link") `[](#__codelineno-7-1)cat <---`. The first number indicates `ServingGroup`. The second (`405b`) is the `Role`. The remaining indices identify the pod within the role. * * * ## 6\. Accessing the vLLM OpenAI-Compatible API[¶](#6-accessing-the-vllm-openai-compatible-api "Permanent link") Expose the entry via a Service: `[](#__codelineno-12-1)apiVersion: v1 [](#__codelineno-12-2)kind: Service [](#__codelineno-12-3)metadata: [](#__codelineno-12-4) name: llama-multinode-openai [](#__codelineno-12-5) namespace: default [](#__codelineno-12-6)spec: [](#__codelineno-12-7) selector: [](#__codelineno-12-8) modelserving.volcano.sh/name: llama-multinode [](#__codelineno-12-9) modelserving.volcano.sh/entry: "true" [](#__codelineno-12-10) # optionally further narrow to leader role if you label it [](#__codelineno-12-11) ports: [](#__codelineno-12-12) - name: http [](#__codelineno-12-13) port: 80 [](#__codelineno-12-14) targetPort: 8080 [](#__codelineno-12-15) type: ClusterIP` Port-forward from your local machine: `[](#__codelineno-13-1)kubectl port-forward svc/llama-multinode-openai 30080:80 -n default` Then: - List models: `[](#__codelineno-14-1)curl -s http://localhost:30080/v1/models` - Send a completion request (mirroring vLLM production stack docs): `[](#__codelineno-15-1)curl -X POST http://localhost:30080/v1/completions \ [](#__codelineno-15-2) -H "Content-Type: application/json" \ [](#__codelineno-15-3) -d '{ [](#__codelineno-15-4) "model": "meta-llama/Llama-3.1-405B-Instruct", [](#__codelineno-15-5) "prompt": "Once upon a time,", [](#__codelineno-15-6) "max_tokens": 10 [](#__codelineno-15-7) }'` You should see an OpenAI-style response from vLLM. * * * ## 7\. Clean Up[¶](#7-clean-up "Permanent link") To remove the deployment and its resources: `[](#__codelineno-16-1)kubectl delete modelserving llama-multinode -n default` If you’re done with the entire stack: `[](#__codelineno-17-1)helm uninstall kthena -n kthena-system # or your Kthena release name [](#__codelineno-17-2)helm uninstall volcano -n volcano-system` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/kubeai.md "Edit this page") [KubeAI](https://github.com/substratusai/kubeai) is a Kubernetes operator that enables you to deploy and manage AI models on Kubernetes. It provides a simple and scalable way to deploy vLLM in production. Functionality such as scale-from-zero, load based autoscaling, model caching, and much more is provided out of the box with zero external dependencies. Please see the Installation Guides for environment specific instructions: - [Any Kubernetes Cluster](https://www.kubeai.org/installation/any/) - [AKS](https://www.kubeai.org/installation/aks/) - [EKS](https://www.kubeai.org/installation/eks/) - [GKE](https://www.kubeai.org/installation/gke/) Once you have KubeAI installed, you can [configure text generation models](https://www.kubeai.org/how-to/configure-text-generation-models/) using vLLM. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/kuberay.md "Edit this page") [KubeRay](https://github.com/ray-project/kuberay) provides a Kubernetes-native way to run vLLM workloads on Ray clusters. A Ray cluster can be declared in YAML, and the operator then handles pod scheduling, networking configuration, restarts, and blue-green deployments — all while preserving the familiar Kubernetes experience. ## Why KubeRay instead of manual scripts?[¶](#why-kuberay-instead-of-manual-scripts "Permanent link") Feature Manual scripts KubeRay Cluster bootstrap Manually SSH into every node and run a script One command to create or update the whole cluster: `kubectl apply -f cluster.yaml` Autoscaling Manual Automatically patches CRDs for adjusting cluster size Upgrades Tear down & re-create manually Blue/green deployment updates supported Declarative config Bash flags & environment variables Git-ops-friendly YAML CRDs (RayCluster/RayService) Using KubeRay reduces the operational burden and simplifies integration of Ray + vLLM with existing Kubernetes workflows (CI/CD, secrets, storage classes, etc.). ## Learn more[¶](#learn-more "Permanent link") - ["Serve a Large Language Model using Ray Serve LLM on Kubernetes"](https://docs.ray.io/en/master/cluster/kubernetes/examples/rayserve-llm-example.html) - An end-to-end example of how to serve a model using vLLM, KubeRay, and Ray Serve. - [KubeRay documentation](https://docs.ray.io/en/latest/cluster/kubernetes/index.html) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/llamastack.md "Edit this page") vLLM is also available via [Llama Stack](https://github.com/llamastack/llama-stack). To install Llama Stack, run `[](#__codelineno-0-1)pip install llama-stack -q` ## Inference using OpenAI-Compatible API[¶](#inference-using-openai-compatible-api "Permanent link") Then start the Llama Stack server and configure it to point to your vLLM server with the following settings: `[](#__codelineno-1-1)inference: [](#__codelineno-1-2) - provider_id: vllm0 [](#__codelineno-1-3) provider_type: remote::vllm [](#__codelineno-1-4) config: [](#__codelineno-1-5) url: http://127.0.0.1:8000` Please refer to [this guide](https://llama-stack.readthedocs.io/en/latest/providers/inference/remote_vllm.html) for more details on this remote vLLM provider. ## Inference using Embedded vLLM[¶](#inference-using-embedded-vllm "Permanent link") An [inline provider](https://github.com/llamastack/llama-stack/tree/main/llama_stack/providers/inline/inference) is also available. This is a sample of configuration using that method: `[](#__codelineno-2-1)inference: [](#__codelineno-2-2) - provider_type: vllm [](#__codelineno-2-3) config: [](#__codelineno-2-4) model: Llama3.1-8B-Instruct [](#__codelineno-2-5) tensor_parallel_size: 4` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/llm-d.md "Edit this page") vLLM can be deployed with [llm-d](https://github.com/llm-d/llm-d), a Kubernetes-native distributed inference serving stack providing well-lit paths for anyone to serve large generative AI models at scale. It helps achieve the fastest "time to state-of-the-art (SOTA) performance" for key OSS models across most hardware accelerators and infrastructure providers. You can use vLLM with llm-d directly by following [the official guides](https://llm-d.ai/docs/guides) or via [KServe's LLMInferenceService](https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview). --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Deployment](https://docs.vllm.ai/en/latest/docker/) 4. [Integrations](https://docs.vllm.ai/en/latest/deployment/aibrix/) [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/llmaz.md "Edit this page") [llmaz](https://github.com/InftyAI/llmaz) is an easy-to-use and advanced inference platform for large language models on Kubernetes, aimed for production use. It uses vLLM as the default model serving backend. Please refer to the [Quick Start](https://github.com/InftyAI/llmaz?tab=readme-ov-file#quick-start) for more details. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/cuda_graphs_multimodal.md "Edit this page") The [CUDA Graphs](https://docs.vllm.ai/en/latest/cuda_graphs/) infrastructure in vLLM primarily targets the **decoder** (language model) forward pass. vLLM also supports capturing the **encoder** (vision transformer) forward pass as CUDA Graphs, independently from the decoder. This is based on [Pull Request #35963](https://github.com/vllm-project/vllm/pull/35963). Note Encoder CUDA Graphs are orthogonal to decoder CUDA Graphs — both can be enabled simultaneously. Encoder graphs capture the vision encoder execution (e.g., ViT in Qwen3-VL), while decoder graphs capture the language model execution as described in the [CUDA Graphs design document](https://docs.vllm.ai/en/latest/cuda_graphs/). ## Motivation[¶](#motivation "Permanent link") Vision encoder inference incurs CUDA kernel launch overhead on the host side. The overhead is more significant when the batch size is small or image size is small. Encoder CUDA Graphs eliminate this overhead by pre-capturing the full encoder forward pass at multiple token budget levels during model initialization, then replaying the appropriate graph at runtime. ## Design[¶](#design "Permanent link") The encoder CUDA Graph system uses a **budget-based capture/replay** strategy, managed by [EncoderCudaGraphManager](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph/#vllm.v1.worker.encoder_cudagraph.EncoderCudaGraphManager " EncoderCudaGraphManager"). The system contains the following core components: - [EncoderCudaGraphManager](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph/#vllm.v1.worker.encoder_cudagraph.EncoderCudaGraphManager " EncoderCudaGraphManager"): orchestrates capture, replay, greedy packing, and data-parallel execution for encoder CUDA Graphs. - [SupportsEncoderCudaGraph](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph " SupportsEncoderCudaGraph"): a runtime-checkable protocol that models implement to opt-in to encoder CUDA Graphs. - [EncoderItemSpec](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph_defs/#vllm.v1.worker.encoder_cudagraph_defs.EncoderItemSpec " EncoderItemSpec dataclass "): describes a single encoder input item (image or video) with its input size and output token count. - [BudgetGraphMetadata](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph/#vllm.v1.worker.encoder_cudagraph.BudgetGraphMetadata " BudgetGraphMetadata dataclass "): holds the captured CUDA Graph and its associated I/O buffers for a single token budget level. ### Budget-based graph capture[¶](#budget-based-graph-capture "Permanent link") Multiple CUDA Graphs are pre-captured at different **token budget** levels (e.g., `[2048, 4096, 8192, 13824]`). Each budget defines a fixed token capacity, and all budgets share the same maximum batch size (number of images). The [`BudgetGraphMetadata`](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph/#vllm.v1.worker.encoder_cudagraph.BudgetGraphMetadata " BudgetGraphMetadata dataclass ") for each level stores the graph along with pre-allocated input, metadata, and output buffers: `[](#__codelineno-0-1)@dataclass [](#__codelineno-0-2)class BudgetGraphMetadata: [](#__codelineno-0-3) token_budget: int [](#__codelineno-0-4) max_batch_size: int [](#__codelineno-0-5) max_frames_per_batch: int [](#__codelineno-0-6) graph: torch.cuda.CUDAGraph [](#__codelineno-0-7) input_buffers: dict[str, torch.Tensor] # e.g. pixel_values, embeddings, seq metadata [](#__codelineno-0-8) output_buffer: torch.Tensor # encoder hidden states` Budgets are auto-generated as power-of-2 levels from a model-provided range via `get_encoder_cudagraph_budget_range()`, with the maximum budget always included even if it does not fall on a power-of-2 boundary. Budgets can also be explicitly specified by the user via `encoder_cudagraph_token_budgets` in [`CompilationConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.CompilationConfig " CompilationConfig"). ### Greedy bin-packing at runtime[¶](#greedy-bin-packing-at-runtime "Permanent link") When a batch of images arrives, the manager sorts images by output token count (smallest first) and greedily packs as many images as possible into each sub-batch while staying within the **largest** token budget and the maximum batch size. Once a sub-batch is finalized (the next image would overflow either constraint), the manager finds the **smallest** budget that fits the sub-batch's total tokens and replays the corresponding CUDA Graph. This repeats until the batch is exhausted. Images that exceed all budgets fall back to eager execution. For each graph replay: 1. Call `prepare_encoder_cudagraph_replay_buffers()` to compute buffer values (including `pixel_values` and precomputed metadata) from actual batch inputs. 2. Zero the pre-allocated `input_buffers`, then slice-copy the replay values into them. 3. Replay the CUDA Graph. 4. Clone outputs from `output_buffer` (cloning is necessary since the buffer is reused across replays). ### Data-parallel support[¶](#data-parallel-support "Permanent link") When `mm_encoder_tp_mode="data"`, the manager distributes images across TP ranks using load-balanced assignment via `get_load_balance_assignment`, executes locally on each rank, then gathers results back in the original order via `tensor_model_parallel_all_gather`. ### Video inference support[¶](#video-inference-support "Permanent link") Following [Pull Request #35963](https://github.com/vllm-project/vllm/pull/35963) (ViT full CUDA graph support for image inference), [Pull Request #38061](https://github.com/vllm-project/vllm/pull/38061) extends the encoder CUDA graph framework to support video inference for Qwen3-VL. Previously, the CUDA graph capture/replay path only handled image inputs (`pixel_values` + `image_grid_thw`). Video inputs use different keys (`pixel_values_videos` + `video_grid_thw`) and require larger `cu_seqlens` buffers because each video item contributes multiple frames (`T` attention sequences). This PR generalizes the protocol and manager to handle both modalities through a single shared graph manager. Note Video CUDA graphs are automatically disabled when EVS (Efficient Video Sampling) pruning is enabled, since EVS makes the token count data-dependent and incompatible with CUDA graph capture. Mixed inputs (image+video) per prompt are also supported now. ## Model integration via [`SupportsEncoderCudaGraph`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph " SupportsEncoderCudaGraph")[¶](#model-integration-via-supportsencodercudagraph "Permanent link") Models opt-in to encoder CUDA Graphs by implementing the [SupportsEncoderCudaGraph](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph " SupportsEncoderCudaGraph") protocol. This protocol encapsulates all model-specific logic so that the manager remains model-agnostic. The protocol defines the following methods: - `get_encoder_cudagraph_config()` — returns static configuration (supported modalities, buffer keys, output hidden size, padding logics, max frames per video). - `get_encoder_cudagraph_budget_range(vllm_config)` — returns `(min_budget, max_budget)` for auto-inference of token budgets. - `get_encoder_cudagraph_item_specs(mm_kwargs)` — returns `list[EncoderItemSpec]` describing each item with its input size and output token count. Replaces the former three separate methods (`get_num_items`, `get_per_item_output_tokens`, `get_per_item_input_sizes`). - `select_encoder_cudagraph_items(mm_kwargs, indices)` — extracts a sub-batch of items by index, used during greedy packing and DP sharding. - `prepare_encoder_cudagraph_capture_inputs(...)` — creates dummy inputs for graph capture. Returns [`EncoderCudaGraphCaptureInputs`](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph_defs/#vllm.v1.worker.encoder_cudagraph_defs.EncoderCudaGraphCaptureInputs " EncoderCudaGraphCaptureInputs dataclass ") with a single `values: dict[str, torch.Tensor]` that contains all buffers to be recorded into the graph. - `prepare_encoder_cudagraph_replay_buffers(mm_kwargs, max_batch_size, max_frames_per_batch)` — computes buffer values from actual batch inputs. Returns [`EncoderCudaGraphReplayBuffers`](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph_defs/#vllm.v1.worker.encoder_cudagraph_defs.EncoderCudaGraphReplayBuffers " EncoderCudaGraphReplayBuffers dataclass ") with a `values` dict whose keys match `buffer_keys` in the config. - `encoder_cudagraph_forward(inputs: dict[str, torch.Tensor])` — forward pass accepting only fixed-shaped input tensors (the captured `values` dict). Called during both capture and replay. The `pixel_values` tensor is included in `inputs` alongside metadata buffers. - `encoder_eager_forward(mm_kwargs)` — fallback eager forward when no graph fits. - `postprocess_encoder_output(...)` — post-process encoder output, delegates to `scatter_output_slices` by default. Note The [`SupportsEncoderCudaGraph`](https://docs.vllm.ai/en/api/vllm/model_executor/models/interfaces/#vllm.model_executor.models.interfaces.SupportsEncoderCudaGraph " SupportsEncoderCudaGraph") protocol is designed to be model-agnostic. New vision encoder models can opt-in by implementing the protocol methods without modifying the manager. **Supported models:** Architecture Models CG for Image CG for Video `InternVLChatModel` `InternVL3.5`, `InternVL3`, `InternVL2.5`, `InternVL2` ✅︎ ✅︎ `Qwen2VLForConditionalGeneration` `Qwen2-VL` ✅︎ ✅︎ `Qwen2_5_VLForConditionalGeneration` `Qwen2.5-VL` ✅︎ ✅︎ `Qwen3VLForConditionalGeneration` `Qwen3-VL` ✅︎ ✅︎ `Qwen3_5ForConditionalGeneration` `Qwen3.5` ✅︎ ✅︎ `Step3VLForConditionalGeneration` `Step3-VL` ✅︎ ❌︎ Note Encoder CUDA Graphs have currently been tested with `--mm-encoder-attn-backend=FLASH_ATTN` and `--mm-encoder-attn-backend=FLASHINFER` on Blackwell GPUs. For Qwen2-VL and Qwen2.5-VL only FA2 and FA3 has been tested. ## Configuration[¶](#configuration "Permanent link") Three fields in [`CompilationConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.CompilationConfig " CompilationConfig") control encoder CUDA Graphs: - `cudagraph_mm_encoder` (`bool`, default `False`) — enable CUDA Graph capture for multimodal encoder. When enabled, captures the full encoder forward as a CUDA Graph for each token budget level. - `encoder_cudagraph_token_budgets` (`list[int]`, default `[]`) — token budget levels for capture. If empty (default), auto-inferred from model architecture as power-of-2 levels. User-provided values override auto-inference. - `encoder_cudagraph_max_vision_items_per_batch` (`int`, default `0`) — maximum number of images/videos per batch during capture. If 0 (default), auto-inferred as `max_budget // min_budget`. - `encoder_cudagraph_max_frames_per_batch` (`int`, default `None`) — maximum number of video frames per batch during capture. If `None` (default), auto-inferred as `encoder_cudagraph_max_vision_items_per_batch * max_frames_per_video` (`max_frames_per_video` is a model-specific value from [`EncoderCudaGraphConfig`](https://docs.vllm.ai/en/api/vllm/v1/worker/encoder_cudagraph_defs/#vllm.v1.worker.encoder_cudagraph_defs.EncoderCudaGraphConfig " EncoderCudaGraphConfig dataclass "), computed by `get_max_frames_per_video()` on the model). If we limit the video count per prompt to `0`, it will also be set to `0` (i.e., fall back to image-only mode). ## Usage guide[¶](#usage-guide "Permanent link") ### Image inference[¶](#image-inference "Permanent link") Enable encoder CUDA Graphs via `compilation_config`: `[](#__codelineno-1-1)vllm serve Qwen/Qwen3-VL-32B \ [](#__codelineno-1-2) --compilation-config '{"cudagraph_mm_encoder": true}'` With explicit budgets: `[](#__codelineno-2-1)vllm serve Qwen/Qwen3-VL-32B \ [](#__codelineno-2-2) --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], "encoder_cudagraph_max_vision_items_per_batch": 8}'` Python example: `[](#__codelineno-3-1)import vllm [](#__codelineno-3-2)[](#__codelineno-3-3)compilation_config = { [](#__codelineno-3-4) "cudagraph_mm_encoder": True, [](#__codelineno-3-5) # Optional: override auto-inferred budgets [](#__codelineno-3-6) # "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], [](#__codelineno-3-7) # "encoder_cudagraph_max_vision_items_per_batch": 8, [](#__codelineno-3-8)} [](#__codelineno-3-9)[](#__codelineno-3-10)model = vllm.LLM( [](#__codelineno-3-11) model="Qwen/Qwen3-VL-32B", [](#__codelineno-3-12) compilation_config=compilation_config, [](#__codelineno-3-13))` The manager tracks hit/miss statistics and logs them periodically. A "hit" means an image was processed via CUDA Graph replay; a "miss" means eager fallback (image exceeded all budgets). ### Video inference[¶](#video-inference "Permanent link") Enable encoder CUDA Graphs via `compilation_config`: `[](#__codelineno-4-1)vllm serve Qwen/Qwen3-VL-32B \ [](#__codelineno-4-2) --compilation-config '{"cudagraph_mm_encoder": true}'` With explicit budgets: `[](#__codelineno-5-1)vllm serve Qwen/Qwen3-VL-32B \ [](#__codelineno-5-2) --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], "encoder_cudagraph_max_vision_items_per_batch": 8, "encoder_cudagraph_max_frames_per_batch": 64}'` Python example: `[](#__codelineno-6-1)import vllm [](#__codelineno-6-2)[](#__codelineno-6-3)compilation_config = { [](#__codelineno-6-4) "cudagraph_mm_encoder": True, [](#__codelineno-6-5) # Optional: override auto-inferred budgets [](#__codelineno-6-6) # "encoder_cudagraph_token_budgets": [2048, 4096, 8192, 13824], [](#__codelineno-6-7) # "encoder_cudagraph_max_vision_items_per_batch": 8, [](#__codelineno-6-8) # "encoder_cudagraph_max_frames_per_batch": 64, [](#__codelineno-6-9)} [](#__codelineno-6-10)[](#__codelineno-6-11)model = vllm.LLM( [](#__codelineno-6-12) model="Qwen/Qwen3-VL-32B", [](#__codelineno-6-13) compilation_config=compilation_config, [](#__codelineno-6-14))` ## About the Performance[¶](#about-the-performance "Permanent link") The following benchmarks were run on Blackwell GPUs (GB200) using `vllm bench mm-processor`. See [#35963](https://github.com/vllm-project/vllm/pull/35963) for full details. ### Single GPU (1x GB200)[¶](#single-gpu-1x-gb200 "Permanent link") Model: `Qwen/Qwen3-VL-30B-A3B-Instruct`, dataset: `lmarena-ai/VisionArena-Chat` (3000 prompts, 300 warmup), `max_model_len=32768`. Backend Mean latency improvement P99 latency improvement FLASH\_ATTN +11.8% (5.13→4.52ms) +31.6% (9.16→6.26ms) FLASHINFER +19.6% (5.42→4.36ms) +40.3% (10.87→6.49ms) To reproduce: `[](#__codelineno-7-1)vllm bench mm-processor \ [](#__codelineno-7-2) --model Qwen/Qwen3-VL-30B-A3B-Instruct \ [](#__codelineno-7-3) --dataset-name hf --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-7-4) --num-prompts 3000 --num-warmups 300 \ [](#__codelineno-7-5) --max-model-len 32768 --seed 42 \ [](#__codelineno-7-6) --mm-encoder-attn-backend FLASH_ATTN \ [](#__codelineno-7-7) --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4864], "encoder_cudagraph_max_vision_items_per_batch": 8}'` ### Multi-GPU (4x GB200, TP=4, DP=4)[¶](#multi-gpu-4x-gb200-tp4-dp4 "Permanent link") Model: `Qwen/Qwen3-VL-32B-Instruct`, dataset: `random-mm` (1000 prompts, 200 warmup, 20 images/request at 336x336), `max_model_len=8192`. Backend Mean latency improvement P99 latency improvement FLASH\_ATTN +18.4% (28.39→23.16ms) +14.0% (238.78→205.28ms) FLASHINFER +44.4% (23.24→12.91ms) +84.9% (172.41→26.05ms) To reproduce: `[](#__codelineno-8-1)vllm bench mm-processor \ [](#__codelineno-8-2) --model Qwen/Qwen3-VL-32B-Instruct \ [](#__codelineno-8-3) --dataset-name random-mm \ [](#__codelineno-8-4) --random-mm-base-items-per-request 20 \ [](#__codelineno-8-5) --random-mm-num-mm-items-range-ratio 0.0 \ [](#__codelineno-8-6) --random-mm-bucket-config '{"(336,336,1)": 1.0}' \ [](#__codelineno-8-7) --num-prompts 1000 --num-warmups 200 \ [](#__codelineno-8-8) --max-model-len 8192 --seed 42 \ [](#__codelineno-8-9) --mm-encoder-attn-backend FLASHINFER \ [](#__codelineno-8-10) --tensor-parallel-size 4 --mm-encoder-tp-mode data \ [](#__codelineno-8-11) --compilation-config '{"cudagraph_mm_encoder": true, "encoder_cudagraph_token_budgets": [512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4864], "encoder_cudagraph_max_vision_items_per_batch": 8}'` Note Find more details about benchmarks on GPUs (A100) for video inference at [#38061](https://github.com/vllm-project/vllm/pull/38061). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/arch_overview.md "Edit this page") This document provides an overview of the vLLM architecture. - [Architecture Overview](#architecture-overview) - [Entrypoints](#entrypoints) - [LLM Class](#llm-class) - [Online Serving](#online-serving) - [V1 Process Architecture](#v1-process-architecture) - [API Server Process](#api-server-process) - [Engine Core Process](#engine-core-process) - [GPU Worker Processes](#gpu-worker-processes) - [DP Coordinator Process (conditional)](#dp-coordinator-process-conditional) - [Process Count Summary](#process-count-summary) - [LLM Engine](#llm-engine) - [LLMEngine](#llmengine) - [AsyncLLMEngine](#asyncllmengine) - [Worker](#worker) - [Model Runner](#model-runner) - [Model](#model) - [Class Hierarchy](#class-hierarchy) ## Entrypoints[¶](#entrypoints "Permanent link") vLLM provides a number of entrypoints for interacting with the system. The following diagram shows the relationship between them. [![Entrypoints Diagram](https://docs.vllm.ai/en/assets/design/arch_overview/entrypoints.excalidraw.png)](https://docs.vllm.ai/en/assets/design/arch_overview/entrypoints.excalidraw.png) ### LLM Class[¶](#llm-class "Permanent link") The LLM class provides the primary Python interface for doing offline inference, which is interacting with a model without using a separate model inference server. Here is a sample of [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class usage: Code `[](#__codelineno-0-1)from vllm import LLM, SamplingParams [](#__codelineno-0-2)[](#__codelineno-0-3)# Define a list of input prompts [](#__codelineno-0-4)prompts = [ [](#__codelineno-0-5) "Hello, my name is", [](#__codelineno-0-6) "The capital of France is", [](#__codelineno-0-7) "The largest ocean is", [](#__codelineno-0-8)] [](#__codelineno-0-9)[](#__codelineno-0-10)# Define sampling parameters [](#__codelineno-0-11)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-0-12)[](#__codelineno-0-13)# Initialize the LLM engine with the OPT-125M model [](#__codelineno-0-14)llm = LLM(model="facebook/opt-125m") [](#__codelineno-0-15)[](#__codelineno-0-16)# Generate outputs for the input prompts [](#__codelineno-0-17)outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-18)[](#__codelineno-0-19)# Print the generated outputs [](#__codelineno-0-20)for output in outputs: [](#__codelineno-0-21) prompt = output.prompt [](#__codelineno-0-22) generated_text = output.outputs[0].text [](#__codelineno-0-23) print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")` More API details can be found in the [Offline Inference](https://docs.vllm.ai/en/api/#offline-inference) section of the API docs. The code for the [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class can be found in [vllm/entrypoints/llm.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/llm.py). ### Online Serving[¶](#online-serving "Permanent link") The second primary interface to vLLM is via its online server. This server can be started using the `vllm serve` command. `[](#__codelineno-1-1)vllm serve ` The code for the `vllm` CLI can be found in [vllm/entrypoints/cli/main.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/cli/main.py). Sometimes you may see the API server entrypoint used directly instead of via the `vllm` CLI command. For example: `[](#__codelineno-2-1)python -m vllm.entrypoints.openai.api_server --model ` Warning `python -m vllm.entrypoints.openai.api_server` is deprecated and may become unsupported in a future release. That code can be found in [vllm/entrypoints/openai/api\_server.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/api_server.py). More details on the API server can be found in the [Online Serving](https://docs.vllm.ai/en/serving/online_serving/) document. ## V1 Process Architecture[¶](#v1-process-architecture "Permanent link") vLLM V1 uses a multi-process architecture to separate concerns and maximize throughput. Understanding this architecture is important for properly sizing CPU resources in your deployment. The key processes are: ### API Server Process[¶](#api-server-process "Permanent link") The API server process handles HTTP requests (e.g., the OpenAI-compatible API), performs input processing (tokenization, multi-modal data loading), and streams results back to clients. It communicates with the engine core process(es) via ZMQ sockets. By default, there is **1 API server process**, but when data parallelism is used, the API server count automatically scales to match the data parallel size. This can also be manually configured with the `--api-server-count` flag. Each API server connects to **all** engine cores via ZMQ in a many-to-many topology, enabling any API server to route requests to any engine core. Each API server process uses multiple CPU threads for media loading (controlled by `VLLM_MEDIA_LOADING_THREAD_COUNT`, default 8). The code can be found in [vllm/entrypoints/openai/api\_server.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/api_server.py) and [vllm/v1/utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/utils.py). ### Engine Core Process[¶](#engine-core-process "Permanent link") The engine core process runs the scheduler, manages KV cache, and coordinates model execution across GPU workers. It runs a busy loop that continuously schedules requests and dispatches work to the GPU workers. There is **1 engine core process per data parallel rank**. For example, with `--data-parallel-size 4`, there are 4 engine core processes. The code can be found in [vllm/v1/engine/core.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/engine/core.py) and [vllm/v1/engine/utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/engine/utils.py). ### GPU Worker Processes[¶](#gpu-worker-processes "Permanent link") Each GPU is managed by a dedicated worker process. The worker process loads model weights, executes forward passes, and manages GPU memory. Workers communicate with the engine core process that owns them. There is **1 worker process per GPU**. The total number of GPU worker processes equals `tensor_parallel_size x pipeline_parallel_size` per engine core. The code can be found in [vllm/v1/executor/multiproc\_executor.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/executor/multiproc_executor.py) and [vllm/v1/worker/gpu\_worker.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu_worker.py). ### DP Coordinator Process (conditional)[¶](#dp-coordinator-process-conditional "Permanent link") When using data parallelism (`--data-parallel-size > 1`), an additional coordinator process manages load balancing across DP ranks and coordinates synchronized forward passes for MoE models. There is **1 DP coordinator process** (only when data parallelism is enabled). The code can be found in [vllm/v1/engine/coordinator.py](https://github.com/vllm-project/vllm/blob/main/vllm/v1/engine/coordinator.py). ### Process Count Summary[¶](#process-count-summary "Permanent link") For a deployment with `N` GPUs, `TP` tensor parallel size, `DP` data parallel size, and `A` API server count: Process Type Count Notes API Server `A` (default `DP`) Handles HTTP requests and input processing Engine Core `DP` (default 1) Scheduler and KV cache management GPU Worker `N` (= `DP x PP x TP`) One per GPU, executes model forward passes DP Coordinator 1 if `DP > 1`, else 0 Load balancing across DP ranks **Total** **`A + DP + N` (+ 1 if DP > 1)** For example, a typical single-node deployment with 4 GPUs (`vllm serve -tp=4`) has: - 1 API server + 1 engine core + 4 GPU workers = **6 processes** [![V1 Process Architecture - TP=4](https://docs.vllm.ai/en/assets/design/arch_overview/v1_process_architecture_tp4.png)](https://docs.vllm.ai/en/assets/design/arch_overview/v1_process_architecture_tp4.png) A data parallel deployment with 8 GPUs (`vllm serve -tp=2 -dp=4`) has: - 4 API servers + 4 engine cores + 8 GPU workers + 1 DP coordinator = **17 processes** [![V1 Process Architecture - TP=2, DP=4](https://docs.vllm.ai/en/assets/design/arch_overview/v1_process_architecture_tp2_dp4.png)](https://docs.vllm.ai/en/assets/design/arch_overview/v1_process_architecture_tp2_dp4.png) For CPU resource sizing recommendations, see [CPU Resources for GPU Deployments](https://docs.vllm.ai/en/configuration/optimization/#cpu-resources-for-gpu-deployments). ## LLM Engine[¶](#llm-engine "Permanent link") The [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine") and `AsyncLLMEngine` classes are central to the functioning of the vLLM system, handling model inference and asynchronous request processing. [![LLMEngine Diagram](https://docs.vllm.ai/en/assets/design/arch_overview/llm_engine.excalidraw.png)](https://docs.vllm.ai/en/assets/design/arch_overview/llm_engine.excalidraw.png) ### LLMEngine[¶](#llmengine "Permanent link") The [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine") class is the core component of the vLLM engine. It is responsible for receiving requests from clients and generating outputs from the model. The [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine") includes input processing, model execution (possibly distributed across multiple hosts and/or GPUs), scheduling, and output processing. - **Input Processing**: Handles tokenization of input text using the specified tokenizer. - **Scheduling**: Chooses which requests are processed in each step. - **Model Execution**: Manages the execution of the language model, including distributed execution across multiple GPUs. - **Output Processing**: Processes the outputs generated by the model, decoding the token IDs from a language model into human-readable text. The code for [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine") can be found in [vllm/engine/llm\_engine.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/llm_engine.py). ### AsyncLLMEngine[¶](#asyncllmengine "Permanent link") The `AsyncLLMEngine` class is an asynchronous wrapper for the [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine") class. It uses `asyncio` to create a background loop that continuously processes incoming requests. The `AsyncLLMEngine` is designed for online serving, where it can handle multiple concurrent requests and stream outputs to clients. The OpenAI-compatible API server uses the `AsyncLLMEngine`. There is also a demo API server that serves as a simpler example in [vllm/entrypoints/api\_server.py](https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/api_server.py). The code for `AsyncLLMEngine` can be found in [vllm/engine/async\_llm\_engine.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/async_llm_engine.py). ## Worker[¶](#worker "Permanent link") A worker is a process that runs the model inference. vLLM follows the common practice of using one process to control one accelerator device, such as GPUs. For example, if we use tensor parallelism of size 2 and pipeline parallelism of size 2, we will have 4 workers in total. Workers are identified by their `rank` and `local_rank`. `rank` is used for global orchestration, while `local_rank` is mainly used for assigning the accelerator device and accessing local resources such as the file system and shared memory. ## Model Runner[¶](#model-runner "Permanent link") Every worker has one model runner object, responsible for loading and running the model. Much of the model execution logic resides here, such as preparing input tensors and capturing cudagraphs. ## Model[¶](#model "Permanent link") Every model runner object has one model object, which is the actual `torch.nn.Module` instance. See [huggingface\_integration](https://docs.vllm.ai/en/latest/huggingface_integration/) for how various configurations affect the class we ultimately get. ## Class Hierarchy[¶](#class-hierarchy "Permanent link") The following figure shows the class hierarchy of vLLM: [![Class Hierarchy](https://docs.vllm.ai/en/assets/design/hierarchy.png)](https://docs.vllm.ai/en/assets/design/hierarchy.png) There are several important design choices behind this class hierarchy: 1\. **Extensibility**: All classes in the hierarchy accept a configuration object containing all the necessary information. The [VllmConfig](https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/config.py#L2036) class is the main configuration object that is passed around. The class hierarchy is quite deep, and every class needs to read the configuration it is interested in. By encapsulating all configurations in one object, we can easily pass the configuration object around and access the configuration we need. Suppose we want to add a new feature (this is often the case given how fast the field of LLM inference is evolving) that only touches the model runner. We will have to add a new configuration option in the [`VllmConfig`](https://docs.vllm.ai/en/api/vllm/config/vllm/#vllm.config.vllm.VllmConfig " VllmConfig") class. Since we pass the whole config object around, we only need to add the configuration option to the [`VllmConfig`](https://docs.vllm.ai/en/api/vllm/config/vllm/#vllm.config.vllm.VllmConfig " VllmConfig") class, and the model runner can access it directly. We don't need to change the constructor of the engine, worker, or model class to pass the new configuration option. 2\. **Uniformity**: The model runner needs a unified interface to create and initialize the model. vLLM supports more than 50 types of popular open-source models. Each model has its own initialization logic. If the constructor signature varies with models, the model runner does not know how to call the constructor accordingly, without complicated and error-prone inspection logic. By making the constructor of the model class uniform, the model runner can easily create and initialize the model without knowing the specific model type. This is also useful for composing models. Vision-language models often consist of a vision model and a language model. By making the constructor uniform, we can easily create a vision model and a language model and compose them into a vision-language model. Note To support this change, all vLLM models' signatures have been updated to: `[](#__codelineno-3-1)def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""):` To avoid accidentally passing incorrect arguments, the constructor is now keyword-only. This ensures that the constructor will raise an error if old configurations are passed. vLLM developers have already made this change for all models within vLLM. For out-of-tree registered models, developers need to update their models, for example by adding shim code to adapt the old constructor signature to the new one: Code `[](#__codelineno-4-1)class MyOldModel(nn.Module): [](#__codelineno-4-2) def __init__( [](#__codelineno-4-3) self, [](#__codelineno-4-4) config, [](#__codelineno-4-5) cache_config: Optional[CacheConfig] = None, [](#__codelineno-4-6) quant_config: Optional[QuantizationConfig] = None, [](#__codelineno-4-7) lora_config: Optional[LoRAConfig] = None, [](#__codelineno-4-8) prefix: str = "", [](#__codelineno-4-9) ) -> None: [](#__codelineno-4-10) ... [](#__codelineno-4-11)[](#__codelineno-4-12)from vllm.config import VllmConfig [](#__codelineno-4-13)class MyNewModel(MyOldModel): [](#__codelineno-4-14) def __init__(self, *, vllm_config: VllmConfig, prefix: str = ""): [](#__codelineno-4-15) config = vllm_config.model_config.hf_config [](#__codelineno-4-16) cache_config = vllm_config.cache_config [](#__codelineno-4-17) quant_config = vllm_config.quant_config [](#__codelineno-4-18) lora_config = vllm_config.lora_config [](#__codelineno-4-19) super().__init__(config, cache_config, quant_config, lora_config, prefix) [](#__codelineno-4-20)[](#__codelineno-4-21)from packaging import version [](#__codelineno-4-22)if version.parse(__version__) >= version.parse("0.6.4"): [](#__codelineno-4-23) MyModel = MyNewModel [](#__codelineno-4-24)else: [](#__codelineno-4-25) MyModel = MyOldModel` This way, the model can work with both old and new versions of vLLM. 3\. **Sharding and Quantization at Initialization**: Certain features require changing the model weights. For example, tensor parallelism needs to shard the model weights, and quantization needs to quantize the model weights. There are two possible ways to implement this feature. One way is to change the model weights after the model is initialized. The other way is to change the model weights during the model initialization. vLLM chooses the latter. The first approach is not scalable to large models. Suppose we want to run a 405B model (with roughly 810GB weights) with 16 H100 80GB GPUs. Ideally, every GPU should only load 50GB weights. If we change the model weights after the model is initialized, we need to load the full 810GB weights to every GPU and then shard the weights, leading to a huge memory overhead. Instead, if we shard the weights during the model initialization, every layer will only create a shard of the weights it needs, leading to a much smaller memory overhead. The same idea applies to quantization. Note that we also add an additional argument `prefix` to the model's constructor so that the model can initialize itself differently based on the prefix. This is useful for non-uniform quantization, where different parts of the model are quantized differently. The `prefix` is usually an empty string for the top-level model and a string like `"vision"` or `"language"` for the sub-models. In general, it matches the name of the module's state dict in the checkpoint file. One disadvantage of this design is that it is hard to write unit tests for individual components in vLLM because every component needs to be initialized by a complete config object. We solve this problem by providing a default initialization function that creates a default config object with all fields set to `None`. If the component we want to test only cares about a few fields in the config object, we can create a default config object and set the fields we care about. This way, we can test the component in isolation. Note that many tests in vLLM are end-to-end tests that test the whole system, so this is not a big problem. In summary, the complete config object [`VllmConfig`](https://docs.vllm.ai/en/api/vllm/config/vllm/#vllm.config.vllm.VllmConfig " VllmConfig") can be treated as an engine-level global state that is shared among all vLLM classes. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/deployment/integrations/production-stack.md "Edit this page") Deploying vLLM on Kubernetes is a scalable and efficient way to serve machine learning models. This guide walks you through deploying vLLM using the [vLLM production stack](https://github.com/vllm-project/production-stack). Born out of a Berkeley-UChicago collaboration, [vLLM production stack](https://github.com/vllm-project/production-stack) is an officially released, production-optimized codebase under the [vLLM project](https://github.com/vllm-project), designed for LLM deployment with: - **Upstream vLLM compatibility** – It wraps around upstream vLLM without modifying its code. - **Ease of use** – Simplified deployment via Helm charts and observability through Grafana dashboards. - **High performance** – Optimized for LLM workloads with features like multimodel support, model-aware and prefix-aware routing, fast vLLM bootstrapping, and KV cache offloading with [LMCache](https://github.com/LMCache/LMCache), among others. If you are new to Kubernetes, don't worry: in the vLLM production stack [repo](https://github.com/vllm-project/production-stack), we provide a step-by-step [guide](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) and a [short video](https://www.youtube.com/watch?v=EsTJbQtzj0g) to set up everything and get started in **4 minutes**! ## Pre-requisite[¶](#pre-requisite "Permanent link") Ensure that you have a running Kubernetes environment with GPU (you can follow [this tutorial](https://github.com/vllm-project/production-stack/blob/main/tutorials/00-install-kubernetes-env.md) to install a Kubernetes environment on a bare-metal GPU machine). ## Deployment using vLLM production stack[¶](#deployment-using-vllm-production-stack "Permanent link") The standard vLLM production stack is installed using a Helm chart. You can run this [bash script](https://github.com/vllm-project/production-stack/blob/main/utils/install-helm.sh) to install Helm on your GPU server. To install the vLLM production stack, run the following commands on your desktop: `[](#__codelineno-0-1)sudo helm repo add vllm https://vllm-project.github.io/production-stack [](#__codelineno-0-2)sudo helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml` This will instantiate a vLLM-production-stack-based deployment named `vllm` that runs a small LLM (Facebook opt-125M model). ### Validate Installation[¶](#validate-installation "Permanent link") Monitor the deployment status using: `[](#__codelineno-1-1)sudo kubectl get pods` And you will see that pods for the `vllm` deployment will transit to `Running` state. `[](#__codelineno-2-1)NAME READY STATUS RESTARTS AGE [](#__codelineno-2-2)vllm-deployment-router-859d8fb668-2x2b7 1/1 Running 0 2m38s [](#__codelineno-2-3)vllm-opt125m-deployment-vllm-84dfc9bd7-vb9bs 1/1 Running 0 2m38s` Note It may take some time for the containers to download the Docker images and LLM weights. ### Send a Query to the Stack[¶](#send-a-query-to-the-stack "Permanent link") Forward the `vllm-router-service` port to the host machine: `[](#__codelineno-3-1)sudo kubectl port-forward svc/vllm-router-service 30080:80` And then you can send out a query to the OpenAI-compatible API to check the available models: `[](#__codelineno-4-1)curl -o- http://localhost:30080/v1/models` Output `[](#__codelineno-5-1){ [](#__codelineno-5-2) "object": "list", [](#__codelineno-5-3) "data": [ [](#__codelineno-5-4) { [](#__codelineno-5-5) "id": "facebook/opt-125m", [](#__codelineno-5-6) "object": "model", [](#__codelineno-5-7) "created": 1737428424, [](#__codelineno-5-8) "owned_by": "vllm", [](#__codelineno-5-9) "root": null [](#__codelineno-5-10) } [](#__codelineno-5-11) ] [](#__codelineno-5-12)}` To send an actual chatting request, you can issue a curl request to the OpenAI `/completion` endpoint: `[](#__codelineno-6-1)curl -X POST http://localhost:30080/v1/completions \ [](#__codelineno-6-2) -H "Content-Type: application/json" \ [](#__codelineno-6-3) -d '{ [](#__codelineno-6-4) "model": "facebook/opt-125m", [](#__codelineno-6-5) "prompt": "Once upon a time,", [](#__codelineno-6-6) "max_tokens": 10 [](#__codelineno-6-7) }'` Output `[](#__codelineno-7-1){ [](#__codelineno-7-2) "id": "completion-id", [](#__codelineno-7-3) "object": "text_completion", [](#__codelineno-7-4) "created": 1737428424, [](#__codelineno-7-5) "model": "facebook/opt-125m", [](#__codelineno-7-6) "choices": [ [](#__codelineno-7-7) { [](#__codelineno-7-8) "text": " there was a brave knight who...", [](#__codelineno-7-9) "index": 0, [](#__codelineno-7-10) "finish_reason": "length" [](#__codelineno-7-11) } [](#__codelineno-7-12) ] [](#__codelineno-7-13)}` ### Uninstall[¶](#uninstall "Permanent link") To remove the deployment, run: `[](#__codelineno-8-1)sudo helm uninstall vllm` * * * ### (Advanced) Configuring vLLM production stack[¶](#advanced-configuring-vllm-production-stack "Permanent link") The core vLLM production stack configuration is managed with YAML. Here is the example configuration used in the installation above: Yaml `[](#__codelineno-9-1)servingEngineSpec: [](#__codelineno-9-2) runtimeClassName: "" [](#__codelineno-9-3) modelSpec: [](#__codelineno-9-4) - name: "opt125m" [](#__codelineno-9-5) repository: "vllm/vllm-openai" [](#__codelineno-9-6) tag: "latest" [](#__codelineno-9-7) modelURL: "facebook/opt-125m" [](#__codelineno-9-8) [](#__codelineno-9-9) replicaCount: 1 [](#__codelineno-9-10) [](#__codelineno-9-11) requestCPU: 6 [](#__codelineno-9-12) requestMemory: "16Gi" [](#__codelineno-9-13) requestGPU: 1 [](#__codelineno-9-14) [](#__codelineno-9-15) pvcStorage: "10Gi"` In this YAML configuration: - **`modelSpec`** includes: - `name`: A nickname that you prefer to call the model. - `repository`: Docker repository of vLLM. - `tag`: Docker image tag. - `modelURL`: The LLM model that you want to use. - **`replicaCount`**: Number of replicas. - **`requestCPU` and `requestMemory`**: Specifies the CPU and memory resource requests for the pod. - **`requestGPU`**: Specifies the number of GPUs required. - **`pvcStorage`**: Allocates persistent storage for the model. Note If you intend to set up two pods, please refer to this [YAML file](https://github.com/vllm-project/production-stack/blob/main/tutorials/assets/values-01-2pods-minimal-example.yaml). Tip vLLM production stack offers many more features (_e.g._ CPU offloading and a wide range of routing algorithms). Please check out these [examples and tutorials](https://github.com/vllm-project/production-stack/tree/main/tutorials) and our [repo](https://github.com/vllm-project/production-stack) for more details! --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/fusions.md "Edit this page") vLLM applies a set of kernel/operator fusions at compile time (via custom [`torch.compile`](https://docs.vllm.ai/en/latest/torch_compile/) Inductor passes) to separate optimizations from model definitions and avoid breaking layer abstractions in model code. These fusions are controlled by fields in [`PassConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.PassConfig " PassConfig") and are automatically enabled at appropriate [optimization levels](https://docs.vllm.ai/en/latest/optimization_levels/). ## Quick Reference[¶](#quick-reference "Permanent link") The table below maps each fusion to its controlling flag/config knob, the operations it fuses, what level enables it by default, and an indicative speedup. The Fullgraph column indicates whether the fusion requires the entire model graph to be visible (either via Inductor partition or `splitting_ops=[]`), and the last column indicates whether the fusion activates for all `num_tokens` or just on the low or high end. Info Speedup depends heavily on the exact model, batch size, and hardware. If tuning performance by hand, always benchmark your exact use-case with and without the fusion to verify the impact. Fusion [`PassConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.PassConfig " PassConfig") flag Fused operations Default at E2E Speedup Fullgraph `num_tokens` [AllReduce + RMSNorm](#allreduce--rmsnorm-fuse_allreduce_rms) `fuse_allreduce_rms` All-reduce → RMSNorm (+residual\_add) (→ quant) O2 (Hopper/Blackwell + TP > 1) 5-20% No Low [Attention + Quant](#attention--quantization-fuse_attn_quant) `fuse_attn_quant` Attention output → FP8/NVFP4 quant Off by default 3-7% Yes Always [MLA Attention + Quant](#attention--quantization-fuse_attn_quant) `fuse_attn_quant` MLA Attention output → FP8/NVFP4 quant Off by default TBD Yes Always [RoPE + KV-Cache Update](#rope--kv-cache-update-fuse_rope_kvcache) `fuse_rope_kvcache` Rotary embedding → KV cache write O2 (ROCm/AITER only) 2-4% No Low [QK Norm + RoPE](#qk-norm--rope-enable_qk_norm_rope_fusion) `enable_qk_norm_rope_fusion` Q/K RMSNorm → rotary embedding Off by default 2-3% No Low [Sequence Parallelism](#sequence-parallelism-enable_sp) `enable_sp` AllReduce → ReduceScatter + AllGather Off by default Prereq for AsyncTP Yes High [AsyncTP GEMM + collective](#asynctp-gemm--collective-overlap-fuse_gemm_comms) `fuse_gemm_comms` GEMM → reduce-scatter / all-gather → GEMM Off by default 7-10% Yes High [RMSNorm + Quant](#rmsnorm--quantization-fuse_norm_quant) `fuse_norm_quant` RMSNorm (+residual add) → FP8/FP4 quant O1 (conditional) 1-4% No Always [SiLU+Mul + Quant](#silumul--quantization-fuse_act_quant) `fuse_act_quant` SiLU+Mul activation → FP8/FP4 quant O1 (conditional) 1-4% No Always [RMSNorm + Padding](#rmsnorm--padding-fuse_act_padding) `fuse_act_padding` Residual add + RMSNorm → padding O1 (ROCm/AITER only) TBD No Always [MLA Dual RMSNorm](#mla-dual-rmsnorm-fuse_mla_dual_rms_norm) `fuse_mla_dual_rms_norm` Paired Q + KV RMSNorm → single kernel O1 (ROCm/AITER only) ~2% No Always ## Support Matrix[¶](#support-matrix "Permanent link") The table below lists the quantization schemes supported by each fusion on each platform. **—** means the fusion is not available on that platform. The latest and in-progress work is available in the tracking issue: [#36066](https://github.com/vllm-project/vllm/issues/36066) Fusion SM100 (Blackwell) SM90 (Hopper) SM89 (Ada) SM80 (Ampere) ROCm `fuse_allreduce_rms` FP16/BF16, FP8 static, NVFP4 FP16/BF16, FP8 static — — — `fuse_attn_quant`\* FP8 static\*, NVFP4\* FP8 static\* FP8 static\* — FP8 static\* `fuse_attn_quant` (MLA)\* FP8 static\*, FP8 per-group\*, NVFP4\* FP8 static\*, FP8 per-group\* FP8 static\*, FP8 per-group\* — FP8 static\* (untested) `fuse_rope_kvcache` — — — — FP16/BF16 `enable_qk_norm_rope_fusion` FP16/BF16 FP16/BF16 FP16/BF16† FP16/BF16† — `enable_sp` FP16/BF16, FP8 static† FP16/BF16, FP8 static FP16/BF16† FP16/BF16† — `fuse_gemm_comms` FP16/BF16, FP8 static† FP16/BF16, FP8 static FP16/BF16† FP16/BF16† — `fuse_norm_quant` FP8 static, FP8 per-token, FP8 per-group FP8 static, FP8 per-token, FP8 per-group FP8 static, FP8 per-token, FP8 per-group — FP8 static, FP8 per-token, FP8 per-group `fuse_act_quant` FP8 static, NVFP4 FP8 static, FP8 per-group (128/64) FP8 static, FP8 per-group (128/64) — FP8 per-group `fuse_act_padding` — — — — FP16/BF16 `fuse_mla_dual_rms_norm` — — — — BF16 \* `fuse_attn_quant` support depends on the attention backend in use; not all backends support fused quantization output. See the [`fuse_attn_quant` section](#attention--quantization-fuse_attn_quant) for per-backend details. † `enable_sp` and `fuse_gemm_comms` are only autoconfigured for SM90 today; other architectures support requires setting `PassConfig.sp_min_token_num` explicitly. SM100 support also requires setting `VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel`. ## Enabling / Disabling Fusions[¶](#enabling-disabling-fusions "Permanent link") Fusions are exposed through [`PassConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.PassConfig " PassConfig"), which is nested inside [`CompilationConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.CompilationConfig " CompilationConfig"): `[](#__codelineno-0-1)from vllm import LLM [](#__codelineno-0-2)from vllm.config import CompilationConfig, PassConfig [](#__codelineno-0-3)[](#__codelineno-0-4)llm = LLM( [](#__codelineno-0-5) model="...", [](#__codelineno-0-6) optimization_level=2, # Default optimization level [](#__codelineno-0-7) compilation_config=CompilationConfig( [](#__codelineno-0-8) pass_config=PassConfig( [](#__codelineno-0-9) fuse_norm_quant=True, [](#__codelineno-0-10) fuse_act_quant=True, [](#__codelineno-0-11) fuse_allreduce_rms=False, # disable a specific fusion [](#__codelineno-0-12) ) [](#__codelineno-0-13) ), [](#__codelineno-0-14))` Fusions can also be enabled using command-line flags with any `vllm ...` command: `[](#__codelineno-1-1)# Enable O2 defaults, but turn off allreduce fusion [](#__codelineno-1-2)vllm serve meta-llama/Llama-3.1-8B-Instruct -O2 -cc.pass_config.fuse_allreduce_rms=False [](#__codelineno-1-3)[](#__codelineno-1-4)# The above is equivalent to the more verbose: [](#__codelineno-1-5)vllm serve meta-llama/Llama-3.1-8B-Instruct -O2 --compilation-config '{"pass_config": {"fuse_allreduce_rms": false}}' [](#__codelineno-1-6)[](#__codelineno-1-7)# Same syntax in other commands, e.g. vllm bench: [](#__codelineno-1-8)vllm bench latency --model=meta-llama/Llama-3.1-8B-Instruct -O2 -cc.pass_config.fuse_allreduce_rms=False` Fields set explicitly by the user always take precedence over optimization-level defaults. ## Fusion Details[¶](#fusion-details "Permanent link") ### AllReduce + RMSNorm (`fuse_allreduce_rms`)[¶](#allreduce-rmsnorm-fuse_allreduce_rms "Permanent link") Warning TP+DP and TP+PP combinations are currently broken ( [#34458](https://github.com/vllm-project/vllm/issues/34458) and [#35426](https://github.com/vllm-project/vllm/issues/35426)). Only supported on NVIDIA Hopper (SM90) and Blackwell (SM100) with FlashInfer installed. **What it fuses.** Fuses the tensor-parallel all-reduce collective with the subsequent residual add, RMSNorm, and optionally a quantization step into a single FlashInfer / TRT-LLM communication kernel. This fusion is only profitable for small `num_tokens`, so the fusion is only performed in the lower compiled range. Patterns covered: - `AllReduce → RMSNorm(+residual_add)`: CUDA sm90+ with FlashInfer - `AllReduce → RMSNorm(+residual_add) → FP8 static quant`: CUDA sm90+ with FlashInfer - `AllReduce → RMSNorm(+residual_add) → NVFP4 dynamic quant`: CUDA sm100+ with FlashInfer The maximum tensor size below which the fused kernel is used is hardware-dependent (64 MB for TP=2 on SM90/SM100) and configurable via `PassConfig.fi_allreduce_fusion_max_size_mb`. **Code locations.** - Pass: [`vllm/compilation/passes/fusion/allreduce_rms_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/allreduce_rms_fusion.py) - FlashInfer all-reduce: [`vllm/distributed/device_communicators/flashinfer_all_reduce.py`](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/device_communicators/flashinfer_all_reduce.py) - Benchmark: [`benchmarks/kernels/benchmark_fused_collective.py`](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_fused_collective.py) ### Attention + Quantization (`fuse_attn_quant`)[¶](#attention-quantization-fuse_attn_quant "Permanent link") Info `fuse_attn_quant` is currently not enabled at any optimization level by default and must be set explicitly. It requires the full model graph to be visible (Inductor partition or `splitting_ops=[]`). **What it fuses.** Fuses the attention output quantization directly after the attention computation, eliminating a full-precision memory round-trip of the attention output. This fusion supports both standard [`Attention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/attention/#vllm.model_executor.layers.attention.attention.Attention " Attention") and [`MLAAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mla_attention/#vllm.model_executor.layers.attention.mla_attention.MLAAttention " MLAAttention") (used by DeepSeek-V2/V3/R1 models). Patterns covered: `Attention → FP8 static quant`: - `TRITON_ATTN`: CUDA, ROCm - `FLASHINFER`: CUDA sm100+ with FlashInfer installed - `ROCM_ATTN`: ROCm - `ROCM_AITER_UNIFIED_ATTN`: ROCm with AITER `Attention → NVFP4 dynamic quant`: - `FLASHINFER`: CUDA sm100+ with FlashInfer installed `MLAAttention → FP8 static, FP8 per-group, NVFP4 dynamic quant` The MLA fusion operates at the graph level on the `unified_mla_attention_with_output` op and works with all MLA decode and prefill backend combinations. Unlike standard [`Attention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/attention/#vllm.model_executor.layers.attention.attention.Attention " Attention") backends (where the kernel writes FP8 output directly), no MLA prefill or decode backend currently supports direct FP8/FP4 output. The fusion writes to an intermediate buffer and quantizes in a separate step, so there is no memory round-trip elimination yet. Info The MLA attention fusion is not expected to yield a measurable speedup yet. This will improve once MLA prefill/decode kernels support direct FP8/FP4 output. Other attention backends do not support fused output quantization yet. **Code locations.** - Pass (Attention): [`vllm/compilation/passes/fusion/attn_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/attn_quant_fusion.py) - Pass (MLAAttention): [`vllm/compilation/passes/fusion/mla_attn_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/mla_attn_quant_fusion.py) - Attention backends: [`vllm/v1/attention/backends/`](https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/) ### RoPE + KV-Cache Update (`fuse_rope_kvcache`)[¶](#rope-kv-cache-update-fuse_rope_kvcache "Permanent link") Info ROCm/AITER-only. Not available on NVIDIA CUDA or CPU. The fusion is only enabled for `num_tokens ≤ 256` by default due to AITER fused kernel performance issues. This threshold is configurable via `PassConfig.rope_kvcache_fusion_max_token_num`. **What it fuses.** Fuses the rotary positional embedding kernel with the KV-cache scatter/write into a single kernel, avoiding separate reads and writes of the key and value tensors. Requires: AMD ROCm with AITER enabled, the `rotary_embedding` custom op active (automatic), and the `kv_cache` update op visible in the graph: either by using Inductor graph partition or removed from `splitting_ops`. If these conditions are set, the fusion is enabled automatically for optimization level O1 and above. **Code locations.** - Pass: [`vllm/compilation/passes/fusion/rope_kvcache_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rope_kvcache_fusion.py) ### Sequence Parallelism (`enable_sp`)[¶](#sequence-parallelism-enable_sp "Permanent link") **What it fuses.** Replaces all-reduce collectives with reduce-scatter + local RMSNorm + all-gather, splitting the sequence dimension across TP ranks. This restructures the graph so the subsequent AsyncTP pass can fuse the reduce-scatter / all-gather with the surrounding GEMMs. Sequence Parallelism itself does not directly improve performance; it is a prerequisite for the AsyncTP pass (`fuse_gemm_comms`). SP is only applied above a minimum token threshold that is autoconfigured based on device capability and model `hidden_size`. Currently only active on H100/SM90 for models with `hidden_size >= 8192`. The threshold is configurable via `PassConfig.sp_min_token_num`. The general transformation: `[](#__codelineno-2-1)Input → AllReduce → RMSNorm → Output [](#__codelineno-2-2)becomes: [](#__codelineno-2-3)Input → ReduceScatter → local RMSNorm → AllGather → Output` Patterns covered: - First block: `AllReduce → RMSNorm` → `ReduceScatter → RMSNorm → AllGather` - Middle blocks: `AllReduce → fused_add_RMSNorm` → `ReduceScatter → fused_add_RMSNorm → AllGather` - Both with optional `→ FP8 static quant` suffix Requires: `use_inductor_graph_partition=True` **or** piecewise compilation with static sizes divisible by `tensor_parallel_size`. Supported hardware: Only tested on NVIDIA CUDA, possibly works on ROCm. FP8 all-gather requires sm90+. **Code locations.** - Pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py) ### AsyncTP GEMM + Collective Overlap (`fuse_gemm_comms`)[¶](#asynctp-gemm-collective-overlap-fuse_gemm_comms "Permanent link") Info Requires `enable_sp=True` (enabled automatically). This pass is a no-op if Sequence Parallelism has not been applied. **What it fuses.** After Sequence Parallelism transforms the graph, fuses GEMM kernels with the surrounding reduce-scatter (output projection) and all-gather (input projection) using `torch.ops.symm_mem` symmetric-memory primitives, overlapping communication and computation. This overlap is only profitable for large `num_tokens`, so the fusion (and preceding SP) is only performed in the higher compiled range above `PassConfig.sp_min_token_num`. Patterns covered: - `GEMM → reduce-scatter` → `fused_matmul_reduce_scatter` - `all-gather → GEMM` → `all_gather_matmul` - FP8 scaled variants of both patterns Supported hardware: NVIDIA CUDA with symmetric-memory (`torch.distributed._symmetric_memory`) support. On B200, pattern-matching fp8 FlashInfer scaled MM is not supported, so it must be disabled ( [#27893](https://github.com/vllm-project/vllm/issues/27893)) `[](#__codelineno-3-1)VLLM_DISABLED_KERNELS=FlashInferFP8ScaledMMLinearKernel ...` **Code locations.** - Pass: [`vllm/compilation/passes/fusion/collective_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/collective_fusion.py) - Sequence parallelism pass: [`vllm/compilation/passes/fusion/sequence_parallelism.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/sequence_parallelism.py) ### QK Norm + RoPE (`enable_qk_norm_rope_fusion`)[¶](#qk-norm-rope-enable_qk_norm_rope_fusion "Permanent link") Info Only applicable to models that apply per-head RMSNorm to Q and K before rotary positional embedding (e.g. Qwen). Not enabled by default at any optimization level due to perf issues on H100: [#34391](https://github.com/vllm-project/vllm/issues/34391) **What it fuses.** Fuses the sequence: split QKV → reshape → Q/K RMSNorm → reshape → rotary embedding into a single `fused_qk_norm_rope` CUDA kernel. `[](#__codelineno-4-1)# Unfused: [](#__codelineno-4-2)q, k, v = split(qkv) [](#__codelineno-4-3)q_norm = rms_norm(q.view(heads)) [](#__codelineno-4-4)k_norm = rms_norm(k.view(kv_heads)) [](#__codelineno-4-5)q_rope, k_rope = rotary_embedding(q_norm, k_norm, ...) [](#__codelineno-4-6)[](#__codelineno-4-7)# Fused: [](#__codelineno-4-8)fused_qk_norm_rope(qkv, ...)` Supported hardware: CUDA (sm80+) only, tested only on sm90 and sm100. **Code locations.** - Pass: [`vllm/compilation/passes/fusion/qk_norm_rope_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/qk_norm_rope_fusion.py) - CUDA kernel: [`csrc/ops.h`](https://github.com/vllm-project/vllm/blob/main/csrc/ops.h) (`fused_qk_norm_rope`) ### RMSNorm + Quantization (`fuse_norm_quant`)[¶](#rmsnorm-quantization-fuse_norm_quant "Permanent link") Warning On NVIDIA, Inductor actually generates a faster fused kernel than our custom CUDA kernel. Hence, this fusion is only enabled when either `rms_norm` or `quant_fp8` is using a custom kernel. **What it fuses.** Combines the custom `rms_norm` / `fused_add_rms_norm` operations with subsequent quantization into a single fused kernel, eliminating an intermediate read/write of the full-precision activation tensor. Two variants are fused: - _Plain RMSNorm + quant_: `rms_norm(x) → quant_fp8(y)` - _Fused-add RMSNorm + quant_: `fused_add_rms_norm(x, residual) → quant_fp8(y)` — also updates the residual in-place. Note that AITER fusions are currently in a separate pass in `vllm.compilation.passes.fusion.rocm_aiter_fusion`. Supported quantization scheme/hardware combinations: - FP8 static per-tensor: CUDA & HIP kernel - FP8 dynamic per-token: CUDA & HIP kernel, AITER - FP8 dynamic per-token-group (128/64): CUDA & HIP kernel, AITER **Code locations.** - Pass: [`vllm/compilation/passes/fusion/rms_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rms_quant_fusion.py) - ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py) - CUDA/HIP kernels: [`csrc/layernorm_quant_kernels.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/layernorm_quant_kernels.cu) ### SiLU+Mul + Quantization (`fuse_act_quant`)[¶](#silumul-quantization-fuse_act_quant "Permanent link") Warning Same as `fuse_norm_quant`: on NVIDIA, Inductor generates a faster fused kernel than our custom ops. This fusion is only enabled when either `silu_and_mul` or `quant_fp8` are using a custom kernel, or for NVFP4-quantized models (where FP4 quant is always a custom op). **What it fuses.** Fuses the `silu_and_mul` gate-up projection activation with subsequent quantization into a single kernel, avoiding materialization of the full-precision post-activation tensor. Note that AITER fusions are in a separate pass in `vllm.compilation.passes.fusion.rocm_aiter_fusion`. Supported quantization scheme/hardware combinations: - FP8 static per-tensor: CUDA & HIP kernel - FP8 dynamic per-group (128/64): CUDA kernel (sm89+, not active when DeepGemm is used on sm100+) - NVFP4 dynamic: CUDA sm100+ only with FlashInfer - FP8 per-token-group (128): ROCm AITER only **Code locations.** - Pass: [`vllm/compilation/passes/fusion/act_quant_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/act_quant_fusion.py) - ROCm AITER pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py) - CUDA/HIP kernels: [`csrc/quantization/`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/) - Fused SiLU+Mul+BlockQuant kernel: [`csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu`](https://github.com/vllm-project/vllm/blob/main/csrc/quantization/fused_kernels/fused_silu_mul_block_quant.cu) ### RMSNorm + Padding (`fuse_act_padding`)[¶](#rmsnorm-padding-fuse_act_padding "Permanent link") Info ROCm/AITER-only. Targeted at GPT-OSS models. **What it fuses.** Fuses a residual add + RMSNorm with a subsequent padding operation that pads the hidden dimension to a multiple required by downstream AITER Triton GEMM kernels. Requires: AMD ROCm with AITER RMSNorm enabled. Enabled by default in optimization level O1 and above when the hidden size is 2880 and AITER Triton GEMMs _not_ enabled. **Code locations.** - Pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py) ([`RocmAiterTritonAddRMSNormPadFusionPass`](https://docs.vllm.ai/en/api/vllm/compilation/passes/fusion/rocm_aiter_fusion/#vllm.compilation.passes.fusion.rocm_aiter_fusion.RocmAiterTritonAddRMSNormPadFusionPass " RocmAiterTritonAddRMSNormPadFusionPass")) ### MLA Dual RMSNorm (`fuse_mla_dual_rms_norm`)[¶](#mla-dual-rmsnorm-fuse_mla_dual_rms_norm "Permanent link") Info ROCm/AITER-only. Targeted at DeepSeek-V3 / Kimi-K2 MLA attention. Note When the native implementation of `rms_norm` is used (the default on CUDA and ROCm for now), Inductor's built-in fusion already handles merging these norms automatically. This explicit pass targets the case where AITER's custom `rms_norm` op is active, which Inductor cannot fuse on its own. **What it fuses.** Fuses the paired `q_a_layernorm` and `kv_a_layernorm` RMS norm operations in MLA attention into a single `fused_qk_rmsnorm` HIP kernel call via AITER, reducing kernel launch overhead from 2 launches to 1 per MLA layer. `[](#__codelineno-5-1)# Unfused: [](#__codelineno-5-2)q_c, kv_lora = split(projected, [q_dim, kv_dim]) [](#__codelineno-5-3)kv_c, k_pe = split(kv_lora, [kv_c_dim, k_pe_dim]) [](#__codelineno-5-4)q_c = rms_norm(q_c, q_weight, eps) [](#__codelineno-5-5)kv_c = rms_norm(kv_c, kv_weight, eps) [](#__codelineno-5-6)[](#__codelineno-5-7)# Fused: [](#__codelineno-5-8)q_c, kv_lora = split(projected, [q_dim, kv_dim]) [](#__codelineno-5-9)kv_c, k_pe = split(kv_lora, [kv_c_dim, k_pe_dim]) [](#__codelineno-5-10)q_normed, kv_normed = fused_mla_dual_rms_norm( [](#__codelineno-5-11) q_c, q_weight, kv_c, kv_weight, eps1, eps2)` Requires: AMD ROCm with AITER enabled. Enabled by default at optimization level O1 and above when AITER is available. **Code locations.** - Pass: [`vllm/compilation/passes/fusion/rocm_aiter_fusion.py`](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/passes/fusion/rocm_aiter_fusion.py) ([`MLADualRMSNormFusionPass`](https://docs.vllm.ai/en/api/vllm/compilation/passes/fusion/rocm_aiter_fusion/#vllm.compilation.passes.fusion.rocm_aiter_fusion.MLADualRMSNormFusionPass " MLADualRMSNormFusionPass")) - Custom op: [`vllm/_aiter_ops.py`](https://github.com/vllm-project/vllm/blob/main/vllm/_aiter_ops.py) (`fused_mla_dual_rms_norm`) - AITER kernel: [`fused_qk_rmsnorm`](https://github.com/ROCm/aiter/pull/2442) ## See Also[¶](#see-also "Permanent link") - [Optimization Levels](https://docs.vllm.ai/en/latest/optimization_levels/) — high-level presets that set fusion defaults. - [torch.compile in vLLM](https://docs.vllm.ai/en/latest/torch_compile/) — how the Inductor pass pipeline works. - [Attention Backends](https://docs.vllm.ai/en/latest/attention_backends/) — attention-specific kernel selection. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/custom_op.md "Edit this page") [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") is an abstract class used for dispatching the forward method of various operations to the appropriate backend. It also offers a mechanism for both vLLM and OOT (Out-Of-Tree) plugins to register their custom operations. This document will introduce how CustomOp works in vLLM and how to implement a new [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp"). ## How CustomOp Works in vLLM[¶](#how-customop-works-in-vllm "Permanent link") [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") manages two dictionaries of all custom ops (i.e., op classes, indexed by registered name) in its class, for vLLM and OOT plugins respectively. We can use `@CustomOp.register("op_name")` to register an op class to the [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") system. After this, the `op_name` and its class will be added into the `op_registry` dictionary. In addition, We can also register an OOT op by `@CustomOp.register_oot("op_name")`. We will introduce this mechanism in detail later. When a [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") is called (i.e., call its `forward()` method), if it is enabled (i.e., with `--compilation_config.custom_ops '["+op_name"]'`), it will automatically dispatch the forward method to the appropriate backend according to `current_platform`. Otherwise (i.e., it is disabled), it will only call the `forward_native()` method to use PyTorch-native implementation of this forward method. - **CPU platform:** dispatch to `forward_cpu()`. - **CUDA platform:** dispatch to `forward_cuda()`. - **ROCm platform:** dispatch to `forward_hip()`. If `forward_hip()` is not implemented, it will use `forward_cuda()` as a fallback. - **XPU platform:** dispatch to `forward_xpu()`. - **TPU platform:** dispatch to `forward_tpu()`. - **OOT platform:** dispatch to `forward_oot()`. This will only be called on OOT platforms. - **Default:** dispatch to `forward_native()` as a final fallback for all platforms. Note Note that the dispatching logic might not be absolute because of class inheritance. Derived class might override the behavior. Furthermore, vLLM decides whether to enable or disable a [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") based on `compilation_config.custom_ops`. To be specific, if a [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") is not registered in `compilation_config.custom_ops` (i.e., uses the default config), it will be enabled if `compilation_config.custom_ops` contains `all`, or will be disabled if it contains `none`. Note Note that `all` and `none` cannot coexist in `compilation_config.custom_ops`. By default, if `compilation_config.backend == "inductor"` and `compilation_config.mode != CompilationMode.NONE`, a `none` will be appended into `compilation_config.custom_ops`, otherwise a `all` will be appended. In other words, this means [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") will be disabled in some platforms (i.e., those use `inductor` as default backend for `torch.compile`) when running with torch compile mode. In this case, Inductor generates (fused) Triton kernels for those disabled custom ops. Note For multi-modal models, vLLM has enforced the enabling of some custom ops to use device-specific deep-optimized kernels for better performance in ViT part, such as [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention") and `ApplyRotaryEmb`. We can also pass a `enforce_enable=True` param to the `__init__()` method of the [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") to enforce enable itself at object-level. Note that this `enforce_enable` mechanism will be removed after we add a separate `compilation_config` for multi-modal part. ## How to Customise Your Configuration for CustomOp[¶](#how-to-customise-your-configuration-for-customop "Permanent link") vLLM also offers fine-grained control over which custom ops to enable or disable for users, by manually passing a `--compilation_config.custom_ops '["..."]'` when launching a server. For example: - Use `--compilation_config.custom_ops '["all"]'` to enable all custom ops. - Use `--compilation_config.custom_ops '["none"]'` to disable all custom ops. - Use `--compilation_config.custom_ops '["all,-op1"]'` to enable all custom ops except op1 (i.e., prefixed with a `-` means "disable"). - Use `--compilation_config.custom_ops '["none,+op1,+op2"]'` to only enable op1 and op2 (i.e., prefixed with a `+` means "enable"). ## Types of Supported CustomOp in vLLM[¶](#types-of-supported-customop-in-vllm "Permanent link") **1\. Attention:** `[](#__codelineno-0-1)@PluggableLayer.register("multi_head_latent_attention") [](#__codelineno-0-2)class MultiHeadLatentAttentionWrapper(PluggableLayer): [](#__codelineno-0-3) """Pluggable MLA layer which allows OOT backends to add [](#__codelineno-0-4) custom implementations of the outer MLA layer (including rope & o_proj). [](#__codelineno-0-5) Note that currently oot platforms can still use CustomOp.register_oot to [](#__codelineno-0-6) replace MLA layer entirely, although we use PluggableLayer to register [](#__codelineno-0-7) this layer now. [](#__codelineno-0-8) [](#__codelineno-0-9) This class takes positions and hidden_states as input. [](#__codelineno-0-10) The input tensors can either contain prefill tokens or decode tokens. [](#__codelineno-0-11) The class does the following: [](#__codelineno-0-12) [](#__codelineno-0-13) 1. MLA Preprocess. [](#__codelineno-0-14) 2. Perform multi-head attention to prefill tokens and [](#__codelineno-0-15) multi-query attention to decode tokens separately. [](#__codelineno-0-16) 3. Return the output tensor. [](#__codelineno-0-17) """` **2\. Activation:** `[](#__codelineno-1-1)@CustomOp.register("silu_and_mul") [](#__codelineno-1-2)class SiluAndMul(CustomOp): [](#__codelineno-1-3) """An activation function for SwiGLU. [](#__codelineno-1-4) [](#__codelineno-1-5) The function computes x -> silu(x[:d]) * x[d:] where d = x.shape[-1] // 2. [](#__codelineno-1-6) [](#__codelineno-1-7) Shapes: [](#__codelineno-1-8) x: (num_tokens, 2 * d) or (batch_size, seq_len, 2 * d) [](#__codelineno-1-9) return: (num_tokens, d) or (batch_size, seq_len, d) [](#__codelineno-1-10) """ [](#__codelineno-1-11) [](#__codelineno-1-12)[](#__codelineno-1-13)@CustomOp.register("mul_and_silu") [](#__codelineno-1-14)class MulAndSilu(CustomOp): [](#__codelineno-1-15) """An activation function for SwiGLU. [](#__codelineno-1-16) [](#__codelineno-1-17) The function computes x -> x[:d] * silu(x[d:]) where d = x.shape[-1] // 2. [](#__codelineno-1-18) [](#__codelineno-1-19) Shapes: [](#__codelineno-1-20) x: (num_tokens, 2 * d) or (batch_size, seq_len, 2 * d) [](#__codelineno-1-21) return: (num_tokens, d) or (batch_size, seq_len, d) [](#__codelineno-1-22) """ [](#__codelineno-1-23) [](#__codelineno-1-24)[](#__codelineno-1-25)@CustomOp.register("gelu_new") [](#__codelineno-1-26)class NewGELU(CustomOp): [](#__codelineno-1-27)[](#__codelineno-1-28)@CustomOp.register("gelu_fast") [](#__codelineno-1-29)class FastGELU(CustomOp): [](#__codelineno-1-30)[](#__codelineno-1-31)@CustomOp.register("quick_gelu") [](#__codelineno-1-32)class QuickGELU(CustomOp): [](#__codelineno-1-33) # https://github.com/huggingface/transformers/blob/main/src/transformers/activations.py#L90 [](#__codelineno-1-34)[](#__codelineno-1-35)@CustomOp.register("gelu_and_mul") [](#__codelineno-1-36)class GeluAndMul(CustomOp): [](#__codelineno-1-37) """An activation function for GeGLU. [](#__codelineno-1-38) [](#__codelineno-1-39) The function computes x -> GELU(x[:d]) * x[d:] where d = x.shape[-1] // 2. [](#__codelineno-1-40) [](#__codelineno-1-41) Shapes: [](#__codelineno-1-42) x: (batch_size, seq_len, 2 * d) or (num_tokens, 2 * d) [](#__codelineno-1-43) return: (batch_size, seq_len, d) or (num_tokens, d) [](#__codelineno-1-44) """ [](#__codelineno-1-45) [](#__codelineno-1-46)[](#__codelineno-1-47)@CustomOp.register("gelu_and_mul_sparse") [](#__codelineno-1-48)class GeluAndMulSparse(CustomOp): [](#__codelineno-1-49) """An activation function for GeluAndMulSparse. [](#__codelineno-1-50) This activation function is used in Gemma3n. It computes: [](#__codelineno-1-51) up_proj = self.up_proj(x) [](#__codelineno-1-52) gate_proj = self.gate_proj(x) [](#__codelineno-1-53) gate_proj = self._gaussian_topk(gate_proj) # sparsity [](#__codelineno-1-54) activations = self.act_fn(gate_proj) # gelu [](#__codelineno-1-55) down_proj = self.down_proj(activations * up_proj) [](#__codelineno-1-56) Shapes: [](#__codelineno-1-57) x: (num_tokens, 2 * d) or (batch_size, seq_len, 2 * d) [](#__codelineno-1-58) return: (num_tokens, d) or (batch_size, seq_len, d) [](#__codelineno-1-59) """ [](#__codelineno-1-60) [](#__codelineno-1-61)[](#__codelineno-1-62)@CustomOp.register("relu2") [](#__codelineno-1-63)class ReLUSquaredActivation(CustomOp): [](#__codelineno-1-64) """ [](#__codelineno-1-65) Applies the relu^2 activation introduced in https://arxiv.org/abs/2109.08668v2 [](#__codelineno-1-66) """ [](#__codelineno-1-67) [](#__codelineno-1-68)[](#__codelineno-1-69)@CustomOp.register("xielu") [](#__codelineno-1-70)class XIELU(CustomOp): [](#__codelineno-1-71) """ [](#__codelineno-1-72) Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010 [](#__codelineno-1-73) If the user has installed the nickjbrowning/XIELU, we import xIELU CUDA [](#__codelineno-1-74) Otherwise, we emit a single warning and use xIELU Python [](#__codelineno-1-75) """ [](#__codelineno-1-76) [](#__codelineno-1-77)[](#__codelineno-1-78)@CustomOp.register("swigluoai_and_mul") [](#__codelineno-1-79)class SwigluOAIAndMul(CustomOp): [](#__codelineno-1-80) # https://github.com/huggingface/transformers/blob/v4.55.0/src/transformers/models/gpt_oss/modeling_gpt_oss.py#L106-L110 [](#__codelineno-1-81)[](#__codelineno-1-82)@CustomOp.register("fatrelu_and_mul") [](#__codelineno-1-83)class FatreluAndMul(CustomOp): [](#__codelineno-1-84) """An activation function for FATReLU. [](#__codelineno-1-85) [](#__codelineno-1-86) The function computes x -> FATReLU(x[:d]) * x[d:] where [](#__codelineno-1-87) d = x.shape[-1] // 2. [](#__codelineno-1-88) This is used in openbmb/MiniCPM-S-1B-sft. [](#__codelineno-1-89) [](#__codelineno-1-90) Shapes: [](#__codelineno-1-91) x: (num_tokens, 2 * d) or (batch_size, seq_len, 2 * d) [](#__codelineno-1-92) return: (num_tokens, d) or (batch_size, seq_len, d) [](#__codelineno-1-93) """` **3\. MM-Conv:** `[](#__codelineno-2-1)@CustomOp.register("conv2d") [](#__codelineno-2-2)class Conv2dLayer(ConvLayerBase): [](#__codelineno-2-3) """Conv layer with Conv2d.""" [](#__codelineno-2-4) [](#__codelineno-2-5)[](#__codelineno-2-6)@CustomOp.register("conv3d") [](#__codelineno-2-7)class Conv3dLayer(ConvLayerBase): [](#__codelineno-2-8) """Conv layer with Conv3d."""` **4\. Embedding:** `[](#__codelineno-3-1)@PluggableLayer.register("vocab_parallel_embedding") [](#__codelineno-3-2)class VocabParallelEmbedding(PluggableLayer): [](#__codelineno-3-3) """Embedding parallelized in the vocabulary dimension. [](#__codelineno-3-4) [](#__codelineno-3-5) Adapted from torch.nn.Embedding, note that we pad the vocabulary size to [](#__codelineno-3-6) make sure it is divisible by the number of model parallel GPUs. [](#__codelineno-3-7) [](#__codelineno-3-8) In order to support various loading methods, we ensure that LoRA-added [](#__codelineno-3-9) embeddings are always at the end of TP-sharded tensors. In other words, [](#__codelineno-3-10) we shard base embeddings and LoRA embeddings separately (both padded), [](#__codelineno-3-11) and place them in the same tensor. [](#__codelineno-3-12) In this example, we will have the original vocab size = 1010, [](#__codelineno-3-13) added vocab size = 16 and padding to 64. Therefore, the total [](#__codelineno-3-14) vocab size with padding will be 1088 (because we first pad 1010 to [](#__codelineno-3-15) 1024, add 16, and then pad to 1088). [](#__codelineno-3-16) Therefore, the tensor format looks like the following: [](#__codelineno-3-17) TP1, rank 0 (no sharding): [](#__codelineno-3-18) |< --------BASE-------- >|< -BASE PADDING-- >|< -----LORA------ >|< -LORA PADDING-- >| [](#__codelineno-3-19) corresponding token_id: | 0 | 1 | ... | 1009 | -1 | ... | -1 | 1010 | ... | 1025 | -1 | ... | -1 | [](#__codelineno-3-20) index: | 0 | 1 | ... | 1009 | 1010 | ... | 1023 | 1024 | ... | 1039 | 1040 | ... | 1087 | [](#__codelineno-3-21) [](#__codelineno-3-22) TP2, rank 0: [](#__codelineno-3-23) |< --------------------BASE--------------------- >|< -----LORA------ >|< -LORA PADDING- >| [](#__codelineno-3-24) corresponding token_id: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 1010 | ... | 1025 | -1 | ... | -1 | [](#__codelineno-3-25) index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 528 | ... | 543 | [](#__codelineno-3-26) TP2, rank 1: [](#__codelineno-3-27) |< -----------BASE----------- >|< -BASE PADDING- >|< -----------LORA PADDING----------- >| [](#__codelineno-3-28) corresponding token_id: | 512 | 513 | 514 | ... | 1009 | -1 | ... | -1 | -1 | ... | -1 | -1 | ... | -1 | [](#__codelineno-3-29) index: | 0 | 1 | 2 | ... | 497 | 498 | ... | 511 | 512 | ... | 527 | 528 | ... | 543 | [](#__codelineno-3-30) [](#__codelineno-3-31) Args: [](#__codelineno-3-32) num_embeddings: vocabulary size. [](#__codelineno-3-33) embedding_dim: size of hidden state. [](#__codelineno-3-34) params_dtype: type of the parameters. [](#__codelineno-3-35) org_num_embeddings: original vocabulary size (without LoRA). [](#__codelineno-3-36) padding_size: padding size for the vocabulary. [](#__codelineno-3-37) quant_config: quant config for the layer [](#__codelineno-3-38) prefix: full name of the layer in the state dict [](#__codelineno-3-39) """ # noqa: E501 [](#__codelineno-3-40) [](#__codelineno-3-41)[](#__codelineno-3-42)@PluggableLayer.register("parallel_lm_head") [](#__codelineno-3-43)class ParallelLMHead(VocabParallelEmbedding): [](#__codelineno-3-44) """Parallelized LM head. [](#__codelineno-3-45) [](#__codelineno-3-46) Output logits weight matrices used in the Sampler. The weight and bias [](#__codelineno-3-47) tensors are padded to make sure they are divisible by the number of [](#__codelineno-3-48) model parallel GPUs. [](#__codelineno-3-49) [](#__codelineno-3-50) Args: [](#__codelineno-3-51) num_embeddings: vocabulary size. [](#__codelineno-3-52) embedding_dim: size of hidden state. [](#__codelineno-3-53) bias: whether to use bias. [](#__codelineno-3-54) params_dtype: type of the parameters. [](#__codelineno-3-55) org_num_embeddings: original vocabulary size (without LoRA). [](#__codelineno-3-56) padding_size: padding size for the vocabulary. [](#__codelineno-3-57) """` **5\. Linear:** `[](#__codelineno-4-1)@PluggableLayer.register("row_parallel_linear") [](#__codelineno-4-2)class RowParallelLinear(LinearBase): [](#__codelineno-4-3) """Linear layer with row parallelism. [](#__codelineno-4-4) [](#__codelineno-4-5) The linear layer is defined as Y = XA + b. A is parallelized along [](#__codelineno-4-6) its first dimension and X along its second dimension as: [](#__codelineno-4-7) - - [](#__codelineno-4-8) | A_1 | [](#__codelineno-4-9) | . | [](#__codelineno-4-10) A = | . | X = [X_1, ..., X_p] [](#__codelineno-4-11) | . | [](#__codelineno-4-12) | A_p | [](#__codelineno-4-13) - - [](#__codelineno-4-14) Arguments: [](#__codelineno-4-15) input_size: first dimension of matrix A. [](#__codelineno-4-16) output_size: second dimension of matrix A. [](#__codelineno-4-17) bias: If true, add bias. Note that bias is not parallelized. [](#__codelineno-4-18) input_is_parallel: If true, we assume that the input is already [](#__codelineno-4-19) split across the GPUs and we do not split [](#__codelineno-4-20) again. [](#__codelineno-4-21) skip_bias_add: This was added to enable performance optimization where [](#__codelineno-4-22) bias can be fused with other element-wise operations. [](#__codelineno-4-23) We skip adding bias but instead return it. [](#__codelineno-4-24) params_dtype: Data type for the parameters. [](#__codelineno-4-25) reduce_results: If true, call all-reduce on output and make Y available [](#__codelineno-4-26) to all GPUs, otherwise, every GPU will have its output [](#__codelineno-4-27) which is Y = X_iA_i [](#__codelineno-4-28) quant_config: Quantization configure. [](#__codelineno-4-29) prefix: The name of the layer in the state dict, including all parents [](#__codelineno-4-30) (e.g. model.layers.0.down_proj) [](#__codelineno-4-31) return_bias: If true, return bias together with outputs in forward pass. [](#__codelineno-4-32) disable_tp: If true, weights matrix won't be sharded through tp rank. [](#__codelineno-4-33) """ [](#__codelineno-4-34) [](#__codelineno-4-35)[](#__codelineno-4-36)@PluggableLayer.register("column_parallel_linear") [](#__codelineno-4-37)class ColumnParallelLinear(LinearBase): [](#__codelineno-4-38) """Linear layer with column parallelism. [](#__codelineno-4-39) [](#__codelineno-4-40) The linear layer is defined as Y = XA + b. A is parallelized along [](#__codelineno-4-41) its second dimension as A = [A_1, ..., A_p]. [](#__codelineno-4-42) [](#__codelineno-4-43) Args: [](#__codelineno-4-44) input_size: first dimension of matrix A. [](#__codelineno-4-45) output_size: second dimension of matrix A. [](#__codelineno-4-46) bias: If true, add bias. [](#__codelineno-4-47) gather_output: If true, call all-gather on output and make Y available [](#__codelineno-4-48) to all GPUs, otherwise, every GPU will have its output [](#__codelineno-4-49) which is Y_i = XA_i [](#__codelineno-4-50) skip_bias_add: This was added to enable performance optimizations where [](#__codelineno-4-51) bias can be fused with other element-wise operations. we [](#__codelineno-4-52) skip adding bias but instead return it. [](#__codelineno-4-53) params_dtype: Data type for the parameters. [](#__codelineno-4-54) quant_config: Quantization configure. [](#__codelineno-4-55) prefix: The name of the layer in the state dict, including all parents [](#__codelineno-4-56) (e.g. model.layers.0.qkv_proj) [](#__codelineno-4-57) return_bias: If true, return bias together with outputs in forward pass. [](#__codelineno-4-58) disable_tp: If true, weights matrix won't be sharded through tp rank. [](#__codelineno-4-59) """ [](#__codelineno-4-60) [](#__codelineno-4-61)[](#__codelineno-4-62)@PluggableLayer.register("replicated_linear") [](#__codelineno-4-63)class ReplicatedLinear(LinearBase): [](#__codelineno-4-64) """Replicated linear layer. [](#__codelineno-4-65) [](#__codelineno-4-66) Args: [](#__codelineno-4-67) input_size: input dimension of the linear layer. [](#__codelineno-4-68) output_size: output dimension of the linear layer. [](#__codelineno-4-69) bias: If true, add bias. [](#__codelineno-4-70) skip_bias_add: If true, skip adding bias but instead return it. [](#__codelineno-4-71) params_dtype: Data type for the parameters. [](#__codelineno-4-72) quant_config: Quantization configure. [](#__codelineno-4-73) prefix: The name of the layer in the state dict, including all parents [](#__codelineno-4-74) (e.g. model.layers.0.qkv_proj) [](#__codelineno-4-75) return_bias: If true, return bias together with outputs in forward pass. [](#__codelineno-4-76) disable_tp: Take no effect for replicated linear layers. [](#__codelineno-4-77) """` **6\. Logits Processor:** `[](#__codelineno-5-1)@PluggableLayer.register("logits_processor") [](#__codelineno-5-2)class LogitsProcessor(PluggableLayer): [](#__codelineno-5-3) """Process logits and apply logits processors from sampling metadata. [](#__codelineno-5-4) [](#__codelineno-5-5) This layer does the following: [](#__codelineno-5-6) 1. Gather logits from model hidden_states. [](#__codelineno-5-7) 2. Scale logits if needed. [](#__codelineno-5-8) 3. Apply logits processors (if any). [](#__codelineno-5-9) """` **7\. Mamba:** ``[](#__codelineno-6-1)@PluggableLayer.register("mamba_mixer") [](#__codelineno-6-2)class MambaMixer(MambaBase, PluggableLayer): [](#__codelineno-6-3) """ [](#__codelineno-6-4) Compute ∆, A, B, C, and D the state space parameters and compute [](#__codelineno-6-5) the `contextualized_states`. A, D are input independent [](#__codelineno-6-6) (see Mamba paper [1] Section 3.5.2 "Interpretation of A" [](#__codelineno-6-7) for why A isn't selective) ∆, B, C are input-dependent [](#__codelineno-6-8) (this is a key difference between Mamba and the linear time [](#__codelineno-6-9) invariant S4, and is why Mamba is called [](#__codelineno-6-10) **selective** state spaces) [](#__codelineno-6-11) """ [](#__codelineno-6-12) [](#__codelineno-6-13)[](#__codelineno-6-14)@PluggableLayer.register("mamba_mixer2") [](#__codelineno-6-15)class MambaMixer2(MambaBase, PluggableLayer): [](#__codelineno-6-16) """ [](#__codelineno-6-17) Compute ∆, A, B, C, and D the state space parameters and compute [](#__codelineno-6-18) the `contextualized_states`. A, D are input independent [](#__codelineno-6-19) (see Mamba paper [1] Section 3.5.2 "Interpretation of A" [](#__codelineno-6-20) for why A isn't selective) ∆, B, C are input-dependent [](#__codelineno-6-21) (this is a key difference between Mamba and the linear time [](#__codelineno-6-22) invariant S4, and is why Mamba is called [](#__codelineno-6-23) **selective** state spaces) [](#__codelineno-6-24) """ [](#__codelineno-6-25) [](#__codelineno-6-26)[](#__codelineno-6-27)@CustomOp.register("mixer2_gated_rms_norm") [](#__codelineno-6-28)class Mixer2RMSNormGated(CustomOp): [](#__codelineno-6-29)[](#__codelineno-6-30)@PluggableLayer.register("plamo2_mamba_mixer") [](#__codelineno-6-31)class Plamo2MambaMixer(MambaBase, PluggableLayer): [](#__codelineno-6-32)[](#__codelineno-6-33)@CustomOp.register("short_conv") [](#__codelineno-6-34)class ShortConv(MambaBase, CustomOp):`` **8\. MoE:** ``[](#__codelineno-7-1)@PluggableLayer.register("fused_moe") [](#__codelineno-7-2)class FusedMoE(PluggableLayer): [](#__codelineno-7-3) """FusedMoE layer for MoE models. [](#__codelineno-7-4) [](#__codelineno-7-5) This layer contains both MergedColumnParallel weights (gate_up_proj / [](#__codelineno-7-6) w13) and RowParallelLinear weights (down_proj/ w2). [](#__codelineno-7-7) [](#__codelineno-7-8) Note: Mixtral uses w1, w2, and w3 for gate, up, and down_proj. We [](#__codelineno-7-9) copy that naming convention here and handle any remapping in the [](#__codelineno-7-10) load_weights function in each model implementation. [](#__codelineno-7-11) [](#__codelineno-7-12) Args: [](#__codelineno-7-13) num_experts: Number of experts in the model [](#__codelineno-7-14) top_k: Number of experts selected for each token [](#__codelineno-7-15) hidden_size: Input hidden state size of the transformer [](#__codelineno-7-16) intermediate_size: Intermediate size of the experts [](#__codelineno-7-17) params_dtype: Data type for the parameters. [](#__codelineno-7-18) renormalize: Whether to renormalize the logits in the fused_moe kernel [](#__codelineno-7-19) quant_config: Quantization configure. [](#__codelineno-7-20) enable_eplb: Whether to enable expert parallelism load balancer. [](#__codelineno-7-21) router_logits_dtype: Data type for router logits buffers. [](#__codelineno-7-22) routed_scaling_factor: A scaling factor that is applied to the topk_weights [](#__codelineno-7-23) by the router or the output of the layer depending [](#__codelineno-7-24) on the value of `apply_routed_scale_to_output` [](#__codelineno-7-25) apply_routed_scale_to_output: Determine whether or not `routed_scaling_factor` [](#__codelineno-7-26) is applied to the topk_weights or to the experts [](#__codelineno-7-27) output. It is applied to the experts output [](#__codelineno-7-28) instead of the topk_weights when this feature is [](#__codelineno-7-29) not supported by the router (or the experts). [](#__codelineno-7-30) """ [](#__codelineno-7-31) [](#__codelineno-7-32)[](#__codelineno-7-33)@CustomOp.register("modular_fused_moe") [](#__codelineno-7-34)class FusedMoEModularMethod(FusedMoEMethodBase, CustomOp): [](#__codelineno-7-35)[](#__codelineno-7-36)@CustomOp.register("unquantized_fused_moe") [](#__codelineno-7-37)class UnquantizedFusedMoEMethod(FusedMoEMethodBase, CustomOp): [](#__codelineno-7-38) """MoE method without quantization.""" [](#__codelineno-7-39) [](#__codelineno-7-40)[](#__codelineno-7-41)@PluggableLayer.register("transformers_fused_moe") [](#__codelineno-7-42)class TransformersFusedMoE(FusedMoE): [](#__codelineno-7-43) """Custom FusedMoE for the Transformers modeling backend.""" [](#__codelineno-7-44) [](#__codelineno-7-45)[](#__codelineno-7-46)@CustomOp.register("grouped_topk") [](#__codelineno-7-47)class GroupedTopk(CustomOp): [](#__codelineno-7-48) """GroupedTopk used by the Deepseek-V2 and Deepseek-V3 model."""`` **9\. Norm:** `[](#__codelineno-8-1)@CustomOp.register("rms_norm") [](#__codelineno-8-2)class RMSNorm(CustomOp): [](#__codelineno-8-3) """Root mean square normalization. [](#__codelineno-8-4) [](#__codelineno-8-5) Computes x -> w * x / sqrt(E[x^2] + eps) where w is the learned weight. [](#__codelineno-8-6) Refer to https://arxiv.org/abs/1910.07467 [](#__codelineno-8-7) """ [](#__codelineno-8-8) [](#__codelineno-8-9)[](#__codelineno-8-10)@CustomOp.register("rms_norm_gated") [](#__codelineno-8-11)class RMSNormGated(CustomOp): [](#__codelineno-8-12) """RMS Normalization with optional gating. [](#__codelineno-8-13) [](#__codelineno-8-14) This is a native PyTorch implementation that supports: [](#__codelineno-8-15) - Standard RMS normalization [](#__codelineno-8-16) - Group RMS normalization [](#__codelineno-8-17) - Optional gating with SiLU activation [](#__codelineno-8-18) """ [](#__codelineno-8-19) [](#__codelineno-8-20)[](#__codelineno-8-21)@CustomOp.register("gemma_rms_norm") [](#__codelineno-8-22)class GemmaRMSNorm(CustomOp): [](#__codelineno-8-23) """RMS normalization for Gemma. [](#__codelineno-8-24) [](#__codelineno-8-25) Two differences from the above RMSNorm: [](#__codelineno-8-26) 1. x * (1 + w) instead of x * w. [](#__codelineno-8-27) 2. (x * w).to(orig_dtype) instead of x.to(orig_dtype) * w. [](#__codelineno-8-28) """` **10\. Quantization:** `[](#__codelineno-9-1)@CustomOp.register("quant_fp8") [](#__codelineno-9-2)class QuantFP8(CustomOp): [](#__codelineno-9-3) """ [](#__codelineno-9-4) Quantize input tensor to FP8 (per-tensor, per-token, per-channel, or per-group). [](#__codelineno-9-5) This CustomOp supports both static and dynamic quantization. [](#__codelineno-9-6) """` **11\. Rope:** `[](#__codelineno-10-1)@CustomOp.register("rotary_embedding") [](#__codelineno-10-2)class RotaryEmbeddingBase(CustomOp): [](#__codelineno-10-3) """Original rotary positional embedding.""" [](#__codelineno-10-4) [](#__codelineno-10-5)[](#__codelineno-10-6)@CustomOp.register("dual_chunk_rotary_embedding") [](#__codelineno-10-7)class DualChunkRotaryEmbedding(CustomOp): [](#__codelineno-10-8) """Rotary positional embedding for Dual Chunk Attention.""" [](#__codelineno-10-9) [](#__codelineno-10-10)[](#__codelineno-10-11)@CustomOp.register("apply_rotary_emb") [](#__codelineno-10-12)class ApplyRotaryEmb(CustomOp):` **12\. Encoder:** `[](#__codelineno-11-1)@PluggableLayer.register("qwen2_decoder") [](#__codelineno-11-2)class CustomQwen2Decoder(PluggableLayer): [](#__codelineno-11-3) """ [](#__codelineno-11-4) Qwen2 visual encoder [](#__codelineno-11-5) non-causal attention + causal attention [](#__codelineno-11-6) token_type_ids ：0=non-causal, 1=causal [](#__codelineno-11-7) """ [](#__codelineno-11-8) [](#__codelineno-11-9)[](#__codelineno-11-10)@CustomOp.register("mm_encoder_attn") [](#__codelineno-11-11)class MMEncoderAttention(CustomOp): [](#__codelineno-11-12) """Multi-headed attention without any cache, used for multimodal encoder.""" [](#__codelineno-11-13) [](#__codelineno-11-14)[](#__codelineno-11-15)@PluggableLayer.register("rel_pos_attention") [](#__codelineno-11-16)class RelPosAttention(PluggableLayer): [](#__codelineno-11-17) """Multi-head Attention block with relative position embeddings."""` ## Guidelines for Implementing a New CustomOp[¶](#guidelines-for-implementing-a-new-customop "Permanent link") ### Implement a New CustomOp in vLLM[¶](#implement-a-new-customop-in-vllm "Permanent link") This part is a tutorial of how to implement a New [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") in vLLM. Steps: 1. Implement a new op class, which extends from [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") base class. 2. Add the `@CustomOp.register("op_name")` decorator on this op class to register it into [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") system. 3. Implement different `forward_xxx()` method according to your needs. Taking [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention") as an example: Code `[](#__codelineno-12-1)@CustomOp.register("mm_encoder_attn") [](#__codelineno-12-2)class MMEncoderAttention(CustomOp): [](#__codelineno-12-3) [](#__codelineno-12-4) def __init__( [](#__codelineno-12-5) self, [](#__codelineno-12-6) num_heads: int, [](#__codelineno-12-7) head_size: int, [](#__codelineno-12-8) scale: float | None = None, [](#__codelineno-12-9) num_kv_heads: int | None = None, [](#__codelineno-12-10) prefix: str = "", [](#__codelineno-12-11) multimodal_config: MultiModalConfig | None = None, [](#__codelineno-12-12) ) -> None: [](#__codelineno-12-13) super().__init__() [](#__codelineno-12-14) # Init... [](#__codelineno-12-15) [](#__codelineno-12-16) def forward_native( [](#__codelineno-12-17) self, [](#__codelineno-12-18) query: torch.Tensor, [](#__codelineno-12-19) key: torch.Tensor, [](#__codelineno-12-20) value: torch.Tensor, [](#__codelineno-12-21) cu_seqlens: torch.Tensor | None = None, [](#__codelineno-12-22) max_seqlen: torch.Tensor | None = None, # Only used for Flash Attention [](#__codelineno-12-23) ) -> torch.Tensor: [](#__codelineno-12-24) # Call TORCH_SDPA implementation... [](#__codelineno-12-25) [](#__codelineno-12-26) def forward_cuda( [](#__codelineno-12-27) self, [](#__codelineno-12-28) query: torch.Tensor, [](#__codelineno-12-29) key: torch.Tensor, [](#__codelineno-12-30) value: torch.Tensor, [](#__codelineno-12-31) cu_seqlens: torch.Tensor | None = None, [](#__codelineno-12-32) max_seqlen: torch.Tensor | None = None, # Only used for Flash Attention [](#__codelineno-12-33) ) -> torch.Tensor: [](#__codelineno-12-34) # Call FA or TORCH_SDPA implementation... [](#__codelineno-12-35) [](#__codelineno-12-36) def forward_cpu( [](#__codelineno-12-37) self, [](#__codelineno-12-38) query: torch.Tensor, [](#__codelineno-12-39) key: torch.Tensor, [](#__codelineno-12-40) value: torch.Tensor, [](#__codelineno-12-41) cu_seqlens: torch.Tensor | None = None, [](#__codelineno-12-42) max_seqlen: torch.Tensor | None = None, # Only used for Flash Attention [](#__codelineno-12-43) ) -> torch.Tensor: [](#__codelineno-12-44) # Call TORCH_SDPA implementation... [](#__codelineno-12-45) [](#__codelineno-12-46) def forward_xpu( [](#__codelineno-12-47) self, [](#__codelineno-12-48) query: torch.Tensor, [](#__codelineno-12-49) key: torch.Tensor, [](#__codelineno-12-50) value: torch.Tensor, [](#__codelineno-12-51) cu_seqlens: torch.Tensor | None = None, [](#__codelineno-12-52) max_seqlen: torch.Tensor | None = None, # Only used for Flash Attention [](#__codelineno-12-53) ) -> torch.Tensor: [](#__codelineno-12-54) # Call FA implementation... [](#__codelineno-12-55) [](#__codelineno-12-56) def forward_tpu( [](#__codelineno-12-57) self, [](#__codelineno-12-58) query: torch.Tensor, [](#__codelineno-12-59) key: torch.Tensor, [](#__codelineno-12-60) value: torch.Tensor, [](#__codelineno-12-61) cu_seqlens: torch.Tensor | None = None, [](#__codelineno-12-62) max_seqlen: torch.Tensor | None = None, # Only used for Flash Attention [](#__codelineno-12-63) ) -> torch.Tensor: [](#__codelineno-12-64) # Call PALLAS implementation...` ### Register a New CustomOp in OOT Device Plugins[¶](#register-a-new-customop-in-oot-device-plugins "Permanent link") Currently, thanks to [vLLM's hardware-plugin mechanism](https://docs.vllm.ai/en/latest/plugin_system/), there are various OOT device plugins emerging out to enable vLLM seamlessly runs on different hardwares. You can also find more details about this mechanism at [Introducing vLLM Hardware Plugin, Best Practice from Ascend NPU](https://blog.vllm.ai/2025/05/12/hardware-plugin.html). - **Official device plugins:** [vllm-ascend](https://github.com/vllm-project/vllm-ascend) (for Huawei Ascend NPU), [vllm-spyre](https://github.com/vllm-project/vllm-spyre) (for Spyre), [vllm-gaudi](https://github.com/vllm-project/vllm-gaudi) (for Intel Gaudi), [vllm-neuron](https://github.com/vllm-project/vllm-neuron) (for AWS Neuron), [vllm-meta](https://github.com/vllm-project/vllm-metal) (for Apple Silicon), etc. - **Non-official device plugins:** [vllm-metax](https://github.com/MetaX-MACA/vLLM-metax) (for MetaX GPU), [vllm-kunlun](https://github.com/baidu/vLLM-Kunlun) (for Baidu Kunlun XPU), [vllm-musa](https://github.com/MooreThreads/vllm-musa) (for Moore Threads GPU), etc. In this case, [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") can enable these hardware manufacturers to seamlessly replace vLLM's operations with their deep-optimized kernels for specific devices at runtime, by just registering an OOT [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") and implementing the `forward_oot()` method. Now, this part will show you how to register an OOT [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") for a device plugin. Taking [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention") as an example: 1. Implement a `CustomMMEncoderAttention` class which extends from [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention") and implement its `forward_oot()` method. 2. Register your `CustomMMEncoderAttention` into vLLM to replace [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention"). Code `[](#__codelineno-13-1)from vllm.model_executor.layers.attention import MMEncoderAttention [](#__codelineno-13-2)from vllm.model_executor.custom_op import CustomOp [](#__codelineno-13-3) [](#__codelineno-13-4)[](#__codelineno-13-5)@CustomOp.register_oot("MMEncoderAttention") [](#__codelineno-13-6)class CustomMMEncoderAttention(MMEncoderAttention): [](#__codelineno-13-7) [](#__codelineno-13-8) def __init__(...): [](#__codelineno-13-9) super().__init__(...) [](#__codelineno-13-10) [](#__codelineno-13-11) def forward_oot(...): [](#__codelineno-13-12) # Call optimized device-specific kernels. [](#__codelineno-13-13) ...` In this case, a new item `{"MMEncoderAttention": CustomMMEncoderAttention}` will be added into `op_registry_oot`. When initializing a [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention") op object, if the class name (i.e., [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention")) is contained in the keys of `op_registry_oot`, vLLM will replace it with our registered class (i.e., `CustomMMEncoderAttention`) and instantiate it. After that, when this [`MMEncoderAttention`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/attention/mm_encoder_attention/#vllm.model_executor.layers.attention.mm_encoder_attention.MMEncoderAttention " MMEncoderAttention") op is called, your `forward_oot()` will be called if it is enabled. Thus, you will get expected performance on your hardwares without directly modify vLLM. In addition, you can also register all your [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") at one place for better management. Code `[](#__codelineno-14-1)from vllm.model_executor.custom_op import CustomOp [](#__codelineno-14-2) [](#__codelineno-14-3)[](#__codelineno-14-4)REGISTERED_CUSTOM_OPS = { [](#__codelineno-14-5) "CustomOP1": YourCustomOp1, [](#__codelineno-14-6) "CustomOP2": YourCustomOp2, [](#__codelineno-14-7) "CustomOP3": YourCustomOp3, [](#__codelineno-14-8)} [](#__codelineno-14-9)[](#__codelineno-14-10)for op_name, op_cls in REGISTERED_CUSTOM_OPS.items(): [](#__codelineno-14-11) CustomOp.register_oot(_decorated_op_cls=op_cls, name=op_name)` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/fused_moe_modular_kernel.md "Edit this page") ## Introduction[¶](#introduction "Permanent link") FusedMoEModularKernel is implemented [here](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/modular_kernel.py) Based on the format of the input activations, FusedMoE implementations are broadly classified into 2 types. - Contiguous / Standard / Non-Batched, and - Batched Note The terms Contiguous, Standard, and Non-Batched are used interchangeably throughout the document. The input activation format completely depends on the All2All Dispatch being used. - In the Contiguous variant, the All2All Dispatch returns the activations as a contiguous tensor of shape (M, K) along with TopK Ids and TopK weights of shape (M, num\_topk). Look at [`DeepEPHTPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ht/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht.DeepEPHTPrepareAndFinalize " DeepEPHTPrepareAndFinalize") for an example. - In the Batched variant, the All2All Dispatch returns the activations as a tensor of shape (num\_experts, max\_tokens, K). Here, the activations/tokens that subscribe to the same expert are batched together. Note that not all entries of the tensor are valid. The activations tensor is typically accompanied by an `expert_num_tokens` tensor of size `num_experts`, where `expert_num_tokens[i]` indicates the number of valid tokens that subscribe to the ith expert. Look at [`DeepEPLLPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ll/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll.DeepEPLLPrepareAndFinalize " DeepEPLLPrepareAndFinalize") for an example. The FusedMoE operation is generally made of multiple operations, in both the Contiguous and Batched variants, as described in the diagrams below [![FusedMoE Non-Batched](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/fused_moe_non_batched.png)](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/fused_moe_non_batched.png) [![FusedMoE Batched](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/fused_moe_batched.png)](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/fused_moe_batched.png) Note The main difference, in terms of operations, between the Batched and Non-Batched cases is the Permute / Unpermute operations. All other operations remain. ## Motivation[¶](#motivation "Permanent link") As can be seen from the diagrams, there are a lot of operations and there can be a variety of implementations for each operation. The set of ways the operations can be put together to make a valid FusedMoE implementation quickly becomes intractable. The Modular Kernel framework addresses this issue, by grouping the operations into logical components. This broad categorization makes the combinations manageable and prevents code-duplication. This also decouples the All2All Dispatch & Combine implementations from the FusedMoE implementations and allows for their independent development and testing. Furthermore, the Modular Kernel framework introduces Abstract classes for the different components thus providing a well-defined skeleton for future implementations. The rest of the document will focus on the Contiguous / Non-Batched case. Extrapolating to the Batched case should be straight-forward. ## ModularKernel Components[¶](#modularkernel-components "Permanent link") FusedMoEModularKernel splits the FusedMoE operation into 3 parts, 1. TopKWeightAndReduce 2. FusedMoEPrepareAndFinalizeModular 3. FusedMoEExpertsModular ### TopKWeightAndReduce[¶](#topkweightandreduce "Permanent link") The TopK Weight Application and Reduction components happen right after the Unpermute operation and before the All2All Combine. Note that the [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") is responsible for the Unpermute and [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") is responsible for the All2All Combine. There is value in doing the TopK Weight Application and Reduction in the [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular"). But some implementations choose to do it [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular"). In order to enable this flexibility, we have a TopKWeightAndReduce abstract class. Please find the implementations of TopKWeightAndReduce [here](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/fused_moe/topk_weight_and_reduce.py). `FusedMoEPrepareAndFinalizeModular::finalize()` method accepts a [`TopKWeightAndReduce`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.TopKWeightAndReduce " TopKWeightAndReduce") argument that is invoked inside the method. The `FusedMoEModularKernel` acts as a bridge between the [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") and [`FusedMoEPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalize " FusedMoEPrepareAndFinalize") implementations to determine where the TopK Weight Application and Reduction happens. - `FusedMoEExpertsModular::finalize_weight_and_reduce_impl` method returns `TopKWeightAndReduceNoOp` if the [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") implementation does the weight application and reduction itself. - `FusedMoEExpertsModular::finalize_weight_and_reduce_impl` method returns [`TopKWeightAndReduceContiguous`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/topk_weight_and_reduce/#vllm.model_executor.layers.fused_moe.topk_weight_and_reduce.TopKWeightAndReduceContiguous " TopKWeightAndReduceContiguous") / [`TopKWeightAndReduceNaiveBatched`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/topk_weight_and_reduce/#vllm.model_executor.layers.fused_moe.topk_weight_and_reduce.TopKWeightAndReduceNaiveBatched " TopKWeightAndReduceNaiveBatched") / [`TopKWeightAndReduceDelegate`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/topk_weight_and_reduce/#vllm.model_executor.layers.fused_moe.topk_weight_and_reduce.TopKWeightAndReduceDelegate " TopKWeightAndReduceDelegate") if the [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") implementation needs the `FusedMoEPrepareAndFinalizeModular::finalize()` to do the weight application and reduction. ### FusedMoEPrepareAndFinalizeModular[¶](#fusedmoeprepareandfinalizemodular "Permanent link") The [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") abstract class exposes `prepare`, `prepare_no_receive` and `finalize` functions. The `prepare` function is responsible for input activation Quantization and All2All Dispatch. If implemented, The `prepare_no_receive` is like `prepare` except it does not wait to receive results from other workers. Instead it returns a "receiver" callback that must be invoked to wait for the final results of worker. It is not required that this method is supported by all [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") classes, but if it is available, it can be used to interleave work with the initial all to all communication, e.g. interleaving shared experts with fused experts. The `finalize` function is responsible for invoking the All2All Combine. Additionally the `finalize` function may or may not do the TopK weight application and reduction (Please refer to the TopKWeightAndReduce section) [![FusedMoEPrepareAndFinalizeModular Blocks](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/prepare_and_finalize_blocks.png)](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/prepare_and_finalize_blocks.png) ### FusedMoEExpertsModular[¶](#fusedmoeexpertsmodular "Permanent link") The [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") class is where the crux of the MoE operations happen. The [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") abstract class exposes a few important functions, - apply() - workspace\_shapes() - finalize\_weight\_and\_reduce\_impl() #### apply()[¶](#apply "Permanent link") The `apply` method is where the implementations perform - Permute - Matmul with weight W1 - Act + Mul - Quantization - Matmul with weight W2 - Unpermute - Maybe TopK Weight Application + Reduction #### workspace\_shapes()[¶](#workspace_shapes "Permanent link") The core FusedMoE implementation performs a series of operations. It would be inefficient to create output memory for each of these operations separately. To that effect, implementations are required to declare 2 workspace shapes, the workspace datatype and the FusedMoE output shape as outputs of the workspace\_shapes() method. This information is used to allocate the workspace tensors and the output tensor in `FusedMoEModularKernel::forward()` and passed on to the `FusedMoEExpertsModular::apply()` method. The workspaces could then be used as intermediate buffers in the FusedMoE implementation. #### finalize\_weight\_and\_reduce\_impl()[¶](#finalize_weight_and_reduce_impl "Permanent link") It is sometimes efficient to perform TopK weight application and Reduction inside the `FusedMoEExpertsModular::apply()`. Find an example [here](https://github.com/vllm-project/vllm/pull/20228). We have a [`TopKWeightAndReduce`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.TopKWeightAndReduce " TopKWeightAndReduce") abstract class to facilitate such implementations. Please refer to the TopKWeightAndReduce section. `FusedMoEExpertsModular::finalize_weight_and_reduce_impl()` returns the [`TopKWeightAndReduce`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.TopKWeightAndReduce " TopKWeightAndReduce") object that the implementation wants the `FusedMoEPrepareAndFinalizeModular::finalize()` to use. [![FusedMoEExpertsModular Blocks](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/fused_experts_blocks.png)](https://docs.vllm.ai/en/assets/design/fused_moe_modular_kernel/fused_experts_blocks.png) ### FusedMoEModularKernel[¶](#fusedmoemodularkernel "Permanent link") `FusedMoEModularKernel` is composed of the [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") and [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") objects. `FusedMoEModularKernel` pseudocode/sketch, `[](#__codelineno-0-1)class FusedMoEModularKernel: [](#__codelineno-0-2) def __init__(self, [](#__codelineno-0-3) prepare_finalize: FusedMoEPrepareAndFinalizeModular, [](#__codelineno-0-4) fused_experts: FusedMoEExpertsModular): [](#__codelineno-0-5) [](#__codelineno-0-6) self.prepare_finalize = prepare_finalize [](#__codelineno-0-7) self.fused_experts = fused_experts [](#__codelineno-0-8) [](#__codelineno-0-9) def forward(self, DP_A): [](#__codelineno-0-10) [](#__codelineno-0-11) Aq, A_scale, _, _, _ = self.prepare_finalize.prepare(DP_A, ...) [](#__codelineno-0-12) [](#__codelineno-0-13) workspace13_shape, workspace2_shape, _, _ = self.fused_experts.workspace_shapes(...) [](#__codelineno-0-14) [](#__codelineno-0-15) # allocate workspaces [](#__codelineno-0-16) workspace_13 = torch.empty(workspace13_shape, ...) [](#__codelineno-0-17) workspace_2 = torch.empty(workspace2_shape, ...) [](#__codelineno-0-18) [](#__codelineno-0-19) # execute fused_experts [](#__codelineno-0-20) fe_out = self.fused_experts.apply(Aq, A_scale, workspace13, workspace2, ...) [](#__codelineno-0-21) [](#__codelineno-0-22) # war_impl is an object of type TopKWeightAndReduceNoOp if the fused_experts implementations [](#__codelineno-0-23) # performs the TopK Weight Application and Reduction. [](#__codelineno-0-24) war_impl = self.fused_experts.finalize_weight_and_reduce_impl() [](#__codelineno-0-25) [](#__codelineno-0-26) output = self.prepare_finalize.finalize(fe_out, war_impl,...) [](#__codelineno-0-27) [](#__codelineno-0-28) return output` ## How-To[¶](#how-to "Permanent link") ### How To Add a FusedMoEPrepareAndFinalizeModular Type[¶](#how-to-add-a-fusedmoeprepareandfinalizemodular-type "Permanent link") Typically a FusedMoEPrepareAndFinalizeModular type is backed by an All2All Dispatch & Combine implementation / kernel. For example, - DeepEPHTPrepareAndFinalize type is backed by DeepEP High-Throughput All2All kernels, and - DeepEPLLPrepareAndFinalize type is backed by DeepEP Low-Latency All2All kernels. #### Step 1: Add an All2All manager[¶](#step-1-add-an-all2all-manager "Permanent link") The purpose of the All2All Manager is to set up the All2All kernel implementations. The [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") implementations typically fetch a kernel-implementation "handle" from the All2All Manager to invoke the Dispatch and Combine functions. Please look at the All2All Manager implementations [here](https://github.com/vllm-project/vllm/blob/main/vllm/distributed/device_communicators/all2all.py). #### Step 2: Add a FusedMoEPrepareAndFinalizeModular Type[¶](#step-2-add-a-fusedmoeprepareandfinalizemodular-type "Permanent link") This section describes the significance of the various functions exposed by the [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") abstract class. `FusedMoEPrepareAndFinalizeModular::prepare()`: The prepare method implements the Quantization and All2All Dispatch. Typically the Dispatch function from the relevant All2All Manager is invoked. `FusedMoEPrepareAndFinalizeModular::has_prepare_no_receive()`: Indicates whether or not this subclass implements `prepare_no_receive`. Defaults to False. `FusedMoEPrepareAndFinalizeModular::prepare_no_receive()`: The prepare\_no\_receive method implements the Quantization and All2All Dispatch. It does not wait for the result of the dispatch operation but instead returns a thunk that can be invoked to wait for the final results. Typically the Dispatch function from the relevant All2All Manager is invoked. `FusedMoEPrepareAndFinalizeModular::finalize()`: Maybe perform TopK Weight Application and Reduction and All2All Combine. Typically the Combine function from the relevant All2AllManager is invoked. `FusedMoEPrepareAndFinalizeModular::activation_format()`: Return `FusedMoEActivationFormat.BatchedExperts` if the output of the prepare method (i.e. the All2All dispatch) is Batched. Return `FusedMoEActivationFormat.Standard` otherwise. `FusedMoEPrepareAndFinalizeModular::topk_indices_dtype()`: Data type of the TopK ids. Some All2All kernels have strict requirements pertaining to the data type of the TopK ids. This requirement is passed on to the `FusedMoe::select_experts` function so it could be respected. If there are no strict requirements return None. `FusedMoEPrepareAndFinalizeModular::max_num_tokens_per_rank()`: This is the maximum number of tokens that would be submitted to the All2All Dispatch at once. `FusedMoEPrepareAndFinalizeModular::num_dispatchers()`: Total number of dispatching units. This value determines the size of the Dispatch output. The Dispatch output is of shape (num\_local\_experts, max\_num\_tokens, K). Here max\_num\_tokens = num\_dispatchers() \* max\_num\_tokens\_per\_rank(). We suggest picking an already existing [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") implementation that matches your All2All implementation closely and using it as a reference. ### How To Add a FusedMoEExpertsModular Type[¶](#how-to-add-a-fusedmoeexpertsmodular-type "Permanent link") FusedMoEExpertsModular performs the core of the FusedMoE operations. The various functions exposed by the abstract class and their significance is as follows, `FusedMoEExpertsModular::activation_formats()`: Return the supported Input and Output activation formats. i.e. Contiguous / Batched format. `FusedMoEExpertsModular::supports_expert_map()`: Return True if the implementation supports expert map. `FusedMoEExpertsModular::workspace_shapes()` / `FusedMoEExpertsModular::finalize_weight_and_reduce_impl` / `FusedMoEExpertsModular::apply`: Refer to [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") section above. ### FusedMoEModularKernel Initialization[¶](#fusedmoemodularkernel-initialization "Permanent link") `FusedMoEMethodBase` class has 3 methods that are collectively responsible in creating the `FusedMoEModularKernel` object. They are, - maybe\_make\_prepare\_finalize, - select\_gemm\_impl, and - init\_prepare\_finalize #### maybe\_make\_prepare\_finalize[¶](#maybe_make_prepare_finalize "Permanent link") The `maybe_make_prepare_finalize` method is responsible for constructing an instance of [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") when appropriate based on the current all2all backend, e.g. when EP + DP is enabled. The base class method currently constructs all the [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") objects for the EP+DP case. Derived classes can override this method to construct prepare/finalize objects for different scenarios, e.g. [`ModelOptNvFp4FusedMoE`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/modelopt/#vllm.model_executor.layers.quantization.modelopt.ModelOptNvFp4FusedMoE " ModelOptNvFp4FusedMoE") can construct a `FlashInferCutlassMoEPrepareAndFinalize` for the EP+TP case. Please refer to the implementations in, - [`ModelOptNvFp4FusedMoE`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/modelopt/#vllm.model_executor.layers.quantization.modelopt.ModelOptNvFp4FusedMoE " ModelOptNvFp4FusedMoE") #### select\_gemm\_impl[¶](#select_gemm_impl "Permanent link") The `select_gemm_impl` method is undefined in the base class. It is the responsibility of the derived class to implement a method that constructs a valid/appropriate [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") object. Please refer to the implementations in, - [`UnquantizedFusedMoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method/#vllm.model_executor.layers.fused_moe.unquantized_fused_moe_method.UnquantizedFusedMoEMethod " UnquantizedFusedMoEMethod") - [`CompressedTensorsW8A8Fp8MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w8a8_fp8/#vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe.compressed_tensors_moe_w8a8_fp8.CompressedTensorsW8A8Fp8MoEMethod " CompressedTensorsW8A8Fp8MoEMethod") - `CompressedTensorsW8A8Fp8MoECutlassMethod` - [`Fp8MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/fp8/#vllm.model_executor.layers.quantization.fp8.Fp8MoEMethod " Fp8MoEMethod") - [`ModelOptNvFp4FusedMoE`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/modelopt/#vllm.model_executor.layers.quantization.modelopt.ModelOptNvFp4FusedMoE " ModelOptNvFp4FusedMoE") derived classes. #### init\_prepare\_finalize[¶](#init_prepare_finalize "Permanent link") Based on the input and env settings, the `init_prepare_finalize` method creates the appropriate [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") object. The method then queries `select_gemm_impl` for the appropriate [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") object and builds the `FusedMoEModularKernel` object Please take a look at [init\_prepare\_finalize](https://github.com/vllm-project/vllm/blob/1cbf951ba272c230823b947631065b826409fa62/vllm/model_executor/layers/fused_moe/layer.py#L188). **Important**: The `FusedMoEMethodBase` derived classes use the `FusedMoEMethodBase::fused_experts` object in their `apply` methods. When settings permit the construction of a valid `FusedMoEModularKernel` object, we override `FusedMoEMethodBase::fused_experts` with it. This essentially makes the derived classes agnostic to what FusedMoE implementation is used. ### How To Unit Test[¶](#how-to-unit-test "Permanent link") We have `FusedMoEModularKernel` unit tests at [test\_modular\_kernel\_combinations.py](https://github.com/vllm-project/vllm/blob/main/tests/kernels/moe/test_modular_kernel_combinations.py). The unit test iterates through all combinations of [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") and `FusedMoEPremuteExpertsUnpermute` types and if they are compatible, runs some correctness tests. If you are adding some [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") / [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") implementations, 1. Add the implementation type to `MK_ALL_PREPARE_FINALIZE_TYPES` and `MK_FUSED_EXPERT_TYPES` in [mk\_objects.py](https://github.com/vllm-project/vllm/blob/main/tests/kernels/moe/modular_kernel_tools/mk_objects.py) respectively. 2. Update `Config::is_batched_prepare_finalize()`, `Config::is_batched_fused_experts()`, `Config::is_standard_fused_experts()`, `Config::is_fe_16bit_supported()`, `Config::is_fe_fp8_supported()`, `Config::is_fe_block_fp8_supported()` methods in [/tests/kernels/moe/modular\_kernel\_tools/common.py](https://github.com/vllm-project/vllm/blob/main/tests/kernels/moe/modular_kernel_tools/common.py) Doing this will add the new implementation to the test suite. ### How To Check [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") & [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") Compatibility[¶](#how-to-check-fusedmoeprepareandfinalizemodular-fusedmoeexpertsmodular-compatibility "Permanent link") The unit test file [test\_modular\_kernel\_combinations.py](https://github.com/vllm-project/vllm/blob/main/tests/kernels/moe/test_modular_kernel_combinations.py) can also be executed as a standalone script. Example: `python3 -m tests.kernels.moe.test_modular_kernel_combinations --pf-type DeepEPLLPrepareAndFinalize --experts-type BatchedTritonExperts` As a side effect, this script can be used to test [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") & [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") compatibility. When invoked with incompatible types, the script will error. ### How To Profile[¶](#how-to-profile "Permanent link") Please take a look at [profile\_modular\_kernel.py](https://github.com/vllm-project/vllm/blob/main/tests/kernels/moe/modular_kernel_tools/profile_modular_kernel.py) The script can be used to generate Torch traces for a single `FusedMoEModularKernel::forward()` call for any compatible [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") and [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") types. Example: `python3 -m tests.kernels.moe.modular_kernel_tools.profile_modular_kernel --pf-type DeepEPLLPrepareAndFinalize --experts-type BatchedTritonExperts` ## FusedMoEPrepareAndFinalizeModular Implementations[¶](#fusedmoeprepareandfinalizemodular-implementations "Permanent link") See [Fused MoE Kernel features](https://docs.vllm.ai/en/latest/moe_kernel_features/#fused-moe-modular-all2all-backends) for a list of all the available modular prepare and finalize subclasses. ## FusedMoEExpertsModular[¶](#fusedmoeexpertsmodular_1 "Permanent link") See [Fused MoE Kernel features](https://docs.vllm.ai/en/latest/moe_kernel_features/#fused-moe-experts-kernels) for a list of all the available modular experts. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/attention_backends.md "Edit this page") This document is auto-generated by `tools/pre_commit/generate_attention_backend_docs.py`. It shows the feature support for each registered attention backend based on the checks in `AttentionBackend.validate_configuration()`. **Do not edit this file manually.** Run the following command to regenerate it: `[](#__codelineno-0-1)python tools/pre_commit/generate_attention_backend_docs.py` ## Setting the Attention Backend[¶](#setting-the-attention-backend "Permanent link") ### Command Line[¶](#command-line "Permanent link") There are two ways to specify the backend from the command line: **Option 1: Using `--attention-backend` (simple)** `[](#__codelineno-1-1)vllm serve --attention-backend FLASH_ATTN` **Option 2: Using `--attention-config.backend` / `-ac.backend` (structured config)** `[](#__codelineno-2-1)# Dot notation [](#__codelineno-2-2)vllm serve --attention-config.backend FLASH_ATTN [](#__codelineno-2-3)vllm serve -ac.backend FLASH_ATTN [](#__codelineno-2-4)[](#__codelineno-2-5)# JSON format [](#__codelineno-2-6)vllm serve --attention-config '{"backend": "FLASH_ATTN"}' [](#__codelineno-2-7)vllm serve -ac '{"backend": "FLASH_ATTN"}'` > **Note:** `--attention-backend` and `--attention-config.backend` are mutually exclusive. Use one or the other, not both. ### Python API[¶](#python-api "Permanent link") Use [`AttentionConfig`](https://docs.vllm.ai/en/api/vllm/config/attention/#vllm.config.attention.AttentionConfig " AttentionConfig") with the [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class: `[](#__codelineno-3-1)from vllm import LLM [](#__codelineno-3-2)from vllm.config import AttentionConfig [](#__codelineno-3-3)from vllm.v1.attention.backends.registry import AttentionBackendEnum [](#__codelineno-3-4)[](#__codelineno-3-5)# Method 1: Using AttentionConfig with enum [](#__codelineno-3-6)llm = LLM( [](#__codelineno-3-7) model="Qwen/Qwen3-0.6B", [](#__codelineno-3-8) attention_config=AttentionConfig(backend=AttentionBackendEnum.FLASH_ATTN), [](#__codelineno-3-9)) [](#__codelineno-3-10)[](#__codelineno-3-11)# Method 2: Using attention_backend parameter with string [](#__codelineno-3-12)llm = LLM( [](#__codelineno-3-13) model="Qwen/Qwen3-0.6B", [](#__codelineno-3-14) attention_backend="FLASH_ATTN", [](#__codelineno-3-15))` ## Backend Selection Behavior[¶](#backend-selection-behavior "Permanent link") ### Manual Selection[¶](#manual-selection "Permanent link") When you explicitly set a backend via `--attention-backend` or [`AttentionConfig`](https://docs.vllm.ai/en/api/vllm/config/attention/#vllm.config.attention.AttentionConfig " AttentionConfig"): 1. The backend is **validated** against your configuration (model dtype, head size, compute capability, etc.) 2. If the backend **doesn't support** your configuration, an error is raised with the specific reason 3. If valid, the backend is used Example error when selecting an incompatible backend: `[](#__codelineno-4-1)ValueError: Selected backend FLASHMLA is not valid for this configuration. [](#__codelineno-4-2)Reason: ['compute capability not supported']` ### Automatic Selection[¶](#automatic-selection "Permanent link") When no backend is specified (the default): 1. vLLM iterates through backends in **priority order** (see tables below) 2. Each backend is validated against your configuration 3. The **first compatible backend** is selected 4. If no backend is compatible, an error is raised listing all backends and their incompatibility reasons ## Backend Priority (CUDA)[¶](#backend-priority-cuda "Permanent link") When no backend is explicitly selected, vLLM chooses the first compatible backend from these priority-ordered lists. Priority is **1 = highest** (tried first). ### Standard Attention (MHA, MQA, GQA)[¶](#standard-attention-mha-mqa-gqa "Permanent link") **Blackwell (SM 10.x):** Priority Backend 1 `FLASHINFER` 2 `FLASH_ATTN` 3 `TRITON_ATTN` 4 `FLEX_ATTENTION` 5 `TURBOQUANT` **Ampere/Hopper (SM 8.x-9.x):** Priority Backend 1 `FLASH_ATTN` 2 `FLASHINFER` 3 `TRITON_ATTN` 4 `FLEX_ATTENTION` 5 `TURBOQUANT` ### MLA Attention (DeepSeek-style)[¶](#mla-attention-deepseek-style "Permanent link") **Blackwell (SM 10.x):** Priority Backend 1 `FLASHINFER_MLA` 2 `TOKENSPEED_MLA` 3 `CUTLASS_MLA` 4 `FLASH_ATTN_MLA` 5 `FLASHMLA` 6 `TRITON_MLA` 7 `FLASHINFER_MLA_SPARSE`**\*** 8 `FLASHMLA_SPARSE` **Ampere/Hopper (SM 8.x-9.x):** Priority Backend 1 `FLASH_ATTN_MLA` 2 `FLASHMLA` 3 `FLASHINFER_MLA` 4 `TRITON_MLA` 5 `FLASHMLA_SPARSE` > **\*** For sparse MLA, FP8 KV cache always prefers `FLASHINFER_MLA_SPARSE`. With BF16 KV cache, `FLASHINFER_MLA_SPARSE` is preferred for low query-head counts (<= 16), while `FLASHMLA_SPARSE` is preferred otherwise. > > **Note:** ROCm and CPU platforms have their own selection logic. See the platform-specific documentation for details. ## Legend[¶](#legend "Permanent link") Column Description **Dtypes** Supported model data types (fp16, bf16, fp32) **KV Dtypes** Supported KV cache data types (`auto`, `fp8`, `fp8_e4m3`, etc.) **Block Sizes** Supported KV cache block sizes (%N means multiples of N) **Head Sizes** Supported attention head sizes **Sink** Attention sink support (for StreamingLLM) **Non-Causal** Non-causal (bidirectional) attention support for decoder models **Sparse** Sparse attention support (MLA only) **MM Prefix** Multimodal prefix full attention support **DCP** Decode Context Parallelism support (`--decode-context-parallel-size`) **Attention Types** Supported attention patterns (Decoder, Encoder, Enc-Dec) **Compute Cap.** Required CUDA compute capability (N/A for non-CUDA backends) **Symbols:** ✅ = Supported, ❌ = Not supported ## Standard Attention (MHA, MQA, GQA) Backends[¶](#standard-attention-mha-mqa-gqa-backends "Permanent link") Backend Version Dtypes KV Dtypes Block Sizes Head Sizes Sink Non-Causal MM Prefix DCP Attention Types Compute Cap. `CPU_ATTN` fp16, bf16, fp32 `auto`, `fp8`, `fp8_e4m3`, `fp8_e5m2` %16 32, 64, 80, 96, 112, 128, 160, 192, 224, 256, 512 ❌ ❌ ❌ ❌ All N/A `FLASHINFER` Native† fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` 16, 32, 64 64, 128, 256, 512 ❌ ❌ ❌ ✅ Decoder 7.x-9.x `FLASHINFER` TRTLLM† fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2`, `nvfp4` 16, 32, 64 64, 128, 256, 512 ✅ ❌ ❌ ✅ Decoder 10.x `FLASH_ATTN` FA2\* fp16, bf16 `auto`, `float16`, `bfloat16` %16 Any ❌ ✅ ❌ ✅ All ≥8.0 `FLASH_ATTN` FA3\* fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` %16 Any ✅ ✅ ❌ ✅ All 9.x `FLASH_ATTN` FA4\* fp16, bf16 `auto`, `float16`, `bfloat16` %16 Any ✅ ✅ ❌ ✅ All ≥10.0 `FLASH_ATTN_DIFFKV` fp16, bf16 `auto` Any Any ❌ ❌ ❌ ✅ Decoder Any `FLEX_ATTENTION` fp16, bf16, fp32 `auto`, `float16`, `bfloat16` %16 Any ❌ ✅ ✅ ❌ Decoder, Encoder Only Any `ROCM_AITER_FA` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` 16, 32 64, 128, 256 ✅ ✅ ❌ ❌ Decoder N/A `ROCM_AITER_UNIFIED_ATTN` fp16, bf16 `auto` %16 Any ✅ ❌ ✅ ❌ All N/A `ROCM_ATTN` fp16, bf16, fp32 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` %16 32, 64, 80, 96, 128, 160, 192, 224, 256 ❌ ✅ ✅ ❌ Decoder, Encoder, Encoder Only N/A `TRITON_ATTN` fp16, bf16, fp32 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2`, `int8_per_token_head`, `fp8_per_token_head` %16 Any ✅ ❌ ✅ ❌ All Any `TURBOQUANT` fp16, bf16 `turboquant_k8v4`, `turboquant_4bit_nc`, `turboquant_k3v4_nc`, `turboquant_3bit_nc` 16, 32, 64, 128 Any ❌ ❌ ❌ ❌ Decoder Any > **†** FlashInfer uses TRTLLM attention on Blackwell (SM100), which supports sinks. Disable via `--attention-config.use_trtllm_attention=0`. > > **\*** Specify the FlashAttention version via `--attention-config.flash_attn_version=2`, `3`, or `4`. Default is FA4 on SM100+ (Blackwell), FA3 on SM90 (Hopper), FA2 otherwise. ## MLA (Multi-head Latent Attention) Backends[¶](#mla-multi-head-latent-attention-backends "Permanent link") MLA uses separate backends for prefill and decode phases. ### Prefill Backends[¶](#prefill-backends "Permanent link") To explicitly select a prefill backend, use `-ac.mla_prefill_backend=` (e.g., `FLASH_ATTN`, `FLASHINFER`). Otherwise, the prefill backend is selected automatically at runtime based on hardware and configuration. Backend Description Dtypes Compute Cap. Notes `FLASH_ATTN`‡ FlashAttention varlen (FA2/FA3/FA4) fp16, bf16 Any FA4 on SM100+, FA3 on SM90, FA2 otherwise `TRTLLM_RAGGED` TensorRT-LLM ragged attention fp16, bf16 10.x DeepSeek R1 dims only `FLASHINFER` FlashInfer CUTLASS backend fp16, bf16 10.x DeepSeek R1 dims only `TOKENSPEED_MLA` fp16, bf16 10.x DeepSeek R1 dims only > **‡** Automatic selection tries FlashAttention first. On Blackwell (SM100), the fallback order is TRT-LLM Ragged, FlashInfer, then TokenSpeed MLA. On other GPUs, only FlashAttention is considered. ### Decode Backends[¶](#decode-backends "Permanent link") MLA decode backends are selected using the standard `-ac.backend=` argument (e.g., `FLASHMLA`, `TRITON_MLA`). Backend Dtypes KV Dtypes Block Sizes Head Sizes Sink Non-Causal Sparse MM Prefix DCP Attention Types Compute Cap. `CUTLASS_MLA` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3` 128 Any ❌ ❌ ❌ ❌ ✅ Decoder 10.x `FLASHINFER_MLA` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3` 32, 64 Any ❌ ❌ ❌ ❌ ❌ Decoder 10.x `FLASHINFER_MLA_SPARSE` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3` 32, 64 576 ❌ ❌ ✅ ❌ ❌ Decoder 10.x `FLASHMLA` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3` 64 Any ❌ ❌ ❌ ❌ ✅ Decoder 9.x-10.x `FLASHMLA_SPARSE` bf16 `auto`, `bfloat16`, `fp8_ds_mla` 64 576 ❌ ❌ ✅ ❌ ❌ Decoder 9.x-10.x `FLASH_ATTN_MLA` fp16, bf16 `auto`, `float16`, `bfloat16` %16 Any ❌ ❌ ❌ ❌ ✅ Decoder 9.x `ROCM_AITER_MLA` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3`, `fp8_e5m2` %1 Any ❌ ❌ ❌ ❌ ❌ Decoder N/A `ROCM_AITER_MLA_SPARSE` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3` 1, 64 Any ❌ ❌ ✅ ❌ ❌ Decoder N/A `ROCM_AITER_TRITON_MLA` fp16, bf16 `auto` Any Any ❌ ❌ ❌ ❌ ❌ Decoder N/A `TOKENSPEED_MLA` fp16, bf16 `fp8`, `fp8_e4m3` 32, 64 Any ❌ ❌ ❌ ❌ ❌ Decoder 10.x `TRITON_MLA` fp16, bf16 `auto`, `float16`, `bfloat16`, `fp8`, `fp8_e4m3` %16 Any ❌ ❌ ❌ ❌ ✅ Decoder Any `XPU_MLA_SPARSE` fp16, bf16 `auto`, `float16`, `bfloat16` Any 576 ❌ ❌ ✅ ❌ ❌ Decoder Any ### DeepSeek V4 Decode Backends[¶](#deepseek-v4-decode-backends "Permanent link") DeepSeek V4 sparse MLA uses its own decode backends, selected via `--attention-backend=` (e.g., `FLASHMLA_SPARSE_DSV4`, `FLASHINFER_MLA_SPARSE_DSV4`). They share the V4 sparse-index pipeline (compressor + SWA + indexer, 256-token blocks, head 512); default on NVIDIA is `FLASHMLA_SPARSE_DSV4`. Backend Dtypes KV Dtypes Block Sizes Head Sizes Sink Non-Causal Sparse MM Prefix DCP Attention Types Compute Cap. `FLASHINFER_MLA_SPARSE_DSV4` fp16, bf16 `auto` Any Any ❌ ❌ ❌ ❌ ❌ Decoder Any `FLASHMLA_SPARSE_DSV4` bf16 `auto`, `bfloat16`, `fp8_ds_mla`, `fp8` 256 512 ❌ ❌ ✅ ❌ ❌ Decoder 9.x-10.x `ROCM_FLASHMLA_SPARSE_DSV4` fp16, bf16 `auto` Any Any ❌ ❌ ❌ ❌ ❌ Decoder N/A --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/dbo.md "Edit this page") ## Motivation[¶](#motivation "Permanent link") The core motivation of the DBO system in vLLM is to overlap the sparse all-to-all communication in the MoE layer with the surrounding computation. This system currently only targets DP+EP deployments. ## Introduction[¶](#introduction "Permanent link") The Dual Batch Overlap system works by splitting the batch in the model runner, creating two worker threads, and then running the model on each of these worker threads. When DBO is enabled, yield points within the `FusedMoEModularKernel` allow the two CPU worker threads (also called UBatch threads) to ping-pong between each other so that when one is running compute, the other is waiting on communication. Throughout the code, ubatch may be used as a short form of microbatch; this is an ASCII-friendly version of the short form µ-batch. The DBO system includes modifications to `GpuModelRunner` and `ModularKernel`, and defines two utility classes: `UBatchWrapper` and [`UBatchContext`](https://docs.vllm.ai/en/api/vllm/v1/worker/ubatching/#vllm.v1.worker.ubatching.UBatchContext " UBatchContext"). `UBatchWrapper` manages thread lifecycle and CUDA graph execution of the model. [`UBatchContext`](https://docs.vllm.ai/en/api/vllm/v1/worker/ubatching/#vllm.v1.worker.ubatching.UBatchContext " UBatchContext") wraps `ForwardContext` to coordinate synchronization between the two UBatch threads. Below is the overlap schedule that is currently implemented in vLLM. `[](#__codelineno-0-1)# Schedule notation legend: [](#__codelineno-0-2)# S = Shared expert [](#__codelineno-0-3)# A0 = MLA qkv proj, [](#__codelineno-0-4)# A1 = Core attn + out proj + MoE gate [](#__codelineno-0-5)# D = Dispatch [](#__codelineno-0-6)# C = Combine [](#__codelineno-0-7)[](#__codelineno-0-8)# Comp: |-A0₀-A1₀-||-MLP₁-||-S₁-MLP₀-||-S₀-A0₁-A1₁-| [](#__codelineno-0-9)# Comm: |----D₁---||--D₀--||----C₁---||-----C₀-----| [](#__codelineno-0-10)# Order: D₁ send, A0₀, A1₀, D₁ recv, D₀ send, MLP₁, D₀ recv, [](#__codelineno-0-11)# C₁ send, S₁, MLP₀, C₁ recv, C₀ send, S₀, A0₁, A1₁, C₀ recv. [](#__codelineno-0-12)# MLP_SHARED_OVERLAP = "mlp_shared_overlap"` ## Running with DBO[¶](#running-with-dbo "Permanent link") To enable the DBO system pass in the `--enable-dbo` argument to your vllm serve command. This must be run in conjunction with `--data-parallel-size N` where N is greater than 1 and `--enable-expert-parallel`. Additionally, there are two configuration knobs. - `--dbo-decode-token-threshold` the minimum number of tokens in a decode-only batch required to enable DBO for that batch - `--dbo-prefill-token-threshold` the minimum number of tokens in a batch containing at least one prefill required to enable DBO for that batch Currently, DBO is only supported with DeepEP, so DeepEP must be installed and the `--all2all-backend` argument must be set to `deepep_low_latency` if your workload is primarily decode requests, or `deepep_high_throughput` if your workload is primarily prefill requests. Below is a command that will spin up a two DP rank server with expert parallelism and DBO enabled. EX: `vllm serve deepseek-ai/DeepSeek-V2-Lite --trust-remote-code --data-parallel-size 2 --enable-expert-parallel --enable-dbo --all2all-backend deepep_low_latency` Note that there must be at least two GPUs visible in `CUDA_VISIBLE_DEVICES` ## DBO Components[¶](#dbo-components "Permanent link") - GPUModelRunner - UBatchWrapper - UBatchContext ### GPU Model Runner[¶](#gpu-model-runner "Permanent link") The batch is split into microbatches by the `GPUModelRunner` class. This is accomplished in two steps. First, coordination across all DP ranks is performed to determine whether microbatching will be applied. Microbatching must be uniform across all DP ranks. If microbatching is not feasible for any DP rank, it is disabled for all ranks. If all DP ranks are going to microbatch, the total number of tokens is padded up to the max number of tokens amongst all ranks. If any rank would end up with an empty second microbatch after the padding is applied, microbatching will be aborted and no ranks will microbatch. Once microbatching has been initiated by all ranks, the second step is performed. The [`CommonAttentionMetadata`](https://docs.vllm.ai/en/api/vllm/v1/attention/backend/#vllm.v1.attention.backend.CommonAttentionMetadata " CommonAttentionMetadata dataclass ") is sliced in half by the `GPUModelRunner` so that there is one attention metadata per-microbatch. ### UBatchWrapper[¶](#ubatchwrapper "Permanent link") gpu\_ubatch\_wrapper The `UBatchWrapper` class is a model wrapper that's responsible for all of the thread, UBatchContext, and CUDA graph management for DBO. It's designed to be relatively transparent to the GPU Model Runner. The implementation runs the model twice, once for each microbatch. Each model invocation occurs within a UBatch thread. These threads are launched in parallel and are synchronized using the [`UBatchContext`](https://docs.vllm.ai/en/api/vllm/v1/worker/ubatching/#vllm.v1.worker.ubatching.UBatchContext " UBatchContext"). Each thread is provided with a sliced version of the attention metadata that is used to run its half of the batch. CUDA graphs for DBO are entirely managed by the `UBatchWrapper`. Because of this, DBO only supports running with Full CUDA graphs. However, once a DBO CUDA graph has been captured, it can be replayed without any multithreading or CPU synchronization. #### Interfaces[¶](#interfaces "Permanent link") The `__init__` method takes in the model, VllmConfig, CUDAGraphMode, and device. The `forward` method exclusively takes in model arguments. It determines whether or not to run with DBO based on whether a `ubatch_slices` object is present in the `forward_context`. Otherwise, the model is run without DBO. ### UBatchContext[¶](#ubatchcontext "Permanent link") ubatch\_context The [`UBatchContext`](https://docs.vllm.ai/en/api/vllm/v1/worker/ubatching/#vllm.v1.worker.ubatching.UBatchContext " UBatchContext") class is a `ForwardContext` wrapper class that is used by the `UBatchWrapper` class to synchronize the two UBatch threads. It should only be instantiated by using `make_ubatch_contexts`. When one of the UBatch threads reaches a `dbo_yield` call, it pauses, and starts the other thread which will run until it reaches the same `dbo_yield` call. This "ping-pong" dynamic continues, with threads swapping at each `dbo_yield call`, until the model's execution is complete. The current implementation has all `dbo_yield` and `dbo_maybe_run_recv_hook` calls in the `FusedMoEModularKernel.forward` method. #### Interfaces[¶](#interfaces_1 "Permanent link") The `make_ubatch_context` function initializes two `UBatchContexts`, one for each UBatch thread. It takes two CUDA streams, the preexisting `ForwardContexts` and a CPU thread barrier. This function should be used exclusively to instantiate `UBatchContexts`. It will handle all of the event initialization. The `dbo_register_recv_hook` method registers a callback that can be returned by the [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") class in the other UBatch thread’s [`UBatchContext`](https://docs.vllm.ai/en/api/vllm/v1/worker/ubatching/#vllm.v1.worker.ubatching.UBatchContext " UBatchContext"). The callback will be run when the other thread calls `dbo_maybe_run_recv_hook`. This is typically used to wait on an all-to-all kernel. The `dbo_maybe_run_recv_hook` method runs a callback that’s set by the `dbo_register_recv_hook` function if that callback exists. The `dbo_yield` method puts the current thread to sleep and wakes up the other UBatch thread. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/debug_vllm_compile.md "Edit this page") TL;DR: - use tlparse to acquire torch.compile logs. Include these logs in bug reports and/or support asks. - The vLLM-torch.compile integration is multiple pieces. vLLM exposes flags to turn off each piece: Online Flag Offline Flag Result \--enforce-eager enforce\_eager=True Turn off torch.compile and CUDAGraphs \-cc.mode=0 compilation\_config=CompilationConfig(mode=CompilationMode.NONE) Turn off torch.compile only \-cc.mode=1 compilation\_config=CompilationConfig(mode=CompilationMode.STOCK\_TORCH\_COMPILE) Turn off vLLM-compile modifications to torch.compile \-cc.cudagraph\_mode=NONE compilation\_config=CompilationConfig(cudagraph\_mode=CUDAGraphMode.NONE) Turn off CUDAGraphs only \-cc.backend=eager compilation\_config=CompilationConfig(backend='eager') Turn off TorchInductor \-cc.ir\_enable\_torch\_wrap=False compilation\_config=CompilationConfig(ir\_enable\_torch\_wrap=False) Turn off vLLM IR wrapping ## vLLM-torch.compile overview[¶](#vllm-torchcompile-overview "Permanent link") To improve performance, vLLM leverages torch.compile and CUDAGraphs to speed things up. torch.compile generates optimized kernels for PyTorch code while CUDAGraphs eliminates overhead. Most notably, vLLM-compile is NOT torch.compile, it is a custom compiler built using internal PyTorch Compile APIs. [![vLLM-compile diagram](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/design_diagram.png)](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/design_diagram.png) - Given a model, we do a full graph capture via TorchDynamo that is dynamic on the batch size (number of tokens) - vLLM then optionally splits and/or specializes this graph and then uses TorchInductor to compile each graph into a compiled artifact. This step may use vLLM custom Inductor passes to further optimize the graph. This includes vLLM IR lowering to remove dispatch overhead. - The compiled artifact is saved to vLLM's compile cache so that it can be loaded in the future. - vLLM applies CUDAGraphs to reduce CPU overheads. Things can go wrong in each of the four steps. When something does go wrong, please try to isolate the subsystem that went wrong -- this will allow you to turn off the minimal number of things to keep reliability goals while minimizing impact to performance and also helps us (vLLM) when you open a bug report. For more details on the design, please see the following resources: - [Introduction to vLLM-torch.compile blogpost](https://blog.vllm.ai/2025/08/20/torch-compile.html) - [vLLM-torch.compile integration design](https://docs.vllm.ai/en/latest/torch_compile/) - [vLLM IR design](https://docs.vllm.ai/en/latest/vllm_ir/) - [vLLM Office Hours #26](https://www.youtube.com/live/xLyxc7hxCJc?si=Xulo9pe53C6ywf0V&t=561) - [Talk at PyTorch Conference 2025](https://youtu.be/1wV1ESbGrVQ?si=s1GqymUfwiwOrDTg&t=725) ## Use tlparse[¶](#use-tlparse "Permanent link") Use [tlparse](https://github.com/meta-pytorch/tlparse) to view torch.compile logs. These logs show all stages of the compilation process, including the fused kernels that torch.compile produces. Install tlparse: `[](#__codelineno-0-1)pip install tlparse` To enable the torch.compile logs, you can set the envvar `TORCH_TRACE=`. During tracing, a file per rank will be created inside of that directory, with each file containing the artifacts during compilation. If you can, we recommend sending these log files along with bug reports -- they are very helpful. Usage (offline inference) `[](#__codelineno-1-1)TORCH_TRACE=~/trace_dir python my_script.py [](#__codelineno-1-2)tlparse ~/trace_dir/` Usage (serving) `[](#__codelineno-2-1)TORCH_TRACE=~/trace_dir vllm serve [](#__codelineno-2-2)# ctrl-c out of the server [](#__codelineno-2-3)tlparse ~/trace_dir/` Given one of the log files, the `tlparse` command outputs some HTML files (perhaps into e.g. `./tl_out/index.html`). Open it to see the logs. It'll look something like the following: [![tlparse example](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/tlparse_inductor.png)](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/tlparse_inductor.png) ## Turn off vLLM-torch.compile integration[¶](#turn-off-vllm-torchcompile-integration "Permanent link") Pass `--enforce-eager` to turn off the vLLM-torch.compile integration and run entirely in eager mode. This includes turning off CUDAGraphs. `[](#__codelineno-3-1)# Online [](#__codelineno-3-2)vllm serve --enforce-eager` `[](#__codelineno-4-1)# Offline [](#__codelineno-4-2)LLM(model, enforce_eager=True)` To turn off just torch.compile, pass `mode = NONE` to the compilation config. (`-cc` is short for `--compilation_config`): `[](#__codelineno-5-1)# Online [](#__codelineno-5-2)vllm serve -cc.mode=0` `[](#__codelineno-6-1)# Offline [](#__codelineno-6-2)from vllm.config.compilation import CompilationConfig, CompilationMode [](#__codelineno-6-3)LLM(model, compilation_config=CompilationConfig(mode=CompilationMode.NONE))` To turn off just CUDAGraphs, pass `cudagraph_mode = NONE`: `[](#__codelineno-7-1)# Online [](#__codelineno-7-2)vllm serve -cc.cudagraph_mode=NONE` `[](#__codelineno-8-1)# Offline [](#__codelineno-8-2)from vllm.config.compilation import CompilationConfig, CUDAGraphMode [](#__codelineno-8-3)LLM(model, compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE))` vLLM IR makes heavy use of the compilation pipeline, from functionalization, custom fusions, and lowering. To turn that off and capture eager-mode dispatching behavior of vLLM IR, run with `ir_enable_torch_wrap=False`. IR torch wrap is only enabled by default when using `mode=VLLM_COMPILE` and `backend="inductor"` (default). `[](#__codelineno-9-1)# Online [](#__codelineno-9-2)vllm serve -cc.ir_enable_torch_wrap=False` `[](#__codelineno-10-1)# Offline [](#__codelineno-10-2)from vllm.config.compilation import CompilationConfig [](#__codelineno-10-3)LLM(model, compilation_config=CompilationConfig(ir_enable_torch_wrap=False))` ## Debugging TorchDynamo[¶](#debugging-torchdynamo "Permanent link") vLLM requires model code be capturable into a full graph via TorchDynamo (torch.compile's frontend). TorchDynamo does not support all of Python. It will error (in fullgraph mode) if it cannot support a feature (this is sometimes known as a graph break). If you encounter a graph break, please [open an issue to pytorch/pytorch](https://github.com/pytorch/pytorch) so the PyTorch devs can prioritize. Then, try your best to rewrite the code to avoid the graph break. For more information, see this [Dynamo guide](https://docs.pytorch.org/docs/stable/compile/programming_model.dynamo_core_concepts.html). ## Debugging Dynamic Shape full graph capture[¶](#debugging-dynamic-shape-full-graph-capture "Permanent link") vLLM requires that the model's forward pass be capturable into a full graph that is dynamic on the batch size (i.e. the number of tokens). It (by default) compiles this one graph into one artifact and uses this artifact for all batch sizes. If your code cannot be captured with Dynamic Shapes, you may see silent incorrectness, loud errors, or CUDA illegal memory accesses. For example, the following is not capturable into a single graph: `[](#__codelineno-11-1)if data.size[0] % 128 == 0: [](#__codelineno-11-2) foo(...) [](#__codelineno-11-3)else: [](#__codelineno-11-4) bar(...)` This problem is easy to diagnose. Use tlparse and click on `compilation_metrics`: it will tell you symbolic constraints on the batch size. If there is any constraint that restricts the batch sizes, then we've got a problem. [![Bad tlparse example](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/dynamic_shapes.png)](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/dynamic_shapes.png) To avoid this, please either: 1. avoid branching on the number of tokens 2. wrap the branching logic into a custom operator. TorchDynamo does not trace into custom operators. ## Debugging constraint violations and dynamic shapes guards issues[¶](#debugging-constraint-violations-and-dynamic-shapes-guards-issues "Permanent link") Dynamic-shape guards are a specific category of Dynamo guards. They are constraints that `torch.compile` attaches to dynamic dimensions (e.g., `seq_len`) to ensure the compiled artifact remains valid. These guards typically appear when framework code, custom passes, or user code branches based on dynamic shape values. **Example:** `[](#__codelineno-12-1)if x > 10: [](#__codelineno-12-2) # path A [](#__codelineno-12-3)else: [](#__codelineno-12-4) # path B` This creates a guard `x > 10` or `x <= 10` depending on which path was traced. **vLLM's Assumption:** vLLM assumes that all guards added by torch.compile are safe to drop and will not constrain the compiled graph to specific input shapes. When this assumption is violated, it can cause issues that users need to debug. Some side effects that indicates this assumption is violated are runtime errors or `ConstraintViolationErrors`. A `ConstraintViolationErrors` will be thrown if a dynamic shape gets constrained to a single value. If you encounter a constraint violation error or suspect that a dynamic shapes guard is being added incorrectly, you can use stricter dynamic shape modes to help debug the issue: `[](#__codelineno-13-1)# Online - using unbacked mode [](#__codelineno-13-2)vllm serve meta-llama/Llama-3.2-1B -cc.dynamic_shapes_config.type=unbacked [](#__codelineno-13-3)[](#__codelineno-13-4)# Online - using backed_size_oblivious mode [](#__codelineno-13-5)vllm serve meta-llama/Llama-3.2-1B -cc.dynamic_shapes_config.type=backed_size_oblivious` `[](#__codelineno-14-1)# Offline - using unbacked mode [](#__codelineno-14-2)from vllm.config.compilation import CompilationConfig, DynamicShapesConfig, DynamicShapesType [](#__codelineno-14-3)LLM(model, compilation_config=CompilationConfig( [](#__codelineno-14-4) dynamic_shapes_config=DynamicShapesConfig(type=DynamicShapesType.UNBACKED) [](#__codelineno-14-5))) [](#__codelineno-14-6)[](#__codelineno-14-7)# Offline - using backed_size_oblivious mode [](#__codelineno-14-8)from vllm.config.compilation import CompilationConfig, DynamicShapesConfig, DynamicShapesType [](#__codelineno-14-9)LLM(model, compilation_config=CompilationConfig( [](#__codelineno-14-10) dynamic_shapes_config=DynamicShapesConfig(type=DynamicShapesType.BACKED_SIZE_OBLIVIOUS) [](#__codelineno-14-11)))` These modes are stricter and reduce or eliminate the need of dynamic shapes guarding, which can help isolate issues: - `unbacked`: Uses unbacked symints which don't allow guards, making it easier to identify where guards are being incorrectly added - `backed_size_oblivious`: Uses a mode that is stricter about guarding. For more details on dynamic shapes modes, see [Dynamic shapes and vLLM guard dropping](https://docs.vllm.ai/en/latest/torch_compile/#dynamic-shapes-and-vllm-guard-dropping). ### Printing guards[¶](#printing-guards "Permanent link") To see all guards that are being added during compilation, you can use `TORCH_LOGS=+dynamic`: `[](#__codelineno-15-1)TORCH_LOGS=+dynamic vllm serve meta-llama/Llama-3.2-1B` Look for `[guard added]` in the logs to see where guards are being added. This can help you identify which operations are causing guards to be added incorrectly. ## Debugging TorchInductor[¶](#debugging-torchinductor "Permanent link") TorchInductor takes a captured graph and then compiles it down to some Python code that may call 1+ triton kernels. On rare (but unfortunate) occasions, it may produce an incorrect triton kernel. This may manifest as silent incorrectness, CUDA illegal memory accesses, or loud errors. ### Inductor runtime assertions[¶](#inductor-runtime-assertions "Permanent link") By default (on torch < 2.12), vLLM disables Inductor's runtime assertions (`assert_size_stride`, `assert_alignment`) to avoid ~2ms overhead per forward pass on large models. Setting `VLLM_LOGGING_LEVEL=DEBUG` automatically re-enables them so debugging sessions get full shape/stride validation: `[](#__codelineno-16-1)VLLM_LOGGING_LEVEL=DEBUG vllm serve ` You can also override them explicitly via `--compilation-config`: `[](#__codelineno-17-1)vllm serve -cc.inductor_compile_config='{"size_asserts": true, "alignment_asserts": true, "scalar_asserts": true}'` On torch >= 2.12, PyTorch uses an efficient assert-once strategy and these flags are no longer suppressed by vLLM. To debug if TorchInductor is at fault, you can disable it by passing `backend='eager'` to the compilation config: `[](#__codelineno-18-1)# online [](#__codelineno-18-2)vllm serve -cc.backend=eager` `[](#__codelineno-19-1)# offline [](#__codelineno-19-2)LLM(compilation_config=CompilationConfig(backend='eager'))` If Inductor is at fault, [file a bug to PyTorch](https://github.com/pytorch/pytorch). If you're feeling adventurous, you can debug the triton kernels in the Inductor output code (that you can locate via using tlparse). [![tlparse example](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/tlparse_inductor.png)](https://docs.vllm.ai/en/assets/design/debug_vllm_compile/tlparse_inductor.png) You can also use `TORCH_LOGS=output_code ` to print the Inductor output code. ### Editable TorchInductor code[¶](#editable-torchinductor-code "Permanent link") You can edit the TorchInductor code that gets run by setting `VLLM_COMPILE_CACHE_SAVE_FORMAT=unpacked` or passing `-cc.compile_cache_save_format=unpacked`. The default is `binary`, which means it is not editable. This is a useful technique: you can put breakpoints (e.g. `torch.distributed.breakpoint()`) and print statements in the output code. ## Debugging vLLM-compile cache[¶](#debugging-vllm-compile-cache "Permanent link") vLLM built its own cache for torch.compile artifacts. The idea is that the artifacts can be compiled once and then reused after they have been compiled. This is a layer on top of [torch.compile's compiler cache](https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html). While torch.compile's compiler cache is rock-stable, vLLM's compiler cache is unfortunately not always correct. You can disable it via setting `VLLM_DISABLE_COMPILE_CACHE=1`. You can also manually remove this cache. - Remove vLLM's compile cache with `rm -rf ~/.cache/vllm` (look at logs to see if the location changed) - Remove torch.compile's built-in caches with `rm -rf /tmp/torchinductor_$(whoami)` vLLM's cache is a mapping from cache key to a compiled artifact. vLLM computes the cache key via combining multiple factors (e.g. config flags and model name). If vLLM's compile cache is wrong, this usually means that a factor is missing. Please see [this example](https://github.com/vllm-project/vllm/blob/18b39828d90413d05d770dfd2e2f48304f4ca0eb/vllm/config/model.py#L310) of how vLLM computes part of the cache key. vLLM's compilation cache requires that the code being compiled ends up being serializable. If this is not the case, then it will error out on save. Usually the fixes are to either: - rewrite the non-serializable pieces (perhaps difficult because it's difficult to tell right now what is serializable and what isn't) - file a bug report - ignore the error by setting `VLLM_DISABLE_COMPILE_CACHE=1` (note that this will make warm server starts a lot slower). ## Debugging CUDAGraphs[¶](#debugging-cudagraphs "Permanent link") CUDAGraphs is a feature that allows one to: - Capture a callable that launches 1+ CUDA kernels into a CUDAGraph - Replay the CUDAGraph The captured CUDAGraph contains all of the memory used during the capture process. The replay of the CUDAGraph reads and writes to exactly the same regions of memory. This leads to some restrictions: 1. In order to use CUDAGraphs on new data, you'll need to copy the data into a buffer that the CUDAGraph is reading from 2. CUDAGraphs only capture CUDA kernels, they don't capture work done on CPU. vLLM uses the raw CUDAGraphs API, which is unsafe when used incorrectly. To turn off just CUDAGraphs, pass `cudagraph_mode = NONE`: `[](#__codelineno-20-1)# Online [](#__codelineno-20-2)vllm serve -cc.cudagraph_mode=NONE` `[](#__codelineno-21-1)# Offline [](#__codelineno-21-2)from vllm.config.compilation import CompilationConfig, CUDAGraphMode [](#__codelineno-21-3)LLM(model, compilation_config=CompilationConfig(cudagraph_mode=CUDAGraphMode.NONE))` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/cuda_graphs.md "Edit this page") This write-up introduces the new CUDA Graphs modes in vLLM v1 beyond previous [torch.compile integration](https://docs.vllm.ai/en/latest/torch_compile/). To summarize, we: 1. Added flexible `cudagraph_mode` configuration 2. Made full CUDA Graphs support orthogonal to compilation 3. Introduced a CUDA Graphs dispatcher as a central controller that picks the desired runtime mode and CUDA Graphs per batch automatically In this document we will discuss the: - [Motivation](#motivation) - [CUDA Graphs modes](#cudagraphmodes) - [Detailed design](#detailed-design) - [Example usage of the different CUDA Graphs modes](#usage-guide) - [Vision Encoder (ViT) CUDA Graphs](https://docs.vllm.ai/en/latest/cuda_graphs_multimodal/) Note In this document, we refer to pure decode (`max_query_len=1`) or speculative decode (`max_query_len =1+num_spec_tokens`) as **uniform decode** batches, and the opposite would be **non-uniform** batches (i.e., prefill or mixed prefill-decode batches). ## Motivation[¶](#motivation "Permanent link") Initial piecewise compilation was built to allow piecewise cudagraph capture, excluding cudagraph-unsupported operations (mainly attention). This allowed some speedup from cudagraphs while maintaining compatibility with all attention backends. We later added support for "full cudagraphs" by not compiling piecewise, so that we could further reduce the latency in cases where attention supported cudagraphs. However, this tight coupling between compilation and cudagraph capture led to an all-or-nothing experience with little flexibility. Many attention backends also weren’t ready for unified "full" CUDA Graphs capture (e.g., only FlashAttention 3 supports it currently) or only support CUDA Graphs for pure decode batches (e.g., Flashinfer, FlashMLA, and Mamba, etc.). That led to confusing performance/compatibility tradeoffs, inconsistent CUDA Graphs support, and increasingly complex code structure. This led us to seek a more fine-grained CUDA Graphs solution with the following features: - Explicitly aware of CUDA Graphs for prefill/mixed or (uniform-)decode batch and capture them separately. - Separate CUDAGraph capture logic from compilation (as much as feasible) for feature orthogonality, which suggest: - Capturing piecewise and full cudagraphs using the same compiled graph, and - Full cudagraph capture without compilation. - Dispatch between full and piecewise cudagraph at runtime depending on batch composition. - Centralized control of CUDAGraph behavior for reduced code complexity and allowed more extendibility. These features allow the most flexibility for cudagraph capture and compilation for all kinds of startup/performance tradeoffs and feature support. ## `CudagraphModes`[¶](#cudagraphmodes "Permanent link") [CUDAGraphMode](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.CUDAGraphMode " CUDAGraphMode") is the single knob you tune in `CompilationConfig.cudagraph_mode`: - `NONE` — turn CUDA Graphs off. Good for debugging. - `PIECEWISE` — a single-mode strategy (and past default). It is the most flexible: attention or other CUDA Graphs-incompatible operations stay eager, everything else goes into CUDA Graphs. Requires piecewise compilation. - `FULL` — a single-mode strategy, which only captures full CUDA Graphs for non-uniform batches, then uniform-decode batches reuse the CUDA Graph of non-uniform batch of the same batch\_size, since they are compatible; can be good for small models or workloads with small prompts. - `FULL_DECODE_ONLY` — full CUDA Graph for uniform decode, no cudagraph for prefill/mixed etc.; suitable for decode instances in a P/D setup where prefill is not as important, this way we can save the memory needed for `PIECEWISE` CUDA Graphs. - `FULL_AND_PIECEWISE` — (default mode) full CUDA Graph for uniform decode, piecewise CUDA Graphs for others; generally the most performant setting, especially for low latency with small models or MoEs, but also requires the most memory and takes the longest to capture. Defaults: If you’re on v1 with piecewise compilation, we default to `FULL_AND_PIECEWISE` for better performance, (for pooling models, it's still `PIECEWISE`). Otherwise, e.g. if piecewise compilation unavailable, we default to `NONE`. While `NONE` , `PIECEWISE`, and `FULL` are single-mode configurations and simply equivalent to past implementations of eager execution, piecewise CUDA Graphs, and full CUDA Graphs respectively, `FULL_DECODE_ONLY` and `FULL_AND_PIECEWISE` are newly appended dual-mode configurations, which require dispatching to switch between concrete runtime modes according to runtime batches dynamically. Note Here, the single-modes `NONE`, `PIECEWISE`, and `FULL` are treated as the runtime modes for CUDA Graphs dispatching. If using a dual-mode, the dispatcher will always dispatch to one of its member modes (plus a potential `NONE` if no suitable CUDA Graph available), depending on the batch composition. While cascade attention is not cudagraph compatible, it is now compatible with all possible cudagraph mode configurations. If a batch uses cascade attention, it always gets dispatched to `PIECEWISE` mode if available (otherwise `NONE`). Note Not all CUDA Graph modes are compatible with every attention backend. We automatically "downgrade" modes to the closest supported mode. For example, if a backend only supports CUDA Graphs for pure decode/uniform batches, we convert `FULL` to `FULL_AND_PIECEWISE` if piecewise compilation is enabled, and `FULL_DECODE_ONLY` otherwise. ## Detailed Design[¶](#detailed-design "Permanent link") ### Overview[¶](#overview "Permanent link") The new CUDA Graphs logic is built on top of piecewise compilation and supports dual CUDA Graphs runtime mode switching. The system contains the following core components: - [CUDAGraphWrapper](https://docs.vllm.ai/en/api/vllm/compilation/cuda_graph/#vllm.compilation.cuda_graph.CUDAGraphWrapper " CUDAGraphWrapper"): wrapper that handles CUDAGraph capture & replay on the wrapped callable - [CudagraphDispatcher](https://docs.vllm.ai/en/api/vllm/v1/cudagraph_dispatcher/#vllm.v1.cudagraph_dispatcher.CudagraphDispatcher " CudagraphDispatcher"): the central controller that contains the single source of truth about CUDA Graphs and handles dispatching between them. - [CUDAGraphMode](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.CUDAGraphMode " CUDAGraphMode"): enum describing the supported and runtime modes (introduced above). - [BatchDescriptor](https://docs.vllm.ai/en/api/vllm/forward_context/#vllm.forward_context.BatchDescriptor " BatchDescriptor dataclass "), serving as a unique representation of the runtime batch used for dispatching. See the following figures for a quick comparison between the previous and current design patterns of CUDA Graphs with inductor compilation. We can see that previously the CUDA Graphs logic and compilation logic were tightly coupled into the vllm `PiecewiseBackend`, and CUDA Graphs was implicitly dispatched by `batch_size` idly. Now the CUDA Graphs logic is separated into the [`CUDAGraphWrapper`](https://docs.vllm.ai/en/api/vllm/compilation/cuda_graph/#vllm.compilation.cuda_graph.CUDAGraphWrapper " CUDAGraphWrapper") class, responsible for both full and piecewise CUDA Graphs abilities, and dispatching is **explicitly** done via **runtime mode** plus the [`BatchDescriptor`](https://docs.vllm.ai/en/api/vllm/forward_context/#vllm.forward_context.BatchDescriptor " BatchDescriptor dataclass ") as the **dispatch key** via [`CudagraphDispatcher`](https://docs.vllm.ai/en/api/vllm/v1/cudagraph_dispatcher/#vllm.v1.cudagraph_dispatcher.CudagraphDispatcher " CudagraphDispatcher"). **Before:** [![previous_design](https://docs.vllm.ai/en/assets/design/cuda_graphs/previous_design.png)](https://docs.vllm.ai/en/assets/design/cuda_graphs/previous_design.png) **After:** [![new_design](https://docs.vllm.ai/en/assets/design/cuda_graphs/current_design.png)](https://docs.vllm.ai/en/assets/design/cuda_graphs/current_design.png) ### [`BatchDescriptor`](https://docs.vllm.ai/en/api/vllm/forward_context/#vllm.forward_context.BatchDescriptor " BatchDescriptor dataclass ")[¶](#batchdescriptor "Permanent link") [BatchDescriptor](https://docs.vllm.ai/en/api/vllm/forward_context/#vllm.forward_context.BatchDescriptor " BatchDescriptor dataclass ") is a component within `ForwardContext`, alongside the CUDA Graphs runtime modes, serving as the core structure for dispatching keys at runtime. The prototype is: `[](#__codelineno-0-1)class BatchDescriptor(NamedTuple): [](#__codelineno-0-2) num_tokens: int [](#__codelineno-0-3) num_reqs: int [](#__codelineno-0-4) uniform: bool = False [](#__codelineno-0-5) has_lora: bool = False` where `num_tokens` can be the padded token length, and `uniform` indicates if all the requests have the same query lengths. Many attention backends only support full cudagraphs when the batches are uniform; pure decode batches are uniform but may not be query length 1 (i.e. `num_tokens == num_reqs`), this occurs in the validation pass of spec-decode where "decode" batches will have a query length of `1+num_spec_tokens`. The goal of this structure is to uniquely identify a (padded) batch with minimal possible items corresponding to a CUDA Graphs item. Note The prototype of [`BatchDescriptor`](https://docs.vllm.ai/en/api/vllm/forward_context/#vllm.forward_context.BatchDescriptor " BatchDescriptor dataclass ") may be extended for more general situations in the future, e.g., include more items, like `uniform_query_len` to support multiple different uniform decode lengths settings ( [Pull Request #23679](https://github.com/vllm-project/vllm/pull/23679)), or other modifications needed to support CUDA Graphs for models whose inputs are not necessarily token length aware (for example, some multi-modal inputs). ### [`CudagraphDispatcher`](https://docs.vllm.ai/en/api/vllm/v1/cudagraph_dispatcher/#vllm.v1.cudagraph_dispatcher.CudagraphDispatcher " CudagraphDispatcher")[¶](#cudagraphdispatcher "Permanent link") The [CudagraphDispatcher](https://docs.vllm.ai/en/api/vllm/v1/cudagraph_dispatcher/#vllm.v1.cudagraph_dispatcher.CudagraphDispatcher " CudagraphDispatcher") takes responsibility for maintaining two sets of valid dispatching keys, one set for `FULL` runtime mode and one set for `PIECEWISE` runtime mode, and dispatches the correct runtime mode and the dispatching keys before executing the model's forwards. It will take in the initial key (a rough batch\_descriptor for the padded input) and return the selected runtime mode and the final batch\_descriptor, then tell the CUDAGraphWrapper instances that decision through forward contexts. Notice that [`CudagraphDispatcher`](https://docs.vllm.ai/en/api/vllm/v1/cudagraph_dispatcher/#vllm.v1.cudagraph_dispatcher.CudagraphDispatcher " CudagraphDispatcher") is the only source of truth for available CUDA Graph keys and [`CUDAGraphWrapper`](https://docs.vllm.ai/en/api/vllm/compilation/cuda_graph/#vllm.compilation.cuda_graph.CUDAGraphWrapper " CUDAGraphWrapper") instances can blindly trust the forward context on what CUDA Graphs to dispatch to. This lets us simplify the wrapper code and centralize the logic in the dispatcher. The dispatching keys are initialized through the dispatcher's `initialize_cudagraph_keys` method, which is called by the gpu\_model\_runner after all possible attention backends are initialized. This is where we can get much fancier in the future and “prepare” all kinds of CUDA Graphs combinations. For now, we just append available keys based on the valid combos of `decode_mode`/`mixed_mode` of `cudagraph_mode` and `cudagraph_capture_sizes` in the compilation config. The dispatch code looks like: `[](#__codelineno-1-1)batch_descriptor=BatchDescriptor(num_tokens=num_input_tokens, uniform_decode=...) [](#__codelineno-1-2)runtime_mode, batch_descriptor = cudagraphdispatcher.dispatch(batch_descriptor) [](#__codelineno-1-3)# execution [](#__codelineno-1-4)with set_forward_context( [](#__codelineno-1-5) ..., [](#__codelineno-1-6) cudagraph_runtime_mode=runtime_mode, [](#__codelineno-1-7) batch_descriptor=batch_descriptor, [](#__codelineno-1-8)): [](#__codelineno-1-9) output = self.model(...)` Inside the `dispatch()` method, the dispatcher will search the proper CUDA Graphs runtime mode and existing dispatching keys for a return. We basically search the existing keys following the priority: `FULL`\>`PIECEWISE`\>`None`. If the dispatching key does not exist, default to return `NONE` mode for eager execution. The implementations can be found [here](https://github.com/vllm-project/vllm/blob/main/vllm/v1/cudagraph_dispatcher.py#L91). Here is a simplified illustration of the workflow at runtime in the model executor: [![executor_runtime](https://docs.vllm.ai/en/assets/design/cuda_graphs/executor_runtime.png)](https://docs.vllm.ai/en/assets/design/cuda_graphs/executor_runtime.png) ### [`CUDAGraphWrapper`](https://docs.vllm.ai/en/api/vllm/compilation/cuda_graph/#vllm.compilation.cuda_graph.CUDAGraphWrapper " CUDAGraphWrapper")[¶](#cudagraphwrapper "Permanent link") A [CUDAGraphWrapper](https://docs.vllm.ai/en/api/vllm/compilation/cuda_graph/#vllm.compilation.cuda_graph.CUDAGraphWrapper " CUDAGraphWrapper") instance wraps a runnable and simply mimics the runnable with appended CUDA Graphs abilities. Each wrapper instance is bound to a specific `runtime_mode`, which is restricted to `PIECEWISE` and `FULL` mode, and takes responsibility for capturing/replaying and passing through (directly calling) the runnable. At runtime, each wrapper would: 1. inspect the runtime\_mode and batch\_descriptor(dispatching key) from the global forward context. 2. If runtime\_mode is `NONE` or runtime\_mode does not match the mode of the wrapper, just call the runnable directly. 3. Otherwise, i.e., the runtime\_mode matches the mode of the wrapper, the wrapper will perform CUDA Graphs capture (if key does not exist, create a new entry and cache it) or replay (if key exists in the cache). The above steps are based on the assumption that the CUDA Graphs wrapper would directly trust what’s in the forward context (controlled by the dispatcher). This lets us simplify and centralize the logic, reducing the complexity as well as the risk of mismatched state between the wrappers and the dispatcher. It also allows reusing the wrapper class for both `FULL` and `PIECEWISE` runtime modes. See the implementation [here](https://github.com/vllm-project/vllm/blob/f751e50b7a2aae3110d83ed0d88202fc91b3e78a/vllm/compilation/cuda_graph.py#L106). #### Nested Wrapper design[¶](#nested-wrapper-design "Permanent link") The core mechanism of making a full CUDA Graphs and piecewise CUDA Graphs coexist and compatible is the nested CUDA Graphs wrapper design, building on top of piecewise compilation with only a single piecewise FX graph. We wrap a FULL mode wrapper outside the entire model for the full CUDA Graphs functionality; meanwhile, each piecewise backend is wrapped via a `PIECEWISE` mode wrapper inside the compilation. The flow chart below should clearly describe how it works. [![wrapper_flow](https://docs.vllm.ai/en/assets/design/cuda_graphs/wrapper_flow.png)](https://docs.vllm.ai/en/assets/design/cuda_graphs/wrapper_flow.png) Therefore, for a `FULL` runtime mode, it is safe to capture/replay a full CUDA Graph since the piecewise wrapper is not activated. The situation is similar for `PIECEWISE` mode, as there are no conflicts between the `FULL` mode wrapper and `PIECEWISE` mode wrappers. For the `NONE` runtime mode, both `FULL` and `PIECEWISE` wrappers would not be activated, so we simply fall through to eager execution. ### Full CUDA Graph capturing & warm-up[¶](#full-cuda-graph-capturing-warm-up "Permanent link") The CUDA Graphs capturing happens when the runner first calls the model forward (using `_dummy_run`) with a non-`NONE` runtime mode. For full CUDA Graph capture, we explicitly capture different cases (i.e., prefill/mixed batch or uniform\_decode batch) by properly setting attention metadata to make sure the underlying attention backends launch the desired kernel routines. To distinguish prefill/mixed batch or uniform\_decode batch, the most important property is the `max_query_len` in attn\_metadata (true for most attention backends). We set it to the desired `uniform_query_len` for uniform\_decode otherwise we make it just the `num_tokens` for a non-uniform\_decode batch. The CUDA Graphs wrapper no longer manages the warm-up logic. The warm-up process is now controlled directly by the GPU model runner, where the `NONE` runtime mode is assigned to play an eager execution for warm-up. When warming up for a full CUDA Graph, it is also important to explicitly run attention during the warmup `dummy_run` call. ## CUDA Graphs Compatibility of Attention Backends[¶](#cuda-graphs-compatibility-of-attention-backends "Permanent link") To signal the CUDA Graphs compatibility of the attention backends, we introduce a new enum type [AttentionCGSupport](https://docs.vllm.ai/en/api/vllm/v1/attention/backend/#vllm.v1.attention.backend.AttentionCGSupport " AttentionCGSupport"), which is an enum type that tracks the capability of the attention backend to support CUDA Graphs. The value is sorted in the order of the capability, i.e., `ALWAYS`\> `UNIFORM_BATCH`\> `UNIFORM_SINGLE_TOKEN_DECODE`\> `NEVER`. `[](#__codelineno-2-1)class AttentionCGSupport(enum.Enum): [](#__codelineno-2-2) """ Constants for the CUDA Graphs support of the attention backend [](#__codelineno-2-3) Here we do not consider the cascade attention, as currently [](#__codelineno-2-4) it is never CUDA Graphs supported.""" [](#__codelineno-2-5) [](#__codelineno-2-6) ALWAYS = 3 [](#__codelineno-2-7) """CUDA Graphs always supported; supports mixed-prefill-decode""" [](#__codelineno-2-8) UNIFORM_BATCH = 2 [](#__codelineno-2-9) """CUDA Graphs supported for batches the only contain query lengths that are [](#__codelineno-2-10) the same, this can be used for spec-decode [](#__codelineno-2-11) i.e. "decodes" are 1 + num_speculative_tokens""" [](#__codelineno-2-12) UNIFORM_SINGLE_TOKEN_DECODE = 1 [](#__codelineno-2-13) """CUDA Graphs supported for batches the only contain query_len==1 decodes""" [](#__codelineno-2-14) NEVER = 0 [](#__codelineno-2-15) """NO CUDA Graphs support"""` Suppose we have hybrid attention backends (e.g., in mamba mixer models). In that case, we seek the minimum capability of all backends to determine the final capability of the model, and we might resolve the incompatible CUDA Graphs mode by downgrading the mode to the best fit one. For example, downgrading `FULL` mode to `FULL_AND_PIECEWISE` mode if the minimum capability is `UNIFORM_BATCH`, or `PIECEWISE` mode if the minimum capability is `NEVER` for -O3 compilation mode. For the complete fallback policy, please see the code for [this](https://docs.vllm.ai/en/api/vllm/v1/worker/gpu_model_runner/#vllm.v1.worker.gpu_model_runner.GPUModelRunner._check_and_update_cudagraph_mode " _check_and_update_cudagraph_mode(attention_backends, kv_cache_groups, is_profiling=False)"). The following table lists backends that support full CUDA Graphs at the time of writing. Attention Backend cudagraph\_support Comments FlashAttention v2 `UNIFORM_BATCH` Actually `ALWAYS` but workaround to fallback to `FULL_AND_PIECEWISE` for performance reason FlashAttention v3 `ALWAYS` has unified routine for both batches, so `FULL` mode is good Triton Attention `ALWAYS` prefer `FULL_AND_PIECEWISE` since it has different kernels for prefill/mixed and pure decode batches AITER FlashAttention `UNIFORM_BATCH` FlashInfer `UNIFORM_SINGLE_TOKEN_DECODE` Will be set to `UNIFORM_BATCH` when using TRTLLM attention on Blackwell FlashMLA `UNIFORM_BATCH` FlashInferMLA `UNIFORM_BATCH` FlashInferMLASparse `UNIFORM_BATCH` AITER MLA `UNIFORM_SINGLE_TOKEN_DECODE` CUTLASS MLA `UNIFORM_SINGLE_TOKEN_DECODE` Mamba attention `UNIFORM_SINGLE_TOKEN_DECODE` Unlisted backends are all declared as `NEVER`. ## Usage guide[¶](#usage-guide "Permanent link") Now the CLI is directly using the uppercase string of cudagraph\_mode for compilation\_config: `--compilation-config '{"cudagraph_mode": "..."}'`, where `...` should be one of `NONE`, `PIECEWISE`, `FULL`, `FULL_DECODE_ONLY`, and `FULL_AND_PIECEWISE`. Note that all `PIECEWISE` related modes require piecewise compilation, and all `FULL` related modes need CUDA Graphs support of attention backends. For example: `[](#__codelineno-3-1)vllm serve --model meta-llama/Llama-3.1-8B-Instruct --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}'` ### Python examples[¶](#python-examples "Permanent link") `[](#__codelineno-4-1)import os [](#__codelineno-4-2)os.environ.setdefault("VLLM_LOGGING_LEVEL", "DEBUG") [](#__codelineno-4-3)[](#__codelineno-4-4)import vllm [](#__codelineno-4-5)from vllm.config import CUDAGraphMode [](#__codelineno-4-6)[](#__codelineno-4-7)compilation_config = {"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE"} [](#__codelineno-4-8)model = vllm.LLM( [](#__codelineno-4-9) model="meta-llama/Llama-3.1-8B-Instruct", [](#__codelineno-4-10) dtype="auto", [](#__codelineno-4-11) compilation_config=compilation_config, [](#__codelineno-4-12)) [](#__codelineno-4-13)sampling_params = vllm.SamplingParams( [](#__codelineno-4-14) temperature=0, # greedy decoding [](#__codelineno-4-15) max_tokens=1024, [](#__codelineno-4-16)) [](#__codelineno-4-17)outputs = model.generate( [](#__codelineno-4-18) ["My name is John and"], [](#__codelineno-4-19) sampling_params=sampling_params, [](#__codelineno-4-20))` ### Piecewise compilation and full graph custom passes (attention fusion, sequence parallelism)[¶](#piecewise-compilation-and-full-graph-custom-passes-attention-fusion-sequence-parallelism "Permanent link") Unfortunately, some custom compile passes have to see the whole graph to be effective and hence aren't compatible with piecewise compilation. This includes [`AttnQuantFusionPass`](https://docs.vllm.ai/en/api/vllm/compilation/passes/fusion/attn_quant_fusion/#vllm.compilation.passes.fusion.attn_quant_fusion.AttnQuantFusionPass " AttnQuantFusionPass") and [`SequenceParallelismPass`](https://docs.vllm.ai/en/api/vllm/compilation/passes/fusion/sequence_parallelism/#vllm.compilation.passes.fusion.sequence_parallelism.SequenceParallelismPass " SequenceParallelismPass"). As a short-term solution, we automatically disable piecewise compilation (by setting `splitting_ops=[]`) when attention fusion is enabled. We use CUDA Graph modes `FULL` or `FULL_DECODE_ONLY` (depending on backend support). However, this leads to another optimization incompatibility and confusing performance tradeoffs. Long term, we've added the ability to partition the graph in Inductor instead of right after Dynamo. It can be enabled with `CompilationConfig.use_inductor_graph_partition=True` but is currently experimental and only available with `torch>=2.9`. This also increases compilation time as it has to compile the whole graph and cannot reuse piecewise compilation artifacts. Once vLLM supports 2.9, we plan to make this the default approach as it will also speed up piecewise cudagraph capture. ## About the Performance[¶](#about-the-performance "Permanent link") See the following links for examples: - [20059#issuecomment-3160858458](https://github.com/vllm-project/vllm/pull/20059#issuecomment-3160858458) - [20059#issuecomment-3188735226](https://github.com/vllm-project/vllm/pull/20059#issuecomment-3188735226) - [20059#issuecomment-3219888738](https://github.com/vllm-project/vllm/pull/20059#issuecomment-3219888738) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/huggingface_integration.md "Edit this page") This document describes how vLLM integrates with Hugging Face libraries. We will explain step by step what happens under the hood when we run `vllm serve`. Let's say we want to serve the popular Qwen model by running `vllm serve Qwen/Qwen2-7B`. 1. The `model` argument is `Qwen/Qwen2-7B`. vLLM determines whether this model exists by checking for the corresponding config file `config.json`. See this [code snippet](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/transformers_utils/config.py#L162-L182) for the implementation. Within this process: - If the `model` argument corresponds to an existing local path, vLLM will load the config file directly from this path. - If the `model` argument is a Hugging Face model ID consisting of a username and model name, vLLM will first try to use the config file from the Hugging Face local cache, using the `model` argument as the model name and the `--revision` argument as the revision. See [their website](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables#hfhome) for more information on how the Hugging Face cache works. - If the `model` argument is a Hugging Face model ID but it is not found in the cache, vLLM will download the config file from the Hugging Face model hub. Refer to [this function](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/transformers_utils/config.py#L91) for the implementation. The input arguments include the `model` argument as the model name, the `--revision` argument as the revision, and the environment variable `HF_TOKEN` as the token to access the model hub. In our case, vLLM will download the [config.json](https://huggingface.co/Qwen/Qwen2-7B/blob/main/config.json) file. 2. After confirming the existence of the model, vLLM loads its config file and converts it into a dictionary. See this [code snippet](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/transformers_utils/config.py#L185-L186) for the implementation. 3. Next, vLLM [inspects](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/transformers_utils/config.py#L189) the `model_type` field in the config dictionary to [generate](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/transformers_utils/config.py#L190-L216) the config object to use. There are some `model_type` values that vLLM directly supports; see [here](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/transformers_utils/config.py#L48) for the list. If the `model_type` is not in the list, vLLM will use [AutoConfig.from\_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoConfig.from_pretrained) to load the config class, with `model`, `--revision`, and `--trust_remote_code` as the arguments. Please note that: - Hugging Face also has its own logic to determine the config class to use. It will again use the `model_type` field to search for the class name in the transformers library; see [here](https://github.com/huggingface/transformers/tree/main/src/transformers/models) for the list of supported models. If the `model_type` is not found, Hugging Face will use the `auto_map` field from the config JSON file to determine the class name. Specifically, it is the `AutoConfig` field under `auto_map`. See [DeepSeek](https://huggingface.co/deepseek-ai/DeepSeek-V2.5/blob/main/config.json) for an example. - The `AutoConfig` field under `auto_map` points to a module path in the model's repository. To create the config class, Hugging Face will import the module and use the `from_pretrained` method to load the config class. This can generally cause arbitrary code execution, so it is only executed when `--trust_remote_code` is enabled. 4. Subsequently, vLLM applies some historical patches to the config object. These are mostly related to RoPE configuration; see [here](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/config.py#L244) for the implementation. 5. Finally, vLLM can reach the model class we want to initialize. vLLM uses the `architectures` field in the config object to determine the model class to initialize, as it maintains the mapping from architecture name to model class in [its registry](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/model_executor/models/registry.py#L80). If the architecture name is not found in the registry, it means this model architecture is not supported by vLLM. For `Qwen/Qwen2-7B`, the `architectures` field is `["Qwen2ForCausalLM"]`, which corresponds to the `Qwen2ForCausalLM` class in [vLLM's code](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/model_executor/models/qwen2.py#L364). This class will initialize itself depending on various configs. Beyond that, there are two more things vLLM depends on Hugging Face for. 1. **Tokenizer**: vLLM uses the tokenizer from Hugging Face to tokenize the input text. The tokenizer is loaded using [AutoTokenizer.from\_pretrained](https://huggingface.co/docs/transformers/en/model_doc/auto#transformers.AutoTokenizer.from_pretrained) with the `model` argument as the model name and the `--revision` argument as the revision. It is also possible to use a tokenizer from another model by specifying the `--tokenizer` argument in the `vllm serve` command. Other relevant arguments are `--tokenizer-revision` and `--tokenizer-mode`. Setting `VLLM_USE_FASTOKENS=1` swaps in a drop-in Rust BPE backend for any HF fast tokenizer loaded by vLLM (see [fastokens Backend](https://docs.vllm.ai/en/configuration/optimization/#fastokens-backend)). Please check Hugging Face's documentation for the meaning of these arguments. This part of the logic can be found in the [get\_tokenizer](https://github.com/vllm-project/vllm/blob/127c07480ecea15e4c2990820c457807ff78a057/vllm/transformers_utils/tokenizer.py#L87) function. After obtaining the tokenizer, notably, vLLM will cache some expensive attributes of the tokenizer in [vllm.tokenizers.hf.get\_cached\_tokenizer](https://docs.vllm.ai/en/api/vllm/tokenizers/hf/#vllm.tokenizers.hf.get_cached_tokenizer " get_cached_tokenizer(tokenizer)"). 2. **Model weight**: vLLM downloads the model weight from the Hugging Face model hub using the `model` argument as the model name and the `--revision` argument as the revision. vLLM provides the argument `--load-format` to control what files to download from the model hub. By default, it will try to load the weights in the safetensors format and fall back to the PyTorch bin format if the safetensors format is not available. We can also pass `--load-format dummy` to skip downloading the weights. - It is recommended to use the safetensors format, as it is efficient for loading in distributed inference and also safe from arbitrary code execution. See the [documentation](https://huggingface.co/docs/safetensors/en/index) for more information on the safetensors format. This part of the logic can be found [here](https://github.com/vllm-project/vllm/blob/10b67d865d92e376956345becafc249d4c3c0ab7/vllm/model_executor/model_loader/loader.py#L385). Please note that: This completes the integration between vLLM and Hugging Face. In summary, vLLM reads the config file `config.json`, tokenizer, and model weight from the Hugging Face model hub or a local directory. It uses the config class from either vLLM, Hugging Face transformers, or loads the config class from the model's repository. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/hybrid_kv_cache_manager.md "Edit this page") Warning This document was written based on commit [458e74](https://github.com/vllm-project/vllm/commit/458e74eb907f96069e6d8a4f3c9f457001fef2ea). This feature is still in its early stage and things may change. ## What is a hybrid model?[¶](#what-is-a-hybrid-model "Permanent link") Many recent "hybrid" LLMs combine multiple attention types within one model. For example: 1. Sliding window attention (sw) + full attention (full): gpt-oss, Gemma 2/3, Ministral, cohere, etc. 2. Mamba + full: Bamba, Jamba, Minimax, etc. 3. Local chunked attention + full: Llama4 To serve these models efficiently, our [KVCacheManager](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_manager/#vllm.v1.core.kv_cache_manager.KVCacheManager " KVCacheManager") must: 1. Allocate different slots to different layer type, for example: - Full attention layers: reserve slots for **all** tokens. - Sliding window layers: reserve slots only for the most recent **`sliding_window_size`** tokens. 2. Support layer-specific prefix-cache rules, for example: - Full attention: a cache hit prefix requires **all** tokens remain in the KV cache. - Sliding window: a cache hit prefix only requires the last **`sliding_window_size`** tokens remain in the KV cache. ## Definitions[¶](#definitions "Permanent link") 1. **kv hidden size**: The number of bytes to store one token's KV cache for a single layer. 2. **block**: the memory reserved for kv cache are divided into multiple _blocks_ with the same _page size_ (defined below) 3. **block size**: number of tokens inside a block 4. **page size**: the physical memory size of a block, defined as: \\\[ \\text{num\_layers} \\times \\text{block\_size} \\times \\text{kv\_hidden\_size} \\\] `num_layers` doesn't mean the total number of layers in the model. The exact number depends on the context in this doc. Note This is different from `KVCacheSpec.page_size_bytes` in the code, which is defined as: \\\[ \\text{block\_size} \\times \\text{kv\_hidden\_size} \\\] ## Allocation[¶](#allocation "Permanent link") ### High level idea[¶](#high-level-idea "Permanent link") We use a single memory pool for all layer types. The memory pool is split into multiple blocks with the same page size. [KVCacheManager](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_manager/#vllm.v1.core.kv_cache_manager.KVCacheManager " KVCacheManager") allocates different numbers of blocks to different layers according to its attention type. The core challenge is ensuring every layer type uses the same **page size**. For full-attention-only models, the page size is straightforward, defined as: \\\[ \\text{page\_size} = \\text{block\_size} \\times \\text{num\_hidden\_layers} \\times \\text{kv\_hidden\_size} \\\] However, in hybrid models, `num_hidden_layers` varies by attention type, which would normally produce mismatched page sizes. The cases below show how we unify them. ### Case 1: toy model[¶](#case-1-toy-model "Permanent link") Let's start with a toy example: a model has 1 full attention layer and 3 sliding window attention layers. All layers have the same `kv_hidden_size`. We let each block to hold `block_size` tokens for one layer, so: \\\[ \\text{page\_size} = \\text{kv\_hidden\_size} \\times \\text{block\_size} \\\] [KVCacheManager](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_manager/#vllm.v1.core.kv_cache_manager.KVCacheManager " KVCacheManager") allocates a different number of blocks to each layer. This case is only a toy example. For real models, please refer to the following cases. ### Case 2: same `kv_hidden_size` and a regular pattern[¶](#case-2-same-kv_hidden_size-and-a-regular-pattern "Permanent link") When the model has more layers, e.g., 20 sliding window attention layers and 10 full attention layers with the same `kv_hidden_size`. Calling the allocator once per layer (30 calls) is OK but becomes inefficient. As a solution, we group the allocation of layers that need the same number of blocks to reduce the number of calls. The grouping is feasible because there is usually a beautiful ratio between the number of different types of layers. For example: - Gemma-2: 1 sw : 1 full - Llama 4: 3 local : 1 full Our example can be regarded as 2 sw : 1 full. We can allocate blocks as if there are 2 sw and 1 full in the model, and repeat the result by 10 times to generate the `block_ids` for the 30 layers. The page size becomes: \\\[ 10 \\times \\text{kv\_hidden\_size} \\times \\text{block\_size} \\\] Assume `block_size` 16, sliding window size 32, request length 112, then for the above example model, we need to allocate 11 blocks (0-6 for full, 7-8 for sw group 1, 9-10 for sw group 2). [![Allocation Result](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/basic_grouping_example.png)](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/basic_grouping_example.png) Here, "/" denotes no block needed (sliding‑window layers don't need slots for early tokens). See the formal definition below. The layers are divided into multiple _KV Cache Groups_ so that there is: 1. **Identical attention type inside each group**: Each group only contains layers with the same attention type and thus need the same number of blocks for a given request. This enables layers in the same group share the same block ids without memory waste. 2. **Identical page size across groups**: Because our memory pool only have one page size. Our example model is divided into 3 KV cache groups: - Group 0: 10 full attention layers (full.0 - full.9) - Group 1: 10 sliding window attention layers (sw.0 - sw.9) - Group 2: 10 sliding window attention layers (sw.10 - sw.19) Obviously, it satisfies rule 1. For rule 2, all 3 groups have \\\[ 10 \\times \\text{kv\_hidden\_size} \\times \\text{block\_size} \\\] as their page size. ### Case 3: same `kv_hidden_size` and no regular pattern[¶](#case-3-same-kv_hidden_size-and-no-regular-pattern "Permanent link") Unfortunately, not all models have such a beautiful ratio, and approach in Case 2 will produce too many small groups. For example, Gemma-3-27b has 52 sliding window attention layers and 10 full attention layers. With the constraints in case 2, it would be 26 sliding window groups and 5 full attention groups, each contains 2 layers. The allocation is still inefficient. To reduce the number of kv cache groups, we group layers using the smallest layer count among all attention types. For example, min(52, 10)=10 layers per group in Gemma-3-27b. Then the grouping result is: - Group 0: 10 full attention layers (full.0 - full.9) - Group 1: 10 sliding window attention layers (sw.0 - sw.9) - Group 2: 10 sliding window attention layers (sw.10 - sw.19) - ... - Group 6: 10 sliding window attention layers (sw.40 - sw.49) - Group 7: 2 sliding window attention layers (sw.50 - sw.51) and 8 padding layers We will update this algorithm if this heuristic leads to a bad result when a new model comes out (e.g., 20 full + 30 sw, the group size should be 10 instead of 20). This case happens in Gemma-3 series models, and models in case 2 but with eagle speculative decoding which introduce one full attention layer. The solution has some memory waste and is not perfect. Please report any cases where padding overhead becomes unacceptable so we can refine the algorithm. ### Case 4: different `kv_hidden_size` (mainly hybrid mamba models)[¶](#case-4-different-kv_hidden_size-mainly-hybrid-mamba-models "Permanent link") Some architectures (e.g., Bamba, Jamba, Minimax) interleave standard attention layers with Mamba layers, where each Mamba layer's state size per token can be much larger than the attention layers' `kv_hidden_size`. Because we only support a single page size across all groups, we must reconcile these differing hidden sizes. The current algorithm is: 1. Increase the `block_size` of attention layers until $$ \\text{block\_size} \\times \\text{kv\_hidden\_size}_{\\text{att}} \\ge \\text{state\_size}_ $$} 2. Pad the mamba state per layer to $$ \\text{block\_size} \\times \\text{kv\_hidden\_size}\_{\\text{att}} $$ 3. Apply the grouping strategy in case 3. Note This can lead to more than 400 `block_size` for attention layers, which is too large. Another padding strategy is to increase `block_size` until \\\[ \\text{block\_size} \\times \\text{kv\_hidden\_size}\_{\\text{att}} \\times \\text{num\_attn\_layers} \\ge \\text{state\_size}\_{\\text{mamba}} \\\] This padding strategy is still a work in progress. ### Case 5: KV sharing[¶](#case-5-kv-sharing "Permanent link") KV sharing refers to a layer using the KV cache of another layer, e.g., gemma-3n. In these models, [KVCacheManager](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_manager/#vllm.v1.core.kv_cache_manager.KVCacheManager " KVCacheManager") ignores all layers with kv sharing and only allocates KV cache for layers that need kv cache, and some patches are made in model runner to apply the allocation result to kv sharing layers. ## Prefix caching[¶](#prefix-caching "Permanent link") For simplicity, we assume `block_size=1` in this section. ### High level idea[¶](#high-level-idea_1 "Permanent link") The block pool uses a dict similar to `tuple(block_hash, group_id) -> block` to catch the full blocks. That means the same tokens of different groups are cached and evicted independently. When a new request comes in, we check the cache hit prefix of each group, and return the intersection of these groups as the cached prefix of the request. See below for the detailed algorithm for checking the cache hit of one group & performing the intersection. ### Case 0: full attention only models[¶](#case-0-full-attention-only-models "Permanent link") For full attention layers, blocks are allocated for all tokens in the request. For details on the underlying design, see [Prefix Caching](https://docs.vllm.ai/en/latest/prefix_caching/) To find the longest cache hit prefix of a request, we enumerate from left (the first block) to right (the last block), checking whether the block is cached, and exit when cache misses. For example, we will return the first 7 tokens (0-6) as the cache hit prefix in the below example (blue blocks are cached): [![Prefix Caching of Full Attention](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/full_attn.png)](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/full_attn.png) ### Case 1: sliding window attention only models[¶](#case-1-sliding-window-attention-only-models "Permanent link") For sliding window attention layers, a naive implementation for memory allocation is to allocate `sliding_window_size` blocks and fill in the blocks in a round-robin way. But this naive implementation is not compatible with prefix caching so we didn't pick this design. In vLLM, we allocate different blocks for different tokens and free blocks that are outside the sliding window. For a new request, the cache hit prefix only requires the last `sliding_window_size - 1` tokens being cached. Let's say `sliding_window_size = 4` and `block_size = 1`, and the request is a 15-token prompt (blue blocks are cached): [![Prefix Caching of Sliding Window Attention](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/sw_attn.png)](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/sw_attn.png) There are 3 possible cache hit prefixes: - cache hit length 5, compute prefill with \[2, 3, 4\] → \[5, 6, …, 14\] - cache hit length 6, compute prefill with \[3, 4, 5\] → \[6, 7, …, 14\] - cache hit length 14, compute prefill with \[11, 12, 13\] → \[14\] (most efficient) We can check the cache hit from right to left, and early exit when we find a match.This is opposite from full attention, where we check from left to right and early exit when the match fails. One potential cons (compared to full attention) is that we end up iterating over the entire list of tokens when there's no match, which is often a common case. This could potentially cause non-negligible overheads, but fine with full + swa, as discussed below. ### Case 2: sliding window attention + full attention models[¶](#case-2-sliding-window-attention-full-attention-models "Permanent link") The first problem is how to find the cache hit prefix. We need to "intersect" the cache hits of global and sliding window attention layers by: 1. Get the longest cache hit for full attention (scanning from left to right) 2. Get the longest cache hit for sliding window attention that is within that length. Implemented by checking cache hits from right to left starting from the cache hit length of full attention. It can be ensured that the resulting cache hit of sliding window attention layers is also a cache hit of full attention layers. This is more efficient than finding all possible prefixes of each group and doing the intersection, because our approach can exit early if there is no cache hit. The algorithm applies to models with exactly two attention types full attention + X, where X can be an arbitrary efficient attention algorithm like sliding window, llama 4 local attention, and mamba. It doesn't support models without full attention layers, and models with more than 2 types of attention. This is enough for most hybrid models at the moment of writing this doc. The second question is the cache eviction policy. For now, we use one LRU queue for all kv cache groups. The blocks are added to the LRU queue when freed, either because the request is finished or the block is out of the sliding window. ### Case 3: mamba models[¶](#case-3-mamba-models "Permanent link") The prefix caching support of the mamba model is work in progress. Once implemented, models with mamba layer + full attention layer can be supported via the full attention + X algorithm in case 2. ## Implementation[¶](#implementation "Permanent link") ### Overview[¶](#overview "Permanent link") [![Overview of Hybrid KV Cache Manager](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/overview.png)](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/overview.png) The `KVCacheManager` is organized into 3 layers: - **[KVCacheManager](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_manager/#vllm.v1.core.kv_cache_manager.KVCacheManager " KVCacheManager")**: The interface between the scheduler and kv cache management system. - **[KVCacheCoordinator](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_coordinator/#vllm.v1.core.kv_cache_coordinator.KVCacheCoordinator " KVCacheCoordinator")**: coordinate per-group SingleTypeKVCacheManagers to generate the allocation result of a request. Depending on the model's configuration, one of these coordinators is chosen: - **[KVCacheCoordinatorNoPrefixCache](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_coordinator/#vllm.v1.core.kv_cache_coordinator.KVCacheCoordinatorNoPrefixCache " KVCacheCoordinatorNoPrefixCache")**: Used when prefix caching is disabled. - **[UnitaryKVCacheCoordinator](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_coordinator/#vllm.v1.core.kv_cache_coordinator.UnitaryKVCacheCoordinator " UnitaryKVCacheCoordinator")**: If only one KV cache group. The prefix caching logic is simplified as no intersection is needed. - **[HybridKVCacheCoordinator](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_coordinator/#vllm.v1.core.kv_cache_coordinator.HybridKVCacheCoordinator " HybridKVCacheCoordinator")**: Handles exactly two KV cache groups (must include one full‑attention group plus one other efficient‑attention group). Other cases are not implemented. You can disable prefix caching to use the KVCacheCoordinatorNoPrefixCache. - **[SingleTypeKVCacheManager](https://docs.vllm.ai/en/api/vllm/v1/core/single_type_kv_cache_manager/#vllm.v1.core.single_type_kv_cache_manager.SingleTypeKVCacheManager " SingleTypeKVCacheManager")**: Each instance manages allocation and prefix caching for one KV cache group, implementing the attention‑type–specific logic (e.g., full attention, sliding window, Mamba). The blue box in the above figure shows the case with 10 full attention layers and 20 sliding window attention layers, thus: - use [`HybridKVCacheCoordinator`](https://docs.vllm.ai/en/api/vllm/v1/core/kv_cache_coordinator/#vllm.v1.core.kv_cache_coordinator.HybridKVCacheCoordinator " HybridKVCacheCoordinator") - use 1 `FullAttentionManager` and 2 `SlidingWindowManager` for the 3 `KVCacheGroup`s. ### Memory Layout[¶](#memory-layout "Permanent link") For a model with n `KVCacheGroup`s, each with m layers, we allocate m buffers. Each buffer is shared by n layers, one from each group. The following figure is for a model with 10 full attention layers (full.0 - full.9) and 20 sliding window attention layers (sw.0-sw.19). It follows "case 2" in "Allocation" section and is divided into 3 groups: - Group 0: 10 full attention layers (full.0 - full.9) - Group 1: 10 sliding window attention layers (sw.0 - sw.9) - Group 2: 10 sliding window attention layers (sw.10 - sw.19) And for a request, we allocate 11 blocks with `block_id` 0-6 to group 0, 7-8 to group 1, and 9-10 to group 2. With such an example, the physical memory is divided into 10 buffers ([`KVCacheTensor`](https://docs.vllm.ai/en/api/vllm/v1/kv_cache_interface/#vllm.v1.kv_cache_interface.KVCacheTensor " KVCacheTensor dataclass ") 0 - [`KVCacheTensor`](https://docs.vllm.ai/en/api/vllm/v1/kv_cache_interface/#vllm.v1.kv_cache_interface.KVCacheTensor " KVCacheTensor dataclass ") 9). Each buffer is shared by 3 layers (e.g., [`KVCacheTensor`](https://docs.vllm.ai/en/api/vllm/v1/kv_cache_interface/#vllm.v1.kv_cache_interface.KVCacheTensor " KVCacheTensor dataclass ") 0 is shared by full.0 from group 0, sw.0 from group 1, and sw.10 from group 2) and is divided into pieces with size `block_size * kv_hidden_size`. The KV cache of these 3 attention layers are saved to different pieces of the buffer based on the allocated `block_ids`: [![Example Memory Layout](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/memory_layout.png)](https://docs.vllm.ai/en/assets/design/hybrid_kv_cache_manager/memory_layout.png) Note One logic "block" is mapped to 10 pieces in the 10 buffers of the physical memory. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/io_processor_plugins.md "Edit this page") IO Processor plugins are a feature that allows pre- and post-processing of the model input and output for pooling models. The idea is that users are allowed to pass a custom input to vLLM that is converted into one or more model prompts and fed to the model `encode` method. One potential use-case of such plugins is that of using vLLM for generating multi-modal data. Say users feed an image to vLLM and get an image in output. When performing an inference with IO Processor plugins, the prompt type is defined by the plugin and the same is valid for the final request output. vLLM does not perform any validation of input/output data, and it is up to the plugin to ensure the correct data is being fed to the model and returned to the user. As of now these plugins support only pooling models and can be triggered via the `encode` method in [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") and [`AsyncLLM`](https://docs.vllm.ai/en/api/vllm/v1/engine/async_llm/#vllm.v1.engine.async_llm.AsyncLLM " AsyncLLM"), or in online serving mode via the `/pooling` endpoint. ## Writing an IO Processor Plugin[¶](#writing-an-io-processor-plugin "Permanent link") IO Processor plugins implement the [`IOProcessor`](https://docs.vllm.ai/en/api/vllm/plugins/io_processors/interface/#vllm.plugins.io_processors.interface.IOProcessor " IOProcessor") interface: `[](#__codelineno-0-1)IOProcessorInput = TypeVar("IOProcessorInput") [](#__codelineno-0-2)IOProcessorOutput = TypeVar("IOProcessorOutput") [](#__codelineno-0-3)[](#__codelineno-0-4)class IOProcessor(ABC, Generic[IOProcessorInput, IOProcessorOutput]): [](#__codelineno-0-5) """Abstract interface for pre/post-processing of engine I/O.""" [](#__codelineno-0-6) [](#__codelineno-0-7) def __init__(self, vllm_config: VllmConfig, renderer: BaseRenderer): [](#__codelineno-0-8) super().__init__() [](#__codelineno-0-9) [](#__codelineno-0-10) self.vllm_config = vllm_config [](#__codelineno-0-11) [](#__codelineno-0-12) def parse_data(self, data: object) -> IOProcessorInput: [](#__codelineno-0-13) raise NotImplementedError [](#__codelineno-0-14) [](#__codelineno-0-15) def merge_sampling_params( [](#__codelineno-0-16) self, [](#__codelineno-0-17) params: SamplingParams | None = None, [](#__codelineno-0-18) ) -> SamplingParams: [](#__codelineno-0-19) return params or SamplingParams() [](#__codelineno-0-20) [](#__codelineno-0-21) def merge_pooling_params( [](#__codelineno-0-22) self, [](#__codelineno-0-23) params: PoolingParams | None = None, [](#__codelineno-0-24) ) -> PoolingParams: [](#__codelineno-0-25) return params or PoolingParams(task="plugin") [](#__codelineno-0-26) [](#__codelineno-0-27) @abstractmethod [](#__codelineno-0-28) def pre_process( [](#__codelineno-0-29) self, [](#__codelineno-0-30) prompt: IOProcessorInput, [](#__codelineno-0-31) request_id: str | None = None, [](#__codelineno-0-32) **kwargs, [](#__codelineno-0-33) ) -> PromptType | Sequence[PromptType]: [](#__codelineno-0-34) raise NotImplementedError [](#__codelineno-0-35) [](#__codelineno-0-36) async def pre_process_async( [](#__codelineno-0-37) self, [](#__codelineno-0-38) prompt: IOProcessorInput, [](#__codelineno-0-39) request_id: str | None = None, [](#__codelineno-0-40) **kwargs, [](#__codelineno-0-41) ) -> PromptType | Sequence[PromptType]: [](#__codelineno-0-42) return self.pre_process(prompt, request_id, **kwargs) [](#__codelineno-0-43) [](#__codelineno-0-44) @abstractmethod [](#__codelineno-0-45) def post_process( [](#__codelineno-0-46) self, [](#__codelineno-0-47) model_output: Sequence[PoolingRequestOutput], [](#__codelineno-0-48) request_id: str | None = None, [](#__codelineno-0-49) **kwargs, [](#__codelineno-0-50) ) -> IOProcessorOutput: [](#__codelineno-0-51) raise NotImplementedError [](#__codelineno-0-52) [](#__codelineno-0-53) async def post_process_async( [](#__codelineno-0-54) self, [](#__codelineno-0-55) model_output: AsyncGenerator[tuple[int, PoolingRequestOutput]], [](#__codelineno-0-56) request_id: str | None = None, [](#__codelineno-0-57) **kwargs, [](#__codelineno-0-58) ) -> IOProcessorOutput: [](#__codelineno-0-59) # We cannot guarantee outputs are returned in the same order they were [](#__codelineno-0-60) # fed to vLLM. [](#__codelineno-0-61) # Let's sort them by id before post_processing [](#__codelineno-0-62) sorted_output = sorted( [](#__codelineno-0-63) [(i, item) async for i, item in model_output], key=lambda output: output[0] [](#__codelineno-0-64) ) [](#__codelineno-0-65) collected_output = [output[1] for output in sorted_output] [](#__codelineno-0-66) return self.post_process(collected_output, request_id=request_id, **kwargs)` The `parse_data` method is used for validating the user data and converting it into the input expected by the `pre_process*` methods. The `merge_sampling_params` and `merge_pooling_params` methods merge input [`SamplingParams`](https://docs.vllm.ai/en/api/vllm/sampling_params/#vllm.sampling_params.SamplingParams " SamplingParams") or [`PoolingParams`](https://docs.vllm.ai/en/api/vllm/pooling_params/#vllm.pooling_params.PoolingParams " PoolingParams") (if any) with the default one. The `pre_process*` methods take the validated plugin input to generate vLLM's model prompts for regular inference. The `post_process*` methods take [`PoolingRequestOutput`](https://docs.vllm.ai/en/api/vllm/outputs/#vllm.outputs.PoolingRequestOutput " PoolingRequestOutput") objects as input and generate a custom plugin output. An example implementation of a plugin that enables generating geotiff images with the PrithviGeospatialMAE model is available [here](https://github.com/IBM/terratorch/tree/main/terratorch/vllm/plugins/segmentation). Please, also refer to our online ( [examples/pooling/plugin/prithvi\_geospatial\_mae\_online.py](https://github.com/vllm-project/vllm/blob/main/examples/pooling/plugin/prithvi_geospatial_mae_online.py)) and offline ( [examples/pooling/plugin/prithvi\_geospatial\_mae\_io\_processor.py](https://github.com/vllm-project/vllm/blob/main/examples/pooling/plugin/prithvi_geospatial_mae_io_processor.py)) inference examples. ## Using an IO Processor plugin[¶](#using-an-io-processor-plugin "Permanent link") IO Processor plugins are loaded at engine startup and there are two methods for specifying the name of the plugin to be loaded: 1. Via vLLM's [`EngineArgs`](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.EngineArgs " EngineArgs dataclass "): setting the `io_processor_plugin` argument in the [`EngineArgs`](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.EngineArgs " EngineArgs dataclass ") used to initialize the [`AsyncLLM`](https://docs.vllm.ai/en/api/vllm/v1/engine/async_llm/#vllm.v1.engine.async_llm.AsyncLLM " AsyncLLM"). The same can be achieved by passing the `io_processor_plugin` argument to [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") in offline mode, or by passing the `--io-processor-plugin` argument in serving mode. 2. Via the model HF configuration: adding an `io_processor_plugin` field to the model config (config.json). The order also determines method priority. i.e., setting the plugin name via [`EngineArgs`](https://docs.vllm.ai/en/api/vllm/engine/arg_utils/#vllm.engine.arg_utils.EngineArgs " EngineArgs dataclass ") will override any plugin name specified in the model HF config (config.json). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/logits_processors.md "Edit this page") Important Some logits processors design changes are still in progress and the API may change in the near future. We hope to stabilize this part of the API soon This document describes how the vLLM engine interacts with logits processors, and the programming model which vLLM supports for implementing logits processors. ## Logits Processors Background[¶](#logits-processors-background "Permanent link") A logits processor adjusts the next-token probability distribution, usually with the intention of steering the model towards a desired type of behavior. In vLLM, logits processors operate at batch granularity. During a given engine step, the logits processor consumes a `(num_requests) x (vocab_size)` tensor of raw logits output by the model. For all requests which enable the logits processor, the logits processor applies a transformation to the corresponding row of the logits tensor, while leaving other rows unmodified. The transformed logits tensor is then passed to softmax. ## Logits Processors in the vLLM engine[¶](#logits-processors-in-the-vllm-engine "Permanent link") The vLLM engine's persistent batch data structure maintains a list of loaded logits processors. In order to operate on the entire batch at once, each logits processor may maintain metadata about the requests in the batch (i.e. each request's logits-processor-specific configuration settings). Therefore, logits processors are stateful. In each engine step, the vLLM engine will (1) update each logits processor's internal state and (2) apply logits processors to the model output logits. ### Updating Logits Processor Internal State[¶](#updating-logits-processor-internal-state "Permanent link") At the beginning of each engine step, the persistent batch may add, discard and/or reorder requests in response to the scheduler output. After the persistent batch has reorganized, the vLLM engine invokes each logits processor's `update_state()` method. This is necessary to ensure that logits processors' internal states are reorganized to match the new persistent batch state at the beginning of the engine step. The pseudocode below shows the process by which the vLLM persistent batch notifies each logits processor of changes in batch state: Model Runner Updates Logits Processor States `[](#__codelineno-0-1)# gpu_model_runner.py [](#__codelineno-0-2)[](#__codelineno-0-3)class GPUModelRunner(...): [](#__codelineno-0-4) [](#__codelineno-0-5) ... [](#__codelineno-0-6) [](#__codelineno-0-7) def execute_model(self, scheduler_output, ...): [](#__codelineno-0-8) self._update_states(scheduler_output) [](#__codelineno-0-9) [](#__codelineno-0-10) ... [](#__codelineno-0-11) [](#__codelineno-0-12) def _update_states(...): [](#__codelineno-0-13) [](#__codelineno-0-14) ... [](#__codelineno-0-15) [](#__codelineno-0-16) # ...update persistent batch to reflect new/finished requests & reordering [](#__codelineno-0-17) # of requests within batch... [](#__codelineno-0-18) [](#__codelineno-0-19) ... [](#__codelineno-0-20) [](#__codelineno-0-21) self.input_batch.refresh_metadata() [](#__codelineno-0-22) [](#__codelineno-0-23)[](#__codelineno-0-24)# gpu_input_batch.py [](#__codelineno-0-25)[](#__codelineno-0-26)class InputBatch: [](#__codelineno-0-27) [](#__codelineno-0-28) ... [](#__codelineno-0-29) [](#__codelineno-0-30) def refresh_metadata(self): [](#__codelineno-0-31) [](#__codelineno-0-32) ... [](#__codelineno-0-33) [](#__codelineno-0-34) # Update each logits processor's state to reflect persistent batch state [](#__codelineno-0-35) batch_update = self.batch_update_builder.get_and_reset(self.num_reqs) [](#__codelineno-0-36) for logit_proc in self.logitsprocs.all: [](#__codelineno-0-37) logit_proc.update_state(batch_update) [](#__codelineno-0-38) [](#__codelineno-0-39) ... [](#__codelineno-0-40) [](#__codelineno-0-41)[](#__codelineno-0-42)# vllm/v1/sample/logits_processor/interface.py [](#__codelineno-0-43)[](#__codelineno-0-44)@dataclass(frozen=True) [](#__codelineno-0-45)class BatchUpdate: [](#__codelineno-0-46) # Batch state-change data structure which is passed to logits processors' [](#__codelineno-0-47) # update_state() methods [](#__codelineno-0-48) [](#__codelineno-0-49) batch_size: int [](#__codelineno-0-50) [](#__codelineno-0-51) removed: Sequence[RemovedRequest] [](#__codelineno-0-52) added: Sequence[AddedRequest] [](#__codelineno-0-53) moved: Sequence[MovedRequest]` ### Applying Logits Processors to the Model Output Logits[¶](#applying-logits-processors-to-the-model-output-logits "Permanent link") After updating persistent batch state, the vLLM model runner performs model inference to obtain logits. Then, the model runner invokes the sampler against the logits. In turn, part of the sampler's operation is to invoke the logits processors' `apply()` methods against the model output logit processors, yielding transformed logits (the `apply()` methods may modify the logits in-place or out-of-place, although in-place is more memory-efficient). This process is shown in the pseudocode below. Note that the sampler will access the logits processors via `SamplingMetadata.logitsprocs`. When the vLLM engine constructs `SamplingMetadata` (not shown in the code below), the reference to the list of logits processors is passed from the persistent batch data structure to `SamplingMetadata`. Apply logits processors to model output logits `[](#__codelineno-1-1)# gpu_model_runner.py [](#__codelineno-1-2)[](#__codelineno-1-3)class GPUModelRunner(...): [](#__codelineno-1-4) [](#__codelineno-1-5) ... [](#__codelineno-1-6) [](#__codelineno-1-7) def execute_model(self, scheduler_output, ...): [](#__codelineno-1-8) # (discussed in previous section) [](#__codelineno-1-9) self._update_states(scheduler_output) [](#__codelineno-1-10) [](#__codelineno-1-11) ... [](#__codelineno-1-12) [](#__codelineno-1-13) # ...run model inference to obtain logits... [](#__codelineno-1-14) [](#__codelineno-1-15) ... [](#__codelineno-1-16) [](#__codelineno-1-17) # Invoke sampler, which applies logits processors [](#__codelineno-1-18) sampler_output = self.sampler(logits=logits, [](#__codelineno-1-19) sampling_metadata=sampling_metadata) [](#__codelineno-1-20) [](#__codelineno-1-21) ... [](#__codelineno-1-22) [](#__codelineno-1-23)[](#__codelineno-1-24)# sampler.py [](#__codelineno-1-25)[](#__codelineno-1-26)class Sampler(nn.Module): [](#__codelineno-1-27) [](#__codelineno-1-28) ... [](#__codelineno-1-29) [](#__codelineno-1-30) def forward(self, logits, sampling_metadata): [](#__codelineno-1-31) [](#__codelineno-1-32) ... [](#__codelineno-1-33) [](#__codelineno-1-34) # Apply non-argmax-invariant logits processors to model output logits [](#__codelineno-1-35) for processor in (sampling_metadata.logitsprocs.non_argmax_invariant): [](#__codelineno-1-36) logits = processor.apply(logits) [](#__codelineno-1-37) [](#__codelineno-1-38) sampled = self.sample(logits, sampling_metadata) [](#__codelineno-1-39) [](#__codelineno-1-40) ... [](#__codelineno-1-41) [](#__codelineno-1-42) # ...return sampler output data structure... [](#__codelineno-1-43) [](#__codelineno-1-44) [](#__codelineno-1-45) def sample(self, logits, sampling_metadata) [](#__codelineno-1-46) [](#__codelineno-1-47) ... [](#__codelineno-1-48) [](#__codelineno-1-49) # ...exit early if all requests are greedy-sampling... [](#__codelineno-1-50) [](#__codelineno-1-51) ... [](#__codelineno-1-52) [](#__codelineno-1-53) # Apply argmax-invariant logits processors [](#__codelineno-1-54) for processor in sampling_metadata.logitsprocs.argmax_invariant: [](#__codelineno-1-55) logits = processor.apply(logits) [](#__codelineno-1-56) [](#__codelineno-1-57) ... [](#__codelineno-1-58) [](#__codelineno-1-59) # ...perform sampling and return sampling result...` At sampling time, the sampler checks whether all requests in the persistent batch employ greedy sampling. If that is the case, the sampler saves compute by skipping "argmax-invariant" logits processors. Here, "argmax" is shorthand for the token ID with the highest logit value in a given row of the logits tensor (i.e. the token which the model weighted the highest for a given request). - An **argmax-invariant logits processor** is a logits processor (such as Min-P) which does not modify the argmax. For example, a logits processor which masks out the lowest-probability tokens will not change which token ID has the max logit. Greedy sampling always picks the highest-logit-value token ID, and so conceptually an argmax-invariant logits processor can be skipped for greedy sampling requests. - A **non-argmax-invariant logits processor** is a logits processor which may modify the argmax. For example, a logits processor which masks all tokens except for EOS after a certain number of steps in order to force decoding to terminate might end up masking the max-logit-value token and therefore change the argmax. Conceptually, these logits processors cannot be skipped for greedy sampling requests. The vLLM logits processor abstraction requires the engine to apply logits processors at batch granularity; therefore in practice the argmax-invariant logits processors can only be skipped when the entire batch uses greedy sampling. ## Logits Processor Programming Model[¶](#logits-processor-programming-model "Permanent link") The previous sections alluded to the interfaces which vLLM logits processors must support. This section introduces in full the programming model for implementing logits processors that are compatible with the vLLM engine, including the [`LogitsProcessor`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/logits_processor/#vllm.model_executor.layers.logits_processor.LogitsProcessor " LogitsProcessor") base class and its interface methods as well as the [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure for representing persistent batch state changes, both of which are shown in the code below: [`LogitsProcessor`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/logits_processor/#vllm.model_executor.layers.logits_processor.LogitsProcessor " LogitsProcessor") base class and [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure ``[](#__codelineno-2-1)from abc import ABC, abstractmethod [](#__codelineno-2-2)from collections.abc import Sequence [](#__codelineno-2-3)from dataclasses import dataclass [](#__codelineno-2-4)from enum import Enum, auto [](#__codelineno-2-5)from typing import TYPE_CHECKING [](#__codelineno-2-6)[](#__codelineno-2-7)import torch [](#__codelineno-2-8)[](#__codelineno-2-9)from vllm import SamplingParams [](#__codelineno-2-10)[](#__codelineno-2-11)if TYPE_CHECKING: [](#__codelineno-2-12) from vllm.config import VllmConfig [](#__codelineno-2-13) [](#__codelineno-2-14)[](#__codelineno-2-15)class MoveDirectionality(Enum): [](#__codelineno-2-16) # One-way i1->i2 req move within batch [](#__codelineno-2-17) UNIDIRECTIONAL = auto() [](#__codelineno-2-18) # Two-way i1<->i2 req swap within batch [](#__codelineno-2-19) SWAP = auto() [](#__codelineno-2-20) [](#__codelineno-2-21)[](#__codelineno-2-22)# (index, params, prompt_tok_ids, output_tok_ids) tuples for new [](#__codelineno-2-23)# requests added to the batch. [](#__codelineno-2-24)AddedRequest = tuple[int, SamplingParams, list[int], list[int]] [](#__codelineno-2-25)[](#__codelineno-2-26)# (index 1, index 2, directionality) tuples representing [](#__codelineno-2-27)# one-way moves or two-way swaps of requests in batch [](#__codelineno-2-28)MovedRequest = tuple[int, int, MoveDirectionality] [](#__codelineno-2-29)[](#__codelineno-2-30)# Batch indices of any removed requests. [](#__codelineno-2-31)RemovedRequest = int [](#__codelineno-2-32) [](#__codelineno-2-33)[](#__codelineno-2-34)@dataclass(frozen=True) [](#__codelineno-2-35)class BatchUpdate: [](#__codelineno-2-36) """Persistent batch state change info for logitsprocs""" [](#__codelineno-2-37) batch_size: int # Current num reqs in batch [](#__codelineno-2-38) [](#__codelineno-2-39) # Metadata for requests added to, removed from, and moved [](#__codelineno-2-40) # within the persistent batch. [](#__codelineno-2-41) # [](#__codelineno-2-42) # Key assumption: the `output_tok_ids` list (which is an element of each [](#__codelineno-2-43) # tuple in `added`) is a reference to the request's running output tokens [](#__codelineno-2-44) # list; via this reference, the logits processors always see the latest [](#__codelineno-2-45) # list of generated output tokens [](#__codelineno-2-46) removed: Sequence[RemovedRequest] [](#__codelineno-2-47) moved: Sequence[MovedRequest] [](#__codelineno-2-48) added: Sequence[AddedRequest] [](#__codelineno-2-49) [](#__codelineno-2-50)[](#__codelineno-2-51)class LogitsProcessor(ABC): [](#__codelineno-2-52) [](#__codelineno-2-53) @abstractmethod [](#__codelineno-2-54) def __init__(self, vllm_config: "VllmConfig", device: torch.device, [](#__codelineno-2-55) is_pin_memory: bool) -> None: [](#__codelineno-2-56) raise NotImplementedError [](#__codelineno-2-57) [](#__codelineno-2-58) @abstractmethod [](#__codelineno-2-59) def apply(self, logits: torch.Tensor) -> torch.Tensor: [](#__codelineno-2-60) raise NotImplementedError [](#__codelineno-2-61) [](#__codelineno-2-62) @abstractmethod [](#__codelineno-2-63) def is_argmax_invariant(self) -> bool: [](#__codelineno-2-64) """True if logits processor has no impact on the [](#__codelineno-2-65) argmax computation in greedy sampling. [](#__codelineno-2-66) NOTE: may or may not have the same value for all [](#__codelineno-2-67) instances of a given LogitsProcessor subclass, [](#__codelineno-2-68) depending on subclass implementation. [](#__codelineno-2-69) """ [](#__codelineno-2-70) raise NotImplementedError [](#__codelineno-2-71) [](#__codelineno-2-72) @abstractmethod [](#__codelineno-2-73) def update_state( [](#__codelineno-2-74) self, [](#__codelineno-2-75) batch_update: "BatchUpdate" | None, [](#__codelineno-2-76) ) -> None: [](#__codelineno-2-77) """Called when there are new output tokens, prior [](#__codelineno-2-78) to each forward pass. [](#__codelineno-2-79) [](#__codelineno-2-80) Args: [](#__codelineno-2-81) batch_update is non-None iff there have been [](#__codelineno-2-82) changes to the batch makeup. [](#__codelineno-2-83) """ [](#__codelineno-2-84) raise NotImplementedError [](#__codelineno-2-85) [](#__codelineno-2-86) @classmethod [](#__codelineno-2-87) def validate_params(cls, sampling_params: SamplingParams): [](#__codelineno-2-88) """Validate sampling params for this logits processor. [](#__codelineno-2-89) [](#__codelineno-2-90) Raise ValueError for invalid ones. [](#__codelineno-2-91) """ [](#__codelineno-2-92) return None`` A vLLM logits processor must subclass [`LogitsProcessor`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/logits_processor/#vllm.model_executor.layers.logits_processor.LogitsProcessor " LogitsProcessor") and define (at minimum) the following methods: - `__init__(self, vllm_config: VllmConfig, device: torch.device, is_pin_memory: bool)` - `vllm_config`: engine configuration data structure - `device`: hardware accelerator device info - `is_pin_memory`: flag indicating whether pin memory is available to support logits processor implementation - `apply(self, logits: torch.Tensor) -> torch.Tensor`: - Consume a `(num_requests) x (vocab_size)` logits tensor (`logits`) - Apply logits processor transformation at batch granularity - Return a transformed `(num_requests) x (vocab_size)` logits tensor - You can modify the input logits processors in-place or out-of-place; in-place is more memory-efficient - `is_argmax_invariant(self) -> bool`: - Return `True` if the logits processor is argmax invariant (never changes what is the highest-logit-value token ID for a given request), `False` if the logits processor may modify argmax - `is_argmax_invariant()` is evaluated once at startup; if `True`, vLLM will skip applying this logits processor in a given step when all requests use greedy sampling - `update_state(self, batch_update: "BatchUpdate" | None) -> None`: - Consume a [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure representing persistent batch state changes at the beginning of the current engine step - Use the [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") members to update logits processor internal state - **Note:** batch update data structure may be `None`, signaling no change to the batch constituents. In this case, the LogitsProcessor might still want to update its state based on the updated `output_token_ids` lists that it could have retained when they were added. - `validate_params(cls, sampling_params: SamplingParams)`: - Raise `ValueError` if [`SamplingParams`](https://docs.vllm.ai/en/api/vllm/sampling_params/#vllm.sampling_params.SamplingParams " SamplingParams") has invalid arguments (especially custom arguments) used by logits processor. - When request is sent to entrypoint, `validate_params()` will validate [`SamplingParams`](https://docs.vllm.ai/en/api/vllm/sampling_params/#vllm.sampling_params.SamplingParams " SamplingParams") and refuse request with invalid arguments. ### [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure[¶](#batchupdate-data-structure "Permanent link") The [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") abstraction models the persistent batch as a list of requests, supporting the following operations to change batch state (note that the order in which the operations are mentioned below reflects the order in which they should be processed in `update_state()`): - **Remove:** remove (without replacement) request at index `i` - A Remove is represented in `Batchupdate.removed` by an `int` (representing `i`) - Effect of remove-at-index on batch: `[](#__codelineno-3-1)Batch: [A,B,C] [](#__codelineno-3-2)Remove @ i: 1 [](#__codelineno-3-3)[](#__codelineno-3-4)=> [](#__codelineno-3-5)[](#__codelineno-3-6)New Batch: [A,x,C] # Discard B and leave an empty slot` - **Add:** add (or replace existing request with) a new request at index `i`. If a request is replaced, its associated state should be discarded. - An Add is represented in `Batchupdate.added` as a tuple of `[](#__codelineno-4-1)(index, new request SamplingParams, prompt token ids, output token ids)` - `prompt token ids` and `output token ids` are references to the request's prompt token ids and output token ids lists, respectively. Note that the output token ids list grows with each engine step, and this growth is visible to the logits processor because output token ids are passed by reference. **This is important for LogitsProcessors that take into account the tokens generated so far**. - The implementation of the particular logits processor subclass determines whether or how the fields in the added request tuple are digested into an internal representation. For example, a logits processor that does not utilize prompt or output token ids may only need to utilize `index` and [`SamplingParams`](https://docs.vllm.ai/en/api/vllm/sampling_params/#vllm.sampling_params.SamplingParams " SamplingParams") and discard the other tuple fields - If index `i` currently holds a request, a replacement occurs: `[](#__codelineno-5-1)Batch: [A,B,C] [](#__codelineno-5-2)New request to be added @ i: D @ 1 [](#__codelineno-5-3)[](#__codelineno-5-4)=> [](#__codelineno-5-5)[](#__codelineno-5-6)New Batch: [A,D,C] # Add D, discard B` - If index `i` does not currently hold a request (because `i` is out of bounds of the current batch size): `[](#__codelineno-6-1)Batch: [A,B,C] [](#__codelineno-6-2)New request to be added @ i: D @ 3 [](#__codelineno-6-3)[](#__codelineno-6-4)=> [](#__codelineno-6-5)[](#__codelineno-6-6)New Batch: [A,B,C,D] # Add D, extending batch` - **Move:** move request at index `s` to index `d` OR swap requests at indices `s` and `d` - A Move is represented in `Batchupdate.moved` as a tuple of `[](#__codelineno-7-1)(s, d, UNIDIRECTIONAL or SWAP)` - If the Move specifies `UNIDIRECTIONAL`: - The request at index `s` is moved to index `d`; index `s` becomes an empty slot `[](#__codelineno-8-1)Batch: [A,x,C,D] [](#__codelineno-8-2)Unidirectionally Move s -> d: 3 -> 1 [](#__codelineno-8-3)[](#__codelineno-8-4)=> [](#__codelineno-8-5)[](#__codelineno-8-6)New Batch: [A,D,C,x] # Move D to 1, leaving empty slot at 3` - If another request already resided at index `d`, it is replaced and discarded `[](#__codelineno-9-1)Batch: [A,B,C,D] [](#__codelineno-9-2)Unidirectionally Move s -> d: 3 -> 1 [](#__codelineno-9-3)[](#__codelineno-9-4)=> [](#__codelineno-9-5)[](#__codelineno-9-6)New Batch: [A,D,C,x] # Move D to 1, discarding B and leaving empty slot at 3` - If the Move specifies `SWAP`, the requests at `s` and `d` exchange indices `[](#__codelineno-10-1)Batch: [A,B,C,D] [](#__codelineno-10-2)Swap Move s <-> d: 3 <-> 1 [](#__codelineno-10-3)[](#__codelineno-10-4)=> [](#__codelineno-10-5)[](#__codelineno-10-6)New Batch: [A,D,C,B] # Swap B and D` Additionally, the [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure includes a representation (`batch_size`) of the size of the persistent batch at the beginning of the engine step. ### How the vLLM engine builds the [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure[¶](#how-the-vllm-engine-builds-the-batchupdate-data-structure "Permanent link") Logits processor `update_state()` implementations should assume the following model for how the model runner updates persistent batch state (expressed here in terms of the [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") abstraction): 1. Identify indices of requests which finished in the current engine step 2. Identify new requests introduced in the current step 3. Use Add operations to replace as many finished requests with new requests, in order of increasing index of the replaced request starting with the lowest index 4. Based on the relative number of new and finished requests: 1. If the numbers of new and finished requests are the same, proceed to next step 2. _If there are more new requests than finished requests:_ apply Add operations to extend the batch with the remaining new requests which did not replace finished requests. Assign consecutive indices to these new requests, starting with `current_max_batch_index + 1` 3. _If there are fewer new requests than finished requests:_ - Apply Remove operations to finished requests which were not replaced with new requests. These removed request indices will necessarily be greater than the greatest index of the finished requests which were replaced in the previous step. The Removes may leave the batch in a non-contiguous state - **"Condense" the batch to be contiguous:** starting with the lowest-index empty slot (which was caused by a Remove), apply a Unidirectional Move from the current highest non-empty slot in the batch to fill the empty slot. Proceed with additional Unidirectional Move operations in order of increasing empty slot destination index and decreasing non-empty slot source index until the batch is contiguous - **Shrink the batch:** a side effect of condensing the batch is that empty slots resulting from Remove operations are grouped in a contiguous block at the end of the batch array. Thus, after condensing, update `BatchUpdate.batch_size` to reflect the number of non-empty slots 5. Reorder the batch for improved efficiency. Depending on the attention backend implementation and the current characteristics of the batch, zero or more Swap Move operations may be applied to reorder the batch Notes: - A logits processor `update_state()` method must process batch update operations in the following order: removes, adds, moves - The index argument for Add operations refers to the index _at the time the Add occurred_, i.e. before any Move operations - Example: if a request is Added at index 5 and then swapped with index 3, the Add operation in `BatchUpdate.added` will be associated with index 5 not 3 - In other words Move operations can be assumed to be applied after Adds and Removes - Move operations can be assumed to be applied in the order in which they appear in `BatchUpdate.moved` - If there are no new/finished requests and there is no batch reordering, then the batch update for the logits processors will be `None` #### Example: Batch Update with Fewer New Requests Than Finished Requests[¶](#example-batch-update-with-fewer-new-requests-than-finished-requests "Permanent link") The following example models an engine step where 1 new request is introduced and 2 finished requests are eliminated, additionally the attention backend performs a swap to optimize the batch ordering. `[](#__codelineno-11-1)Batch state (beginning of engine step): [A,B,C,D] [](#__codelineno-11-2)Batch size: 4 [](#__codelineno-11-3)[](#__codelineno-11-4)New requests: E [](#__codelineno-11-5)[](#__codelineno-11-6)Finished requests: A, C [](#__codelineno-11-7)[](#__codelineno-11-8)Processing steps (using BatchUpdate abstraction): [](#__codelineno-11-9)[](#__codelineno-11-10)1. Add E at index 0 [](#__codelineno-11-11)[](#__codelineno-11-12)[E,B,C,D] # Discard A [](#__codelineno-11-13)Batch size: 4 [](#__codelineno-11-14)[](#__codelineno-11-15)2. Remove at index 2 [](#__codelineno-11-16)[](#__codelineno-11-17)[E,B,x,D] # Discard C, empty slot at index 2 [](#__codelineno-11-18)Batch size: 4 [](#__codelineno-11-19)[](#__codelineno-11-20)3. Condense batch with a Unidirectional Move 3 -> 2 operation and shrink batch [](#__codelineno-11-21)[](#__codelineno-11-22)[E,B,D] x # Empty slot is now outside batch [](#__codelineno-11-23)Batch size: 3 [](#__codelineno-11-24)[](#__codelineno-11-25)4. Attention backend optimization: reorder batch with Swap 0 <-> 1 [](#__codelineno-11-26)[](#__codelineno-11-27)[B,E,D] [](#__codelineno-11-28)Batch size: 3` The resulting [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure will look like `[](#__codelineno-12-1)BatchUpdate instance [](#__codelineno-12-2)* added: [(0,E's SamplingParams,E's prompt tokens ref,E's output tokens ref)] [](#__codelineno-12-3)* removed: [2] # request C was removed without replacement [](#__codelineno-12-4)* moved: [(3,2,UNIDIRECTIONAL),(0,1,SWAP)]` #### Example: Batch Update with More New Requests Than Finished Requests[¶](#example-batch-update-with-more-new-requests-than-finished-requests "Permanent link") The following example models an engine step where 2 new requests are introduced and 1 finished request is eliminated, additionally the attention backend performs a swap to optimize the batch ordering. `[](#__codelineno-13-1)Batch state (beginning of engine step): [A,B,C,D] [](#__codelineno-13-2)Batch size: 4 [](#__codelineno-13-3)[](#__codelineno-13-4)New requests: E,F [](#__codelineno-13-5)[](#__codelineno-13-6)Finished requests: C [](#__codelineno-13-7)[](#__codelineno-13-8)Processing steps (using BatchUpdate abstraction): [](#__codelineno-13-9)[](#__codelineno-13-10)1. Add E at index 2 [](#__codelineno-13-11)[](#__codelineno-13-12)[A,B,E,D] # Discard C [](#__codelineno-13-13)Batch size: 4 [](#__codelineno-13-14)[](#__codelineno-13-15)2. Add F at index 4 (current max batch index + 1) [](#__codelineno-13-16)[](#__codelineno-13-17)[A,B,E,D,F] # Extend batch by 1 [](#__codelineno-13-18)Batch size: 5 [](#__codelineno-13-19)[](#__codelineno-13-20)4. Attention backend optimization: reorder batch with Swap 0 <-> 1 [](#__codelineno-13-21)[](#__codelineno-13-22)[B,A,E,D,F] [](#__codelineno-13-23)Batch size: 5` Note that batch condensation is skipped because there are no empty slots left behind by Remove operations. The resulting [`BatchUpdate`](https://docs.vllm.ai/en/api/vllm/v1/sample/logits_processor/interface/#vllm.v1.sample.logits_processor.interface.BatchUpdate " BatchUpdate dataclass ") data structure will look like `[](#__codelineno-14-1)BatchUpdate instance [](#__codelineno-14-2)* added: [(2,E's SamplingParams,E's prompt tokens ref,E's output tokens ref),(4,F's SamplingParams,F's prompt tokens ref,F's output tokens ref)] [](#__codelineno-14-3)* removed: [] # no requests were removed without replacement [](#__codelineno-14-4)* moved: [(0,1,SWAP)]` ## How to Introduce a New Logits Processor to vLLM[¶](#how-to-introduce-a-new-logits-processor-to-vllm "Permanent link") ### Best Practices for Writing Built-In Logits Processors[¶](#best-practices-for-writing-built-in-logits-processors "Permanent link") - Write efficient `apply()` and `update_state()` implementations in light of the fact that logits processors operate at batch granularity - For example, you may be able to use efficient vectorized operations to implement `apply()` or update internal state vectors in `update_state()` - However, if you think that a logits processor may be used infrequently, it may be appropriate to use a "sparse" representation of request state i.e. the class can represent request configuration using a dictionary which only stores metadata about requests that enable the logits processor - It is up to the logits processor author to determine: 1. **The per-request attributes which configure the logits processor's behavior against that request.** For example, if you are writing a new built-in logits processor for vLLM, you may or may not need to add additional fields to [`SamplingParams`](https://docs.vllm.ai/en/api/vllm/sampling_params/#vllm.sampling_params.SamplingParams " SamplingParams") and the vLLM REST API 2. **The conditions under which the logits processor is or is not enabled on a per-request basis.** Unless your intention is for the built-in logits processor to act on all requests all the time, you should write your logits processor in such a way that it is possible to disable the logits processor for a given request, i.e. by defaulting an argument to `None` or by passing in a specific do-nothing argument value i.e. `0.0`. Try to save compute and memory for requests which disable the logits processor 3. **The conditions under which the logits processor is short-circuited at the batch level.** Even if you have defined a way to disable the built-in logits processor at the request level, it may be difficult to translate this into compute savings i.e. if your `update_state()` and `apply()` implementations use efficient vectorized implementations that operate on the whole persistent batch in a single command. For example, you cannot skip an entire vectorized operation in `apply()` just because one request disabled the logits processor. To save compute in the edge-case where no running requests utilize the built-in logits processor, we recommend designing `apply()` to return the unmodified input tensor if all requests have the logits processor disabled. Similarly, consider whether steps can be skipped in `update_state()` if no requests enable the logits processor - Additionally, an easy way to save compute in `update_state()` is to exit early when the batch\_update is `None` - Ensure that the logits processor `update_state` method discards information about finished requests (i.e. requests which are replaced by an Add or which are subject to a Remove) - `is_argmax_invariant()` can be hard-coded to `True` or `False` if the logits processor has consistent behavior. However the argmax invariance may also be determined programmatically (i.e. if your logits processor is user-customizable in some way that impacts whether the logits processor is argmax invariant). For this reason, `is_argmax_invariant()` is not a class method ### Built-In Logits Processors[¶](#built-in-logits-processors "Permanent link") Built-in logits processors are always loaded when the vLLM engine starts. See the existing vLLM built-in logits processors in `vllm/v1/sample/logits_processor/builtin.py` for examples of how to write a new built-in vLLM logits processor. It makes sense to write a PR to introduce a new logits processor as a built-in if it is likely to be useful to a wide audience. vLLM currently employs the following built-in logits processors based on the programming model described above: - Min-P - Logit bias - Min-tokens Review these logits processor implementations for guidance on writing built-in logits processors. Additionally, the following logits-processor-like functionalities are hard-coded into the sampler and do not yet utilize the programming model described above. Most of them will be refactored to use the aforementioned logits processor programming model. - Allowed token IDs - Bad words - Repetition penalty - Frequency penalty - Presence penalty - Temperature - Top-K - Top-P ### Custom Logits Processors[¶](#custom-logits-processors "Permanent link") vLLM can be augmented with [user-provided custom logits processors](https://docs.vllm.ai/en/features/custom_logitsprocs/). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/lora_resolver_plugins.md "Edit this page") This directory contains vLLM's LoRA resolver plugins built on the [`LoRAResolver`](https://docs.vllm.ai/en/api/vllm/lora/resolver/#vllm.lora.resolver.LoRAResolver " LoRAResolver") framework. They automatically discover and load LoRA adapters from a specified local storage path, eliminating the need for manual configuration or server restarts. ## Overview[¶](#overview "Permanent link") LoRA Resolver Plugins provide a flexible way to dynamically load LoRA adapters at runtime. When vLLM receives a request for a LoRA adapter that hasn't been loaded yet, the resolver plugins will attempt to locate and load the adapter from their configured storage locations. This enables: - **Dynamic LoRA Loading**: Load adapters on-demand without server restarts - **Multiple Storage Backends**: Support for filesystem, S3, and custom backends. The built-in `lora_filesystem_resolver` requires a local storage path, while the built-in `hf_hub_resolver` will pull LoRA adapters from Huggingface Hub and proceed in an identical manner. In general, custom resolvers can be implemented to fetch from any source. - **Automatic Discovery**: Seamless integration with existing LoRA workflows - **Scalable Deployment**: Centralized adapter management across multiple vLLM instances ## Prerequisites[¶](#prerequisites "Permanent link") Before using LoRA Resolver Plugins, ensure the following environment variables are configured: ### Required Environment Variables[¶](#required-environment-variables "Permanent link") 1. **`VLLM_ALLOW_RUNTIME_LORA_UPDATING`**: Must be set to `true` or `1` to enable dynamic LoRA loading `[](#__codelineno-0-1)export VLLM_ALLOW_RUNTIME_LORA_UPDATING=true` 2. **`VLLM_PLUGINS`**: Must include the desired resolver plugins (comma-separated list) `[](#__codelineno-1-1)export VLLM_PLUGINS=lora_filesystem_resolver` 3. **`VLLM_LORA_RESOLVER_CACHE_DIR`**: Must be set to a valid directory path for filesystem resolver `[](#__codelineno-2-1)export VLLM_LORA_RESOLVER_CACHE_DIR=/path/to/lora/adapters` ### Optional Environment Variables[¶](#optional-environment-variables "Permanent link") - **`VLLM_PLUGINS`**: If not set, all available plugins will be loaded. If set to empty string, no plugins will be loaded. ## Available Resolvers[¶](#available-resolvers "Permanent link") ### lora\_filesystem\_resolver[¶](#lora_filesystem_resolver "Permanent link") The filesystem resolver is installed with vLLM by default and enables loading LoRA adapters from a local directory structure. #### Setup Steps[¶](#setup-steps "Permanent link") 1. **Create the LoRA adapter storage directory**: `[](#__codelineno-3-1)mkdir -p /path/to/lora/adapters` 2. **Set environment variables**: `[](#__codelineno-4-1)export VLLM_ALLOW_RUNTIME_LORA_UPDATING=true [](#__codelineno-4-2)export VLLM_PLUGINS=lora_filesystem_resolver [](#__codelineno-4-3)export VLLM_LORA_RESOLVER_CACHE_DIR=/path/to/lora/adapters` 3. **Start vLLM server**: Your base model can be `meta-llama/Llama-2-7b-hf`. Please make sure you set up the Hugging Face token in your env var `export HF_TOKEN=xxx235`. `[](#__codelineno-5-1)vllm serve your-base-model \ [](#__codelineno-5-2) --enable-lora` #### Directory Structure Requirements[¶](#directory-structure-requirements "Permanent link") The filesystem resolver expects LoRA adapters to be organized in the following structure: `[](#__codelineno-6-1)/path/to/lora/adapters/ [](#__codelineno-6-2)├── adapter1/ [](#__codelineno-6-3)│ ├── adapter_config.json [](#__codelineno-6-4)│ ├── adapter_model.bin [](#__codelineno-6-5)│ └── tokenizer files (if applicable) [](#__codelineno-6-6)├── adapter2/ [](#__codelineno-6-7)│ ├── adapter_config.json [](#__codelineno-6-8)│ ├── adapter_model.bin [](#__codelineno-6-9)│ └── tokenizer files (if applicable) [](#__codelineno-6-10)└── ...` Each adapter directory must contain: - **`adapter_config.json`**: Required configuration file with the following structure: `[](#__codelineno-7-1){ [](#__codelineno-7-2) "peft_type": "LORA", [](#__codelineno-7-3) "base_model_name_or_path": "your-base-model-name", [](#__codelineno-7-4) "r": 16, [](#__codelineno-7-5) "lora_alpha": 32, [](#__codelineno-7-6) "target_modules": ["q_proj", "v_proj"], [](#__codelineno-7-7) "bias": "none", [](#__codelineno-7-8) "modules_to_save": null, [](#__codelineno-7-9) "use_rslora": false, [](#__codelineno-7-10) "use_dora": false [](#__codelineno-7-11)}` - **`adapter_model.bin`**: The LoRA adapter weights file #### Usage Example[¶](#usage-example "Permanent link") 1. **Prepare your LoRA adapter**: `[](#__codelineno-8-1)# Assuming you have a LoRA adapter in /tmp/my_lora_adapter [](#__codelineno-8-2)cp -r /tmp/my_lora_adapter /path/to/lora/adapters/my_sql_adapter` 2. **Verify the directory structure**: `[](#__codelineno-9-1)ls -la /path/to/lora/adapters/my_sql_adapter/ [](#__codelineno-9-2)# Should show: adapter_config.json, adapter_model.bin, etc.` 3. **Make a request using the adapter**: `[](#__codelineno-10-1)curl http://localhost:8000/v1/completions \ [](#__codelineno-10-2) -H "Content-Type: application/json" \ [](#__codelineno-10-3) -d '{ [](#__codelineno-10-4) "model": "my_sql_adapter", [](#__codelineno-10-5) "prompt": "Generate a SQL query for:", [](#__codelineno-10-6) "max_tokens": 50, [](#__codelineno-10-7) "temperature": 0.1 [](#__codelineno-10-8) }'` #### How It Works[¶](#how-it-works "Permanent link") 1. When vLLM receives a request for a LoRA adapter named `my_sql_adapter` 2. The filesystem resolver checks if `/path/to/lora/adapters/my_sql_adapter/` exists 3. If found, it validates the `adapter_config.json` file 4. If the configuration matches the base model and is valid, the adapter is loaded 5. The request is processed normally with the newly loaded adapter 6. The adapter remains available for future requests ## Advanced Configuration[¶](#advanced-configuration "Permanent link") ### Multiple Resolvers[¶](#multiple-resolvers "Permanent link") You can configure multiple resolver plugins to load adapters from different sources: 'lora\_s3\_resolver' is an example of a custom resolver you would need to implement `[](#__codelineno-11-1)export VLLM_PLUGINS=lora_filesystem_resolver,lora_s3_resolver` All listed resolvers are enabled; at request time, vLLM tries them in order until one succeeds. ### Custom Resolver Implementation[¶](#custom-resolver-implementation "Permanent link") To implement your own resolver plugin: 1. **Create a new resolver class**: `[](#__codelineno-12-1)from vllm.lora.resolver import LoRAResolver, LoRAResolverRegistry [](#__codelineno-12-2)from vllm.lora.request import LoRARequest [](#__codelineno-12-3)[](#__codelineno-12-4)class CustomResolver(LoRAResolver): [](#__codelineno-12-5) async def resolve_lora(self, base_model_name: str, lora_name: str) -> Optional[LoRARequest]: [](#__codelineno-12-6) # Your custom resolution logic here [](#__codelineno-12-7) pass` 2. **Register the resolver**: `[](#__codelineno-13-1)def register_custom_resolver(): [](#__codelineno-13-2) resolver = CustomResolver() [](#__codelineno-13-3) LoRAResolverRegistry.register_resolver("Custom Resolver", resolver)` ## Troubleshooting[¶](#troubleshooting "Permanent link") ### Common Issues[¶](#common-issues "Permanent link") 1. **"VLLM\_LORA\_RESOLVER\_CACHE\_DIR must be set to a valid directory"** 2. Ensure the directory exists and is accessible 3. Check file permissions on the directory 4. **"LoRA adapter not found"** 5. Verify the adapter directory name matches the requested model name 6. Check that `adapter_config.json` exists and is valid JSON 7. Ensure `adapter_model.bin` exists in the directory 8. **"Invalid adapter configuration"** 9. Verify `peft_type` is set to "LORA" 10. Check that `base_model_name_or_path` matches your base model 11. Ensure `target_modules` is properly configured 12. **"LoRA rank exceeds maximum"** 13. Check that `r` value in `adapter_config.json` doesn't exceed `max_lora_rank` setting ### Debugging Tips[¶](#debugging-tips "Permanent link") 1. **Enable debug logging**: `[](#__codelineno-14-1)export VLLM_LOGGING_LEVEL=DEBUG` 2. **Verify environment variables**: `[](#__codelineno-15-1)echo $VLLM_ALLOW_RUNTIME_LORA_UPDATING [](#__codelineno-15-2)echo $VLLM_PLUGINS [](#__codelineno-15-3)echo $VLLM_LORA_RESOLVER_CACHE_DIR` 3. **Test adapter configuration**: `[](#__codelineno-16-1)python -c " [](#__codelineno-16-2)import json [](#__codelineno-16-3)with open('/path/to/lora/adapters/my_adapter/adapter_config.json') as f: [](#__codelineno-16-4) config = json.load(f) [](#__codelineno-16-5)print('Config valid:', config) [](#__codelineno-16-6)"` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/metrics.md "Edit this page") vLLM exposes a rich set of metrics to support observability and capacity planning for the V1 engine. ## Objectives[¶](#objectives "Permanent link") - Provide comprehensive coverage of engine and request level metrics to aid production monitoring. - Prioritize Prometheus integrations, as this is what we expect to be used in production environments. - Offer logging support (i.e. printing metrics to the info log) for ad-hoc testing, debugging, development, and exploratory use cases. ## Background[¶](#background "Permanent link") Metrics in vLLM can be categorized as follows: 1. Server-level metrics: Global metrics that track the state and performance of the LLM engine. These are typically exposed as Gauges or Counters in Prometheus. 2. Request-level metrics: Metrics that track the characteristics (e.g. size and timing) of individual requests. These are typically exposed as Histograms in Prometheus and are often the SLOs that an SRE monitoring vLLM will be tracking. The mental model is that server-level metrics help explain the values of request-level metrics. ### Metrics Overview[¶](#metrics-overview "Permanent link") ### v1 Metrics[¶](#v1-metrics "Permanent link") In v1, an extensive set of metrics are exposed via a Prometheus-compatible `/metrics` endpoint using the `vllm:` prefix, for example: - `vllm:num_requests_running` (Gauge) - Number of requests currently running. - `vllm:kv_cache_usage_perc` (Gauge) - Fraction of used KV cache blocks (0–1). - `vllm:prefix_cache_queries` (Counter) - Number of prefix cache queries. - `vllm:prefix_cache_hits` (Counter) - Number of prefix cache hits. - `vllm:prompt_tokens_total` (Counter) - Total number of prompt tokens processed. - `vllm:generation_tokens_total` (Counter) - Total number of generated tokens. - `vllm:request_success_total` (Counter) - Number of finished requests (by finish reason). - `vllm:request_prompt_tokens` (Histogram) - Histogram of input prompt token counts. - `vllm:request_generation_tokens` (Histogram) - Histogram of generation token counts. - `vllm:time_to_first_token_seconds` (Histogram) - Time to first token (TTFT). - `vllm:inter_token_latency_seconds` (Histogram) - Inter-token latency. - `vllm:e2e_request_latency_seconds` (Histogram) - End-to-end request latency. - `vllm:request_prefill_time_seconds` (Histogram) - Request prefill time. - `vllm:request_decode_time_seconds` (Histogram) - Request decode time. These are documented under [Inferencing and Serving -> Production Metrics](https://docs.vllm.ai/en/usage/metrics/). ### Grafana Dashboard[¶](#grafana-dashboard "Permanent link") vLLM also provides [a reference example](https://github.com/vllm-project/vllm/blob/main/examples/observability/prometheus_grafana/README.md) for how to collect and store these metrics using Prometheus and visualize them using a Grafana dashboard. The subset of metrics exposed in the Grafana dashboard gives us an indication of which metrics are especially important: - `vllm:e2e_request_latency_seconds_bucket` - End to end request latency measured in seconds. - `vllm:prompt_tokens` - Prompt tokens. - `vllm:generation_tokens` - Generation tokens. - `vllm:inter_token_latency_seconds` - Inter-token latency (Time Per Output Token, TPOT) in seconds. - `vllm:time_to_first_token_seconds` - Time to First Token (TTFT) latency in seconds. - `vllm:num_requests_running` (also, `_swapped` and `_waiting`) - Number of requests in the RUNNING, WAITING, and SWAPPED states. - `vllm:kv_cache_usage_perc` - Percentage of used cache blocks by vLLM. - `vllm:request_prompt_tokens` - Request prompt length. - `vllm:request_generation_tokens` - Request generation length. - `vllm:request_success` - Number of finished requests by their finish reason: either an EOS token was generated or the max sequence length was reached. - `vllm:request_queue_time_seconds` - Queue time. - `vllm:request_prefill_time_seconds` - Requests prefill time. - `vllm:request_decode_time_seconds` - Requests decode time. - `vllm:request_max_num_generation_tokens` - Max generation tokens in a sequence group. See [the PR which added this Dashboard](https://github.com/vllm-project/vllm/pull/2316) for interesting and useful background on the choices made here. ### Prometheus Client Library[¶](#prometheus-client-library "Permanent link") Prometheus support was initially added [using the aioprometheus library](https://github.com/vllm-project/vllm/pull/1890), but a switch was made quickly to [prometheus\_client](https://github.com/vllm-project/vllm/pull/2730). The rationale is discussed in both linked PRs. During those migrations we briefly lost a `MetricsMiddleware` to track HTTP metrics, but this was reinstated [using prometheus\_fastapi\_instrumentator](https://github.com/vllm-project/vllm/pull/15657): `[](#__codelineno-0-1)$ curl http://0.0.0.0:8000/metrics 2>/dev/null | grep -P '^http_(?!.*(_bucket|_created|_sum)).*' [](#__codelineno-0-2)http_requests_total{handler="/v1/completions",method="POST",status="2xx"} 201.0 [](#__codelineno-0-3)http_request_size_bytes_count{handler="/v1/completions"} 201.0 [](#__codelineno-0-4)http_response_size_bytes_count{handler="/v1/completions"} 201.0 [](#__codelineno-0-5)http_request_duration_highr_seconds_count 201.0 [](#__codelineno-0-6)http_request_duration_seconds_count{handler="/v1/completions",method="POST"} 201.0` ### Multi-process Mode[¶](#multi-process-mode "Permanent link") Historically, metrics were collected in the engine core process and multiprocess mode was used to make them available in the API server process. See [Pull Request #7279](https://github.com/vllm-project/vllm/pull/7279). More recently, metrics are collected in the API server process and multiprocess mode is only used when `--api-server-count > 1`. See [Pull Request #17546](https://github.com/vllm-project/vllm/pull/17546) and details on [API server scale-out](https://docs.vllm.ai/en/serving/data_parallel_deployment/#internal-load-balancing). ### Built in Python/Process Metrics[¶](#built-in-pythonprocess-metrics "Permanent link") The following metrics are supported by default by `prometheus_client`, but they are not exposed when multiprocess mode is used: - `python_gc_objects_collected_total` - `python_gc_objects_uncollectable_total` - `python_gc_collections_total` - `python_info` - `process_virtual_memory_bytes` - `process_resident_memory_bytes` - `process_start_time_seconds` - `process_cpu_seconds_total` - `process_open_fds` - `process_max_fds` Therefore, these metrics are unavailable when `--api-server-count > 1`. It's questionable how relevant these are since they do not aggregate these stats for all processes that make up a vLLM instance. ## Metrics Design[¶](#metrics-design "Permanent link") The ["Even Better Observability"](https://github.com/vllm-project/vllm/issues/3616) feature where was where much of the metrics design was planned. For example, see where [a detailed roadmap was laid out](https://github.com/vllm-project/vllm/issues/3616#issuecomment-2030858781). ### Legacy PRs[¶](#legacy-prs "Permanent link") To help understand the background to the metrics design, here are some of the relevant PRs which added the original, now legacy, metrics: - [Pull Request #1890](https://github.com/vllm-project/vllm/pull/1890) - [Pull Request #2316](https://github.com/vllm-project/vllm/pull/2316) - [Pull Request #2730](https://github.com/vllm-project/vllm/pull/2730) - [Pull Request #4464](https://github.com/vllm-project/vllm/pull/4464) - [Pull Request #7279](https://github.com/vllm-project/vllm/pull/7279) ### Metrics Implementation PRs[¶](#metrics-implementation-prs "Permanent link") For background, here are the relevant PRs relating to the metrics implementation [Issue #10582](https://github.com/vllm-project/vllm/issues/10582): - [Pull Request #11962](https://github.com/vllm-project/vllm/pull/11962) - [Pull Request #11973](https://github.com/vllm-project/vllm/pull/11973) - [Pull Request #10907](https://github.com/vllm-project/vllm/pull/10907) - [Pull Request #12416](https://github.com/vllm-project/vllm/pull/12416) - [Pull Request #12478](https://github.com/vllm-project/vllm/pull/12478) - [Pull Request #12516](https://github.com/vllm-project/vllm/pull/12516) - [Pull Request #12530](https://github.com/vllm-project/vllm/pull/12530) - [Pull Request #12561](https://github.com/vllm-project/vllm/pull/12561) - [Pull Request #12579](https://github.com/vllm-project/vllm/pull/12579) - [Pull Request #12592](https://github.com/vllm-project/vllm/pull/12592) - [Pull Request #12644](https://github.com/vllm-project/vllm/pull/12644) ### Metrics Collection[¶](#metrics-collection "Permanent link") In v1, we wish to move computation and overhead out of the engine core process to minimize the time between each forward pass. The overall idea of V1 EngineCore design is: - EngineCore is the inner loop. Performance is most critical here - AsyncLLM is the outer loop. This is overlapped with GPU execution (ideally), so this is where any "overheads" should be if possible. So AsyncLLM.output\_handler\_loop is the ideal place for the metrics bookkeeping if possible. We will achieve this by collecting metrics in the frontend API server, and base these metrics on information we can glean from the `EngineCoreOutputs` returned by the engine core process to the frontend. ### Interval Calculations[¶](#interval-calculations "Permanent link") Many of our metrics are the time interval between various events in the processing of a request. It is best practice to use timestamps based on "monotonic time" (`time.monotonic()`) rather than "wall-clock time" (`time.time()`) to calculate intervals as the former is unaffected by system clock changes (e.g. from NTP). It's also important to note that monotonic clocks differ between processes - each process has its own reference point. So it is meaningless to compare monotonic timestamps from different processes. Therefore, in order to calculate an interval, we must compare two monotonic timestamps from the same process. ### Scheduler Stats[¶](#scheduler-stats "Permanent link") The engine core process will collect some key statistics from the scheduler - e.g. the number of requests that were scheduled or waiting after the last scheduler pass - and include those statistics in `EngineCoreOutputs`. ### Engine Core Events[¶](#engine-core-events "Permanent link") The engine core will also record the timestamp of certain per-request events so that the frontend can calculate the interval between these events. The events are: - `QUEUED` - when the request was received by the engine core and added to the scheduler queue. - `SCHEDULED` - when the request was first scheduled for execution. - `PREEMPTED` - the request has been put back in the waiting queue in order to make room for other requests to complete. It will be re-scheduled in future and re-start its prefill phase. - `NEW_TOKENS` - when the output included in `EngineCoreOutput` was generated. Since this is common to all requests in a given iteration, we use a single timestamp on `EngineCoreOutputs` to record this event. And the calculated intervals are: - Queue interval - between `QUEUED` and most recent `SCHEDULED`. - Prefill interval - between most recent `SCHEDULED` and the subsequent first `NEW_TOKENS`. - Decode interval - between first (after the most recent `SCHEDULED`) and last `NEW_TOKENS`. - Inference interval - between most recent `SCHEDULED` and last `NEW_TOKENS`. - Inter-token interval - between successive `NEW_TOKENS`. Put another way: [![Interval calculations - common case](https://docs.vllm.ai/en/assets/design/metrics/intervals-1.png)](https://docs.vllm.ai/en/assets/design/metrics/intervals-1.png) We explored the possibility of having the frontend calculate these intervals using the timing of events visible by the frontend. However, the frontend does not have visibility into the timing of the `QUEUED` and `SCHEDULED` events and, since we need to calculate intervals based on monotonic timestamps from the same process ... we need the engine core to record timestamps for all of these events. #### Interval Calculations vs Preemptions[¶](#interval-calculations-vs-preemptions "Permanent link") When a preemption occurs during decode, since any already generated tokens are reused, we consider the preemption as affecting the inter-token, decode, and inference intervals. [![Interval calculations - preempted decode](https://docs.vllm.ai/en/assets/design/metrics/intervals-2.png)](https://docs.vllm.ai/en/assets/design/metrics/intervals-2.png) When a preemption occurs during prefill (assuming such an event is possible), we consider the preemption as affecting the time-to-first-token and prefill intervals. [![Interval calculations - preempted prefill](https://docs.vllm.ai/en/assets/design/metrics/intervals-3.png)](https://docs.vllm.ai/en/assets/design/metrics/intervals-3.png) ### Frontend Stats Collection[¶](#frontend-stats-collection "Permanent link") As the frontend processes a single `EngineCoreOutputs` - i.e. the output from a single engine core iteration - it collects various statistics relating to that iteration: - The total number of new tokens generated in this iteration. - The total number of prompt tokens processed by the prefills that completed in this iteration. - The queue intervals for any requests that were scheduled in this iteration. - The prefill intervals for any requests that completed prefill in this iteration. - The inter-token intervals (Time Per Output Token, TPOT), for all requests included in this iteration. - The Time-To-First-Token (TTFT) for any requests that completed prefill in this iteration. However, we calculate this interval relative to when the request was first received by the frontend (`arrival_time`) in order to account for input processing time. Currently `arrival_time` starts when tokenization begins. For any requests that were completed in a given iteration, we also record: - The inference and decode intervals - relative to the scheduled and first token events, as described above. - End-to-end latency - the interval between frontend `arrival_time` and the frontend receiving the final token. ### KV Cache Residency Metrics[¶](#kv-cache-residency-metrics "Permanent link") We also emit a set of histograms that describe how long sampled KV cache blocks stay resident and how often they are reused. Sampling (`--kv-cache-metrics-sample`) keeps the overhead tiny; when a block is chosen we record: - `lifetime` – allocation ⟶ eviction - `idle before eviction` – last touch ⟶ eviction - `reuse gaps` – the pauses between touches when the block gets reused Those map directly to the Prometheus metrics: - `vllm:kv_block_lifetime_seconds` – how long each sampled block exists. - `vllm:kv_block_idle_before_evict_seconds` – idle tail after the final access. - `vllm:kv_block_reuse_gap_seconds` – time between consecutive touches. The engine core only ships raw eviction events via [`SchedulerStats`](https://docs.vllm.ai/en/api/vllm/v1/metrics/stats/#vllm.v1.metrics.stats.SchedulerStats " SchedulerStats dataclass "); the frontend drains them, turns them into Prometheus observations, and also exposes the same data through `LLM.get_metrics()` when logging is on. Looking at lifetime and idle time on one chart makes it easy to spot stranded cache or workloads that pin prompts for a long decode. ### Metrics Publishing - Logging[¶](#metrics-publishing-logging "Permanent link") The `LoggingStatLogger` metrics publisher outputs a log `INFO` message every 5 seconds with some key metrics: - The current number of running/waiting requests - The current GPU cache usage - The number of prompt tokens processed per second over the past 5 seconds - The number of new tokens generated per second over the past 5 seconds - The prefix cache hit rate over the most recent 1k kv-cache block queries ### Metrics Publishing - Prometheus[¶](#metrics-publishing-prometheus "Permanent link") The `PrometheusStatLogger` metrics publisher makes the metrics available via a `/metrics` HTTP endpoint in a Prometheus-compatible format. A Prometheus instance can then be configured to poll this endpoint (e.g. every second) and record the values in its time-series database. Prometheus is often used via Grafana, allowing these metrics to be graphed over time. Prometheus supports the following metric types: - Counter: a value that will increase over time, never reducing, and generally reset to zero when the vLLM instance restarts. For example, the number of tokens generated over the lifetime of the instance. - Gauge: a value that goes up and down, for example the number of requests currently scheduled for execution. - Histogram: a count of metric samples, recorded in buckets. For example, the number of requests whose TTFT was <1ms, <5ms, <10ms, <20ms, and so on. Prometheus metrics can also be labelled, allowing metrics to be combined according to matching labels. In vLLM, we add a `model_name` label to every metric which includes the name of the model served by that instance. Example output: `[](#__codelineno-1-1)$ curl http://0.0.0.0:8000/metrics [](#__codelineno-1-2)# HELP vllm:num_requests_running Number of requests in model execution batches. [](#__codelineno-1-3)# TYPE vllm:num_requests_running gauge [](#__codelineno-1-4)vllm:num_requests_running{model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.0 [](#__codelineno-1-5)... [](#__codelineno-1-6)# HELP vllm:generation_tokens_total Number of generation tokens processed. [](#__codelineno-1-7)# TYPE vllm:generation_tokens_total counter [](#__codelineno-1-8)vllm:generation_tokens_total{model_name="meta-llama/Llama-3.1-8B-Instruct"} 27453.0 [](#__codelineno-1-9)... [](#__codelineno-1-10)# HELP vllm:request_success_total Count of successfully processed requests. [](#__codelineno-1-11)# TYPE vllm:request_success_total counter [](#__codelineno-1-12)vllm:request_success_total{finished_reason="stop",model_name="meta-llama/Llama-3.1-8B-Instruct"} 1.0 [](#__codelineno-1-13)vllm:request_success_total{finished_reason="length",model_name="meta-llama/Llama-3.1-8B-Instruct"} 131.0 [](#__codelineno-1-14)vllm:request_success_total{finished_reason="abort",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 [](#__codelineno-1-15)... [](#__codelineno-1-16)# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds. [](#__codelineno-1-17)# TYPE vllm:time_to_first_token_seconds histogram [](#__codelineno-1-18)vllm:time_to_first_token_seconds_bucket{le="0.001",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 [](#__codelineno-1-19)vllm:time_to_first_token_seconds_bucket{le="0.005",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 [](#__codelineno-1-20)vllm:time_to_first_token_seconds_bucket{le="0.01",model_name="meta-llama/Llama-3.1-8B-Instruct"} 0.0 [](#__codelineno-1-21)vllm:time_to_first_token_seconds_bucket{le="0.02",model_name="meta-llama/Llama-3.1-8B-Instruct"} 13.0 [](#__codelineno-1-22)vllm:time_to_first_token_seconds_bucket{le="0.04",model_name="meta-llama/Llama-3.1-8B-Instruct"} 97.0 [](#__codelineno-1-23)vllm:time_to_first_token_seconds_bucket{le="0.06",model_name="meta-llama/Llama-3.1-8B-Instruct"} 123.0 [](#__codelineno-1-24)vllm:time_to_first_token_seconds_bucket{le="0.08",model_name="meta-llama/Llama-3.1-8B-Instruct"} 138.0 [](#__codelineno-1-25)vllm:time_to_first_token_seconds_bucket{le="0.1",model_name="meta-llama/Llama-3.1-8B-Instruct"} 140.0 [](#__codelineno-1-26)vllm:time_to_first_token_seconds_count{model_name="meta-llama/Llama-3.1-8B-Instruct"} 140.0` Note The choice of histogram buckets to be most useful to users across a broad set of use cases is not straightforward and will require refinement over time. ### Cache Config Info[¶](#cache-config-info "Permanent link") `prometheus_client` has support for [Info metrics](https://prometheus.github.io/client_python/instrumenting/info/) which are equivalent to a [`Gauge`](https://docs.vllm.ai/en/api/vllm/v1/metrics/reader/#vllm.v1.metrics.reader.Gauge " Gauge dataclass ") whose value is permanently set to 1, but exposes interesting key/value pair information via labels. This is used for information about an instance that does not change - so it only needs to be observed at startup - and allows comparing across instances in Prometheus. We use this concept for the `vllm:cache_config_info` metric: `[](#__codelineno-2-1)# HELP vllm:cache_config_info Information of the LLMEngine CacheConfig [](#__codelineno-2-2)# TYPE vllm:cache_config_info gauge [](#__codelineno-2-3)vllm:cache_config_info{block_size="16",cache_dtype="auto",calculate_kv_scales="False",cpu_offload_gb="0",enable_prefix_caching="False",gpu_memory_utilization="0.9",...} 1.0` However, `prometheus_client` has [never supported Info metrics in multiprocessing mode](https://github.com/prometheus/client_python/pull/300) - for [unclear reasons](gh-pr:7279#discussion_r1710417152). We simply use a [`Gauge`](https://docs.vllm.ai/en/api/vllm/v1/metrics/reader/#vllm.v1.metrics.reader.Gauge " Gauge dataclass ") metric set to 1 and `multiprocess_mode="mostrecent"` instead. ### LoRA Metrics[¶](#lora-metrics "Permanent link") The `vllm:lora_requests_info` [`Gauge`](https://docs.vllm.ai/en/api/vllm/v1/metrics/reader/#vllm.v1.metrics.reader.Gauge " Gauge dataclass ") is somewhat similar, except the value is the current wall-clock time, and is updated every iteration. The label names used are: - `running_lora_adapters`: a per-adapter count of the number requests running using that adapter, formatted as a comma-separated string. - `waiting_lora_adapters`: similar, except counting requests that are waiting to be scheduled. - `max_lora` - the static "max number of LoRAs in a single batch." configuration. Encoding a running/waiting counts for multiple adapters in a comma-separated string seems quite misguided - we could use labels to distinguish between per-adapter counts. This should be revisited. Note that `multiprocess_mode="livemostrecent"` is used - the most recent metric is used, but only from currently running processes. This was added in [Pull Request #9477](https://github.com/vllm-project/vllm/pull/9477) and there is [at least one known user](https://github.com/kubernetes-sigs/gateway-api-inference-extension/pull/54). If we revisit this design and deprecate the old metric, we should coordinate with downstream users so they can migrate before the removal. ### Prefix Cache metrics[¶](#prefix-cache-metrics "Permanent link") The discussion in [Issue #10582](https://github.com/vllm-project/vllm/issues/10582) about adding prefix cache metrics yielded some interesting points which may be relevant to how we approach future metrics. Every time the prefix cache is queried, we record the number of tokens queried and the number of queried tokens present in the cache (i.e. hits). However, the metric of interest is the hit rate - i.e. the number of hits per query. In the case of logging, we expect the user is best served by calculating the hit rate over a fixed number of the most recent queries (the interval is fixed to 1k most recent queries for now). In the case of Prometheus though, we should take advantage of the time-series nature of Prometheus and allow the user to calculate the hit rate over an interval of their choosing. For example, a PromQL query to calculate the hit interval of the past 5 minutes: `[](#__codelineno-3-1)rate(cache_query_hit[5m]) / rate(cache_query_total[5m])` To achieve this, we should record the queries and hits as counters in Prometheus, rather than recording the hit rate as a gauge. ## Deprecated Metrics[¶](#deprecated-metrics "Permanent link") ### How To Deprecate[¶](#how-to-deprecate "Permanent link") Deprecating metrics shouldn't be taken lightly. Users may not notice a metric has been deprecated, and may be quite inconvenienced when it is suddenly (from their perspective) when it is removed, even if there is an equivalent metric for them to use. As an example, see how `vllm:avg_prompt_throughput_toks_per_s` was [deprecated](https://github.com/vllm-project/vllm/pull/2764) (with a comment in the code), [removed](https://github.com/vllm-project/vllm/pull/12383), and then [noticed by a user](https://github.com/vllm-project/vllm/issues/13218). In general: 1. We should be cautious about deprecating metrics, especially since it can be hard to predict the user impact. 2. We should include a prominent deprecation notice in the help string that is included in the \`/metrics' output. 3. We should list deprecated metrics in user-facing documentation and release notes. 4. We should consider hiding deprecated metrics behind a CLI argument in order to give administrators [an escape hatch](https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/#show-hidden-metrics) for some time before deleting them. See the [deprecation policy](https://docs.vllm.ai/en/contributing/deprecation_policy/) for the project-wide deprecation policy. ### Unimplemented - `vllm:tokens_total`[¶](#unimplemented-vllmtokens_total "Permanent link") Added by [Pull Request #4464](https://github.com/vllm-project/vllm/pull/4464), but apparently never implemented. This can just be removed. ### Duplicated - Queue Time[¶](#duplicated-queue-time "Permanent link") The `vllm:time_in_queue_requests` Histogram metric was added by [Pull Request #9659](https://github.com/vllm-project/vllm/pull/9659) and its calculation is: `[](#__codelineno-4-1) self.metrics.first_scheduled_time = now [](#__codelineno-4-2) self.metrics.time_in_queue = now - self.metrics.arrival_time` Two weeks later, [Pull Request #4464](https://github.com/vllm-project/vllm/pull/4464) added `vllm:request_queue_time_seconds` leaving us with: `[](#__codelineno-5-1)if seq_group.is_finished(): [](#__codelineno-5-2) if (seq_group.metrics.first_scheduled_time is not None and [](#__codelineno-5-3) seq_group.metrics.first_token_time is not None): [](#__codelineno-5-4) time_queue_requests.append( [](#__codelineno-5-5) seq_group.metrics.first_scheduled_time - [](#__codelineno-5-6) seq_group.metrics.arrival_time) [](#__codelineno-5-7) ... [](#__codelineno-5-8) if seq_group.metrics.time_in_queue is not None: [](#__codelineno-5-9) time_in_queue_requests.append( [](#__codelineno-5-10) seq_group.metrics.time_in_queue)` This seems duplicative, and one of them should be removed. The latter is used by the Grafana dashboard, so we should deprecate or remove the former. ### Prefix Cache Hit Rate[¶](#prefix-cache-hit-rate "Permanent link") See above - we now expose 'queries' and 'hits' counters rather than a 'hit rate' gauge. ### KV Cache Offloading[¶](#kv-cache-offloading "Permanent link") Two legacy metrics relate to a "swapped" preemption mode that is no longer relevant in v1: - `vllm:num_requests_swapped` - `vllm:cpu_cache_usage_perc` In this mode, when a request was preempted (e.g. to make room in KV cache to complete other requests), kv cache blocks were swapped out to CPU memory. The `--swap-space` flag has been removed as this feature is no longer used in V1. Historically, [vLLM has long supported beam search](https://github.com/vllm-project/vllm/issues/6226). The SequenceGroup encapsulated the idea of N Sequences which all shared the same prompt kv blocks. This enabled KV cache block sharing between requests, and copy-on-write to do branching. CPU swapping was intended for these beam search like cases. Later, the concept of prefix caching was introduced, which allowed KV cache blocks to be shared implicitly. This proved to be a better option than CPU swapping since blocks can be evicted slowly on demand and the part of the prompt that was evicted can be recomputed. SequenceGroup was removed in V1, although a replacement will be required for "parallel sampling" (`n>1`). [Beam search was moved out of the core](https://github.com/vllm-project/vllm/issues/8306). There was a lot of complex code for a very uncommon feature. In V1, with prefix caching being better (zero over head) and therefore on by default, the preemption and recompute strategy should work better. ## Future Work[¶](#future-work "Permanent link") ### Parallel Sampling[¶](#parallel-sampling "Permanent link") Some legacy metrics are only relevant in the context of "parallel sampling". This is where the `n` parameter in a request is used to request multiple completions from the same prompt. As part of adding parallel sampling support in [Pull Request #10980](https://github.com/vllm-project/vllm/pull/10980), we should also add these metrics. - `vllm:request_params_n` (Histogram) Observes the value of the 'n' parameter of every finished request. - `vllm:request_max_num_generation_tokens` (Histogram) Observes the maximum output length of all sequences in every finished sequence group. In the absence of parallel sampling, this is equivalent to `vllm:request_generation_tokens`. ### Speculative Decoding[¶](#speculative-decoding "Permanent link") Some legacy metrics are specific to "speculative decoding". This is where we generate candidate tokens using a faster, approximate method or model and then validate those tokens with the larger model. - `vllm:spec_decode_draft_acceptance_rate` (Gauge) - `vllm:spec_decode_efficiency` (Gauge) - `vllm:spec_decode_num_accepted_tokens` (Counter) - `vllm:spec_decode_num_draft_tokens` (Counter) - `vllm:spec_decode_num_emitted_tokens` (Counter) There is a PR under review ( [Pull Request #12193](https://github.com/vllm-project/vllm/pull/12193)) to add "prompt lookup (ngram)" speculative decoding to v1. Other techniques will follow. We should revisit these metrics in this context. Note We should probably expose acceptance rate as separate accepted and draft counters, like we do for prefix caching hit rate. Efficiency likely also needs similar treatment. ### Autoscaling and Load-balancing[¶](#autoscaling-and-load-balancing "Permanent link") A common use case for our metrics is to support automated scaling of vLLM instances. For related discussion from the [Kubernetes Serving Working Group](https://github.com/kubernetes/community/tree/master/wg-serving), see: - [Standardizing Large Model Server Metrics in Kubernetes](https://docs.google.com/document/d/1SpSp1E6moa4HSrJnS4x3NpLuj88sMXr2tbofKlzTZpk) - [Benchmarking LLM Workloads for Performance Evaluation and Autoscaling in Kubernetes](https://docs.google.com/document/d/1k4Q4X14hW4vftElIuYGDu5KDe2LtV1XammoG-Xi3bbQ) - [Inference Perf](https://github.com/kubernetes-sigs/wg-serving/tree/main/proposals/013-inference-perf) - [Issue #5041](https://github.com/vllm-project/vllm/issues/5041) and [Pull Request #12726](https://github.com/vllm-project/vllm/pull/12726). This is a non-trivial topic. Consider this comment from Rob: > I think this metric should focus on trying to estimate what the max concurrency that will cause the average request length > queries per second ... since this is really what will "saturate" the server. A clear goal is that we should expose the metrics required to detect this saturation point, so administrators can implement auto-scaling rules based on those. However, in order to do so, we need to have a clear view on how an administrator (and automated monitoring system) should judge an instance as approaching saturation: > To identify, what is the saturation point for model server compute (the inflection point where we cannot get more throughput with a higher request rate, but start to incur additional latency) so we can autoscale effectively? ### Metric Naming[¶](#metric-naming "Permanent link") Our approach to naming metrics probably deserves to be revisited: 1. The use of colons in metric names seems contrary to ["colons are reserved for user defined recording rules"](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). 2. Most of our metrics follow the convention of ending with units, but not all do. 3. Some of our metric names end with `_total`: If there is a suffix of `_total` on the metric name, it will be removed. When exposing the time series for counter, a `_total` suffix will be added. This is for compatibility between OpenMetrics and the Prometheus text format, as OpenMetrics requires the `_total` suffix. ### Adding More Metrics[¶](#adding-more-metrics "Permanent link") There is no shortage of ideas for new metrics: - Examples from other projects like [TGI](https://github.com/IBM/text-generation-inference?tab=readme-ov-file#metrics) - Proposals arising from specific use cases, like the Kubernetes auto-scaling topic above - Proposals that might arise out of standardisation efforts like [OpenTelemetry Semantic Conventions for Gen AI](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/gen-ai). We should be cautious in our approach to adding new metrics. While metrics are often relatively straightforward to add: 1. They can be difficult to remove - see the section on deprecation above. 2. They can have a meaningful performance impact when enabled. And metrics are usually of very limited use unless they can be enabled by default and in production. 3. They have an impact on development and maintenance of the project. Every metric added over time has made this effort more time-consuming, and perhaps not all metrics justify this ongoing investment in their maintenance. ## Tracing - OpenTelemetry[¶](#tracing-opentelemetry "Permanent link") Metrics provide an aggregated view over time of the system's performance and health. Tracing, on the other hand, tracks individual requests as they move through different services and components. Both fall under the more general heading of "Observability". vLLM has support for OpenTelemetry tracing: - Added by [Pull Request #4687](https://github.com/vllm-project/vllm/pull/4687) and reinstated by [Pull Request #20372](https://github.com/vllm-project/vllm/pull/20372) - Configured with `--oltp-traces-endpoint` and `--collect-detailed-traces` - [OpenTelemetry blog post](https://opentelemetry.io/blog/2024/llm-observability/) - [User-facing docs](https://github.com/vllm-project/vllm/blob/main/examples/observability/opentelemetry/README.md) - [Blog post](https://medium.com/@ronen.schaffer/follow-the-trail-supercharging-vllm-with-opentelemetry-distributed-tracing-aa655229b46f) - [IBM product docs](https://www.ibm.com/docs/en/instana-observability/current?topic=mgaa-monitoring-large-language-models-llms-vllm-public-preview) OpenTelemetry has a [Gen AI Working Group](https://github.com/open-telemetry/community/blob/main/projects/gen-ai.md). Since metrics is a big enough topic on its own, we consider the topic of tracing to be quite separate from metrics. ### OpenTelemetry Model Forward vs Execute Time[¶](#opentelemetry-model-forward-vs-execute-time "Permanent link") The current implementation exposes the following two metrics: - `vllm:model_forward_time_milliseconds` (Histogram) - The time spent in the model forward pass when this request was in the batch. - `vllm:model_execute_time_milliseconds` (Histogram) - The time spent in the model execute function. This will include model forward, block/sync across workers, cpu-gpu sync time and sampling time. These metrics are only enabled when OpenTelemetry tracing is enabled and if `--collect-detailed-traces=all/model/worker` is used. The documentation for this option states: > collect detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a performance impact. The metrics were added by [Pull Request #7089](https://github.com/vllm-project/vllm/pull/7089) and who up in an OpenTelemetry trace as: `[](#__codelineno-6-1)-> gen_ai.latency.time_in_scheduler: Double(0.017550230026245117) [](#__codelineno-6-2)-> gen_ai.latency.time_in_model_forward: Double(3.151565277099609) [](#__codelineno-6-3)-> gen_ai.latency.time_in_model_execute: Double(3.6468167304992676)` We already have `inference_time` and `decode_time` metrics, so the question is whether there are sufficiently common use cases for the higher-resolution timings to justify the overhead. Since we are going to treat the question of OpenTelemetry support separately, we will include these particular metrics under that topic. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/mm_processing.md "Edit this page") To enable various optimizations in vLLM such as [chunked prefill](https://docs.vllm.ai/en/configuration/optimization/#chunked-prefill) and [prefix caching](https://docs.vllm.ai/en/features/automatic_prefix_caching/), we use [BaseMultiModalProcessor](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor " BaseMultiModalProcessor") to provide the correspondence between placeholder feature tokens (e.g. ``) and multi-modal inputs (e.g. the raw input image) based on the outputs of HF processor. Here are the main features of [BaseMultiModalProcessor](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor " BaseMultiModalProcessor"): ## Prompt Update Detection[¶](#prompt-update-detection "Permanent link") One of the main responsibilities of HF processor is to update the prompt with placeholder tokens. For example: - Insert feature placeholder tokens (e.g. `...`, the number of which equals to the feature size) at the start of the string. - Replace existing input placeholder tokens (e.g. `` for a single image) with feature placeholder tokens (e.g. `...`, the number of which equals to the feature size). The information about which tokens have been updated is key to finding the correspondence between placeholder feature tokens and multi-modal inputs. In vLLM, this information is specified using [PromptUpdate](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.PromptUpdate " PromptUpdate dataclass ") in [\_get\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates " _get_prompt_updates(mm_items, hf_processor_mm_kwargs, out_mm_kwargs) abstractmethod "). We can automatically detect whether HF has updated the prompt by checking the existence of the updated tokens. ## Tokenized Prompt Inputs[¶](#tokenized-prompt-inputs "Permanent link") To enable tokenization in a separate process, we support passing input token IDs alongside multi-modal data. ### The problem[¶](#the-problem "Permanent link") Consider that HF processors follow these main steps: 1. Tokenize the text 2. Process multi-modal inputs 3. Perform prompt updates And we require that: - For text + multi-modal inputs, apply all steps 1--3. - For tokenized + multi-modal inputs, apply only steps 2--3. How can we achieve this without rewriting HF processors? We can try to call the HF processor several times on different inputs: - For text + multi-modal inputs, simply call the HF processor directly. - For tokenized + multi-modal inputs, call the processor only on the multi-modal inputs. While HF processors support text + multi-modal inputs natively, this is not so for tokenized + multi-modal inputs: an error is thrown if the number of input placeholder tokens do not correspond to the number of multi-modal inputs. Moreover, since the tokenized text has not passed through the HF processor, we have to apply Step 3 by ourselves to keep the output tokens and multi-modal data consistent with each other. ### Dummy text[¶](#dummy-text "Permanent link") We work around the first issue by requiring each model to define how to generate dummy text based on the number of multi-modal inputs, via [get\_dummy\_text](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseDummyInputsBuilder.get_dummy_text " get_dummy_text(mm_counts) abstractmethod "). This lets us generate dummy text corresponding to the multi-modal inputs and input them together to obtain the processed multi-modal data. ### Automatic prompt updating[¶](#automatic-prompt-updating "Permanent link") We address the second issue by implementing model-agnostic code in [\_apply\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._apply_prompt_updates " _apply_prompt_updates(token_ids, mm_prompt_updates)") to automatically update the prompt with feature placeholder tokens based on the specification outputted by [\_get\_prompt\_updates](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._get_prompt_updates " _get_prompt_updates(mm_items, hf_processor_mm_kwargs, out_mm_kwargs) abstractmethod "). ### Summary[¶](#summary "Permanent link") With the help of dummy text and automatic prompt updating, our multi-modal processor can finally accept both text and token prompts with multi-modal data. The detailed logic is shown in [\_apply\_hf\_processor\_main](https://docs.vllm.ai/en/api/vllm/multimodal/processing/#vllm.multimodal.processing.BaseMultiModalProcessor._apply_hf_processor_main " _apply_hf_processor_main(prompt, mm_items, hf_processor_mm_kwargs, tokenization_kwargs, *, enable_hf_prompt_update)"). ## Processor Output Caching[¶](#processor-output-caching "Permanent link") Some HF processors, such as the one for Qwen2-VL, are [very slow](https://github.com/vllm-project/vllm/issues/9238). To alleviate this problem, we cache the multi-modal outputs of HF processor to avoid processing the same multi-modal input (e.g. image) again. When new data is passed in, we first check which items are in the cache, and which ones are missing. The missing items are passed into the HF processor in a single batch and cached, before being merged with the existing items in the cache. Since we only process the missing multi-modal data items, the number of input placeholder tokens no longer corresponds to the number of the multi-modal inputs, so they can't be passed alongside the text prompt to HF processor. Therefore, we process the text and multi-modal inputs separately, using [dummy text](#dummy-text) to avoid HF errors. Since this skips HF's prompt updating code, we apply [automatic prompt updating](#automatic-prompt-updating) afterwards to keep the output tokens and multi-modal data consistent with each other. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/model_runner_v2.md "Edit this page") ## Introduction[¶](#introduction "Permanent link") Since vLLM V1 was first implemented, we discovered several fundamental design mistakes and accumulated significant technical debt. Many features were bolted on that were not considered in the original design. We also gained valuable insights into sampling techniques (for example, Gumbel-max sampling), tools (for example, Triton), and CUDA features (for example, UVA). With this knowledge, we implemented Model Runner V2 (MRV2) from first principles to be cleaner, more efficient, and more modular. In hindsight, many of V1's design choices were suboptimal. While MRV2 is not yet feature-complete, not rigorously tested, and still has open design decisions, we believe it is a substantial improvement over V1. This document describes the design of MRV2. ## 1\. Persistent Batch[¶](#1-persistent-batch "Permanent link") One significant source of friction in V1 is its persistent batch implementation. ### Background[¶](#background "Permanent link") V1 introduced persistent batches to minimize CPU overhead during input preparation. When requests are scheduled for a step, the model runner must construct contiguous input tensors (for example, block tables and per-request temperature values) to feed into the model. Building these tensors from scratch each step is often very slow in Python, especially for large tensors like block tables. The persistent batch optimization exploits the fact that request batches in consecutive steps are mostly identical. Only a few requests (if any) join or finish per step. By maintaining persistent state tensors and applying incremental diffs instead of reconstructing inputs from scratch, CPU overhead can be reduced significantly. ### Problems with V1's Approach[¶](#problems-with-v1s-approach "Permanent link") While efficient, V1's persistent batch design introduced unnecessary complexity due to coupling persistent state with input tensors. V1 uses persistent state tensors directly as model and sampler inputs, which imposes strict layout and ordering requirements. When requests join or finish, this often requires complex tensor-wide reordering rather than simple row insertion/removal. V1 also had to maintain `CachedRequestState`, a redundant backup copy of request state, because rows in persistent tensors can be overwritten while requests are still active. The result is complex bookkeeping that becomes more difficult under async scheduling. [![Persistent Batch in V1](https://docs.vllm.ai/en/assets/design/model_runner_v2/persistent_batch_v1.png)](https://docs.vllm.ai/en/assets/design/model_runner_v2/persistent_batch_v1.png) ### MRV2's Solution[¶](#mrv2s-solution "Permanent link") MRV2 decouples persistent state tensors from per-step input tensors. Given request ordering for the step (usually determined by the attention backend), MRV2 gathers input tensors from persistent state. 1. Pre-allocate a fixed-size tensor with `max_num_reqs` rows (1024 by default on most platforms). 2. Assign each request a permanent row for its active lifetime (until finish or preemption). 3. Treat preemption as completion. On resume, re-add request data as fresh state. This removes the need for `CachedRequestState` and simplifies bookkeeping. Large state tensors are mostly stored on GPU memory, so gather runs in parallel on the GPU with low overhead. [![Persistent Batch in MRV2](https://docs.vllm.ai/en/assets/design/model_runner_v2/persistent_batch_mrv2.png)](https://docs.vllm.ai/en/assets/design/model_runner_v2/persistent_batch_mrv2.png) ## 2\. Async-First[¶](#2-async-first "Permanent link") vLLM now relies heavily on asynchronous scheduling. The scheduler and worker prepare inputs for step `N+1` while the GPU executes step `N`, overlapping CPU and GPU work to maximize utilization. V1 was not originally designed with async scheduling in mind, and support required retrofitted behavior and hacks. MRV2 instead assumes the core model execution loop is a CUDA stream with no CPU synchronization points. CPU entrypoints queue work onto the stream. [![Async execution timeline](https://docs.vllm.ai/en/assets/design/model_runner_v2/async_sched.png)](https://docs.vllm.ai/en/assets/design/model_runner_v2/async_sched.png) ## 3\. Removing Async Barrier[¶](#3-removing-async-barrier "Permanent link") A key requirement for async execution is that CPU operations remain non-blocking. Both explicit sync (for example, `torch.accelerator.synchronize`) and implicit sync (for example, unpinned `.to("cuda")`) must be avoided. However, async execution can introduce race conditions when CPU and GPU concurrently touch the same memory. Example (unsafe): `[](#__codelineno-0-1)class ModelRunner: [](#__codelineno-0-2) def __init__(self, ...): [](#__codelineno-0-3) # Pinned buffer [](#__codelineno-0-4) self.states = torch.zeros( [](#__codelineno-0-5) max_num_reqs, dtype=torch.int32, device="cpu", pin_memory=True [](#__codelineno-0-6) ) [](#__codelineno-0-7) [](#__codelineno-0-8) def execute_step(self, ...): [](#__codelineno-0-9) self.states[req_idx] = new_req.data [](#__codelineno-0-10) states = self.states.to("cuda", non_blocking=True)` The CPU may modify `self.states` while GPU is still reading from it via async copy. V1 addresses this with an async barrier around critical sections. That avoids races but has drawbacks: 1. Easy to miss protected buffers (bug-prone). 2. Inflexible organization (all CPU work must stay inside barrier). 3. Potentially less overlap due to synchronization. [![Race condition with shared CPU buffer](https://docs.vllm.ai/en/assets/design/model_runner_v2/async_race_condition.png)](https://docs.vllm.ai/en/assets/design/model_runner_v2/async_race_condition.png) ### MRV2's Solution: Eliminate the Race[¶](#mrv2s-solution-eliminate-the-race "Permanent link") MRV2 separates persistent CPU state from the copied tensor: `[](#__codelineno-1-1)class ModelRunner: [](#__codelineno-1-2) def __init__(self, ...): [](#__codelineno-1-3) # Not pinned [](#__codelineno-1-4) self.states = torch.zeros( [](#__codelineno-1-5) max_num_reqs, dtype=torch.int32, device="cpu", pin_memory=False [](#__codelineno-1-6) ) [](#__codelineno-1-7) [](#__codelineno-1-8) def execute_step(self, ...): [](#__codelineno-1-9) self.states[req_idx] = new_req.data [](#__codelineno-1-10) tmp_states = self.states.pin_memory() [](#__codelineno-1-11) states = tmp_states.to("cuda", non_blocking=True)` Now CPU writes to `self.states` while GPU reads from `tmp_states`, eliminating the race without explicit synchronization. [![No race with temporary pinned copy](https://docs.vllm.ai/en/assets/design/model_runner_v2/async_no_race_condition.png)](https://docs.vllm.ai/en/assets/design/model_runner_v2/async_no_race_condition.png) ## 4\. StagedWriteTensor[¶](#4-stagedwritetensor "Permanent link") For large tensors like block tables, MRV2 avoids full CPU-to-GPU copies each step by using `StagedWriteTensor`: 1. Keep the base tensor on GPU. 2. Stage diffs on CPU. 3. Pack diffs into contiguous buffers. 4. Copy packed diffs to GPU. 5. Launch one kernel to apply diffs. Example usage: `[](#__codelineno-2-1)# Initialize state on GPU [](#__codelineno-2-2)state = StagedWriteTensor(size=(1024, 1000), dtype=torch.int32, device="cuda") [](#__codelineno-2-3)[](#__codelineno-2-4)# Write [3, 1, 2] into row 2, starting at index 3 [](#__codelineno-2-5)state.stage_write(row=2, start=3, value=[3, 1, 2]) [](#__codelineno-2-6)[](#__codelineno-2-7)# Write [-1, -2, -5] into row 0, starting at index 1 [](#__codelineno-2-8)state.stage_write(row=0, start=1, value=[-1, -2, -5]) [](#__codelineno-2-9)[](#__codelineno-2-10)# Apply staged changes [](#__codelineno-2-11)state.apply_write()` This supports ragged updates with no CPU-GPU synchronization and minimal kernel launches. It is especially useful for block tables and mixed CPU/GPU-written states such as `num_computed_tokens`. MRV2 uses Triton kernels to prepare inputs such as `input_ids`, `positions`, `query_start_loc`, and `seq_lens`. Benefits: 1. Better async behavior: GPU can derive values (for example with speculative decoding) that CPU may not know yet. 2. Lower CPU overhead: input prep is very cheap on GPU and avoids Python bottlenecks. ### Universal Virtual Addressing (UVA)[¶](#universal-virtual-addressing-uva "Permanent link") MRV2 uses UVA in some paths to let GPU kernels access large CPU-resident tensors directly (for example `prefill_token_ids`) without duplicating those tensors into GPU memory. ## 6\. Triton-Native Sampler[¶](#6-triton-native-sampler "Permanent link") MRV2 reimplements sampling mostly in Triton for better numeric/memory control and optimization. ### Gumbel Sampling Kernel[¶](#gumbel-sampling-kernel "Permanent link") MRV2 introduces a Triton Gumbel sampling kernel that avoids explicit softmax materialization and uses stateless in-kernel RNG from seed input. ### Efficient Top-K Logprobs[¶](#efficient-top-k-logprobs "Permanent link") V1 materializes full-vocabulary logprobs before top-k. MRV2 identifies top-k tokens from logits first, then computes logprobs only for selected tokens. This reduces peak GPU memory usage. ### Memory-Efficient Prompt Logprobs[¶](#memory-efficient-prompt-logprobs "Permanent link") MRV2 supports finer-grained chunking, including chunking inside a single prompt, to avoid memory spikes on long prompts. ### Better Compatibility with Speculative Decoding[¶](#better-compatibility-with-speculative-decoding "Permanent link") Instead of expanding per-request sampling states to match per-logit shapes, MRV2 uses indirection (`idx_mapping`) inside kernels to map each logits vector to the right request state. This simplifies support for complex sampling parameters and logits processors. ## 7\. Modularity[¶](#7-modularity "Permanent link") MRV2 emphasizes modularity. Compared to V1's large, entangled `gpu_model_runner.py`, MRV2 splits feature logic across dedicated files (for example, `mrope_utils.py`, `penalties.py`, and many others). It also consolidates model inputs into an `InputBatch` class and reduces direct model-runner attribute coupling. ## 8\. No Abuse of `dummy_run`[¶](#8-no-abuse-of-dummy_run "Permanent link") In V1, `dummy_run` handled too many responsibilities: - Initial memory profiling and `torch.compile` - CUDA graph capture - Warmups - Empty DP forward passes for EP+DP MRV2 simplifies this: 1. `execute_model` supports dummy runs without affecting state. 2. `dummy_run` delegates to `execute_model` for profiling, warmup, and empty DP forward passes. 3. CUDA graph capture uses a separate dedicated path. This reduces complexity and removes bugs caused by divergence between `execute_model` and `dummy_run` behavior. ## 9\. Explicit CUDA Graph Management[¶](#9-explicit-cuda-graph-management "Permanent link") V1's CUDA graph handling is implicit and hard to reason about. MRV2 uses a `CUDAGraphManager` that explicitly captures and launches full CUDA graphs through standard PyTorch APIs. This makes graph lifecycle and execution mode decisions more understandable and easier to extend. Example: MRV2 can capture multiple draft-model forward passes into one CUDA graph. ## Development Philosophy[¶](#development-philosophy "Permanent link") MRV2 changes should meet a higher code quality bar. As feature gaps with V1 are filled, features should be reconsidered from first principles in the MRV2 design context instead of quickly porting V1 behavior. A key requirement is preserving modularity and clean abstraction boundaries, even if that requires more upfront design iteration. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/moe_kernel_features.md "Edit this page") The purpose of this document is to provide an overview of the various MoE kernels (both modular and non-modular) so it will be easier to select an appropriate set of kernels for any particular situation. This includes information about the all2all backends used by modular kernels. ## Fused MoE Modular All2All backends[¶](#fused-moe-modular-all2all-backends "Permanent link") There are a number of all2all communication backends that are used to implement expert parallelism (EP) for the [`FusedMoE`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/layer/#vllm.model_executor.layers.fused_moe.layer.FusedMoE " FusedMoE") layer. The different [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") subclasses provide an interface for each all2all backend. The following table describes the relevant features of each backend, i.e. activation format, supported quantization schemes and async support. The output activation format (standard or batched) corresponds to the output of the prepare step of the [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") subclass, and the finalize step requires the same format. All the backend `prepare` methods expect activations in the standard format and all the `finalize` methods return activations in standard format. More details on the formats can be found in the [Fused MoE Modular Kernel](https://docs.vllm.ai/en/latest/fused_moe_modular_kernel/) document. The quantization types and formats enumerate which quantization schemes are supported by each [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") class. The quantization can happen before or after the dispatch based on the format the all2all backend supports, e.g. deepep\_high\_throughput supports only block-quantized fp8 format. Any other format will result in dispatching in higher precision and quantizing afterwards. The output of the prepare step for each backend is the quantized type. The finalize step generally requires the same input type as the original activations, e.g. if the original input is bfloat16 and the quantization scheme is fp8 with per-tensor scales, `prepare` will return fp8/per-tensor scale activations and `finalize` will take bfloat16 activations. See the diagrams in [Fused MoE Modular Kernel](https://docs.vllm.ai/en/latest/fused_moe_modular_kernel/) for more details on the types and formats of activations at each step of the MoE process. If no quantization type is specified, the kernel operates on float16 and/or bfloat16. Async backends support the use of DBO (Dual Batch Overlap) and shared expert overlap (where shared experts are computed during the combine step). Certain models require the topk weights to be applied to the input activations rather than the output activations when topk==1, e.g. Llama. For modular kernels, this feature is supported by the [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") subclass. For non-modular kernels, it is up to the experts function to deal with this flag. Unless otherwise specified, backends are controlled via the `--all2all-backend` command-line argument (or the `all2all_backend` parameter in [`ParallelConfig`](https://docs.vllm.ai/en/api/vllm/config/parallel/#vllm.config.parallel.ParallelConfig " ParallelConfig")). All backends except `flashinfer` only work with EP+DP or EP+TP. `Flashinfer` can work with EP or DP without EP. Backend Output act. format Quant. types Quant. format Async Apply Weight On Input Subclass naive standard all1 G,A,T N 6 [layer.py](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/layer/#vllm.model_executor.layers.fused_moe.layer.FusedMoE " FusedMoE") deepep\_high\_throughput standard fp8 G(128),A,T2 Y Y [`DeepEPHTPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ht/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht.DeepEPHTPrepareAndFinalize " DeepEPHTPrepareAndFinalize") deepep\_low\_latency batched fp8 G(128),A,T3 Y Y [`DeepEPLLPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ll/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll.DeepEPLLPrepareAndFinalize " DeepEPLLPrepareAndFinalize") flashinfer\_nvlink\_two\_sided standard nvfp4,fp8 G,A,T N N [`FlashInferNVLinkTwoSidedPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_two_sided/#vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_two_sided.FlashInferNVLinkTwoSidedPrepareAndFinalize " FlashInferNVLinkTwoSidedPrepareAndFinalize") flashinfer\_nvlink\_one\_sided standard nvfp4,bf16,mxfp8 G,A,T N N [`FlashInferNVLinkOneSidedPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/flashinfer_nvlink_one_sided/#vllm.model_executor.layers.fused_moe.prepare_finalize.flashinfer_nvlink_one_sided.FlashInferNVLinkOneSidedPrepareAndFinalize " FlashInferNVLinkOneSidedPrepareAndFinalize") Table key 1. All types: mxfp4, nvfp4, int4, int8, fp8 2. A,T quantization occurs after dispatch. 3. All quantization happens after dispatch. 4. Controlled by different env vars (`VLLM_FLASHINFER_MOE_BACKEND` "throughput" or "latency") 5. This is a no-op dispatcher that can be used to pair with any modular experts to produce a modular kernel that runs without dispatch or combine. These cannot be selected via environment variable. These are generally use for testing or adapting an expert subclass to the `fused_experts` API. 6. This depends on the experts implementation. * * * - G - Grouped - G(N) - Grouped w/block size N - A - Per activation token - T - Per tensor Modular kernels are supported by the following `FusedMoEMethodBase` classes. - [`ModelOptFp8MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/modelopt/#vllm.model_executor.layers.quantization.modelopt.ModelOptFp8MoEMethod " ModelOptFp8MoEMethod") - [`Fp8MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/fp8/#vllm.model_executor.layers.quantization.fp8.Fp8MoEMethod " Fp8MoEMethod") - [`CompressedTensorsW4A4Nvfp4MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w4a4_nvfp4/#vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe.compressed_tensors_moe_w4a4_nvfp4.CompressedTensorsW4A4Nvfp4MoEMethod " CompressedTensorsW4A4Nvfp4MoEMethod") - [`CompressedTensorsW8A8Fp8MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe/compressed_tensors_moe_w8a8_fp8/#vllm.model_executor.layers.quantization.compressed_tensors.compressed_tensors_moe.compressed_tensors_moe_w8a8_fp8.CompressedTensorsW8A8Fp8MoEMethod " CompressedTensorsW8A8Fp8MoEMethod") - [`GptOssMxfp4MoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/quantization/mxfp4/#vllm.model_executor.layers.quantization.mxfp4.GptOssMxfp4MoEMethod " GptOssMxfp4MoEMethod") - [`UnquantizedFusedMoEMethod`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/unquantized_fused_moe_method/#vllm.model_executor.layers.fused_moe.unquantized_fused_moe_method.UnquantizedFusedMoEMethod " UnquantizedFusedMoEMethod") ## Fused Experts Kernels[¶](#fused-experts-kernels "Permanent link") There are a number of MoE experts kernel implementations for different quantization types and architectures. Most follow the general API of the base Triton [`fused_experts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/fused_moe/#vllm.model_executor.layers.fused_moe.fused_moe.fused_experts " fused_experts(hidden_states, w1, w2, topk_weights, topk_ids, activation=MoEActivation.SILU, apply_router_weight_on_input=False, global_num_experts=-1, expert_map=None, quant_config=None)") function. Many have modular kernel adapters, so they can be used with compatible all2all backends. This table lists each experts kernel and its particular properties. Each kernel must be provided with one of the supported input activation formats. Some flavors of kernels support both standard and batched formats through different entry points, e.g. [`TritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/triton_moe/#vllm.model_executor.layers.fused_moe.experts.triton_moe.TritonExperts " TritonExperts") and [`BatchedTritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe/#vllm.model_executor.layers.fused_moe.experts.fused_batched_moe.BatchedTritonExperts " BatchedTritonExperts"). Batched format kernels are currently only needed for matching with certain all2all backends, e.g. [`DeepEPLLPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ll/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll.DeepEPLLPrepareAndFinalize " DeepEPLLPrepareAndFinalize"). Similar to the backend kernels, each experts kernel only supports certain quantization formats. For non-modular experts, the activations will be in the original type and quantized internally by the kernel. Modular experts will expect the activations to already be in the quantized format. Both types of experts will yield outputs in the original activation type. Each experts kernel supports one or more activation functions, e.g. silu or gelu, which are applied to the intermediate results. As with the backends, some experts support applying topk weights on the input activations. The entries in the column in this table only apply to the non-modular experts. Most experts flavors include an equivalent modular interface which will be a subclass of [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular"). To be used with a particular [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") subclass, MoE kernels must have compatible activation formats, quantization types and quantization formats. Kernel Input act. format Quant. types Quant. format Activation function Apply Weight On Input Modular Source triton standard all1 G,A,T silu, gelu, swigluoai, silu\_no\_mul, gelu\_no\_mul Y Y [`fused_experts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/fused_moe/#vllm.model_executor.layers.fused_moe.fused_moe.fused_experts " fused_experts(hidden_states, w1, w2, topk_weights, topk_ids, activation=MoEActivation.SILU, apply_router_weight_on_input=False, global_num_experts=-1, expert_map=None, quant_config=None)"), [`TritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/triton_moe/#vllm.model_executor.layers.fused_moe.experts.triton_moe.TritonExperts " TritonExperts") triton (batched) batched all1 G,A,T silu, gelu 6 Y [`BatchedTritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe/#vllm.model_executor.layers.fused_moe.experts.fused_batched_moe.BatchedTritonExperts " BatchedTritonExperts") deep gemm standard, batched fp8 G(128),A,T silu, gelu 6 Y [`DeepGemmExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe/#vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe.DeepGemmExperts " DeepGemmExperts"), [`BatchedDeepGemmExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/batched_deep_gemm_moe/#vllm.model_executor.layers.fused_moe.experts.batched_deep_gemm_moe.BatchedDeepGemmExperts " BatchedDeepGemmExperts") cutlass\_fp4 standard, batched nvfp4 A,T silu Y Y [`CutlassExpertsFp4`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/cutlass_moe/#vllm.model_executor.layers.fused_moe.experts.cutlass_moe.CutlassExpertsFp4 " CutlassExpertsFp4") cutlass\_fp8 standard, batched fp8 A,T silu, gelu Y Y [`CutlassExpertsFp8`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/cutlass_moe/#vllm.model_executor.layers.fused_moe.experts.cutlass_moe.CutlassExpertsFp8 " CutlassExpertsFp8"), [`CutlasBatchedExpertsFp8`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/cutlass_moe/#vllm.model_executor.layers.fused_moe.experts.cutlass_moe.CutlassBatchedExpertsFp8 " CutlassBatchedExpertsFp8") flashinfer standard nvfp4, fp8 T 5 N Y [`FlashInferExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe/#vllm.model_executor.layers.fused_moe.experts.flashinfer_cutlass_moe.FlashInferExperts " FlashInferExperts") gpt oss triton standard N/A N/A 5 Y Y [`triton_kernel_fused_experts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe/#vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe.triton_kernel_fused_experts " triton_kernel_fused_experts(output_tensor, hidden_states, w1, w2, routing_data, gather_indx, scatter_indx, topk, activation=MoEActivation.SWIGLUOAI, quant_config=None, swiglu_alpha=1.702, swiglu_limit=7.0, apply_router_weight_on_input=False, global_num_experts=-1, expert_map=None, intermediate_cache=None, a1q_scale=None)"), [`OAITritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/gpt_oss_triton_kernels_moe/#vllm.model_executor.layers.fused_moe.experts.gpt_oss_triton_kernels_moe.OAITritonExperts " OAITritonExperts") marlin standard, batched 3 / N/A 3 / N/A silu, swigluoai Y Y [`fused_marlin_moe`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/marlin_moe/#vllm.model_executor.layers.fused_moe.experts.marlin_moe.fused_marlin_moe " fused_marlin_moe(hidden_states, w1, w2, bias1, bias2, w1_scale, w2_scale, topk_weights, topk_ids, quant_type_id, apply_router_weight_on_input=False, global_num_experts=-1, activation=MoEActivation.SILU, activation_func=apply_moe_activation, moe_sum=None, expert_map=None, input_global_scale1=None, input_global_scale2=None, global_scale1=None, global_scale2=None, g_idx1=None, g_idx2=None, sort_indices1=None, sort_indices2=None, w1_zeros=None, w2_zeros=None, workspace=None, intermediate_cache13=None, intermediate_cache2=None, is_k_full=True, output=None, input_dtype=None, clamp_limit=None)"), [`MarlinExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/marlin_moe/#vllm.model_executor.layers.fused_moe.experts.marlin_moe.MarlinExperts " MarlinExperts"), [`BatchedMarlinExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/marlin_moe/#vllm.model_executor.layers.fused_moe.experts.marlin_moe.BatchedMarlinExperts " BatchedMarlinExperts") trtllm standard mxfp4, nvfp4 G(16),G(32) 5 N Y [`TrtLlmMxfp4ExpertsMonolithic`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/trtllm_mxfp4_moe/#vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe.TrtLlmMxfp4ExpertsMonolithic " TrtLlmMxfp4ExpertsMonolithic"), [`TrtLlmMxfp4ExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/trtllm_mxfp4_moe/#vllm.model_executor.layers.fused_moe.experts.trtllm_mxfp4_moe.TrtLlmMxfp4ExpertsModular " TrtLlmMxfp4ExpertsModular"), [`TrtLlmNvFp4ExpertsMonolithic`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe/#vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe.TrtLlmNvFp4ExpertsMonolithic " TrtLlmNvFp4ExpertsMonolithic"), [`TrtLlmNvfp4ExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe/#vllm.model_executor.layers.fused_moe.experts.trtllm_nvfp4_moe.TrtLlmNvFp4ExpertsModular " TrtLlmNvFp4ExpertsModular") rocm aiter moe standard mxfp4, fp8 G(32),G(128),A,T silu, gelu, swigluoai Y N `rocm_aiter_fused_experts`, `AiterExperts` cpu\_fused\_moe standard N/A N/A silu N N [`CPUFusedMOE`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/cpu_fused_moe/#vllm.model_executor.layers.fused_moe.cpu_fused_moe.CPUFusedMOE " CPUFusedMOE") naive batched4 batched int8, fp8 G,A,T silu, gelu 6 Y [`NaiveBatchedExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe/#vllm.model_executor.layers.fused_moe.experts.fused_batched_moe.NaiveBatchedExperts " NaiveBatchedExperts") Table key 1. All types: mxfp4, nvfp4, int4, int8, fp8 2. A dispatcher wrapper around triton and deep gemm experts. Will select based on type + shape + quantization params 3. uint4, uint8, fp8, fp4 4. This is a naive implementation of experts that supports batched format. Mainly used for testing. 5. The `activation` parameter is ignored and SwiGlu is used by default instead. 6. Only handled by or supported when used with modular kernels. ## Modular Kernel "families"[¶](#modular-kernel-families "Permanent link") The following table shows "families" of modular kernels that are intended to work together. There are some combinations which may work but have not yet been tested, e.g. flashinfer with other fp8 experts. backend [`FusedMoEPrepareAndFinalizeModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEPrepareAndFinalizeModular " FusedMoEPrepareAndFinalizeModular") subclasses [`FusedMoEExpertsModular`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/modular_kernel/#vllm.model_executor.layers.fused_moe.modular_kernel.FusedMoEExpertsModular " FusedMoEExpertsModular") subclasses deepep\_high\_throughput [`DeepEPHTPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ht/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ht.DeepEPHTPrepareAndFinalize " DeepEPHTPrepareAndFinalize") [`DeepGemmExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe/#vllm.model_executor.layers.fused_moe.experts.deep_gemm_moe.DeepGemmExperts " DeepGemmExperts"), [`TritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/triton_moe/#vllm.model_executor.layers.fused_moe.experts.triton_moe.TritonExperts " TritonExperts"), [`TritonOrDeepGemmExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/triton_deep_gemm_moe/#vllm.model_executor.layers.fused_moe.experts.triton_deep_gemm_moe.TritonOrDeepGemmExperts " TritonOrDeepGemmExperts"), [`CutlassExpertsFp8`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/cutlass_moe/#vllm.model_executor.layers.fused_moe.experts.cutlass_moe.CutlassExpertsFp8 " CutlassExpertsFp8"), [`MarlinExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/marlin_moe/#vllm.model_executor.layers.fused_moe.experts.marlin_moe.MarlinExperts " MarlinExperts") deepep\_low\_latency [`DeepEPLLPrepareAndFinalize`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/prepare_finalize/deepep_ll/#vllm.model_executor.layers.fused_moe.prepare_finalize.deepep_ll.DeepEPLLPrepareAndFinalize " DeepEPLLPrepareAndFinalize") `BatchedDeepGemmExperts`, [`BatchedTritonExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/fused_batched_moe/#vllm.model_executor.layers.fused_moe.experts.fused_batched_moe.BatchedTritonExperts " BatchedTritonExperts"), [`CutlassBatchedExpertsFp8`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/cutlass_moe/#vllm.model_executor.layers.fused_moe.experts.cutlass_moe.CutlassBatchedExpertsFp8 " CutlassBatchedExpertsFp8"), [`BatchedMarlinExperts`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/fused_moe/experts/marlin_moe/#vllm.model_executor.layers.fused_moe.experts.marlin_moe.BatchedMarlinExperts " BatchedMarlinExperts") flashinfer `FlashInferCutlassMoEPrepareAndFinalize` `FlashInferExperts` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/multiprocessing.md "Edit this page") ## Debugging[¶](#debugging "Permanent link") Please see the [Troubleshooting](https://docs.vllm.ai/en/usage/troubleshooting/#python-multiprocessing) page for information on known issues and how to solve them. ## Introduction[¶](#introduction "Permanent link") Important The source code references are to the state of the code at the time of writing in December 2024. The use of Python multiprocessing in vLLM is complicated by: - using vLLM as a library, which limits control over its internal code; - incompatibilities between certain multiprocessing methods and vLLM dependencies. This document describes how vLLM deals with these challenges. ## Multiprocessing Methods[¶](#multiprocessing-methods "Permanent link") [Python multiprocessing methods](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods) include: - `spawn` - Spawn a new Python process. The default on Windows and macOS. - `fork` - Use `os.fork()` to fork the Python interpreter. The default on Linux for Python versions prior to 3.14. - `forkserver` - Spawn a server process that will fork a new process on request. The default on Linux for Python version 3.14 and newer. ### Tradeoffs[¶](#tradeoffs "Permanent link") `fork` is the fastest method, but is incompatible with dependencies that use threads. If you are under macOS, using `fork` may cause the process to crash. `spawn` is more compatible with dependencies, but can be problematic when vLLM is used as a library. If the consuming code does not use a `__main__` guard (`if __name__ == "__main__":`), the code will be inadvertently re-executed when vLLM spawns a new process. This can lead to infinite recursion, among other problems. `forkserver` will spawn a new server process that will fork new processes on demand. This unfortunately has the same problem as `spawn` when vLLM is used as a library. The server process is created as a spawned new process, which will re-execute code not protected by a `__main__` guard. For both `spawn` and `forkserver`, the process must not depend on inheriting any global state as would be the case with `fork`. ## Compatibility with Dependencies[¶](#compatibility-with-dependencies "Permanent link") Multiple vLLM dependencies indicate either a preference or requirement for using `spawn`: - [https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing](https://pytorch.org/docs/stable/notes/multiprocessing.html#cuda-in-multiprocessing) - [https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors](https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors) - [https://docs.habana.ai/en/latest/PyTorch/Getting\_Started\_with\_PyTorch\_and\_Gaudi/Getting\_Started\_with\_PyTorch.html?highlight=multiprocessing#torch-multiprocessing-for-dataloaders](https://docs.habana.ai/en/latest/PyTorch/Getting_Started_with_PyTorch_and_Gaudi/Getting_Started_with_PyTorch.html?highlight=multiprocessing#torch-multiprocessing-for-dataloaders) Known issues exist when using `fork` after initializing these dependencies. ## Current State (v0)[¶](#current-state-v0 "Permanent link") The environment variable `VLLM_WORKER_MULTIPROC_METHOD` can be used to control which method is used by vLLM. The current default is `fork`. - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L339-L342](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L339-L342) If the main process is controlled via the `vllm` command, `spawn` is used because it's the most widely compatible. - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/scripts.py#L123-L140](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/scripts.py#L123-L140) The `multiproc_xpu_executor` forces the use of `spawn`. - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/executor/multiproc\_xpu\_executor.py#L14-L18](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/executor/multiproc_xpu_executor.py#L14-L18) There are other miscellaneous places hard-coding the use of `spawn`: - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/distributed/device\_communicators/all\_reduce\_utils.py#L135](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/distributed/device_communicators/all_reduce_utils.py#L135) - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/entrypoints/openai/api\_server.py#L184](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/entrypoints/openai/api_server.py#L184) Related PRs: - [Pull Request #8823](https://github.com/vllm-project/vllm/pull/8823) ## Prior State in v1[¶](#prior-state-in-v1 "Permanent link") There was an environment variable to control whether multiprocessing is used in the v1 engine core, `VLLM_ENABLE_V1_MULTIPROCESSING`. This defaulted to off. - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L452-L454](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/envs.py#L452-L454) When it was enabled, the v1 [`LLMEngine`](https://docs.vllm.ai/en/api/vllm/v1/engine/llm_engine/#vllm.v1.engine.llm_engine.LLMEngine " LLMEngine") would create a new process to run the engine core. - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm\_engine.py#L93-L95](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L93-L95) - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm\_engine.py#L70-L77](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/llm_engine.py#L70-L77) - [https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core\_client.py#L44-L45](https://github.com/vllm-project/vllm/blob/d05f88679bedd73939251a17c3d785a354b2946c/vllm/v1/engine/core_client.py#L44-L45) It was off by default for all the reasons mentioned above - compatibility with dependencies and code using vLLM as a library. ### Changes Made in v1[¶](#changes-made-in-v1 "Permanent link") There is not an easy solution with Python's `multiprocessing` that will work everywhere. As a first step, we can get v1 into a state where it does "best effort" choice of multiprocessing method to maximize compatibility. - Default to `fork`. - Use `spawn` when we know we control the main process (`vllm` was executed). - If we detect `cuda` was previously initialized, force `spawn` and emit a warning. We know `fork` will break, so this is the best we can do. The case that is known to still break in this scenario is code using vLLM as a library that initializes `cuda` before calling vLLM. The warning we emit should instruct users to either add a `__main__` guard or to disable multiprocessing. If that known-failure case occurs, the user will see two messages that explain what is happening. First, a log message from vLLM: ``[](#__codelineno-0-1)WARNING 12-11 14:50:37 multiproc_worker_utils.py:281] CUDA was previously [](#__codelineno-0-2) initialized. We must use the `spawn` multiprocessing start method. Setting [](#__codelineno-0-3) VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See [](#__codelineno-0-4) https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing [](#__codelineno-0-5) for more information.`` Second, Python itself will raise an exception with a nice explanation: `[](#__codelineno-1-1)RuntimeError: [](#__codelineno-1-2) An attempt has been made to start a new process before the [](#__codelineno-1-3) current process has finished its bootstrapping phase. [](#__codelineno-1-4) [](#__codelineno-1-5) This probably means that you are not using fork to start your [](#__codelineno-1-6) child processes and you have forgotten to use the proper idiom [](#__codelineno-1-7) in the main module: [](#__codelineno-1-8) [](#__codelineno-1-9) if __name__ == '__main__': [](#__codelineno-1-10) freeze_support() [](#__codelineno-1-11) ... [](#__codelineno-1-12) [](#__codelineno-1-13) The "freeze_support()" line can be omitted if the program [](#__codelineno-1-14) is not going to be frozen to produce an executable. [](#__codelineno-1-15) [](#__codelineno-1-16) To fix this issue, refer to the "Safe importing of main module" [](#__codelineno-1-17) section in https://docs.python.org/3/library/multiprocessing.html` ## Alternatives Considered[¶](#alternatives-considered "Permanent link") ### Detect if a `__main__` guard is present[¶](#detect-if-a-__main__-guard-is-present "Permanent link") It has been suggested that we could behave better if we could detect whether code using vLLM as a library has a `__main__` guard in place. This [post on Stack Overflow](https://stackoverflow.com/questions/77220442/multiprocessing-pool-in-a-python-class-without-name-main-guard) was from a library author facing the same question. It is possible to detect whether we are in the original, `__main__` process, or a subsequent spawned process. However, it does not appear to be straight forward to detect whether a `__main__` guard is present in the code. This option has been discarded as impractical. ### Use `forkserver`[¶](#use-forkserver "Permanent link") At first it appears that `forkserver` is a nice solution to the problem. However, the way it works presents the same challenges that `spawn` does when vLLM is used as a library. ### Force `spawn` all the time[¶](#force-spawn-all-the-time "Permanent link") One way to clean this up is to just force the use of `spawn` all the time and document that the use of a `__main__` guard is required when using vLLM as a library. This would unfortunately break existing code and make vLLM harder to use, violating the desire to make the [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class as easy as possible to use. Instead of pushing this on our users, we will retain the complexity to do our best to make things work. ## Future Work[¶](#future-work "Permanent link") We may want to consider a different worker management approach in the future that works around these challenges. 1. We could implement something `forkserver`\-like, but have the process manager be something we initially launch by running our own subprocess and a custom entrypoint for worker management (launch a `vllm-manager` process). 2. We can explore other libraries that may better suit our needs. Examples to consider: - [https://github.com/joblib/loky](https://github.com/joblib/loky) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/nixl_kv_cache_lease.md "Edit this page") In disaggregated prefill/decode deployments, the Prefill instance (P) must hold KV cache blocks in GPU memory after completing a prefill, waiting for the Decode instance (D) to read them via RDMA. A mechanism is needed to determine when those blocks can safely be freed when D isn't able to retrieve them. This mechanism was introduced in [PR #41383](https://github.com/vllm-project/vllm/pull/41383). ## Motivation[¶](#motivation "Permanent link") ### The single-timeout problem[¶](#the-single-timeout-problem "Permanent link") The original design used a single, large timeout (`VLLM_NIXL_ABORT_REQUEST_TIMEOUT`, default 480s) to control how long P retains KV blocks. When D crashed or disconnected, P would hold onto potentially several GBs of "dead" blocks for up to 8 minutes before reclaiming them. During this window, subsequent requests hitting P would find reduced cache capacity and experience degraded performance. ### The overloading problem[¶](#the-overloading-problem "Permanent link") Simply lowering the timeout introduces a different failure mode. Under traffic surges, requests can sit in D's waiting queue for a long time before being scheduled. If the fixed timeout on P is too short, blocks get freed before D ever has a chance to read them --- causing unnecessary recomputation and wasted prefill work. ### Solution: lease renewal via heartbeats[¶](#solution-lease-renewal-via-heartbeats "Permanent link") The lease renewal mechanism addresses both problems simultaneously. P grants a **short initial lease** (default 30s) when prefill completes. While a request is **queued or in-flight** on D, D **periodically sends heartbeats** to P extending the lease. If D crashes and stops heartbeating, P reclaims blocks within seconds of the last heartbeat rather than waiting minutes. If D is merely overloaded, the heartbeats keep the blocks alive for as long as needed. ## How It Works[¶](#how-it-works "Permanent link") ### Lease lifecycle[¶](#lease-lifecycle "Permanent link") When P finishes a prefill, it pins the KV blocks with an initial lease duration (`kv_lease_duration`, default 30s). From that point, the blocks are held until either: 1. **D completes the KV transfer** --- P receives a read-completion notification and frees the blocks immediately. 2. **D keeps heartbeating** --- each heartbeat extends the lease by `lease_duration * 2/3` (~20s), keeping blocks alive indefinitely while D is healthy. 3. **No heartbeat arrives** --- the lease expires and P reclaims the blocks. ### Piggybacking on NIXL notifications[¶](#piggybacking-on-nixl-notifications "Permanent link") Rather than introducing a new transport channel, heartbeats reuse NIXL's existing notification system (`send_notif` / `get_new_notifs`). The notification medium is backend-specific, with automatic fallback from IB/RoCE to TCP already handled by NIXL. Each single heartbeat message sent from D to a particular P renews all requests pinned in P on behalf of that D --- in other words, a single batched message per iteration renews the lease of multiple requests. ### Scheduler-side tracking (D)[¶](#scheduler-side-tracking-d "Permanent link") A critical insight is that heartbeating must start **as soon as a request enters D's scheduler** --- not when it gets scheduled for execution. Under heavy load, a request may sit in the waiting queue for much longer than the initial lease duration, and the gap between arrival and scheduling is unbounded. To achieve this, D's connector ([`NixlConnectorScheduler`](https://docs.vllm.ai/en/api/vllm/distributed/kv_transfer/kv_connector/v1/nixl/scheduler/#vllm.distributed.kv_transfer.kv_connector.v1.nixl.scheduler.NixlConnectorScheduler " NixlConnectorScheduler")) hooks into the scheduler via `on_new_request()`. When a request with `do_remote_prefill=True` arrives, the connector immediately starts tracking it for heartbeats. Requests are grouped by `remote_engine_id` for efficient batching. On each scheduler step, heartbeat metadata is packaged into `NixlConnectorMetadata` and sent to the worker, throttled by a heartbeat interval of `lease_duration // 6` (~5s). Tracking stops when either the KV transfer completes (via `update_connector_output`) or the request finishes/aborts (via `request_finished`). ### Timing and simplicity[¶](#timing-and-simplicity "Permanent link") Heartbeat sending and processing happen **in the forward loop**, not in a background thread. This means timing is not millisecond-precise --- a long model forward pass will delay heartbeats. However, the lease durations are configured with sufficient margin: with default settings, the heartbeat interval (~5s) and lease extension (~20s) are at least an order of magnitude larger than a typical forward pass. This avoids lock complexity between threads while keeping the design simple and extensible. ## Happy Path[¶](#happy-path "Permanent link") ``` sequenceDiagram participant R as Routing Proxy participant P as Prefill Instance participant D as Decode Instance R->>P: Request (do_remote_decode=True) P->>P: Run prefill P->>P: Grant lease (30s) P->>R: Response (with kv_transfer_params) R->>D: Request (do_remote_prefill=True) note over D: Request enters waiting queue D->>D: on_new_request() starts tracking loop Every ~5s (heartbeat interval) D->>P: Heartbeat (extend lease) P->>P: Lease extended by ~20s end note over D: Request scheduled for execution D->>P: KV transfer (RDMA read) P-->D: Transfer complete D->>D: Stop heartbeating P->>P: Free KV blocks ``` ## Decode Instance Crash[¶](#decode-instance-crash "Permanent link") ``` sequenceDiagram participant R as Routing Proxy participant P as Prefill Instance participant D as Decode Instance R->>P: Request (do_remote_decode=True) P->>P: Run prefill (holds onto KVs with lease) P->>R: Response R->>D: Request (do_remote_prefill=True) D->>P: Heartbeat (extend lease) D->>P: Heartbeat (extend lease) note over D: D crashes note over P: No heartbeat received P->>P: Lease expires (~20s, not 480s) P->>P: Free KV blocks ``` ### Worker-side sending and receiving[¶](#worker-side-sending-and-receiving "Permanent link") **On D (sending):** During `start_load_kv()` (called every forward pass), the worker reads `metadata.heartbeat_by_engine` and sends batched heartbeat notifications to each remote P engine. If D hasn't yet handshaked with P for a given engine (common for requests still in the waiting queue), it triggers a **proactive handshake** in a background thread. The heartbeat is deferred to the next step once the handshake completes --- the early handshake also **speeds up the eventual KV transfer.** **On P (receiving):** In `_get_new_notifs()`, P's worker checks incoming NIXL notifications. Messages starting with `"HB:"` are routed to `_handle_heartbeat()`, which extends the lease expiry for each referenced request using `max(old_expiry, now + lease_extension)`. This ensures leases are never accidentally shortened. ## Bidirectional KV Transfer[¶](#bidirectional-kv-transfer "Permanent link") For multi-turn conversations, [bidirectional KV transfer](https://docs.vllm.ai/en/features/disagg_prefill/) allows D to cache KV blocks that P can pull from on subsequent turns. Since the timing of the next conversational turn is **client-dependent** (not controlled by the system), the heartbeat-based lease mechanism does not apply here. Instead, a separate `decoder_kv_blocks_ttl` (default 480s) provides a simple fixed timeout for blocks cached on D. If the client takes too long to continue the conversation, the blocks expire and P recomputes. Future work may extend a symmetric heartbeat mechanism to this case. ## Key Design Decisions[¶](#key-design-decisions "Permanent link") - **Per-request leasing, not per-instance.** P has no notion of which D its KV blocks belong to --- block ownership is only resolved after prefill completes and the router selects a D. Leasing at the request level avoids coupling P/D selection in the load balancer. In practice, D batches lease extensions toward the same P by grouping requests with the same `remote_engine_id`. - **NIXL notifications as transport.** Heartbeats reuse the existing `send_notif`/`get_new_notifs` system rather than adding ZMQ connections or API changes. The notification medium is backend-specific with IB/RoCE-to-TCP fallback already handled, making heartbeats work across any NIXL-supported transport. - **No background thread.** Heartbeat sending and processing happen in the forward loop (`start_load_kv` / `get_finished`). This avoids lock complexity between threads. Lease durations provide sufficient margin over forward-pass latency (seconds vs. milliseconds). - **Proactive handshake.** When D needs to heartbeat a P engine it hasn't connected to yet (common for requests still in the waiting queue), it triggers an early handshake in a background thread. This also speeds up the eventual KV transfer. - **Heterogeneous TP support.** When P TP > D TP (e.g., P TP=4, D TP=2), a single D worker pulls from multiple P workers. Heartbeats must be sent to all P workers for a given engine. Conversely, when D TP > P TP, a single P receives notifications from multiple Ds, which simply refreshes the TTL multiple times with no downside. ## Configuration[¶](#configuration "Permanent link") The lease mechanism is controlled through `kv_connector_extra_config` in `--kv-transfer-config`: Parameter Default Description `kv_lease_duration` 30s Initial lease duration on P. Heartbeat interval and extension amount are derived automatically (`interval = duration // 6`, `extension = duration * 2 // 3`). `decoder_kv_blocks_ttl` 480s TTL for KV blocks cached on D in bidirectional transfer mode. Simple fixed timeout, not renewed via heartbeats. `[](#__codelineno-0-1)vllm serve \ [](#__codelineno-0-2) --kv-transfer-config '{ [](#__codelineno-0-3) "kv_connector": "NixlConnector", [](#__codelineno-0-4) "kv_role": "kv_producer", [](#__codelineno-0-5) "kv_connector_extra_config": {"kv_lease_duration": 60} [](#__codelineno-0-6) }'` For full NixlConnector configuration details, see the [NixlConnector Usage Guide](https://docs.vllm.ai/en/features/nixl_connector_usage/). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/optimization_levels.md "Edit this page") ## Overview[¶](#overview "Permanent link") vLLM provides 4 optimization levels (`-O0`, `-O1`, `-O2`, `-O3`) that allow users to trade off startup time for performance: - `-O0`: No optimization. Fastest startup time, but lowest performance. - `-O1`: Fast optimization. Simple compilation and fast fusions, and PIECEWISE cudagraphs. - `-O2`: Default optimization. Additional compilation ranges, additional fusions, FULL\_AND\_PIECEWISE cudagraphs. - `-O3`: Aggressive optimization. Currently equal to `-O2`, but may include additional time-consuming or experimental optimizations in the future. All optimization level defaults can be achieved by manually setting the underlying flags. User-set flags take precedence over optimization level defaults. ## Level Summaries and Usage Examples[¶](#level-summaries-and-usage-examples "Permanent link") `[](#__codelineno-0-1)# CLI usage [](#__codelineno-0-2)vllm serve RedHatAI/Llama-3.2-1B-FP8 -O1 [](#__codelineno-0-3)[](#__codelineno-0-4)# Python API usage [](#__codelineno-0-5)from vllm.entrypoints.llm import LLM [](#__codelineno-0-6)[](#__codelineno-0-7)llm = LLM( [](#__codelineno-0-8) model="RedHatAI/Llama-3.2-1B-FP8", [](#__codelineno-0-9) optimization_level=2 # equivalent to -O2 [](#__codelineno-0-10))` ### `-O0`: No Optimization[¶](#-o0-no-optimization "Permanent link") Startup as fast as possible - no autotuning, no compilation, and no cudagraphs. This level is good for initial phases of development and debugging. Settings: - `-cc.cudagraph_mode=NONE` - `-cc.mode=NONE` (also resulting in `-cc.custom_ops=["none"]`) - `-cc.pass_config.fuse_...=False` (all fusions disabled) - `--kernel-config.enable_flashinfer_autotune=False` ### `-O1`: Fast Optimization[¶](#-o1-fast-optimization "Permanent link") Prioritize fast startup, but still enable basic optimizations like compilation and cudagraphs. This level is a good balance for most development scenarios where you want faster startup but still make sure your code does not break cudagraphs or compilation. Settings: - `-cc.cudagraph_mode=PIECEWISE` - `-cc.mode=VLLM_COMPILE` - `--kernel-config.enable_flashinfer_autotune=True` Fusions: - `-cc.pass_config.fuse_norm_quant=True`\* - `-cc.pass_config.fuse_act_quant=True`\* - `-cc.pass_config.fuse_act_padding=True`† - `-cc.pass_config.fuse_mla_dual_rms_norm=True`† \* These fusions are only enabled when either op is using a custom kernel, otherwise Inductor fusion is better. † These fusions are ROCm-only and require AITER. ### `-O2`: Full Optimization (Default)[¶](#-o2-full-optimization-default "Permanent link") Prioritize performance at the expense of additional startup time. This level is recommended for production workloads and is hence the default. Fusions in this level _may_ take longer due to additional compile ranges. Settings (on top of `-O1`): - `-cc.cudagraph_mode=FULL_AND_PIECEWISE` - `-cc.pass_config.fuse_allreduce_rms=True` - `-cc.pass_config.fuse_rope_kvcache=True`† † These fusions are ROCm-only and require AITER. ### `-O3`: Aggressive Optimization[¶](#-o3-aggressive-optimization "Permanent link") This level is currently the same as `-O2`, but may include additional optimizations in the future that are more time-consuming or experimental. ## Troubleshooting[¶](#troubleshooting "Permanent link") ### Common Issues[¶](#common-issues "Permanent link") 1. **Startup Time Too Long**: Use `-O0` or `-O1` for faster startup 2. **Compilation Errors**: Use `debug_dump_path` for additional debugging information 3. **Performance Issues**: Ensure using `-O2` for production --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/p2p_nccl_connector.md "Edit this page") An implementation of xPyD with dynamic scaling based on point-to-point communication, partly inspired by Dynamo. ## Detailed Design[¶](#detailed-design "Permanent link") ### Overall Process[¶](#overall-process "Permanent link") As shown in Figure 1, the overall process of this **PD disaggregation** solution is described through a request flow: 1. The client sends an HTTP request to the Proxy/Router's `/v1/completions` interface. 2. The Proxy/Router selects a **1P1D (1 Prefill instance + 1 Decode instance)** through either through round-robin or random selection, generates a `request_id` (rules to be introduced later), modifies the `max_tokens` in the HTTP request message to **1**, and then forwards the request to the **P instance**. 3. Immediately afterward, the Proxy/Router forwards the **original HTTP request** to the **D instance**. 4. The **P instance** performs **Prefill** and then **actively sends the generated KV cache** to the D instance (using **PUT\_ASYNC** mode). The D instance's `zmq_addr` can be resolved through the `request_id`. 5. The **D instance** has a **dedicated thread** for receiving the KV cache (to avoid blocking the main process). The received KV cache is saved into the **GPU memory buffer**, the size of which is determined by the vLLM startup parameter `kv_buffer_size`. When the GPU buffer is full, the KV cache is stored in the **local Tensor memory pool**. 6. During the **Decode**, the D instance's main process retrieves the KV cache (transmitted by the P instance) from either the **GPU buffer** or the **memory pool**, thereby **skipping Prefill**. 7. After completing **Decode**, the D instance returns the result to the **Proxy/Router**, which then forwards it to the **client**. [![image1](https://github.com/user-attachments/assets/fb01bde6-755b-49f7-ad45-48a94b1e10a7)](https://github.com/user-attachments/assets/fb01bde6-755b-49f7-ad45-48a94b1e10a7) ### Proxy/Router (Demo)[¶](#proxyrouter-demo "Permanent link") A simple HTTP service acts as the entry point for client requests and starts a background thread to listen for P/D instances reporting their HTTP IP and PORT, as well as ZMQ IP and PORT. It maintains a dictionary of `http_addr -> zmq_addr`. The `http_addr` is the IP:PORT for the vLLM instance's request, while the `zmq_addr` is the address for KV cache handshake and metadata reception. The Proxy/Router is responsible for selecting 1P1D based on the characteristics of the client request, such as the prompt, and generating a corresponding `request_id`, for example: `[](#__codelineno-0-1)cmpl-___prefill_addr_10.0.1.2:21001___decode_addr_10.0.1.3:22001_93923d63113b4b338973f24d19d4bf11-0` Currently, to quickly verify whether xPyD can work, a round-robin selection of 1P1D is used. In the future, it is planned to use a trie combined with the load status of instances to select appropriate P and D. Each P/D instance periodically sends a heartbeat packet to the Proxy/Router (currently every 3 seconds) to register (i.e., report `http_addr -> zmq_addr`) and keep the connection alive. If an instance crashes and fails to send a ping for a certain period of time, the Proxy/Router will remove the timed-out instance (this feature has not yet been developed). ### KV Cache Transfer Methods[¶](#kv-cache-transfer-methods "Permanent link") There are three methods for KVCache transfer: PUT, GET, and PUT\_ASYNC. These methods can be specified using the `--kv-transfer-config` and `kv_connector_extra_config` parameters, specifically through the `send_type` field. Both PUT and PUT\_ASYNC involve the P instance actively sending KVCache to the D instance. The difference is that PUT is a synchronous transfer method that blocks the main process, while PUT\_ASYNC is an asynchronous transfer method. PUT\_ASYNC uses a dedicated thread for sending KVCache, which means it does not block the main process. In contrast, the GET method involves the P instance saving the KVCache to the memory buffer after computing the prefill. The D instance then actively retrieves the computed KVCache from the P instance once it has allocated space for the KVCache. Experimental results have shown that the performance of these methods, from highest to lowest, is as follows: PUT\_ASYNC → GET → PUT. ### P2P Communication via ZMQ & NCCL[¶](#p2p-communication-via-zmq-nccl "Permanent link") As long as the address of the counterpart is known, point-to-point KV cache transfer (using NCCL) can be performed, without being constrained by rank and world size. To support dynamic scaling (expansion and contraction) of instances with PD disaggregation. This means that adding or removing P/D instances does not require a full system restart. Each P/D instance only needs to create a single `P2pNcclEngine` instance. This instance maintains a ZMQ Server, which runs a dedicated thread to listen on the `zmq_addr` address and receive control flow requests from other instances. These requests include requests to establish an NCCL connection and requests to send KVCache metadata (such as tensor shapes and data types). However, it does not actually transmit the KVCache data itself. When a P instance and a D instance transmit KVCache for the first time, they need to establish a ZMQ connection and an NCCL group. For subsequent KVCache transmissions, this ZMQ connection and NCCL group are reused. The NCCL group consists of only two ranks, meaning the world size is equal to 2. This design is intended to support dynamic scaling, which means that adding or removing P/D instances does not require a full system restart. As long as the address of the counterpart is known, point-to-point KVCache transmission can be performed, without being restricted by rank or world size. ### NCCL Group Topology[¶](#nccl-group-topology "Permanent link") Currently, only symmetric TP (Tensor Parallelism) methods are supported for KVCache transmission. Asymmetric TP and PP (Pipeline Parallelism) methods will be supported in the future. Figure 2 illustrates the 1P2D setup, where each instance has a TP (Tensor Parallelism) degree of 2. There are a total of 7 NCCL groups: three vLLM instances each have one NCCL group with TP=2. Additionally, the 0th GPU card of the P instance establishes an NCCL group with the 0th GPU card of each D instance. Similarly, the 1st GPU card of the P instance establishes an NCCL group with the 1st GPU card of each D instance. [![image2](https://github.com/user-attachments/assets/837e61d6-365e-4cbf-8640-6dd7ab295b36)](https://github.com/user-attachments/assets/837e61d6-365e-4cbf-8640-6dd7ab295b36) Each NCCL group occupies a certain amount of GPU memory buffer for communication, the size of which is primarily influenced by the `NCCL_MAX_NCHANNELS` environment variable. When `NCCL_MAX_NCHANNELS=16`, an NCCL group typically occupies 100MB, while when `NCCL_MAX_NCHANNELS=8`, it usually takes up 52MB. For large-scale xPyD configurations—such as DeepSeek's 96P144D—this implementation is currently not feasible. Moving forward, we are considering using RDMA for point-to-point communication and are also keeping an eye on UCCL. ### GPU Memory Buffer and Tensor Memory Pool[¶](#gpu-memory-buffer-and-tensor-memory-pool "Permanent link") The trade-off in the size of the memory buffer is as follows: For P instances, the memory buffer is not required in PUT and PUT\_ASYNC modes, but it is necessary in GET mode. For D instances, a memory buffer is needed in all three modes. The memory buffer for D instances should not be too large. Similarly, for P instances in GET mode, the memory buffer should also not be too large. The memory buffer of D instances is used to temporarily store KVCache sent by P instances. If it is too large, it will reduce the KVCache space available for normal inference by D instances, thereby decreasing the inference batch size and ultimately leading to a reduction in output throughput. The size of the memory buffer is configured by the parameter `kv_buffer_size`, measured in bytes, and is typically set to 5%～10% of the memory size. If the `--max-num-seqs` parameter for P instances is set to a large value, due to the large batch size, P instances will generate a large amount of KVCache simultaneously. This may exceed the capacity of the memory buffer of D instances, resulting in KVCache loss. Once KVCache is lost, D instances need to recompute Prefill, which is equivalent to performing Prefill twice. Consequently, the time-to-first-token (TTFT) will significantly increase, leading to degraded performance. To address the above issues, I have designed and developed a local Tensor memory pool for storing KVCache, inspired by the buddy system used in Linux memory modules. Since the memory is sufficiently large, typically in the TB range on servers, there is no need to consider prefix caching or using block-based designs to reuse memory, thereby saving space. When the memory buffer is insufficient, KVCache can be directly stored in the Tensor memory pool, and D instances can subsequently retrieve KVCache from it. The read and write speed is that of PCIe, with PCIe 4.0 having a speed of approximately 21 GB/s, which is usually faster than the Prefill speed. Otherwise, solutions like Mooncake and lmcache would not be necessary. The Tensor memory pool acts as a flood diversion area, typically unused except during sudden traffic surges. In the worst-case scenario, my solution performs no worse than the normal situation with a Cache store. ## Install vLLM[¶](#install-vllm "Permanent link") `[](#__codelineno-1-1)pip install "vllm>=0.9.2"` ## Run xPyD[¶](#run-xpyd "Permanent link") ### Instructions[¶](#instructions "Permanent link") - The following examples are run on an A800 (80GB) device, using the Meta-Llama-3.1-8B-Instruct model. - Pay attention to the setting of the `kv_buffer_size` (in bytes). The empirical value is 10% of the GPU memory size. This is related to the kvcache size. If it is too small, the GPU memory buffer for temporarily storing the received kvcache will overflow, causing the kvcache to be stored in the tensor memory pool, which increases latency. If it is too large, the kvcache available for inference will be reduced, leading to a smaller batch size and decreased throughput. - For Prefill instances, when using non-GET mode, the `kv_buffer_size` can be set to 1, as Prefill currently does not need to receive kvcache. However, when using GET mode, a larger `kv_buffer_size` is required because it needs to store the kvcache sent to the D instance. - You may need to modify the `kv_buffer_size` and `port` in the following commands (if there is a conflict). - `PUT_ASYNC` offers the best performance and should be prioritized. - The `--port` must be consistent with the `http_port` in the `--kv-transfer-config`. - The `disagg_proxy_p2p_nccl_xpyd.py` script will use port 10001 (for receiving client requests) and port 30001 (for receiving service discovery from P and D instances). - The node running the proxy must have `quart` installed. - Supports multiple nodes; you just need to modify the `proxy_ip` and `proxy_port` in `--kv-transfer-config`. - In the following examples, it is assumed that **the proxy's IP is 10.0.1.1**. ### Run 1P3D[¶](#run-1p3d "Permanent link") #### Proxy (e.g. 10.0.1.1)[¶](#proxy-eg-10011 "Permanent link") `[](#__codelineno-2-1)cd {your vllm directory}/examples/disaggregated/p2p_nccl_xpyd/ [](#__codelineno-2-2)python3 disagg_proxy_p2p_nccl_xpyd.py &` #### Prefill1 (e.g. 10.0.1.2 or 10.0.1.1)[¶](#prefill1-eg-10012-or-10011 "Permanent link") Command `[](#__codelineno-3-1)CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \ [](#__codelineno-3-2) --host 0.0.0.0 \ [](#__codelineno-3-3) --port 20001 \ [](#__codelineno-3-4) --tensor-parallel-size 1 \ [](#__codelineno-3-5) --seed 1024 \ [](#__codelineno-3-6) --served-model-name base_model \ [](#__codelineno-3-7) --dtype float16 \ [](#__codelineno-3-8) --max-model-len 10000 \ [](#__codelineno-3-9) --max-num-batched-tokens 10000 \ [](#__codelineno-3-10) --max-num-seqs 256 \ [](#__codelineno-3-11) --trust-remote-code \ [](#__codelineno-3-12) --gpu-memory-utilization 0.9 \ [](#__codelineno-3-13) --kv-transfer-config \ [](#__codelineno-3-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20001"}}' > /var/vllm.log 2>&1 &` #### Decode1 (e.g. 10.0.1.3 or 10.0.1.1)[¶](#decode1-eg-10013-or-10011 "Permanent link") Command `[](#__codelineno-4-1)CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \ [](#__codelineno-4-2) --host 0.0.0.0 \ [](#__codelineno-4-3) --port 20002 \ [](#__codelineno-4-4) --tensor-parallel-size 1 \ [](#__codelineno-4-5) --seed 1024 \ [](#__codelineno-4-6) --served-model-name base_model \ [](#__codelineno-4-7) --dtype float16 \ [](#__codelineno-4-8) --max-model-len 10000 \ [](#__codelineno-4-9) --max-num-batched-tokens 10000 \ [](#__codelineno-4-10) --max-num-seqs 256 \ [](#__codelineno-4-11) --trust-remote-code \ [](#__codelineno-4-12) --gpu-memory-utilization 0.7 \ [](#__codelineno-4-13) --kv-transfer-config \ [](#__codelineno-4-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20002"}}' > /var/vllm.log 2>&1 &` #### Decode2 (e.g. 10.0.1.4 or 10.0.1.1)[¶](#decode2-eg-10014-or-10011 "Permanent link") Command `[](#__codelineno-5-1)CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \ [](#__codelineno-5-2) --host 0.0.0.0 \ [](#__codelineno-5-3) --port 20003 \ [](#__codelineno-5-4) --tensor-parallel-size 1 \ [](#__codelineno-5-5) --seed 1024 \ [](#__codelineno-5-6) --served-model-name base_model \ [](#__codelineno-5-7) --dtype float16 \ [](#__codelineno-5-8) --max-model-len 10000 \ [](#__codelineno-5-9) --max-num-batched-tokens 10000 \ [](#__codelineno-5-10) --max-num-seqs 256 \ [](#__codelineno-5-11) --trust-remote-code \ [](#__codelineno-5-12) --gpu-memory-utilization 0.7 \ [](#__codelineno-5-13) --kv-transfer-config \ [](#__codelineno-5-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"23001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20003"}}' > /var/vllm.log 2>&1 &` #### Decode3 (e.g. 10.0.1.5 or 10.0.1.1)[¶](#decode3-eg-10015-or-10011 "Permanent link") Command `[](#__codelineno-6-1)CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \ [](#__codelineno-6-2) --host 0.0.0.0 \ [](#__codelineno-6-3) --port 20004 \ [](#__codelineno-6-4) --tensor-parallel-size 1 \ [](#__codelineno-6-5) --seed 1024 \ [](#__codelineno-6-6) --served-model-name base_model \ [](#__codelineno-6-7) --dtype float16 \ [](#__codelineno-6-8) --max-model-len 10000 \ [](#__codelineno-6-9) --max-num-batched-tokens 10000 \ [](#__codelineno-6-10) --max-num-seqs 256 \ [](#__codelineno-6-11) --trust-remote-code \ [](#__codelineno-6-12) --gpu-memory-utilization 0.7 \ [](#__codelineno-6-13) --kv-transfer-config \ [](#__codelineno-6-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"24001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20004"}}' > /var/vllm.log 2>&1 &` ### Run 3P1D[¶](#run-3p1d "Permanent link") #### Proxy (e.g. 10.0.1.1)[¶](#proxy-eg-10011_1 "Permanent link") `[](#__codelineno-7-1)cd {your vllm directory}/examples/disaggregated/p2p_nccl_xpyd/ [](#__codelineno-7-2)python3 disagg_proxy_p2p_nccl_xpyd.py &` #### Prefill1 (e.g. 10.0.1.2 or 10.0.1.1)[¶](#prefill1-eg-10012-or-10011_1 "Permanent link") Command `[](#__codelineno-8-1)CUDA_VISIBLE_DEVICES=0 vllm serve {your model directory} \ [](#__codelineno-8-2) --host 0.0.0.0 \ [](#__codelineno-8-3) --port 20001 \ [](#__codelineno-8-4) --tensor-parallel-size 1 \ [](#__codelineno-8-5) --seed 1024 \ [](#__codelineno-8-6) --served-model-name base_model \ [](#__codelineno-8-7) --dtype float16 \ [](#__codelineno-8-8) --max-model-len 10000 \ [](#__codelineno-8-9) --max-num-batched-tokens 10000 \ [](#__codelineno-8-10) --max-num-seqs 256 \ [](#__codelineno-8-11) --trust-remote-code \ [](#__codelineno-8-12) --gpu-memory-utilization 0.9 \ [](#__codelineno-8-13) --kv-transfer-config \ [](#__codelineno-8-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"21001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20001"}}' > /var/vllm.log 2>&1 &` #### Prefill2 (e.g. 10.0.1.3 or 10.0.1.1)[¶](#prefill2-eg-10013-or-10011 "Permanent link") Command `[](#__codelineno-9-1)CUDA_VISIBLE_DEVICES=1 vllm serve {your model directory} \ [](#__codelineno-9-2) --host 0.0.0.0 \ [](#__codelineno-9-3) --port 20002 \ [](#__codelineno-9-4) --tensor-parallel-size 1 \ [](#__codelineno-9-5) --seed 1024 \ [](#__codelineno-9-6) --served-model-name base_model \ [](#__codelineno-9-7) --dtype float16 \ [](#__codelineno-9-8) --max-model-len 10000 \ [](#__codelineno-9-9) --max-num-batched-tokens 10000 \ [](#__codelineno-9-10) --max-num-seqs 256 \ [](#__codelineno-9-11) --trust-remote-code \ [](#__codelineno-9-12) --gpu-memory-utilization 0.9 \ [](#__codelineno-9-13) --kv-transfer-config \ [](#__codelineno-9-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"22001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20002"}}' > /var/vllm.log 2>&1 &` #### Prefill3 (e.g. 10.0.1.4 or 10.0.1.1)[¶](#prefill3-eg-10014-or-10011 "Permanent link") Command `[](#__codelineno-10-1)CUDA_VISIBLE_DEVICES=2 vllm serve {your model directory} \ [](#__codelineno-10-2) --host 0.0.0.0 \ [](#__codelineno-10-3) --port 20003 \ [](#__codelineno-10-4) --tensor-parallel-size 1 \ [](#__codelineno-10-5) --seed 1024 \ [](#__codelineno-10-6) --served-model-name base_model \ [](#__codelineno-10-7) --dtype float16 \ [](#__codelineno-10-8) --max-model-len 10000 \ [](#__codelineno-10-9) --max-num-batched-tokens 10000 \ [](#__codelineno-10-10) --max-num-seqs 256 \ [](#__codelineno-10-11) --trust-remote-code \ [](#__codelineno-10-12) --gpu-memory-utilization 0.9 \ [](#__codelineno-10-13) --kv-transfer-config \ [](#__codelineno-10-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_buffer_size":"1e1","kv_port":"23001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20003"}}' > /var/vllm.log 2>&1 &` #### Decode1 (e.g. 10.0.1.5 or 10.0.1.1)[¶](#decode1-eg-10015-or-10011 "Permanent link") Command `[](#__codelineno-11-1)CUDA_VISIBLE_DEVICES=3 vllm serve {your model directory} \ [](#__codelineno-11-2) --host 0.0.0.0 \ [](#__codelineno-11-3) --port 20004 \ [](#__codelineno-11-4) --tensor-parallel-size 1 \ [](#__codelineno-11-5) --seed 1024 \ [](#__codelineno-11-6) --served-model-name base_model \ [](#__codelineno-11-7) --dtype float16 \ [](#__codelineno-11-8) --max-model-len 10000 \ [](#__codelineno-11-9) --max-num-batched-tokens 10000 \ [](#__codelineno-11-10) --max-num-seqs 256 \ [](#__codelineno-11-11) --trust-remote-code \ [](#__codelineno-11-12) --gpu-memory-utilization 0.7 \ [](#__codelineno-11-13) --kv-transfer-config \ [](#__codelineno-11-14) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_buffer_size":"8e9","kv_port":"24001","kv_connector_extra_config":{"proxy_ip":"10.0.1.1","proxy_port":"30001","http_port":"20004"}}' > /var/vllm.log 2>&1 &` ## Single request[¶](#single-request "Permanent link") `[](#__codelineno-12-1)curl -X POST -s http://10.0.1.1:10001/v1/completions \ [](#__codelineno-12-2)-H "Content-Type: application/json" \ [](#__codelineno-12-3)-d '{ [](#__codelineno-12-4) "model": "base_model", [](#__codelineno-12-5) "prompt": "San Francisco is a", [](#__codelineno-12-6) "max_tokens": 10, [](#__codelineno-12-7) "temperature": 0 [](#__codelineno-12-8)}'` ## Benchmark[¶](#benchmark "Permanent link") Command `[](#__codelineno-13-1)vllm bench serve \ [](#__codelineno-13-2) --backend vllm \ [](#__codelineno-13-3) --model base_model \ [](#__codelineno-13-4) --tokenizer meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-13-5) --dataset-name "random" \ [](#__codelineno-13-6) --host 10.0.1.1 \ [](#__codelineno-13-7) --port 10001 \ [](#__codelineno-13-8) --random-input-len 1024 \ [](#__codelineno-13-9) --random-output-len 1024 \ [](#__codelineno-13-10) --ignore-eos \ [](#__codelineno-13-11) --burstiness 100 \ [](#__codelineno-13-12) --percentile-metrics "ttft,tpot,itl,e2el" \ [](#__codelineno-13-13) --metric-percentiles "90,95,99" \ [](#__codelineno-13-14) --seed $(date +%s) \ [](#__codelineno-13-15) --trust-remote-code \ [](#__codelineno-13-16) --request-rate 3 \ [](#__codelineno-13-17) --num-prompts 1000` ## Shut down[¶](#shut-down "Permanent link") `[](#__codelineno-14-1)pgrep python | xargs kill -9 && pkill -f python` ## Test data[¶](#test-data "Permanent link") ### **Scenario**: 1K input & 200 output tokens, E2E P99 latency ~2s[¶](#scenario-1k-input-200-output-tokens-e2e-p99-latency-2s "Permanent link") [![testdata](https://github.com/user-attachments/assets/cef0953b-4567-4bf9-b940-405b92a28eb1)](https://github.com/user-attachments/assets/cef0953b-4567-4bf9-b940-405b92a28eb1) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/paged_attention.md "Edit this page") Warning This is a historical document based on the [original paper for vLLM](https://arxiv.org/abs/2309.06180). It no longer describes the code used in vLLM today. Currently, vLLM utilizes its own implementation of a multi-head query attention kernel (`csrc/attention/attention_kernels.cu`). This kernel is designed to be compatible with vLLM's paged KV caches, where the key and value cache are stored in separate blocks (note that this block concept differs from the GPU thread block. So in a later document, I will refer to vLLM paged attention block as "block", while refer to GPU thread block as "thread block"). To achieve high performance, this kernel relies on a specially designed memory layout and access method, specifically when threads read data from global memory to shared memory. The purpose of this document is to provide a high-level explanation of the kernel implementation step by step, aiding those who wish to learn about the vLLM multi-head query attention kernel. After going through this document, users will likely have a better understanding and feel easier to follow the actual implementation. Please note that this document may not cover all details, such as how to calculate the correct index for the corresponding data or the dot multiplication implementation. However, after reading this document and becoming familiar with the high-level logic flow, it should be easier for you to read the actual code and understand the details. ## Inputs[¶](#inputs "Permanent link") The kernel function takes a list of arguments for the current thread to perform its assigned work. The three most important arguments are the input pointers `q`, `k_cache`, and `v_cache`, which point to query, key, and value data on global memory that need to be read and processed. The output pointer `out` points to global memory where the result should be written. These four pointers actually refer to multidimensional arrays, but each thread only accesses the portion of data assigned to it. I have omitted all other runtime parameters here for simplicity. `[](#__codelineno-0-1)template [](#__codelineno-0-2)__device__ void paged_attention_kernel( [](#__codelineno-0-3) ... // Other side args. [](#__codelineno-0-4) const scalar_t* __restrict__ out, // [num_seqs, num_heads, max_num_partitions, head_size] [](#__codelineno-0-5) const scalar_t* __restrict__ q, // [num_seqs, num_heads, head_size] [](#__codelineno-0-6) const scalar_t* __restrict__ k_cache, // [num_blocks, num_kv_heads, head_size/x, block_size, x] [](#__codelineno-0-7) const scalar_t* __restrict__ v_cache, // [num_blocks, num_kv_heads, head_size, block_size] [](#__codelineno-0-8) ... // Other side args. [](#__codelineno-0-9))` There are also a list of template arguments above the function signature that are determined during compilation time. `scalar_t` represents the data type of the query, key, and value data elements, such as FP16. `HEAD_SIZE` indicates the number of elements in each head. `BLOCK_SIZE` refers to the number of tokens in each block. `NUM_THREADS` denotes the number of threads in each thread block. `PARTITION_SIZE` represents the number of tensor parallel GPUs (For simplicity, we assume this is 0 and tensor parallel is disabled). With these arguments, we need to perform a sequence of preparations. This includes calculating the current head index, block index, and other necessary variables. However, for now, we can ignore these preparations and proceed directly to the actual calculations. It will be easier to understand them once we grasp the entire flow. ## Concepts[¶](#concepts "Permanent link") Just before we dive into the calculation flow, I want to describe a few concepts that are needed for later sections. However, you may skip this section and return later if you encounter any confusing terminologies. - **Sequence**: A sequence represents a client request. For example, the data pointed to by `q` has a shape of `[num_seqs, num_heads, head_size]`. That represents there are total `num_seqs` of query sequence data are pointed by `q`. Since this kernel is a single query attention kernel, each sequence only has one query token. Hence, the `num_seqs` equals the total number of tokens that are processed in the batch. - **Context**: The context consists of the generated tokens from the sequence. For instance, `["What", "is", "your"]` are the context tokens, and the input query token is `"name"`. The model might generate the token `"?"`. - **Vec**: The vec is a list of elements that are fetched and calculated together. For query and key data, the vec size (`VEC_SIZE`) is determined so that each thread group can fetch and calculate 16 bytes of data at a time. For value data, the vec size (`V_VEC_SIZE`) is determined so that each thread can fetch and calculate 16 bytes of data at a time. For example, if the `scalar_t` is FP16 (2 bytes) and `THREAD_GROUP_SIZE` is 2, the `VEC_SIZE` will be 4, while the `V_VEC_SIZE` will be 8. - **Thread group**: The thread group is a small group of threads(`THREAD_GROUP_SIZE`) that fetches and calculates one query token and one key token at a time. Each thread handles only a portion of the token data. The total number of elements processed by one thread group is referred as `x`. For example, if the thread group contains 2 threads and the head size is 8, then thread 0 handles the query and key elements at index 0, 2, 4, 6, while thread 1 handles the elements at index 1, 3, 5, 7. - **Block**: The key and value cache data in vLLM are split into blocks. Each block stores data for a fixed number(`BLOCK_SIZE`) of tokens at one head. Each block may contain only a portion of the whole context tokens. For example, if the block size is 16 and the head size is 128, then for one head, one block can store 16 \* 128 = 2048 elements. - **Warp**: A warp is a group of 32 threads(`WARP_SIZE`) that execute simultaneously on a stream multiprocessor (SM). In this kernel, each warp processes the calculation between one query token and key tokens of one entire block at a time (it may process multiple blocks in multiple iterations). For example, if there are 4 warps and 6 blocks for one context, the assignment would be like warp 0 handles the 0th, 4th blocks, warp 1 handles the 1st, 5th blocks, warp 2 handles the 2nd block and warp 3 handles the 3rd block. - **Thread block**: A thread block is a group of threads(`NUM_THREADS`) that can access the same shared memory. Each thread block contains multiple warps(`NUM_WARPS`), and in this kernel, each thread block processes the calculation between one query token and key tokens of a whole context. - **Grid**: A grid is a collection of thread blocks and defines the shape of the collection. In this kernel, the shape is `(num_heads, num_seqs, max_num_partitions)`. Therefore, each thread block only handles the calculation for one head, one sequence, and one partition. ## Query[¶](#query "Permanent link") This section will introduce how query data is stored in memory and fetched by each thread. As mentioned above, each thread group fetches one query token data, while each thread itself only handles a part of one query token data. Within each warp, every thread group will fetch the same query token data, but will multiply it with different key token data. `[](#__codelineno-1-1)const scalar_t* q_ptr = q + seq_idx * q_stride + head_idx * HEAD_SIZE;` [![query](https://docs.vllm.ai/en/assets/design/paged_attention/query.png)](https://docs.vllm.ai/en/assets/design/paged_attention/query.png) Each thread defines its own `q_ptr` which points to the assigned query token data on global memory. For example, if `VEC_SIZE` is 4 and `HEAD_SIZE` is 128, the `q_ptr` points to data that contains total of 128 elements divided into 128 / 4 = 32 vecs. [![q_vecs](https://docs.vllm.ai/en/assets/design/paged_attention/q_vecs.png)](https://docs.vllm.ai/en/assets/design/paged_attention/q_vecs.png) `[](#__codelineno-2-1)__shared__ Q_vec q_vecs[THREAD_GROUP_SIZE][NUM_VECS_PER_THREAD];` Next, we need to read the global memory data pointed to by `q_ptr` into shared memory as `q_vecs`. It is important to note that each vecs is assigned to a different row. For example, if the `THREAD_GROUP_SIZE` is 2, thread 0 will handle the 0th row vecs, while thread 1 handles the 1st row vecs. By reading the query data in this way, neighboring threads like thread 0 and thread 1 can read neighbor memory, achieving the memory coalescing to improve performance. ## Key[¶](#key "Permanent link") Similar to the "Query" section, this section introduces memory layout and assignment for keys. While each thread group only handle one query token one kernel run, it may handle multiple key tokens across multiple iterations. Meanwhile, each warp will process multiple blocks of key tokens in multiple iterations, ensuring that all context tokens are processed by the entire thread group after the kernel run. In this context, "handle" refers to performing the dot multiplication between query data and key data. `[](#__codelineno-3-1)const scalar_t* k_ptr = k_cache + physical_block_number * kv_block_stride [](#__codelineno-3-2) + kv_head_idx * kv_head_stride [](#__codelineno-3-3) + physical_block_offset * x;` Unlike to `q_ptr`, `k_ptr` in each thread will point to different key token at different iterations. As shown above, that `k_ptr` points to key token data based on `k_cache` at assigned block, assigned head and assigned token. [![key](https://docs.vllm.ai/en/assets/design/paged_attention/key.png)](https://docs.vllm.ai/en/assets/design/paged_attention/key.png) The diagram above illustrates the memory layout for key data. It assumes that the `BLOCK_SIZE` is 16, `HEAD_SIZE` is 128, `x` is 8, `THREAD_GROUP_SIZE` is 2, and there are a total of 4 warps. Each rectangle represents all the elements for one key token at one head, which will be processed by one thread group. The left half shows the total 16 blocks of key token data for warp 0, while the right half represents the remaining key token data for other warps or iterations. Inside each rectangle, there are a total 32 vecs (128 elements for one token) that will be processed by 2 threads (one thread group) separately. [![k_vecs](https://docs.vllm.ai/en/assets/design/paged_attention/k_vecs.png)](https://docs.vllm.ai/en/assets/design/paged_attention/k_vecs.png) `[](#__codelineno-4-1)K_vec k_vecs[NUM_VECS_PER_THREAD]` Next, we need to read the key token data from `k_ptr` and store them on register memory as `k_vecs`. We use register memory for `k_vecs` because it will only be accessed by one thread once, whereas `q_vecs` will be accessed by multiple threads multiple times. Each `k_vecs` will contain multiple vectors for later calculation. Each vec will be set at each inner iteration. The assignment of vecs allows neighboring threads in a warp to read neighboring memory together, which again promotes the memory coalescing. For instance, thread 0 will read vec 0, while thread 1 will read vec 1. In the next inner loop, thread 0 will read vec 2, while thread 1 will read vec 3, and so on. You may still be a little confused about the overall flow. Don't worry, please keep reading the next "QK" section. It will illustrate the query and key calculation flow in a clearer and higher-level manner. ## QK[¶](#qk "Permanent link") As shown the pseudocode below, before the entire for loop block, we fetch the query data for one token and store it in `q_vecs`. Then, in the outer for loop, we iterate through different `k_ptrs` that point to different tokens and prepare the `k_vecs` in the inner for loop. Finally, we perform the dot multiplication between the `q_vecs` and each `k_vecs`. `[](#__codelineno-5-1)q_vecs = ... [](#__codelineno-5-2)for ... { [](#__codelineno-5-3) k_ptr = ... [](#__codelineno-5-4) for ... { [](#__codelineno-5-5) k_vecs[i] = ... [](#__codelineno-5-6) } [](#__codelineno-5-7) ... [](#__codelineno-5-8) float qk = scale * Qk_dot::dot(q_vecs[thread_group_offset], k_vecs); [](#__codelineno-5-9)}` As mentioned before, for each thread, it only fetches part of the query and key token data at a time. However, there will be a cross thread group reduction happen in the `Qk_dot<>::dot` . So `qk` returned here is not just between part of the query and key token dot multiplication, but actually a full result between entire query and key token data. For example, if the value of `HEAD_SIZE` is 128 and `THREAD_GROUP_SIZE` is 2, each thread's `k_vecs` will contain total 64 elements. However, the returned `qk` is actually the result of dot multiplication between 128 query elements and 128 key elements. If you want to learn more about the details of the dot multiplication and reduction, you may refer to the implementation of `Qk_dot<>::dot`. However, for the sake of simplicity, I will not cover it in this document. ## Softmax[¶](#softmax "Permanent link") Next, we need to calculate the normalized softmax for all `qk`s, as shown above, where each \\(x\\) represents a `qk`. To do this, we must obtain the reduced value of `qk_max`(\\(m(x)\\)) and the `exp_sum`(\\(\\ell(x)\\)) of all `qk`s. The reduction should be performed across the entire thread block, encompassing results between the query token and all context key tokens. \\\[ \\begin{gather\*} m(x):=\\max \_i \\quad x\_i \\\\ \\quad f(x):=\\left\[\\begin{array}{lll}e^{x\_1-m(x)} & \\ldots & e^{x\_B-m(x)}\\end{array}\\right\]\\\\ \\quad \\ell(x):=\\sum\_i f(x)\_i \\\\ \\quad \\operatorname{softmax}(x):=\\frac{f(x)}{\\ell(x)} \\end{gather\*} \\\] ### `qk_max` and `logits`[¶](#qk_max-and-logits "Permanent link") Just right after we get the `qk` result, we can set the temporary `logits` result with `qk` (In the end, the `logits` should store the normalized softmax result). Also we can compare and collect the `qk_max` for all `qk`s that are calculated by current thread group. `[](#__codelineno-6-1)if (thread_group_offset == 0) { [](#__codelineno-6-2) const bool mask = token_idx >= context_len; [](#__codelineno-6-3) logits[token_idx - start_token_idx] = mask ? 0.f : qk; [](#__codelineno-6-4) qk_max = mask ? qk_max : fmaxf(qk_max, qk); [](#__codelineno-6-5)}` Please note that the `logits` here is on shared memory, so each thread group will set the fields for its own assigned context tokens. Overall, the size of logits should be number of context tokens. `[](#__codelineno-7-1)for (int mask = WARP_SIZE / 2; mask >= THREAD_GROUP_SIZE; mask /= 2) { [](#__codelineno-7-2) qk_max = fmaxf(qk_max, VLLM_SHFL_XOR_SYNC(qk_max, mask)); [](#__codelineno-7-3)} [](#__codelineno-7-4)[](#__codelineno-7-5)if (lane == 0) { [](#__codelineno-7-6) red_smem[warp_idx] = qk_max; [](#__codelineno-7-7)}` Then we need to get the reduced `qk_max` across each warp. The main idea is to make threads in warp to communicate with each other and get the final max `qk` . `[](#__codelineno-8-1)for (int mask = NUM_WARPS / 2; mask >= 1; mask /= 2) { [](#__codelineno-8-2) qk_max = fmaxf(qk_max, VLLM_SHFL_XOR_SYNC(qk_max, mask)); [](#__codelineno-8-3)} [](#__codelineno-8-4)qk_max = VLLM_SHFL_SYNC(qk_max, 0);` Finally, we can get the reduced `qk_max` from whole thread block by compare the `qk_max` from all warps in this thread block. Then we need to broadcast the final result to each thread. ### `exp_sum`[¶](#exp_sum "Permanent link") Similar to `qk_max`, we need to get the reduced sum value from the entire thread block too. `[](#__codelineno-9-1)for (int i = thread_idx; i < num_tokens; i += NUM_THREADS) { [](#__codelineno-9-2) float val = __expf(logits[i] - qk_max); [](#__codelineno-9-3) logits[i] = val; [](#__codelineno-9-4) exp_sum += val; [](#__codelineno-9-5)} [](#__codelineno-9-6)... [](#__codelineno-9-7)exp_sum = block_sum(&red_smem[NUM_WARPS], exp_sum);` Firstly, sum all exp values from each thread group, and meanwhile, convert each entry of `logits` from `qk` to `exp(qk - qk_max)`. Please note, the `qk_max` here is already the max `qk` across the whole thread block. And then we can do reduction for `exp_sum` across whole thread block just like the `qk_max`. `[](#__codelineno-10-1)const float inv_sum = __fdividef(1.f, exp_sum + 1e-6f); [](#__codelineno-10-2)for (int i = thread_idx; i < num_tokens; i += NUM_THREADS) { [](#__codelineno-10-3) logits[i] *= inv_sum; [](#__codelineno-10-4)}` Finally, with the reduced `qk_max` and `exp_sum`, we can obtain the final normalized softmax result as `logits`. This `logits` variable will be used for dot multiplication with the value data in later steps. Now, it should store the normalized softmax result of `qk` for all assigned context tokens. ## Value[¶](#value "Permanent link") [![value](https://docs.vllm.ai/en/assets/design/paged_attention/value.png)](https://docs.vllm.ai/en/assets/design/paged_attention/value.png) [![logits_vec](https://docs.vllm.ai/en/assets/design/paged_attention/logits_vec.png)](https://docs.vllm.ai/en/assets/design/paged_attention/logits_vec.png) [![v_vec](https://docs.vllm.ai/en/assets/design/paged_attention/v_vec.png)](https://docs.vllm.ai/en/assets/design/paged_attention/v_vec.png) Now we need to retrieve the value data and perform dot multiplication with `logits`. Unlike query and key, there is no thread group concept for value data. As shown in diagram, different from key token memory layout, elements from the same column correspond to the same value token. For one block of value data, there are `HEAD_SIZE` of rows and `BLOCK_SIZE` of columns that are split into multiple `v_vecs`. Each thread always fetches `V_VEC_SIZE` elements from the same `V_VEC_SIZE` of tokens at a time. As a result, a single thread retrieves multiple `v_vec`s from different rows and the same columns through multiple inner iterations. For each `v_vec`, it needs to be dot multiplied with the corresponding `logits_vec`, which is also `V_VEC_SIZE` elements from `logits`. Overall, with multiple inner iterations, each warp will process one block of value tokens. And with multiple outer iterations, the whole context value tokens are processed `[](#__codelineno-11-1)float accs[NUM_ROWS_PER_THREAD]; [](#__codelineno-11-2)for ... { // Iteration over different blocks. [](#__codelineno-11-3) logits_vec = ... [](#__codelineno-11-4) for ... { // Iteration over different rows. [](#__codelineno-11-5) v_vec = ... [](#__codelineno-11-6) ... [](#__codelineno-11-7) accs[i] += dot(logits_vec, v_vec); [](#__codelineno-11-8) } [](#__codelineno-11-9)}` As shown in the above pseudocode, in the outer loop, similar to `k_ptr`, `logits_vec` iterates over different blocks and reads `V_VEC_SIZE` elements from `logits`. In the inner loop, each thread reads `V_VEC_SIZE` elements from the same tokens as a `v_vec` and performs dot multiplication. It is important to note that in each inner iteration, the thread fetches different head position elements for the same tokens. The dot result is then accumulated in `accs`. Therefore, each entry of `accs` is mapped to a head position assigned to the current thread. For example, if `BLOCK_SIZE` is 16 and `V_VEC_SIZE` is 8, each thread fetches 8 value elements for 8 tokens at a time. Each element is from different tokens at the same head position. If `HEAD_SIZE` is 128 and `WARP_SIZE` is 32, for each inner loop, a warp needs to fetch `WARP_SIZE * V_VEC_SIZE = 256` elements. This means there are a total of 128 \* 16 / 256 = 8 inner iterations for a warp to handle a whole block of value tokens. And each `accs` in each thread contains 8 elements that accumulated at 8 different head positions. For the thread 0, the `accs` variable will have 8 elements, which are 0th, 32nd … 224th elements of a value head that are accumulated from all assigned 8 tokens. ## LV[¶](#lv "Permanent link") Now, we need to perform reduction for `accs` within each warp. This process allows each thread to accumulate the `accs` for the assigned head positions of all tokens in one block. `[](#__codelineno-12-1)for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) { [](#__codelineno-12-2) float acc = accs[i]; [](#__codelineno-12-3) for (int mask = NUM_V_VECS_PER_ROW / 2; mask >= 1; mask /= 2) { [](#__codelineno-12-4) acc += VLLM_SHFL_XOR_SYNC(acc, mask); [](#__codelineno-12-5) } [](#__codelineno-12-6) accs[i] = acc; [](#__codelineno-12-7)}` Next, we perform reduction for `accs` across all warps, allowing each thread to have the accumulation of `accs` for the assigned head positions of all context tokens. Please note that each `accs` in every thread only stores the accumulation for a portion of elements of the entire head for all context tokens. However, overall, all results for output have been calculated but are just stored in different thread register memory. Code `[](#__codelineno-13-1)float* out_smem = reinterpret_cast(shared_mem); [](#__codelineno-13-2)for (int i = NUM_WARPS; i > 1; i /= 2) { [](#__codelineno-13-3) // Upper warps write to shared memory. [](#__codelineno-13-4) ... [](#__codelineno-13-5) float* dst = &out_smem[(warp_idx - mid) * HEAD_SIZE]; [](#__codelineno-13-6) for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) { [](#__codelineno-13-7) ... [](#__codelineno-13-8) dst[row_idx] = accs[i]; [](#__codelineno-13-9) } [](#__codelineno-13-10) [](#__codelineno-13-11) // Lower warps update the output. [](#__codelineno-13-12) const float* src = &out_smem[warp_idx * HEAD_SIZE]; [](#__codelineno-13-13) for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) { [](#__codelineno-13-14) ... [](#__codelineno-13-15) accs[i] += src[row_idx]; [](#__codelineno-13-16) } [](#__codelineno-13-17) [](#__codelineno-13-18) // Write out the accs. [](#__codelineno-13-19)}` ## Output[¶](#output "Permanent link") Now we can write all of calculated result from local register memory to final output global memory. `[](#__codelineno-14-1)scalar_t* out_ptr = out + seq_idx * num_heads * max_num_partitions * HEAD_SIZE [](#__codelineno-14-2) + head_idx * max_num_partitions * HEAD_SIZE [](#__codelineno-14-3) + partition_idx * HEAD_SIZE;` First, we need to define the `out_ptr` variable, which points to the start address of the assigned sequence and assigned head. `[](#__codelineno-15-1)for (int i = 0; i < NUM_ROWS_PER_THREAD; i++) { [](#__codelineno-15-2) const int row_idx = lane / NUM_V_VECS_PER_ROW + i * NUM_ROWS_PER_ITER; [](#__codelineno-15-3) if (row_idx < HEAD_SIZE && lane % NUM_V_VECS_PER_ROW == 0) { [](#__codelineno-15-4) from_float(*(out_ptr + row_idx), accs[i]); [](#__codelineno-15-5) } [](#__codelineno-15-6)}` Finally, we need to iterate over different assigned head positions and write out the corresponding accumulated result based on the `out_ptr`. ## Citation[¶](#citation "Permanent link") `[](#__codelineno-16-1)@inproceedings{kwon2023efficient, [](#__codelineno-16-2) title={Efficient Memory Management for Large Language Model Serving with PagedAttention}, [](#__codelineno-16-3) author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica}, [](#__codelineno-16-4) booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles}, [](#__codelineno-16-5) year={2023} [](#__codelineno-16-6)}` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/plugin_system.md "Edit this page") The community frequently requests the ability to extend vLLM with custom features. To facilitate this, vLLM includes a plugin system that allows users to add custom features without modifying the vLLM codebase. This document explains how plugins work in vLLM and how to create a plugin for vLLM. ## How Plugins Work in vLLM[¶](#how-plugins-work-in-vllm "Permanent link") Plugins are user-registered code that vLLM executes. Given vLLM's architecture (see [Arch Overview](https://docs.vllm.ai/en/latest/arch_overview/)), multiple processes may be involved, especially when using distributed inference with various parallelism techniques. To enable plugins successfully, every process created by vLLM needs to load the plugin. This is done by the [load\_plugins\_by\_group](https://docs.vllm.ai/en/api/vllm/plugins/#vllm.plugins.load_plugins_by_group " load_plugins_by_group(group)") function in the `vllm.plugins` module. ## How vLLM Discovers Plugins[¶](#how-vllm-discovers-plugins "Permanent link") vLLM's plugin system uses the standard Python `entry_points` mechanism. This mechanism allows developers to register functions in their Python packages for use by other packages. An example of a plugin: Code ``[](#__codelineno-0-1)# inside `setup.py` file [](#__codelineno-0-2)from setuptools import setup [](#__codelineno-0-3)[](#__codelineno-0-4)setup(name='vllm_add_dummy_model', [](#__codelineno-0-5) version='0.1', [](#__codelineno-0-6) packages=['vllm_add_dummy_model'], [](#__codelineno-0-7) entry_points={ [](#__codelineno-0-8) 'vllm.general_plugins': [](#__codelineno-0-9) ["register_dummy_model = vllm_add_dummy_model:register"] [](#__codelineno-0-10) }) [](#__codelineno-0-11)[](#__codelineno-0-12)# inside `vllm_add_dummy_model/__init__.py` file [](#__codelineno-0-13)def register(): [](#__codelineno-0-14) from vllm import ModelRegistry [](#__codelineno-0-15) [](#__codelineno-0-16) if "MyLlava" not in ModelRegistry.get_supported_archs(): [](#__codelineno-0-17) ModelRegistry.register_model( [](#__codelineno-0-18) "MyLlava", [](#__codelineno-0-19) "vllm_add_dummy_model.my_llava:MyLlava", [](#__codelineno-0-20) )`` For more information on adding entry points to your package, please check the [official documentation](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). Every plugin has three parts: 1. **Plugin group**: The name of the entry point group. vLLM uses the entry point group `vllm.general_plugins` to register general plugins. This is the key of `entry_points` in the `setup.py` file. Always use `vllm.general_plugins` for vLLM's general plugins. 2. **Plugin name**: The name of the plugin. This is the value in the dictionary of the `entry_points` dictionary. In the example above, the plugin name is `register_dummy_model`. Plugins can be filtered by their names using the `VLLM_PLUGINS` environment variable. To load only a specific plugin, set `VLLM_PLUGINS` to the plugin name. 3. **Plugin value**: The fully qualified name of the function or module to register in the plugin system. In the example above, the plugin value is `vllm_add_dummy_model:register`, which refers to a function named `register` in the `vllm_add_dummy_model` module. ## Types of supported plugins[¶](#types-of-supported-plugins "Permanent link") - **General plugins** (with group name `vllm.general_plugins`): The primary use case for these plugins is to register custom, out-of-the-tree models into vLLM. This is done by calling `ModelRegistry.register_model` to register the model inside the plugin function. For an example of an official model plugin, see the [bart-plugin](https://github.com/vllm-project/bart-plugin) which adds support for `BartForConditionalGeneration`. - **Platform plugins** (with group name `vllm.platform_plugins`): The primary use case for these plugins is to register custom, out-of-the-tree platforms into vLLM. The plugin function should return `None` when the platform is not supported in the current environment, or the platform class's fully qualified name when the platform is supported. - **IO Processor plugins** (with group name `vllm.io_processor_plugins`): The primary use case for these plugins is to register custom pre-/post-processing of the model prompt and model output for pooling models. The plugin function returns the IOProcessor's class fully qualified name. - **Stat logger plugins** (with group name `vllm.stat_logger_plugins`): The primary use case for these plugins is to register custom, out-of-the-tree loggers into vLLM. The entry point should be a class that subclasses StatLoggerBase. ## Guidelines for Writing Plugins[¶](#guidelines-for-writing-plugins "Permanent link") - **Being re-entrant**: The function specified in the entry point should be re-entrant, meaning it can be called multiple times without causing issues. This is necessary because the function might be called multiple times in some processes. ### Platform plugins guidelines[¶](#platform-plugins-guidelines "Permanent link") 1. Create a platform plugin project, for example, `vllm_add_dummy_platform`. The project structure should look like this: `[](#__codelineno-1-1)vllm_add_dummy_platform/ [](#__codelineno-1-2)├── vllm_add_dummy_platform/ [](#__codelineno-1-3)│ ├── __init__.py [](#__codelineno-1-4)│ ├── my_dummy_platform.py [](#__codelineno-1-5)│ ├── my_dummy_worker.py [](#__codelineno-1-6)│ ├── my_dummy_attention.py [](#__codelineno-1-7)│ ├── my_dummy_device_communicator.py [](#__codelineno-1-8)│ ├── my_dummy_custom_ops.py [](#__codelineno-1-9)├── setup.py` 2. In the `setup.py` file, add the following entry point: `[](#__codelineno-2-1)setup( [](#__codelineno-2-2) name="vllm_add_dummy_platform", [](#__codelineno-2-3) ... [](#__codelineno-2-4) entry_points={ [](#__codelineno-2-5) "vllm.platform_plugins": [ [](#__codelineno-2-6) "my_dummy_platform = vllm_add_dummy_platform:register" [](#__codelineno-2-7) ] [](#__codelineno-2-8) }, [](#__codelineno-2-9) ... [](#__codelineno-2-10))` Please make sure `vllm_add_dummy_platform:register` is a callable function and returns the platform class's fully qualified name. for example: `[](#__codelineno-3-1)def register(): [](#__codelineno-3-2) return "vllm_add_dummy_platform.my_dummy_platform.MyDummyPlatform"` 3. Implement the platform class `MyDummyPlatform` in `my_dummy_platform.py`. The platform class should inherit from `vllm.platforms.interface.Platform`. Please follow the interface to implement the functions one by one. There are some important functions and properties that should be implemented at least: - `_enum`: This property is the device enumeration from [PlatformEnum](https://docs.vllm.ai/en/api/vllm/platforms/interface/#vllm.platforms.interface.PlatformEnum " PlatformEnum"). Usually, it should be `PlatformEnum.OOT`, which means the platform is out-of-tree. - `device_type`: This property should return the type of the device which pytorch uses. For example, `"cpu"`, `"cuda"`, etc. - `device_name`: This property is set the same as `device_type` usually. It's mainly used for logging purposes. - `check_and_update_config`: This function is called very early in the vLLM's initialization process. It's used for plugins to update the vllm configuration. For example, the block size, graph mode config, etc., can be updated in this function. The most important thing is that the **worker\_cls** should be set in this function to let vLLM know which worker class to use for the worker process. - `get_attn_backend_cls`: This function should return the attention backend class's fully qualified name. - `get_device_communicator_cls`: This function should return the device communicator class's fully qualified name. 4. Implement the worker class `MyDummyWorker` in `my_dummy_worker.py`. The worker class should inherit from [WorkerBase](https://docs.vllm.ai/en/api/vllm/v1/worker/worker_base/#vllm.v1.worker.worker_base.WorkerBase " WorkerBase"). Please follow the interface to implement the functions one by one. Basically, all interfaces in the base class should be implemented, since they are called here and there in vLLM. To make sure a model can be executed, the basic functions should be implemented are: - `init_device`: This function is called to set up the device for the worker. - `initialize_cache`: This function is called to set cache config for the worker. - `load_model`: This function is called to load the model weights to device. - `get_kv_cache_spec`: This function is called to generate the kv cache spec for the model. - `determine_available_memory`: This function is called to profiles the peak memory usage of the model to determine how much memory can be used for KV cache without OOMs. - `initialize_from_config`: This function is called to allocate device KV cache with the specified kv\_cache\_config - `execute_model`: This function is called every step to inference the model. Additional functions that can be implemented are: - If the plugin wants to support sleep mode feature, please implement the `sleep` and `wakeup` functions. - If the plugin wants to support graph mode feature, please implement the `compile_or_warm_up_model` function. - If the plugin wants to support speculative decoding feature, please implement the `take_draft_token_ids` function. - If the plugin wants to support lora feature, please implement the `add_lora`,`remove_lora`,`list_loras` and `pin_lora` functions. - If the plugin wants to support data parallelism feature, please implement the `execute_dummy_batch` functions. Please look at the worker base class [WorkerBase](https://docs.vllm.ai/en/api/vllm/v1/worker/worker_base/#vllm.v1.worker.worker_base.WorkerBase " WorkerBase") for more functions that can be implemented. 5. Implement the attention backend class `MyDummyAttention` in `my_dummy_attention.py`. The attention backend class should inherit from [AttentionBackend](https://docs.vllm.ai/en/api/vllm/v1/attention/backend/#vllm.v1.attention.backend.AttentionBackend " AttentionBackend"). It's used to calculate attentions with your device. Take `vllm.v1.attention.backends` as examples, it contains many attention backend implementations. 6. Implement custom ops for high performance. Most ops can be run by pytorch native implementation, while the performance may not be good. In this case, you can implement specific custom ops for your plugins. Currently, there are kinds of custom ops vLLM supports: - pytorch ops there are 3 kinds of pytorch ops: - `communicator ops`: Device communicator op. Such as all-reduce, all-gather, etc. Please implement the device communicator class `MyDummyDeviceCommunicator` in `my_dummy_device_communicator.py`. The device communicator class should inherit from [DeviceCommunicatorBase](https://docs.vllm.ai/en/api/vllm/distributed/device_communicators/base_device_communicator/#vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase " DeviceCommunicatorBase"). - `common ops`: Common ops. Such as matmul, softmax, etc. Please implement the common ops by register oot way. See more detail in [CustomOp](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") class. - `csrc ops`: C++ ops. This kind of ops are implemented in C++ and are registered as torch custom ops. Following csrc module and `vllm._custom_ops` to implement your ops. - triton ops Custom way doesn't work for triton ops now. 7. (optional) Implement other pluggable modules, such as lora, graph backend, quantization, mamba attention backend, etc. ## Compatibility Guarantee[¶](#compatibility-guarantee "Permanent link") vLLM guarantees the interface of documented plugins, such as `ModelRegistry.register_model`, will always be available for plugins to register models. However, it is the responsibility of plugin developers to ensure their plugins are compatible with the version of vLLM they are targeting. For example, `"vllm_add_dummy_model.my_llava:MyLlava"` should be compatible with the version of vLLM that the plugin targets. The interface for the model/module may change during vLLM's development. If you see any deprecation log info, please upgrade your plugin to the latest version. ## Deprecation announcement[¶](#deprecation-announcement "Permanent link") Deprecations - `use_v1` parameter in `Platform.get_attn_backend_cls` is deprecated. It has been removed in v0.13.0. - `_Backend` in `vllm.attention` is deprecated. It has been removed in v0.13.0. Please use `vllm.v1.attention.backends.registry.register_backend` to add new attention backend to [`AttentionBackendEnum`](https://docs.vllm.ai/en/api/vllm/v1/attention/backends/registry/#vllm.v1.attention.backends.registry.AttentionBackendEnum " AttentionBackendEnum") instead. - `seed_everything` platform interface is deprecated. It has been removed in v0.16.0. Please use `vllm.utils.torch_utils.set_random_seed` instead. - `prompt` in `Platform.validate_request` is deprecated. It has been removed in v0.18.0. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/prefix_caching.md "Edit this page") Prefix caching kv-cache blocks is a popular optimization in LLM inference to avoid redundant prompt computations. The core idea is simple – we cache the kv-cache blocks of processed requests, and reuse these blocks when a new request comes in with the same prefix as previous requests. Since prefix caching is almost a free lunch and won’t change model outputs, it has been widely used by many public endpoints (e.g., OpenAI, Anthropic, etc.) and most open source LLM inference frameworks (e.g., SGLang). While there are many ways to implement prefix caching, vLLM chooses a hash-based approach. Specifically, we hash each kv-cache block by the tokens in the block and the tokens in the prefix before the block: `[](#__codelineno-0-1) Block 1 Block 2 Block 3 [](#__codelineno-0-2) [A gentle breeze stirred] [the leaves as children] [laughed in the distance] [](#__codelineno-0-3)Block 1: |<--- block tokens ---->| [](#__codelineno-0-4)Block 2: |<------- prefix ------>| |<--- block tokens --->| [](#__codelineno-0-5)Block 3: |<------------------ prefix -------------------->| |<--- block tokens ---->|` In the example above, the KV cache in the first block can be uniquely identified with the token “A gentle breeze stirred”. The third block can be uniquely identified with the tokens in the block “laughed in the distance”, along with the prefix tokens “A gentle breeze stirred the leaves as children”. Therefore, we can build the block hash of `hash(tuple[components])`, where components are: - Parent hash value: The hash value of the parent hash block. - Block tokens: A tuple of tokens in this block. The reason to include the exact tokens is to reduce potential hash value collision. - Extra hashes: Other values required to make this block unique, such as LoRA IDs, multi-modality input hashes (see the example below), and cache salts to isolate caches in multi-tenant environments. Note 1 We only cache full blocks. Note 2 In previous versions, the hash key was not guaranteed to be collision-free. As of v0.11, the default hashing algorithm is `sha256`, which addresses collision risks. For `vllm serve`, you can control the hashing algorithm via `--prefix-caching-hash-algo`: - `sha256` (default): Uses Python's `pickle` for serialization. Hashes may not be reproducible across different Python or vLLM versions. - `sha256_cbor`: Uses `cbor2` for serialization, providing a reproducible, cross-language compatible hash. This is recommended for deterministic caching across environments. - `xxhash`: `Uses Pickle serialization with xxHash (128-bit) for faster, non-cryptographic hashing. Requires the optional`xxhash`package. IMPORTANT: Use of a hashing algorithm that is not considered cryptographically secure theoretically increases the risk of hash collisions, which can cause undefined behavior or even leak private information in multi-tenant environments. Even if collisions are still very unlikely, it is important to consider your security risk tolerance against the performance benefits before turning this on. -`xxhash\_cbor`combines canonical CBOR serialization with xxHash for reproducible hashing. Requires the optional`xxhash\` package. **A hashing example with multi-modality inputs** In this example, we illustrate how prefix caching works with multi-modality inputs (e.g., images). Assuming we have a request with the following messages: `[](#__codelineno-1-1)messages = [ [](#__codelineno-1-2) {"role": "user", [](#__codelineno-1-3) "content": [ [](#__codelineno-1-4) {"type": "text", [](#__codelineno-1-5) "text": "What's in this image?" [](#__codelineno-1-6) }, [](#__codelineno-1-7) {"type": "image_url", [](#__codelineno-1-8) "image_url": {"url": image_url}, [](#__codelineno-1-9) }, [](#__codelineno-1-10) ]}, [](#__codelineno-1-11)]` It will become the following prompt: `[](#__codelineno-2-1)Prompt: [](#__codelineno-2-2) [INST]What's in this image?\n[IMG][/INST] [](#__codelineno-2-3)[](#__codelineno-2-4)Tokenized prompt: [](#__codelineno-2-5) [1, 3, 7493, 1681, 1294, 1593, 3937, 9551, 10, 4] [](#__codelineno-2-6)[](#__codelineno-2-7)Prompt with placeholders (

): [](#__codelineno-2-8) [1, 3, 7493, 1681, 1294, 1593, 3937, 9551,

,

, ...,

, 4]` As we can see, after the tokenization, the `[IMG]` will be replaced by a sequence of placeholder tokens, and these placeholders will be replaced by image embeddings during prefill. The challenge for prefix caching to support this case is we need to differentiate images from the placeholders. To address this problem, we encode the image hash generated by the frontend image processor. For example, the hash of the blocks in the above prompt would be (assuming block size 16, and we have 41 placeholder tokens): `[](#__codelineno-3-1)Block 0 [](#__codelineno-3-2) Parent hash: None [](#__codelineno-3-3) Token IDs: 1, 3, 7493, 1681, 1294, 1593, 3937, 9551,

, ...,

[](#__codelineno-3-4) Extra hash: [](#__codelineno-3-5)Block 1 [](#__codelineno-3-6) Parent hash: Block 0 hash [](#__codelineno-3-7) Token IDs:

, ...,

[](#__codelineno-3-8) Extra hash: [](#__codelineno-3-9)Block 2 [](#__codelineno-3-10) Parent hash: Block 1 hash [](#__codelineno-3-11) Token IDs:

, ...,

[](#__codelineno-3-12) Extra hash: [](#__codelineno-3-13)Block 3 [](#__codelineno-3-14) Parent hash: Block 2 hash [](#__codelineno-3-15) Token IDs:

, ...,

, 4 [](#__codelineno-3-16) Extra hash: ` In the rest of this document, we first introduce the data structure used for prefix caching in vLLM v1, followed by the prefix caching workflow of major KV cache operators (e.g., allocate, append, free, eviction). Finally, we use an example to illustrate the end to end prefix caching workflow. **Cache Isolation for Security** To improve privacy in shared environments, vLLM supports isolating prefix cache reuse through optional per-request salting. By including a `cache_salt` in the request, this value is injected into the hash of the first block, ensuring that only requests with the same salt can reuse cached KV blocks. This prevents timing-based attacks where an adversary could infer cached content by observing latency differences. This offers protection without compromising performance. `[](#__codelineno-4-1){ [](#__codelineno-4-2) "messages": [ [](#__codelineno-4-3) {"role": "system", "content": "You are a helpful assistant."}, [](#__codelineno-4-4) {"role": "user", "content": "Here is a document with details about the world series: ..."}, [](#__codelineno-4-5) {"role": "user", "content": "Who won the world series in 2020?"} [](#__codelineno-4-6) ], [](#__codelineno-4-7) "cache_salt": "your-cache-salt" [](#__codelineno-4-8)}` With this setup, cache sharing is limited to users or requests that explicitly agree on a common salt, enabling cache reuse within a trust group while isolating others. ## Data Structure[¶](#data-structure "Permanent link") The prefix caching in vLLM v1 is implemented in the KV cache manager. The basic building block is the “Block” data class (simplified): `[](#__codelineno-5-1)class KVCacheBlock: [](#__codelineno-5-2) # The block ID (immutable) [](#__codelineno-5-3) block_id: int [](#__codelineno-5-4) # The block hash (will be assigned when the block is full, [](#__codelineno-5-5) # and will be reset when the block is evicted). [](#__codelineno-5-6) block_hash: BlockHash [](#__codelineno-5-7) # The number of requests using this block now. [](#__codelineno-5-8) ref_cnt: int [](#__codelineno-5-9) [](#__codelineno-5-10) # The pointers to form a doubly linked list for the free queue. [](#__codelineno-5-11) prev_free_block: "KVCacheBlock | None" = None [](#__codelineno-5-12) next_free_block: "KVCacheBlock | None" = None` There are two design points to highlight: 1. We allocate all KVCacheBlock when initializing the KV cache manager to be a block pool. This avoids Python object creation overheads and can easily track all blocks all the time. 2. We introduce doubly linked list pointers directly in the KVCacheBlock, so that we could construct a free queue directly. This gives us two benefits: 1. We could have O(1) complexity moving elements in the middle to the tail. 2. We could avoid introducing another Python queue (e.g., `deque`) which has a wrapper to the elements. As a result, we will have the following components when the KV cache manager is initialized: [![Component Overview](https://docs.vllm.ai/en/assets/design/prefix_caching/overview.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/overview.png) - Block Pool: A list of KVCacheBlock. - Free Block Queue: Only store the pointers of head and tail blocks for manipulations. - Cache blocks: Mapping from hash key to block IDs. - Request blocks: Mapping from request ID to allocated block IDs. ## Operations[¶](#operations "Permanent link") ### Block Allocation[¶](#block-allocation "Permanent link") **New request:** Workflow for the scheduler to schedule a new request with KV cache block allocation: 1. The scheduler calls `kv_cache_manager.get_computed_blocks()` to get a sequence of blocks that have already been computed. This is done by hashing the prompt tokens in the request and looking up cache blocks. 2. The scheduler calls `kv_cache_manager.allocate_slots()`. It does the following steps: 1. Compute the number of new required blocks, and return if there are no sufficient blocks to allocate. 2. “Touch” the computed blocks. It increases the reference count of the computed block by one, and removes the block from the free queue if the block wasn’t used by other requests. This is to avoid these computed blocks being evicted. See the example in the next section for illustration. 3. Allocate new blocks by popping the heads of the free queue. If the head block is a cached block, this also “evicts” the block so that no other requests can reuse it anymore from now on. 4. If an allocated block is already full of tokens, we immediately add it to the cache block, so that the block can be reused by other requests in the same batch. **Running request:** Workflow for the scheduler to schedule a running request with KV cache block allocation: 1. The scheduler calls `kv_cache_manager.allocate_slots()`. It does the following steps: 1. Compute the number of new required blocks, and return if there are no sufficient blocks to allocate. 2. Allocate new blocks by popping the heads of the free queue. If the head block is a cached block, this also “evicts” the block so that no other requests can reuse it anymore from now on. 3. Append token IDs to the slots in existing blocks as well as the new blocks. If a block is full, we add it to the cache block to cache it. **Duplicated blocks** Assuming block size is 4 and you send a request (Request 1) with prompt ABCDEF and decoding length 3: `[](#__codelineno-6-1)Prompt: [A, B, C, D, E, F] [](#__codelineno-6-2)Output: [G, H, I] [](#__codelineno-6-3)[](#__codelineno-6-4)Time 0: [](#__codelineno-6-5) Tokens: [A, B, C, D, E, F, G] [](#__codelineno-6-6) Block Table: [0 (ABCD), 1 (EFG)] [](#__codelineno-6-7) Cache Blocks: 0 [](#__codelineno-6-8)Time 1: [](#__codelineno-6-9) Tokens: [A, B, C, D, E, F, G, H] [](#__codelineno-6-10) Block Table: [0 (ABCD), 1 (EFGH)] [](#__codelineno-6-11) Cache Blocks: 0, 1 [](#__codelineno-6-12)Time 2: [](#__codelineno-6-13) Tokens: [A, B, C, D, E, F, G, H, I] [](#__codelineno-6-14) Block Table: [0 (ABCD), 1 (EFGH), 2 (I)] [](#__codelineno-6-15) Cache Blocks: 0, 1` Now block 0 and block 1 are cached, and we send the same request again (Request 2) with greedy sampling, so that it will produce exactly the same outputs as the Request 1: `[](#__codelineno-7-1)Prompt: [A, B, C, D, E, F] [](#__codelineno-7-2)Output: [G, H, I] [](#__codelineno-7-3)[](#__codelineno-7-4)Time 0: [](#__codelineno-7-5) Tokens: [A, B, C, D, E, F, G] [](#__codelineno-7-6) Block Table: [0 (ABCD), 3 (EFG)] [](#__codelineno-7-7) Cache Blocks: 0, 1 [](#__codelineno-7-8)Time 1: [](#__codelineno-7-9) Tokens: [A, B, C, D, E, F, G, H] [](#__codelineno-7-10) Block Table: [0 (ABCD), 3 (EFGH)] [](#__codelineno-7-11) Cache Blocks: 0, 1, 3` As can be seen, block 3 is a new full block and is cached. However, it is redundant as block 1, meaning that we cached the same block twice. In v0, when detecting block 3 is duplicated, we free block 3 and let Request 2 use block 1 instead, so its block table becomes `[0, 1]` in Time 1. However, the block table in vLLM v1 is append-only, meaning that changing the block table from `[0, 3]` to `[0, 1]` is not allowed. As a result, we will have duplicated blocks for the hash key E-H. This duplication will be eliminated when the request is freed. ### Free[¶](#free "Permanent link") When a request is finished, we free all its blocks if no other requests are using them (reference count = 0). In this example, we free request 1 and block 2, 3, 4, 8 associated with it. We can see that the freed blocks are added to the tail of the free queue in the _reverse_ order. This is because the last block of a request must hash more tokens and is less likely to be reused by other requests. As a result, it should be evicted first. [![Free queue after a request us freed](https://docs.vllm.ai/en/assets/design/prefix_caching/free.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/free.png) ### Eviction (LRU)[¶](#eviction-lru "Permanent link") When the head block (least recently used block) of the free queue is cached, we have to evict the block to prevent it from being used by other requests. Specifically, eviction involves the following steps: 1. Pop the block from the head of the free queue. This is the LRU block to be evicted. 2. Remove the block ID from the cache block. 3. Remove the block hash. ## Example[¶](#example "Permanent link") In this example, we assume the block size is 4 (each block can cache 4 tokens), and we have 10 blocks in the KV-cache manager in total. **Time 1: The cache is empty and a new request comes in.** We allocate 4 blocks. 3 of them are already full and cached. The fourth block is partially full with 3 of 4 tokens. [![Example Time 1](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-1.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-1.png) **Time 2: Request 0 makes the block 3 full and asks for a new block to keep decoding.** We cache block 3 and allocate block 4. [![Example Time 2](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-3.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-3.png) **Time 3: Request 1 comes in with the 14 prompt tokens, where the first 10 tokens are the same as request 0.** We can see that only the first 2 blocks (8 tokens) hit the cache, because the 3rd block only matches 2 of 4 tokens. [![Example Time 3](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-4.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-4.png) **Time 4: Request 0 is finished and free.** Blocks 2, 3 and 4 are added to the free queue in the reverse order (but block 2 and 3 are still cached). Block 0 and 1 are not added to the free queue because they are being used by Request 1. [![Example Time 4](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-5.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-5.png) **Time 5: Request 1 is finished and free.** [![Example Time 5](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-6.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-6.png) **Time 6: Request 2 comes in with the 29 prompt tokens, where the first 12 tokens are the same as request 0.** Note that even the block order in the free queue was `7 - 8 - 9 - 4 - 3 - 2 - 6 - 5 - 1 - 0`, the cache hit blocks (i.e., 0, 1, 2) are touched and removed from the queue before allocation, so the free queue becomes `7 - 8 - 9 - 4 - 3 - 6 - 5`. As a result, the allocated blocks are 0 (cached), 1 (cached), 2 (cached), 7, 8, 9, 4, 3 (evicted). [![Example Time 6](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-7.png)](https://docs.vllm.ai/en/assets/design/prefix_caching/example-time-7.png) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/torch_compile.md "Edit this page") In vLLM's V1 architecture, `torch.compile` is enabled by default and is a critical part of the framework. This document gives a simple walk-through example to show how to understand the `torch.compile` usage. Throughout the example, we will run a common Llama model, and turn on debug level logging to show all the details. The command to be used is `VLLM_LOGGING_LEVEL=DEBUG vllm serve meta-llama/Llama-3.2-1B`. Note For more information and the latest progress of `torch.compile` integration, see this [Blog Post](https://blog.vllm.ai/2025/08/20/torch-compile.html). ## Compilation Cache[¶](#compilation-cache "Permanent link") In the very verbose logs, we can see: `[](#__codelineno-0-1)INFO 03-07 03:06:55 [backends.py:409] Using cache directory: ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0 for vLLM's torch.compile` vLLM will take all the available factors into consideration, and decide a directory to store all the compilation artifact. This means, you can directly copy the whole `~/.cache/vllm/torch_compile_cache` directory in your deployment scenario to save a great amount of compilation time, and hence accelerating the starting time of the vLLM instance. The factors considered include: - All the related configs (see the `compute_hash` functions in their respective configs in the [config folder](https://github.com/vllm-project/vllm/tree/main/vllm/config)) - PyTorch configs (see the `compute_hash` functions in the [compiler\_interface.py](https://github.com/vllm-project/vllm/blob/main/vllm/compilation/compiler_interface.py)) - The model's forward function and the relevant functions called by the forward function (see below) With all these factors taken into consideration, usually we can guarantee that the cache is safe to use, and will not cause any unexpected behavior. Therefore, the cache is enabled by default. If you want to debug the compilation process, or if you suspect the cache is causing some issues, you can disable it by setting the environment variable `VLLM_DISABLE_COMPILE_CACHE=1`. A unique aspect of vLLM's `torch.compile` integration, is that we guarantee all the compilation finishes before we serve any requests. No requests will trigger new compilations. Otherwise, the engine would be blocked on that request, and the response time will have unexpected spikes. By default, the cache saves compiled artifacts as binary files. If you would like to interact with the generated code for debugging purposes, set the field `compile_cache_save_format=unpacked` in the compilation config, or omit this and set the env variable `VLLM_COMPILE_CACHE_SAVE_FORMAT=unpacked`. ## Dynamic shapes and vllm guard dropping[¶](#dynamic-shapes-and-vllm-guard-dropping "Permanent link") `torch.compile` is designed to guard on dynamic shapes with no hesitation when needed. This contradicts with vLLM's `torch.compile` approach of dropping the guards since many of those guards could be material. `torch.compile` provides two kinds of dynamic shapes: `backed` and `unbacked`. `torch.compile` guards on `backed` dynamic shapes and does not provide a guarantee that no guards will be added to them. User code, dynamo, inductor, and autograd all can add guards. Moreover, for 0/1 specializations, backed symbols are specialized unconditionally to 0, 1, or >=2 even without encountering a branching on those ranges. On the contrary, `unbacked` dynamic shapes are guaranteed not to be guarded on and are not 0/1 specialized. However, there is a possibility of throwing a data dependent error when a branch that requires their value is encountered and no explicit unbacked handling is defined. The framework is converging to a state where it won't throw DDE but rather pick general paths. One downside of using unbacked is missed optimization opportunities due to either perf bugs or picking general paths, also using a fixed non-example input-based hint (this will be fixed soon with override\_hint API). An example of picking general paths is assuming input not contiguous in functions call contiguous() and reshape() when can't be symbolically proven with a change of introducing a clone. `backed_size_oblivious` is a flag that enables treating backed symbols as unbacked wherever explicit handling for unbacked is defined. With this mode, 0/1 specializations are mostly avoided in framework code and the default 0/1 specialization does not happen. However, there is still no guarantee that torch.compile won't guard, especially due to user code or custom passes. `backed_size_oblivious` is experimental in PyTorch compile and could be deprecated. That said, it's a safer option to use than `backed` and the probability of reducing performance is lower than `unbacked`. ### Configuring Dynamic Shapes[¶](#configuring-dynamic-shapes "Permanent link") The [`DynamicShapesConfig`](https://docs.vllm.ai/en/api/vllm/config/compilation/#vllm.config.compilation.DynamicShapesConfig " DynamicShapesConfig") allows you to control the dynamic shapes behavior by setting the `type` field. You can choose between three modes: `BACKED`(default), `UNBACKED` , and `BACKED_SIZE_OBLIVIOUS`. #### Offline Inference Example (Using LLM class)[¶](#offline-inference-example-using-llm-class "Permanent link") When using the [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class for offline inference, you can configure dynamic shapes through the `compilation_config` parameter: `[](#__codelineno-1-1)from vllm import LLM, SamplingParams [](#__codelineno-1-2)from vllm.config.compilation import CompilationConfig, DynamicShapesConfig, DynamicShapesType [](#__codelineno-1-3)[](#__codelineno-1-4)# Example: Using backed_size_oblivious (experimental, safer than backed) [](#__codelineno-1-5)llm = LLM( [](#__codelineno-1-6) model="meta-llama/Llama-3.2-1B", [](#__codelineno-1-7) compilation_config=CompilationConfig( [](#__codelineno-1-8) dynamic_shapes_config=DynamicShapesConfig( [](#__codelineno-1-9) type=DynamicShapesType.BACKED_SIZE_OBLIVIOUS [](#__codelineno-1-10) ) [](#__codelineno-1-11) ) [](#__codelineno-1-12)) [](#__codelineno-1-13)[](#__codelineno-1-14)# Example: Using unbacked (strongest guarantee against guards) [](#__codelineno-1-15)llm = LLM( [](#__codelineno-1-16) model="meta-llama/Llama-3.2-1B", [](#__codelineno-1-17) compilation_config=CompilationConfig( [](#__codelineno-1-18) dynamic_shapes_config=DynamicShapesConfig( [](#__codelineno-1-19) type=DynamicShapesType.UNBACKED [](#__codelineno-1-20) ) [](#__codelineno-1-21) ) [](#__codelineno-1-22)) [](#__codelineno-1-23)[](#__codelineno-1-24)# Generate outputs [](#__codelineno-1-25)prompts = ["Hello, my name is", "The future of AI is"] [](#__codelineno-1-26)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-1-27)outputs = llm.generate(prompts, sampling_params)` #### Online Serving Example (Using vllm serve)[¶](#online-serving-example-using-vllm-serve "Permanent link") When using `vllm serve` for online serving, you can configure dynamic shapes through the `--compilation-config` flag: `[](#__codelineno-2-1)# Example: Using unbacked [](#__codelineno-2-2)vllm serve meta-llama/Llama-3.2-1B \ [](#__codelineno-2-3) --compilation-config '{"dynamic_shapes_config": {"type": "unbacked"}}' [](#__codelineno-2-4) [](#__codelineno-2-5)[](#__codelineno-2-6)# Alternative: Using dot notation (simpler for single values) [](#__codelineno-2-7)vllm serve meta-llama/Llama-3.2-1B -cc.dynamic_shapes_config.type=unbacked` #### Choosing the Right Mode[¶](#choosing-the-right-mode "Permanent link") - **BACKED** (default): Use when you're willing to accept potential unsafe dropping of guards for maximal performance. Guard could be unsoundly added and then ignored. - **UNBACKED** Use when you need the strongest guarantee against guards. This is the most conservative option but may miss some optimization opportunities. - **BACKED\_SIZE\_OBLIVIOUS**: Use when you want a balance between avoiding guards and performance. This experimental mode is safer than BACKED but still not as conservative as UNBACKED. ## Python Code Compilation[¶](#python-code-compilation "Permanent link") In the very verbose logs, we can see: Logs `[](#__codelineno-3-1)DEBUG 03-07 03:06:52 [decorators.py:203] Start compiling function [](#__codelineno-3-2)[](#__codelineno-3-3)DEBUG 03-07 03:06:54 [backends.py:370] Traced files (to be considered for compilation cache): [](#__codelineno-3-4)DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/_dynamo/polyfills/builtins.py [](#__codelineno-3-5)DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/nn/modules/container.py [](#__codelineno-3-6)DEBUG 03-07 03:06:54 [backends.py:370] xxx/torch/nn/modules/module.py [](#__codelineno-3-7)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/attention/layer.py [](#__codelineno-3-8)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/distributed/communication_op.py [](#__codelineno-3-9)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/distributed/parallel_state.py [](#__codelineno-3-10)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/custom_op.py [](#__codelineno-3-11)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/activation.py [](#__codelineno-3-12)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/layernorm.py [](#__codelineno-3-13)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/linear.py [](#__codelineno-3-14)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/rotary_embedding.py [](#__codelineno-3-15)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/layers/vocab_parallel_embedding.py [](#__codelineno-3-16)DEBUG 03-07 03:06:54 [backends.py:370] xxx/vllm/model_executor/models/llama.py [](#__codelineno-3-17)[](#__codelineno-3-18)DEBUG 03-07 03:07:07 [backends.py:462] Computation graph saved to ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/computation_graph.py [](#__codelineno-3-19)DEBUG 03-07 03:07:07 [wrapper.py:105] Dynamo transformed code saved to ~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/transformed_code.py` This is about the Python code compilation, i.e. graph capture by Dynamo. It tries to trace the function with code `xxx/vllm/model_executor/models/llama.py:339`, which is the `forward` function of the model we compile. During the forward pass, there are also other functions called and inlined by Dynamo, as shown by the logs, including some PyTorch functions from `xxx/torch/nn/modules/module.py` (used by PyTorch `nn.Module`, because module attribute access will trigger a function call), some communication / attention / activation functions from vLLM. All the traced files will be considered when we decide the cache directory to use. This way, any code change in the above files will trigger compilation cache miss, and therefore recompilation. The result of the Dynamo compilation, is a new function stored in `~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/transformed_code.py`. Usually, this function unpacks tensors from the module, and then pass it to the traced computation graph. The computation graph is stored in `~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/computation_graph.py`. ## Computation Graph Processing[¶](#computation-graph-processing "Permanent link") The computation graph has shape annotations for every tensor. The inputs are input ids, position ids, weights and buffers from the model, and the outputs are the final hidden states. Note that lm head projection and sampling operations are not considered in the graph. Most of the inputs to the computation graph has static shape, since they are model weights and buffers, and will not change during the lifetime of the model. Only the input ids and position ids have symbolic shapes, i.e. the shape can change from batch to batch. However, they will share the same symbolic shapes. That is to say, the only changing size to the computation graph, is the batch size (number of tokens processed in the current forward pass). The attention operation is complicated, and it needs to interact with kv caches, with complicated shapes. Fortunately, the output of the attention operation just share the same shape as the input query of the attention operation. Therefore, we wrap the whole attention operation into a PyTorch custom op `torch.ops.vllm.unified_attention_with_output`, so that Dynamo will not try to inspect any of the internal operations. This way, although attention operation is complicated, we can still capture the model's computation graph as a full-graph, from Dynamo's perspective. The computation graph is further split into pieces, by the `splitting_ops` (usually this is the attention operation). Therefore, in the `~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/computation_graph.py` file, we can see lots of submodules, each submodule is a piece of graph after splitting: - Attention operation itself is a submodule. - The part of computation graph, from one attention operation to the next attention operation, is a submodule. Every submodule can be identified by its index, and will be processed individually. ## Computation Graph Compilation[¶](#computation-graph-compilation "Permanent link") In the very verbose logs, we can also see: `[](#__codelineno-4-1)DEBUG 03-07 03:52:37 [backends.py:134] store the 0-th graph for shape None from inductor via handle ('fpegyiq3v3wzjzphd45wkflpabggdbjpylgr7tta4hj6uplstsiw', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/iw/ciwzrk3ittdqatuzwonnajywvno3llvjcs2vfdldzwzozn3zi3iy.py') [](#__codelineno-4-2)DEBUG 03-07 03:52:39 [backends.py:134] store the 1-th graph for shape None from inductor via handle ('f7fmlodmf3h3by5iiu2c4zarwoxbg4eytwr3ujdd2jphl4pospfd', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/ly/clyfzxldfsj7ehaluis2mca2omqka4r7mgcedlf6xfjh645nw6k2.py') [](#__codelineno-4-3)... [](#__codelineno-4-4)DEBUG 03-07 03:52:45 [backends.py:134] store the 15-th graph for shape None from inductor via handle ('f7fmlodmf3h3by5iiu2c4zarwoxbg4eytwr3ujdd2jphl4pospfd', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/ly/clyfzxldfsj7ehaluis2mca2omqka4r7mgcedlf6xfjh645nw6k2.py') [](#__codelineno-4-5)DEBUG 03-07 03:52:45 [backends.py:134] store the 16-th graph for shape None from inductor via handle ('fvj3ccoi7m34f3dnr4itmu55mmun44l5xymwhrjlwisylsk7q6jy', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/tf/ctfftkglj7b4lcttq5cymx6cew372uoauupqn6ldsvpiucavqcjc.py')` This means the first piece of computation graph (with shape `None` for symbolic shape) is compiled by Inductor (with a key `fpegyiq3v3wzjzphd45wkflpabggdbjpylgr7tta4hj6uplstsiw`). The compiled kernel is stored in `~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/iw/ciwzrk3ittdqatuzwonnajywvno3llvjcs2vfdldzwzozn3zi3iy.py`. You can open the file to see what is the code Inductor finally runs. One more detail: you can see that the 1-th graph and the 15-th graph have the same key, while the 0-th graph and the 16-th graph are different. This is expected, since we split the graph by the attention op, we get 3 unique subgraphs: - the first layer before attention - every middle layer, from one attention operation to the next attention operation - the final layer after attention If we already have the cache directory (e.g. run the same code for the second time), we will see the following logs: `[](#__codelineno-5-1)DEBUG 03-07 04:00:45 [backends.py:86] Directly load the 0-th graph for shape None from inductor via handle ('fpegyiq3v3wzjzphd45wkflpabggdbjpylgr7tta4hj6uplstsiw', '~/.cache/vllm/torch_compile_cache/1517964802/rank_0_0/inductor_cache/iw/ciwzrk3ittdqatuzwonnajywvno3llvjcs2vfdldzwzozn3zi3iy.py')` This time, Inductor compilation is completely bypassed, and we will load from disk to read the compilation artifact we get from the last time. The above example just uses Inductor to compile for a general shape (i.e. symbolic shape). We can also use Inductor to compile for some of the specific shapes, for example: `[](#__codelineno-6-1)vllm serve meta-llama/Llama-3.2-1B \ [](#__codelineno-6-2) --compilation_config '{"compile_sizes": [1, 2, 4, 8]}'` Then it will also compile a specific kernel just for batch size `1, 2, 4, 8`. At this time, all of the shapes in the computation graph are static and known, and we will turn on auto-tuning to tune for max performance. This can be slow when you run it for the first time, but the next time you run it, we can directly bypass the tuning and run the tuned kernel. When all the shapes are known, `torch.compile` can compare different configs, and often find some better configs to run the kernel. For example, we can see the following log: Logs `[](#__codelineno-7-1)AUTOTUNE mm(8x2048, 2048x3072) [](#__codelineno-7-2) triton_mm_4 0.0130 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=2 [](#__codelineno-7-3) triton_mm_8 0.0134 ms 97.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4 [](#__codelineno-7-4) triton_mm_12 0.0148 ms 87.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=4, num_warps=4 [](#__codelineno-7-5) mm 0.0160 ms 81.6% [](#__codelineno-7-6) triton_mm_16 0.0165 ms 78.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=8 [](#__codelineno-7-7) triton_mm_3 0.0199 ms 65.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=2 [](#__codelineno-7-8) triton_mm_1 0.0203 ms 64.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=16, BLOCK_N=32, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=2, num_warps=2 [](#__codelineno-7-9) triton_mm_7 0.0203 ms 64.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4 [](#__codelineno-7-10) triton_mm_2 0.0208 ms 62.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=16, BLOCK_N=64, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=5, num_warps=4 [](#__codelineno-7-11) triton_mm_11 0.0215 ms 60.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=16, BLOCK_N=128, B_PROLOGUE_CAST_TYPE=None, EVEN_K=True, GROUP_M=8, num_stages=3, num_warps=4 [](#__codelineno-7-12)SingleProcess AUTOTUNE benchmarking takes 2.0428 seconds and 7.5727 seconds precompiling` It means, for a matrix multiplication with shape `8x2048x3072`, `torch.compile` tries triton template with various configs, and it is much faster than the default code (which dispatches to cublas library). Unfortunately, because auto-tuning takes quite a long time (from seconds to minutes, depending on the model size and the batch size), even though it can be cached for later use, for the sake of user-friendliness, we turn it off by default. If you want to have max performance, it is recommended to try it, by compiling specific shapes. ## Cudagraph Capture[¶](#cudagraph-capture "Permanent link") vLLM's V1 architecture uses piecewise cudagraph that aligns with the piecewise compilation. The full computation graph is split as mentioned above, and we only capture the cudagraph for the piece of graph between attention operations (including the first graph before any attention operation, and the last graph after all the attention operation). This is based on a common observation: computation between attentions are usually token-wise and easy to deal with for cudagraph; while the attention operation is non-trivial to be cudagraph compatible. Thus, by running the attention operation in eager mode while the rest operations in cudagraph, we keep the flexibility of the attention operation. The piecewise cudagraph also has fine-grained memory management. The purpose is to only exclude the attention kernel from cudagraph, while keeping all the rest modules and the memory allocation operations in the cudagraph. This is why the attention operation in V1 has the output tensor as the input of the attention. The cudagraphs are captured and managed by the compiler backend, and replayed when the batch size has corresponding cudagraph captured. The caller of the model (model runner) only needs to make sure it manages the input buffers correctly. All of the intermediate buffers are managed automatically by the compiler backend. By default, vLLM will try to determine a set of sizes to capture cudagraph. You can also override it using the config `cudagraph_capture_sizes`: `[](#__codelineno-8-1)vllm serve meta-llama/Llama-3.2-1B \ [](#__codelineno-8-2) --compilation-config '{"cudagraph_capture_sizes": [1, 2, 4, 8]}'` Then it will only capture cudagraph for the specified sizes. It can be useful to have fine-grained control over the cudagraph capture. ### Full Cudagraph capture[¶](#full-cudagraph-capture "Permanent link") It is possible to include attention as part of the cudagraph if using an attention backend that is cudagraph compatible. This can improve performance in some cases such as decode speed for smaller models or MOEs. See [CUDA Graphs](https://docs.vllm.ai/en/latest/cuda_graphs/) for more details. --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/torch_compile_multimodal.md "Edit this page") `torch.compile` can now be applied to multimodal encoders and miscellaneous nn modules in vLLM, including vision-language models like LLaMA 4, Qwen-VL, and similar encoder-based architectures. This document covers the basics of how the `torch.compile` integration works for multimodal encoders in vLLM, as well as how to apply the decorator to new models to improve performance. ## Overview[¶](#overview "Permanent link") We have recently enabled the `@support_torch_compile` decorator to work for multiple nn module components within a model type; this enables turning compile on for multimodal encoders, bringing performance improvements to additional components of the stack. When applied to the vision block of [`Qwen2_5_vl`](https://github.com/vllm-project/vllm/pull/23207) we observe ~4.5% e2e perf improvements with some increase in compilation time This feature is off by default, but can be enabled by setting `compile_mm_encoder: true` in the compilation config when models have the `@support_torch_compile` decorator. ## How Compilation Works for Multimodal Components[¶](#how-compilation-works-for-multimodal-components "Permanent link") ### APIs for Enablement[¶](#apis-for-enablement "Permanent link") To compile a multimodal component such as an encoder, we follow the same mechanism as the LLM text backbone, with a few additional scaffoldings: 1. The `@support_torch_compile` decorator should include `enable_if=should_torch_compile_mm_encoder`. This will gate the compilation behind our `compile_mm_encoder` configuration 2. The `@support_torch_compile` decorator should include `is_encoder=True` for encoder components. This is needed for compile range integration (see Compile Range Integration). The decorator automatically uses the class name as the cache directory prefix, avoiding collisions between independently compiled sub-modules (e.g. vision encoder components vs the text backbone). ### CompilationConfig[¶](#compilationconfig "Permanent link") With the exception of `compile_mm_encoder: true`, the multimodal encoder will inherit from the same compilation config as the text LLM. We may extend this for more configuration in the future. ## Applying torch.compile to a New Multimodal Model/Component[¶](#applying-torchcompile-to-a-new-multimodal-modelcomponent "Permanent link") To apply `support_torch_compile` to a new general nn.Module, we advise following the same steps in [`debug_vllm_compile`](https://docs.vllm.ai/en/latest/debug_vllm_compile/); this includes: 1. Applying `support_torch_compile` on initially small modules (such as basic MLP layers), then raising to more general modules until one reaches a good performance tradeoff 2. Leveraging [`tlparse`](https://github.com/meta-pytorch/tlparse) to identify and eliminate the source of recompiles and graph breaks 3. Using `dynamic_arg_dims` and proper `dynamic_shapes_config` to handle dynamism. ### Common pitfalls[¶](#common-pitfalls "Permanent link") ## VllmBackend Feature Support[¶](#vllmbackend-feature-support "Permanent link") ### Compile ranges[¶](#compile-ranges "Permanent link") The torch.compile integration will try to rely on max\_batch\_size to infer compilation ranges for dynamic shapes; however, for modules used in the encoder, this shape can be difficult to infer due to the unspecified range of shapes the encoder may see as input. Therefore, we rely on `is_encoder=True` in the `@support_torch_compile` decorator to alert torch.compile to the fact that this range cannot be inferred, and we default to the range (1, MAX\_INT). Note We may seek to tighten this range for better performance in the future ### Cudagraphs[¶](#cudagraphs "Permanent link") We have not yet explored compilation for multimodal encoders with CUDAGraph integration; behavior is currently unspecified. ## Troubleshooting[¶](#troubleshooting "Permanent link") ### Graph Breaks in Vision Encoders[¶](#graph-breaks-in-vision-encoders "Permanent link") Some vision encoder operations may cause graph breaks. To identify them: `[](#__codelineno-0-1)TORCH_LOGS="+dynamo" vllm serve ` Common causes of graph breaks in multimodal models: - **Dynamic image sizes**: Use `dynamic_shapes_config` to handle variable resolutions - **Untraceable operations**: Some operations (such as to\_list) may not be supported by Dynamo - **Conditional processing**: Data-dependent branching based on image properties ### Compilation Errors[¶](#compilation-errors "Permanent link") If compilation fails for a multimodal model: 1. **Disable and test**: First verify the model works without compilation: `[](#__codelineno-1-1)vllm serve --compilation-config='{"mode":0,"compile_mm_encoder":"false"}'` 2. **Check logs**: Enable debug logging to see compilation details: `[](#__codelineno-2-1)VLLM_LOGGING_LEVEL=DEBUG vllm serve --compilation-config='{"compile_mm_encoder":"true"}'` 3. **Report issues**: If you find a bug, [open an issue on GitHub](https://github.com/vllm-project/vllm/issues/new/choose) ## See Also[¶](#see-also "Permanent link") - [torch.compile Integration](https://docs.vllm.ai/en/latest/torch_compile/) - Core design document - [Debugging torch.compile](https://docs.vllm.ai/en/latest/debug_vllm_compile/) - Detailed debugging guide - [Multimodal Inputs](https://docs.vllm.ai/en/features/multimodal_inputs/) - How to pass multimodal data - [Disaggregated Encoder](https://docs.vllm.ai/en/features/disagg_encoder/) - Scaling vision encoders - [Supported Multimodal Models](https://docs.vllm.ai/en/models/supported_models/#list-of-multimodal-language-models) - Model compatibility --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/design/vllm_ir.md "Edit this page") ## Motivation[¶](#motivation "Permanent link") vLLM IR is a **functional intermediate representation (IR)** that fills the gap between low-level `torch` ops and vLLM layers like [`RMSNorm`](https://docs.vllm.ai/en/api/vllm/model_executor/layers/layernorm/#vllm.model_executor.layers.layernorm.RMSNorm " RMSNorm") and quantization operators, By separating operator **semantics** from the **implementation** and **dispatching**, vLLM IR simplifies both compilation and kernel registration & dispatching simultaneously. It operates as a **dialect** in the torch FX representation, allowing full interoperability with “regular” torch ops & custom torch ops/kernels, as well as a piecewise migration from the previous [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") approach. Key design principles: - **Eager-compile consistency**: identical behavior (barring minor numerics) in eager and compiled modes - **Simple, transparent, yet powerful kernel selection**: good visibility and control allowing easy debugging - **Convention over configuration**: near-zero boilerplate required to register ops and implementations - **Extensibility**: ops and implementations can be registered anywhere, in-tree or out-of-tree - **Interoperability**: fully compatible with “regular” torch ops & custom torch ops/kernels, reducing developer friction and allowing piecewise migration The clean semantics/implementation separation enables a unified and extensible dispatching mechanism, allowing multiple kernels per-platform and powerful kernel selection. The separation also facilitates cleaner testing and benchmarking, removing much of the boilerplate standard for legacy approaches. By delaying kernel selection until late in the compilation process, the compiler can operate on a higher-level representation, which has the following main benefits: - Pattern matching in fusion/transformation passes only requires a single, simple pattern per op - OOT compiler backends can lower from the higher-level representation (in-progress) - The compiler can autotune over available implementations (future feature) ## Quick Overview[¶](#quick-overview "Permanent link") ### Declaring an IR Operation[¶](#declaring-an-ir-operation "Permanent link") IR operations are declared using the `@register_op` decorator with a native PyTorch implementation that defines the op's semantics: `[](#__codelineno-0-1)# vllm/ir/ops/layernorm.py [](#__codelineno-0-2)from torch import Tensor [](#__codelineno-0-3)from vllm.ir import register_op [](#__codelineno-0-4)[](#__codelineno-0-5)@register_op [](#__codelineno-0-6)def rms_norm(x: Tensor, weight: Tensor | None, epsilon: float, variance_size: int | None = None) -> Tensor: [](#__codelineno-0-7) """Weighted root-mean-square layer normalization""" [](#__codelineno-0-8) orig_dtype = x.dtype [](#__codelineno-0-9) x = x.to(torch.float32) [](#__codelineno-0-10) x_var = x if variance_size is None else x[..., :variance_size] [](#__codelineno-0-11) variance = x_var.pow(2).mean(dim=-1, keepdim=True) [](#__codelineno-0-12) x = x * torch.rsqrt(variance + epsilon) [](#__codelineno-0-13) x = x.to(orig_dtype) [](#__codelineno-0-14) if weight is not None: [](#__codelineno-0-15) x = x * weight [](#__codelineno-0-16) return x` The native implementation serves three purposes: 1. **Semantic definition**: Specifies the exact semantics of the operation, including shapes and strides 2. **Default implementation**: Used when no other (better) implementation is available 3. **Reference for testing**: Other implementations must match these semantics ### Registering Implementations[¶](#registering-implementations "Permanent link") Kernel implementations are registered using the `register_impl` decorator on the IR op object: `[](#__codelineno-1-1)# vllm/kernels/vllm_c.py [](#__codelineno-1-2)from vllm import ir [](#__codelineno-1-3)[](#__codelineno-1-4)rms_norm_no_var = lambda x, weight, epsilon, variance_size=None: variance_size is None [](#__codelineno-1-5)[](#__codelineno-1-6)@ir.ops.rms_norm.register_impl("vllm_c", supports_args=rms_norm_no_var, supported=current_platform.is_cuda_alike()) [](#__codelineno-1-7)def rms_norm(x: Tensor, weight: Tensor | None, epsilon: float, variance_size: int | None = None) -> Tensor: [](#__codelineno-1-8) output = torch.empty_like(x) [](#__codelineno-1-9) torch.ops._C.rms_norm(output, x, weight, epsilon) [](#__codelineno-1-10) return output` Implementations can specify: - `supported`: Static boolean indicating if this implementation is available - `supports_args`: Function checking if the implementation supports specific arguments - `inplace`: Whether this implementation reuses input memory for outputs ### Using IR Operations in Models[¶](#using-ir-operations-in-models "Permanent link") IR operations are imported and called directly in model code: `[](#__codelineno-2-1)# vllm/model_executor/layers/layernorm.py [](#__codelineno-2-2)from vllm import ir [](#__codelineno-2-3)[](#__codelineno-2-4)class RMSNorm(nn.Module): [](#__codelineno-2-5) def __init__(self, hidden_size: int, eps: float = 1e-6): [](#__codelineno-2-6) super().__init__() [](#__codelineno-2-7) self.weight = nn.Parameter(torch.ones(hidden_size)) [](#__codelineno-2-8) self.variance_epsilon = eps [](#__codelineno-2-9) [](#__codelineno-2-10) def forward(self, x: Tensor, residual: Tensor | None = None): [](#__codelineno-2-11) if residual is None: [](#__codelineno-2-12) return ir.ops.rms_norm(x, self.weight, self.variance_epsilon) [](#__codelineno-2-13) [](#__codelineno-2-14) # Use maybe_inplace overload to allow implementation to reuse input memory for outputs [](#__codelineno-2-15) # (using x or residual after this call is undefined behavior) [](#__codelineno-2-16) return ir.ops.fused_add_rms_norm.maybe_inplace( [](#__codelineno-2-17) x, residual, self.weight, self.variance_epsilon [](#__codelineno-2-18) )` ### Configuring Kernel Selection[¶](#configuring-kernel-selection "Permanent link") Kernel selection is controlled via priority lists in the configuration. Priority lists specify the order in which implementations are considered, with the first supported implementation being selected. This includes the static support check (`supported=...`) and the dynamic arg support check (`supports_args=...`). #### Command Line Configuration[¶](#command-line-configuration "Permanent link") Use `--ir-op-priority.=,,...`: `[](#__codelineno-3-1)# CUDA: Use vllm_c implementation for rms_norm [](#__codelineno-3-2)vllm serve meta-llama/Llama-3.2-1B \ [](#__codelineno-3-3) --ir-op-priority.rms_norm=vllm_c [](#__codelineno-3-4)[](#__codelineno-3-5)# ROCm: Try aiter first, fall back to vllm_c, then native [](#__codelineno-3-6)vllm serve meta-llama/Llama-3.2-1B \ [](#__codelineno-3-7) --ir-op-priority.rms_norm=aiter,vllm_c,native [](#__codelineno-3-8)[](#__codelineno-3-9)# Configure multiple operations [](#__codelineno-3-10)vllm serve meta-llama/Llama-3.2-1B \ [](#__codelineno-3-11) --ir-op-priority.rms_norm=vllm_c \ [](#__codelineno-3-12) --ir-op-priority.fused_add_rms_norm=vllm_c` #### Python Configuration[¶](#python-configuration "Permanent link") `[](#__codelineno-4-1)from vllm import LLM [](#__codelineno-4-2)from vllm.config import VllmConfig, KernelConfig [](#__codelineno-4-3)[](#__codelineno-4-4)llm = LLM( [](#__codelineno-4-5) model="meta-llama/Llama-3.2-1B", [](#__codelineno-4-6) vllm_config=VllmConfig( [](#__codelineno-4-7) kernel_config=KernelConfig( [](#__codelineno-4-8) ir_op_priority={ [](#__codelineno-4-9) "rms_norm": ["vllm_c", "native"], [](#__codelineno-4-10) "fused_add_rms_norm": ["vllm_c", "native"], [](#__codelineno-4-11) } [](#__codelineno-4-12) ) [](#__codelineno-4-13) ) [](#__codelineno-4-14))` #### Platform Defaults[¶](#platform-defaults "Permanent link") Each platform provides default priority lists that are automatically applied: `[](#__codelineno-5-1)# CUDA/XPU/ROCm platform defaults (when compiling with Inductor) [](#__codelineno-5-2){ [](#__codelineno-5-3) "rms_norm": ["native"], # Native torch is default [](#__codelineno-5-4) "fused_add_rms_norm": ["native"], [](#__codelineno-5-5)} [](#__codelineno-5-6)[](#__codelineno-5-7)# CUDA platform defaults (eager or Dynamo-only) [](#__codelineno-5-8){ [](#__codelineno-5-9) "rms_norm": ["vllm_c", "native"], [](#__codelineno-5-10) "fused_add_rms_norm": ["vllm_c", "native"], [](#__codelineno-5-11)} [](#__codelineno-5-12)[](#__codelineno-5-13)# ROCm platform defaults (future - currently same as CUDA) [](#__codelineno-5-14){ [](#__codelineno-5-15) "rms_norm": ["aiter", "vllm_c", "native"], [](#__codelineno-5-16) "fused_add_rms_norm": ["aiter", "vllm_c", "native"], [](#__codelineno-5-17)} [](#__codelineno-5-18)[](#__codelineno-5-19)# XPU platform defaults (eager or Dynamo-only) [](#__codelineno-5-20){ [](#__codelineno-5-21) "rms_norm": ["xpu_kernels", "native"], [](#__codelineno-5-22) "fused_add_rms_norm": ["xpu_kernels", "native"], [](#__codelineno-5-23)}` User-specified priorities are prepended to platform defaults, so you only need to specify the out-of-order implementations, other implementations are appended automatically. ## Compilation Pipeline[¶](#compilation-pipeline "Permanent link") vLLM IR heavily customizes the `torch.compile`\-based compilation process to allow custom compile passes to operate on high-level IR while still producing efficient low-level code at the end. The compilation pipeline consists of several stages: ### 1\. Dynamo Tracing[¶](#1-dynamo-tracing "Permanent link") When `torch.compile` traces the model's forward pass, vLLM IR operations appear as custom operations in the `vllm_ir` torch library. These operations are opaque to Dynamo, meaning they appear directly in the FX graph without decomposition: `[](#__codelineno-6-1)# Python code (epsilon=1e-5) [](#__codelineno-6-2)x1 = ir.ops.rms_norm(x, weight, epsilon) [](#__codelineno-6-3)x2, residual_out = ir.ops.fused_add_rms_norm.maybe_inplace(x1, residual, weight, epsilon) [](#__codelineno-6-4)[](#__codelineno-6-5)# FX graph after Dynamo tracing [](#__codelineno-6-6)x1 = torch.ops.vllm_ir.rms_norm.default(x, weight, 1e-5); x = None [](#__codelineno-6-7)out = torch.ops.vllm_ir.fused_add_rms_norm.maybe_inplace(x1, residual, weight, 1e-5); x1 = residual = None [](#__codelineno-6-8)x2 = out[0] [](#__codelineno-6-9)residual_out = out[1]` ### 2\. AOTAutograd and Functionalization[¶](#2-aotautograd-and-functionalization "Permanent link") AOTAutograd functionalizes the graph, converting any mutating operations to functional equivalents. For vLLM IR operations with `maybe_inplace` overloads, we perform this manually before AOTAutograd, converting them to the functional `default` overload using the pre-grad custom pass hook. `[](#__codelineno-7-1)# After functionalization [](#__codelineno-7-2)x1 = torch.ops.vllm_ir.rms_norm.default(x, weight, 1e-5); x = None [](#__codelineno-7-3)out = torch.ops.vllm_ir.fused_add_rms_norm.default(x1, residual, weight, 1e-5); x1 = residual = None [](#__codelineno-7-4)x2 = out[0] [](#__codelineno-7-5)residual_out = out[1]` The pass also tracks which inputs were "donated" (passed to `maybe_inplace`), storing this information in vLLM's `PassContext` for later use in clone elimination. ### 3\. IR Fusion and Transformation Passes[¶](#3-ir-fusion-and-transformation-passes "Permanent link") After functionalization, custom vLLM passes operate on the functional FX graph containing high-level IR operations. These passes can perform fusion, distribute operations for sequence parallelism, and other transformations: `[](#__codelineno-8-1)# Example: Sequence Parallelism (see SequenceParallelismPass) [](#__codelineno-8-2)# Before SP pass [](#__codelineno-8-3)[](#__codelineno-8-4)all_reduce = torch.ops.vllm.all_reduce(x, "tp:0") [](#__codelineno-8-5)rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5) [](#__codelineno-8-6)[](#__codelineno-8-7)# after SP pass [](#__codelineno-8-8)reduce_scatter = torch.ops.vllm.reduce_scatter(x, "tp:0") [](#__codelineno-8-9)rms_norm = torch.ops.vllm_ir.rms_norm(all_reduce, weight, 1e-5) [](#__codelineno-8-10)all_gather = torch.ops.vllm.all_gather(x, "tp:0")` Fusion passes benefit from the high-level representation: they don't need to match against low-level PyTorch operations, handle different kernel implementations separately, or deal with functionalization of custom kernels. ### 4\. IR Lowering[¶](#4-ir-lowering "Permanent link") The lowering pass ([`VllmIRLoweringPass`](https://docs.vllm.ai/en/api/vllm/compilation/passes/ir/lowering_pass/#vllm.compilation.passes.ir.lowering_pass.VllmIRLoweringPass " VllmIRLoweringPass")) replaces each vLLM IR operation with its selected implementation. The implementation is chosen based on the priority list and support predicates, using the **fake tensors** in the graph's metadata in place of op arguments: `[](#__codelineno-9-1)# Implementation selection, same in eager dispatch and compile lowering [](#__codelineno-9-2)def dispatch(*args) -> IrOpImpl: [](#__codelineno-9-3) for provider in priority_list: # e.g., ["vllm_c", "native"] [](#__codelineno-9-4) impl = ir_op.impls[provider] [](#__codelineno-9-5) if not impl.supported: [](#__codelineno-9-6) continue [](#__codelineno-9-7) if impl.supports_args and not impl.supports_args(*args): [](#__codelineno-9-8) continue [](#__codelineno-9-9) return impl [](#__codelineno-9-10)[](#__codelineno-9-11)# make_fx uses torch.fx.symbolic_trace [](#__codelineno-9-12)impl_graph = make_fx(selected_impl.impl_fn) [](#__codelineno-9-13)# Replace IR op node with impl_graph's nodes [](#__codelineno-9-14)match.replace_by_example(selected_impl.impl_fn, node.args)` For example, lowering `rms_norm` with the `vllm_c` implementation: `[](#__codelineno-10-1)# Before lowering (IR op) [](#__codelineno-10-2)rms_norm = torch.ops.vllm_ir.rms_norm.default(x, weight, 1e-5) [](#__codelineno-10-3)[](#__codelineno-10-4)# After lowering (vllm_c implementation traced) [](#__codelineno-10-5)# Note: Lowering does not currently functionalize, this will likely change in the future. [](#__codelineno-10-6)empty = torch.ops.aten.empty.memory_format(x.shape, ...) [](#__codelineno-10-7)rms_norm = torch.ops._C.rms_norm(empty, x, weight, 1e-5)` When lowering an implementation that mutates inputs (`inplace=True`), the lowering pass inserts clones to preserve functional semantics: `[](#__codelineno-11-1)# vllm_c implementation for fused_add_rms_norm mutates its first two arguments [](#__codelineno-11-2)# Lowered with clones for safety [](#__codelineno-11-3)clone_default = torch.ops.aten.clone.default(x) [](#__codelineno-11-4)clone_default_1 = torch.ops.aten.clone.default(residual) [](#__codelineno-11-5)fused_add_rms_norm = torch.ops._C.fused_add_rms_norm.default(clone_default, clone_default_1, weight, 1e-5)` ### 5\. Clone Cleanup[¶](#5-clone-cleanup "Permanent link") After lowering, the clone elimination pass ([`UnsafeCloneEliminationPass`](https://docs.vllm.ai/en/api/vllm/compilation/passes/ir/clone_elimination/#vllm.compilation.passes.ir.clone_elimination.UnsafeCloneEliminationPass " UnsafeCloneEliminationPass")) removes unnecessary clones introduced during lowering. This pass is essential for achieving zero-copy behavior when using in-place kernels with `maybe_inplace`. The pass removes a clone if: - the cloned input is created in the graph and not used again in the graph - the cloned input is a graph parameter, marked as donated `[](#__codelineno-12-1)# After cleanup (donated inputs, no subsequent uses) [](#__codelineno-12-2)fused_add_rms_norm = torch.ops._C.fused_add_rms_norm.default(x, residual, weight, 1e-5)` The combination of inplace functionalization (tracking donated inputs) and clone cleanup enables the compiler to safely use in-place kernels without adding redundant copies or increasing the memory usage. ### 6\. Inductor Optimization and Codegen[¶](#6-inductor-optimization-and-codegen "Permanent link") After IR lowering and cleanup, the graph contains only standard PyTorch operations and platform-specific custom ops. Inductor then performs its standard codegen: - **Inductor lowering and pointwise fusion**: Fusing element-wise operations, reductions, etc. - **Memory planning**: Determining buffer allocation and reuse - **Kernel generation**: Generating Triton or C++ code for fused operations - **Autotuning**: Selecting the best kernel configurations ### Pipeline Summary[¶](#pipeline-summary "Permanent link") `[](#__codelineno-13-1)Model Forward Pass [](#__codelineno-13-2) ↓ [](#__codelineno-13-3)[Dynamo Tracing] → FX Graph with vllm_ir.* ops [](#__codelineno-13-4) ↓ [](#__codelineno-13-5)[Pre-grad: Inplace Functionalization] → maybe_inplace → default, track donated inputs [](#__codelineno-13-6) ↓ [](#__codelineno-13-7)[AOTAutograd] → Functionalization [](#__codelineno-13-8) ↓ [](#__codelineno-13-9)[Post-grad: IR Fusion Passes] → Fuse high-level IR ops (e.g., rms_norm + quant) [](#__codelineno-13-10) ↓ [](#__codelineno-13-11)[Post-grad: IR Lowering] → vllm_ir.* ops → impl ops (with clones if needed) [](#__codelineno-13-12) ↓ [](#__codelineno-13-13)[Post-grad: Clone Cleanup] → Remove unnecessary clones using donated input info [](#__codelineno-13-14) ↓ [](#__codelineno-13-15)[Inductor] → Pattern matching, fusion, memory planning, codegen [](#__codelineno-13-16) ↓ [](#__codelineno-13-17)Compiled Code` ## Core vLLM IR Concepts[¶](#core-vllm-ir-concepts "Permanent link") ### Operation Declaration[¶](#operation-declaration "Permanent link") Operations are declared with the `@register_op` decorator, which creates an `IrOp` object: `[](#__codelineno-14-1)@register_op( [](#__codelineno-14-2) name=None, # Operation name (defaults to function name) [](#__codelineno-14-3) activations=None, # List of activation parameters (defaults to params starting with 'x') [](#__codelineno-14-4) allow_inplace=False, # Whether to create a maybe_inplace overload [](#__codelineno-14-5)) [](#__codelineno-14-6)def op_name(...): [](#__codelineno-14-7) ...` **Parameters:** - `activations`: List of parameter names considered "activations" (typically consumed by `maybe_inplace`). Defaults to parameters starting with `x`. - `allow_inplace`: Creates a `maybe_inplace` overload for memory-efficient execution (see below). ### The `maybe_inplace` Overload[¶](#the-maybe_inplace-overload "Permanent link") The `maybe_inplace` overload is a critical feature for memory efficiency in LLM inference. It signals that the caller doesn't need to preserve the activation inputs after the operation, allowing in-place implementations to reuse input memory for outputs. #### Semantics and Usage[¶](#semantics-and-usage "Permanent link") `[](#__codelineno-15-1)# Standard usage: inputs are preserved [](#__codelineno-15-2)out, res_out = ir.ops.fused_add_rms_norm(x, residual, weight, epsilon) [](#__codelineno-15-3)# x and residual are unchanged, out and res_out are new tensors [](#__codelineno-15-4)[](#__codelineno-15-5)# maybe_inplace: inputs may be modified [](#__codelineno-15-6)out, res_out = ir.ops.fused_add_rms_norm.maybe_inplace(x, residual, weight, epsilon) [](#__codelineno-15-7)# x and residual may be modified (undefined behavior to use them after this) [](#__codelineno-15-8)# out and res_out may alias x and residual` Using an activation input after passing it to `maybe_inplace` is **undefined behavior**: `[](#__codelineno-16-1)# WRONG: Using x after donating it [](#__codelineno-16-2)out, res_out = ir.ops.fused_add_rms_norm.maybe_inplace(x, residual, weight, epsilon) [](#__codelineno-16-3)result = out + x # ERROR: x was donated!` If you need to preserve an input, either use the default overload or clone manually: `[](#__codelineno-17-1)# Option 1: Use default overload [](#__codelineno-17-2)out, res_out = ir.ops.fused_add_rms_norm(x, residual, weight, epsilon) [](#__codelineno-17-3)result = out + x # OK: x is preserved [](#__codelineno-17-4)[](#__codelineno-17-5)# Option 2: Clone before maybe_inplace [](#__codelineno-17-6)out, res_out = ir.ops.fused_add_rms_norm.maybe_inplace(x.clone(), residual, weight, epsilon) [](#__codelineno-17-7)result = out + x # OK: x is preserved, clone was donated` #### Compilation Behavior[¶](#compilation-behavior "Permanent link") During compilation, the inplace functionalization pass validates that donated inputs are not used again and converts `maybe_inplace` to the functional `default` overload: `[](#__codelineno-18-1)# Inplace functionalization pass (pre-grad) [](#__codelineno-18-2)for node in graph.nodes: [](#__codelineno-18-3) if node.target == torch.ops.vllm_ir.fused_add_rms_norm.maybe_inplace: [](#__codelineno-18-4) # Check that activation inputs aren't used after this node [](#__codelineno-18-5) for activation_arg in activation_inputs: [](#__codelineno-18-6) for user in activation_arg.users: [](#__codelineno-18-7) if user appears after node: [](#__codelineno-18-8) raise ValueError(f"Input {activation_arg} donated but used again") [](#__codelineno-18-9) [](#__codelineno-18-10) # Convert to default overload [](#__codelineno-18-11) node.target = torch.ops.vllm_ir.fused_add_rms_norm.default [](#__codelineno-18-12) [](#__codelineno-18-13) # Track donated graph inputs for later clone elimination [](#__codelineno-18-14) for i, arg in enumerate(node.args): [](#__codelineno-18-15) if arg.op == "placeholder" and i in activation_indices: [](#__codelineno-18-16) pass_context.donated_input_ids.add(node_to_idx[arg])` The donated input information is then used by the clone cleanup pass to eliminate unnecessary copies when in-place kernels are lowered. #### Eager Mode Behavior[¶](#eager-mode-behavior "Permanent link") In eager mode (without `torch.compile`), `maybe_inplace` enables **maximally memory-efficient** execution by allowing the IR operation to dispatch directly to in-place implementations: `[](#__codelineno-19-1)# Eager dispatch logic for maybe_inplace [](#__codelineno-19-2)impl: IrOpImpl = ir_op.dispatch(*args) [](#__codelineno-19-3)return impl.impl_fn(*args) [](#__codelineno-19-4)[](#__codelineno-19-5)# Eager dispatch logic for default: [](#__codelineno-19-6)impl: IrOpImpl = ir_op.dispatch(*args) [](#__codelineno-19-7)if impl.inplace: [](#__codelineno-19-8) args = [ [](#__codelineno-19-9) arg.clone() if i in ir_op.activations else arg [](#__codelineno-19-10) for i, arg in enumerate(args) [](#__codelineno-19-11) ] [](#__codelineno-19-12)return impl.impl_fn(*args)` The combination of `maybe_inplace` in model code and in-place kernel implementations provides optimal memory efficiency in both eager and compiled modes, with identical semantics in both cases. #### Memory Savings Example[¶](#memory-savings-example "Permanent link") Consider a transformer layer with residual connections: `[](#__codelineno-20-1)# Without maybe_inplace (2 allocations per layer) [](#__codelineno-20-2)hidden_states = self.attention(input) [](#__codelineno-20-3)normed, residual = ir.ops.fused_add_rms_norm(hidden_states, input, weight, eps) [](#__codelineno-20-4)# Memory: input (preserved), hidden_states (preserved), normed (new), residual (new) [](#__codelineno-20-5)[](#__codelineno-20-6)# With maybe_inplace (0 allocations per layer when using in-place kernel) [](#__codelineno-20-7)hidden_states = self.attention(input) [](#__codelineno-20-8)normed, residual = ir.ops.fused_add_rms_norm.maybe_inplace(hidden_states, input, weight, eps) [](#__codelineno-20-9)# Memory: normed (reuses hidden_states), residual (reuses input)` ### Implementation Registration[¶](#implementation-registration "Permanent link") Implementations are registered using the `register_impl` method: `[](#__codelineno-21-1)@ir.ops.op_name.register_impl( [](#__codelineno-21-2) provider="provider_name", # Unique identifier (e.g., "vllm_c", "aiter", "triton") [](#__codelineno-21-3) supported=True, # Static availability check [](#__codelineno-21-4) supports_args=None, # Dynamic argument support check [](#__codelineno-21-5)) [](#__codelineno-21-6)def impl_fn(...): [](#__codelineno-21-7) ...` **Provider naming conventions:** - `native`: Reserved for the native torch implementation (declared with `@register_op`) - `vllm_c`: C++/CUDA kernels via `torch.ops._C` - `aiter`: AMD AITER library - `xpu_kernels`: SYCL/SYCLTLA kernels implemented in `vllm-xpu-kernels` - `triton_*`: Triton kernels - Platform/library names for other implementations **Support checking:** - `supported`: Static boolean, checked once at import time (e.g., `HAS_TRITON`, `is_cuda_alike()`) - `supports_args`: Function `(*args, **kwargs) -> bool` checking argument compatibility - Called with **fake tensors** during compilation for zero-cost checking - Called with **real tensors** during eager mode dispatch - Should NOT check batch sizes or add guards based on values Example support predicate: `[](#__codelineno-22-1)def aiter_rms_norm_supports(x, weight, epsilon, variance_size=None): [](#__codelineno-22-2) # Check dtype (OK: doesn't depend on batch size) [](#__codelineno-22-3) if x.dtype not in [torch.float16, torch.bfloat16]: [](#__codelineno-22-4) return False [](#__codelineno-22-5) # Check optional parameter (OK: static check) [](#__codelineno-22-6) if variance_size is not None: [](#__codelineno-22-7) return False [](#__codelineno-22-8) return True [](#__codelineno-22-9)[](#__codelineno-22-10)@ir.ops.rms_norm.register_impl("aiter", supports_args=aiter_rms_norm_supports) [](#__codelineno-22-11)def rms_norm(...): [](#__codelineno-22-12) ...` Batch-invariant kernels are automatically selected when `VLLM_BATCH_INVARIANT=1` is set. ### Eager Mode vs Compile Mode[¶](#eager-mode-vs-compile-mode "Permanent link") vLLM IR operations behave identically in eager and compile modes: **Eager mode:** - Direct dispatch to implementation based on priority list - Support checked with real tensor arguments - Minimal overhead (can be optimized further if needed) **Compile mode:** - IR ops appear in FX graph as `torch.ops.vllm_ir.*` custom ops - Lowering selects implementation using fake tensors - Full integration with Inductor optimizations This consistency enables: - Prototyping in eager mode with confidence - Debugging by disabling compilation - Gradual migration from eager to compiled execution ## Other Topics[¶](#other-topics "Permanent link") ### Out-of-Tree Implementations[¶](#out-of-tree-implementations "Permanent link") External platforms can register implementations without modifying vLLM: `[](#__codelineno-23-1)# In external package [](#__codelineno-23-2)from vllm import ir [](#__codelineno-23-3)[](#__codelineno-23-4)@ir.ops.rms_norm.register_impl("my_platform", supported=is_my_platform()) [](#__codelineno-23-5)def rms_norm(x, weight, epsilon, variance_size=None): [](#__codelineno-23-6) return my_platform.rms_norm(x, weight, epsilon)` Then configure priority to use your implementation: `[](#__codelineno-24-1)class MyPlatform(Platform): [](#__codelineno-24-2) def get_default_ir_op_priority(self): [](#__codelineno-24-3) return IrOpPriorityConfig(rms_norm=['my_platform', 'native']) [](#__codelineno-24-4)[](#__codelineno-24-5)# Users can still override priority in the same way [](#__codelineno-24-6)llm = LLM(ir_op_priority=IrOpPriorityConfig(rms_norm=['custom_oot_kernel']))` ### Debugging and Observability[¶](#debugging-and-observability "Permanent link") Note Please let us know how observability can be improved for your use-case! Enable debug logging to see kernel selection: `[](#__codelineno-25-1)VLLM_LOGGING_LEVEL=DEBUG vllm serve ...` This logs: - Which implementations are selected for each operation - Why implementations were rejected (unsupported, args not supported) - Compilation cache hits/misses - IR lowering statistics Check selected implementations in compiled graphs: `[](#__codelineno-26-1)# After compilation, inspect the lowering pass [](#__codelineno-26-2)lowering_pass = backend.lowering_pass [](#__codelineno-26-3)print(lowering_pass.selected_impls) [](#__codelineno-26-4)# Output: {'rms_norm': {'node_123': 'vllm_c', 'node_456': 'vllm_c'}}` ## Migration from CustomOp[¶](#migration-from-customop "Permanent link") vLLM IR is designed to coexist with and gradually replace [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp"): 1. **Op declaration**: Convert [`CustomOp`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.CustomOp " CustomOp") class [`PluggableLayer`](https://docs.vllm.ai/en/api/vllm/model_executor/custom_op/#vllm.model_executor.custom_op.PluggableLayer " PluggableLayer") and move `forward_native` to `@register_op` function 2. **Implementation registration**: Use `@ir.ops.op_name.register_impl` instead of overriding methods 3. **Layer usage**: Replace `self.op(...)` with `ir.ops.op_name(...)` 4. **Configuration**: Migrate `--compilation-config.custom-ops` to `--ir-op-priority` The migration can be done incrementally, one operation at a time. ## See Also[¶](#see-also "Permanent link") - [torch.compile Integration](https://docs.vllm.ai/en/latest/torch_compile/) - General compilation infrastructure - [Fusions](https://docs.vllm.ai/en/latest/fusions/) - Custom fusion and transformation passes in vLLM - [Custom Operations](https://docs.vllm.ai/en/latest/custom_op/) - Legacy custom op system --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/README.md "Edit this page") vLLM's examples are organized into the following categories: - **[`basic/`](https://github.com/vllm-project/vllm/tree/main/examples/basic)** – Minimal examples for offline inference and online serving. - **[`generate/`](https://github.com/vllm-project/vllm/tree/main/examples/generate)** – Text generation examples, including multimodal models. - **[`pooling/`](https://github.com/vllm-project/vllm/tree/main/examples/pooling)** – Examples for embedding, classification, scoring, reward, etc. - **[`speech_to_text/`](https://github.com/vllm-project/vllm/tree/main/examples/speech_to_text)** – Speech transcription, translation and real-time audio examples. - **[`features/`](https://github.com/vllm-project/vllm/tree/main/examples/features)** – Demonstrations of individual vLLM features: automatic prefix caching, speculative decoding, LoRA, structured outputs, prompt embedding, pause/resume, batch invariance, KV events, data parallelism, and more. - **[`reasoning/`](https://github.com/vllm-project/vllm/tree/main/examples/reasoning)** – Examples for reasoning with vLLM. - **[`tool_calling/`](https://github.com/vllm-project/vllm/tree/main/examples/tool_calling)** – Examples for function/tool calling with vLLM. - **[`applications/`](https://github.com/vllm-project/vllm/tree/main/examples/applications)** – Application examples such as chatbots and RAG (Retrieval-Augmented Generation). - **[`rl/`](https://github.com/vllm-project/vllm/tree/main/examples/rl)** – Reinforcement learning examples. - **[`deployment/`](https://github.com/vllm-project/vllm/tree/main/examples/deployment)** – Examples for deploying vLLM in production. - **[`ray_serving/`](https://github.com/vllm-project/vllm/tree/main/examples/ray_serving)** – Scalable serving using Ray. - **[`disaggregated/`](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated)** – Examples for disaggregated serving (separate prefill and decode), including various kv cache connectors (LMCache, Mooncake, FlexKV, P2P NCCL) and failure recovery. - **[`observability/`](https://github.com/vllm-project/vllm/tree/main/examples/observability)** – Metrics, logging, tracing (OpenTelemetry), and dashboards (Grafana, Perses). --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/applications/chatbot.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/applications/chatbot](https://github.com/vllm-project/vllm/tree/main/examples/applications/chatbot). ## API Client[¶](#api-client "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)"""Example Python client for `vllm.entrypoints.api_server` [](#__codelineno-0-4)Start the demo server: [](#__codelineno-0-5) python -m vllm.entrypoints.api_server --model [](#__codelineno-0-6)[](#__codelineno-0-7)NOTE: The API server is used only for demonstration and simple performance [](#__codelineno-0-8)benchmarks. It is not intended for production use. [](#__codelineno-0-9)For production use, we recommend `vllm serve` and the OpenAI client API. [](#__codelineno-0-10)""" [](#__codelineno-0-11)[](#__codelineno-0-12)import argparse [](#__codelineno-0-13)import json [](#__codelineno-0-14)from argparse import Namespace [](#__codelineno-0-15)from collections.abc import Iterable [](#__codelineno-0-16)[](#__codelineno-0-17)import requests [](#__codelineno-0-18) [](#__codelineno-0-19)[](#__codelineno-0-20)def clear_line(n: int = 1) -> None: [](#__codelineno-0-21) LINE_UP = "\033[1A" [](#__codelineno-0-22) LINE_CLEAR = "\x1b[2K" [](#__codelineno-0-23) for _ in range(n): [](#__codelineno-0-24) print(LINE_UP, end=LINE_CLEAR, flush=True) [](#__codelineno-0-25) [](#__codelineno-0-26)[](#__codelineno-0-27)def post_http_request( [](#__codelineno-0-28) prompt: str, api_url: str, n: int = 1, stream: bool = False [](#__codelineno-0-29)) -> requests.Response: [](#__codelineno-0-30) headers = {"User-Agent": "Test Client"} [](#__codelineno-0-31) pload = { [](#__codelineno-0-32) "prompt": prompt, [](#__codelineno-0-33) "n": n, [](#__codelineno-0-34) "temperature": 0.0, [](#__codelineno-0-35) "max_tokens": 16, [](#__codelineno-0-36) "stream": stream, [](#__codelineno-0-37) } [](#__codelineno-0-38) response = requests.post(api_url, headers=headers, json=pload, stream=stream) [](#__codelineno-0-39) return response [](#__codelineno-0-40) [](#__codelineno-0-41)[](#__codelineno-0-42)def get_streaming_response(response: requests.Response) -> Iterable[list[str]]: [](#__codelineno-0-43) for chunk in response.iter_lines( [](#__codelineno-0-44) chunk_size=8192, decode_unicode=False, delimiter=b"\n" [](#__codelineno-0-45) ): [](#__codelineno-0-46) if chunk: [](#__codelineno-0-47) data = json.loads(chunk.decode("utf-8")) [](#__codelineno-0-48) output = data["text"] [](#__codelineno-0-49) yield output [](#__codelineno-0-50) [](#__codelineno-0-51)[](#__codelineno-0-52)def get_response(response: requests.Response) -> list[str]: [](#__codelineno-0-53) data = json.loads(response.content) [](#__codelineno-0-54) output = data["text"] [](#__codelineno-0-55) return output [](#__codelineno-0-56) [](#__codelineno-0-57)[](#__codelineno-0-58)def parse_args(): [](#__codelineno-0-59) parser = argparse.ArgumentParser() [](#__codelineno-0-60) parser.add_argument("--host", type=str, default="localhost") [](#__codelineno-0-61) parser.add_argument("--port", type=int, default=8000) [](#__codelineno-0-62) parser.add_argument("--n", type=int, default=1) [](#__codelineno-0-63) parser.add_argument("--prompt", type=str, default="San Francisco is a") [](#__codelineno-0-64) parser.add_argument("--stream", action="store_true") [](#__codelineno-0-65) return parser.parse_args() [](#__codelineno-0-66) [](#__codelineno-0-67)[](#__codelineno-0-68)def main(args: Namespace): [](#__codelineno-0-69) prompt = args.prompt [](#__codelineno-0-70) api_url = f"http://{args.host}:{args.port}/generate" [](#__codelineno-0-71) n = args.n [](#__codelineno-0-72) stream = args.stream [](#__codelineno-0-73) [](#__codelineno-0-74) print(f"Prompt: {prompt!r}\n", flush=True) [](#__codelineno-0-75) response = post_http_request(prompt, api_url, n, stream) [](#__codelineno-0-76) [](#__codelineno-0-77) if stream: [](#__codelineno-0-78) num_printed_lines = 0 [](#__codelineno-0-79) for h in get_streaming_response(response): [](#__codelineno-0-80) clear_line(num_printed_lines) [](#__codelineno-0-81) num_printed_lines = 0 [](#__codelineno-0-82) for i, line in enumerate(h): [](#__codelineno-0-83) num_printed_lines += 1 [](#__codelineno-0-84) print(f"Beam candidate {i}: {line!r}", flush=True) [](#__codelineno-0-85) else: [](#__codelineno-0-86) output = get_response(response) [](#__codelineno-0-87) for i, line in enumerate(output): [](#__codelineno-0-88) print(f"Beam candidate {i}: {line!r}", flush=True) [](#__codelineno-0-89) [](#__codelineno-0-90)[](#__codelineno-0-91)if __name__ == "__main__": [](#__codelineno-0-92) args = parse_args() [](#__codelineno-0-93) main(args)`` ## Gradio OpenAI Chatbot Webserver[¶](#gradio-openai-chatbot-webserver "Permanent link") ``[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)"""Example for starting a Gradio OpenAI Chatbot Webserver [](#__codelineno-1-4)Start vLLM API server: [](#__codelineno-1-5) vllm serve meta-llama/Llama-2-7b-chat-hf [](#__codelineno-1-6)[](#__codelineno-1-7)Start Gradio OpenAI Chatbot Webserver: [](#__codelineno-1-8) python examples/applications/chatbot/gradio_openai_chatbot_webserver.py \ [](#__codelineno-1-9) -m meta-llama/Llama-2-7b-chat-hf [](#__codelineno-1-10)[](#__codelineno-1-11)Note that `pip install --upgrade gradio` is needed to run this example. [](#__codelineno-1-12)More details: https://github.com/gradio-app/gradio [](#__codelineno-1-13)[](#__codelineno-1-14)If your antivirus software blocks the download of frpc for gradio, [](#__codelineno-1-15)you can install it manually by following these steps: [](#__codelineno-1-16)[](#__codelineno-1-17)1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.3/frpc_linux_amd64 [](#__codelineno-1-18)2. Rename the downloaded file to: frpc_linux_amd64_v0.3 [](#__codelineno-1-19)3. Move the file to this location: /home/user/.cache/huggingface/gradio/frpc [](#__codelineno-1-20)""" [](#__codelineno-1-21)[](#__codelineno-1-22)import argparse [](#__codelineno-1-23)[](#__codelineno-1-24)import gradio as gr [](#__codelineno-1-25)from openai import OpenAI [](#__codelineno-1-26) [](#__codelineno-1-27)[](#__codelineno-1-28)def predict(message, history, client, model_name, temp, stop_token_ids): [](#__codelineno-1-29) messages = [ [](#__codelineno-1-30) {"role": "system", "content": "You are a great AI assistant."}, [](#__codelineno-1-31) *history, [](#__codelineno-1-32) {"role": "user", "content": message}, [](#__codelineno-1-33) ] [](#__codelineno-1-34) [](#__codelineno-1-35) # Send request to OpenAI API (vLLM server) [](#__codelineno-1-36) stream = client.chat.completions.create( [](#__codelineno-1-37) model=model_name, [](#__codelineno-1-38) messages=messages, [](#__codelineno-1-39) temperature=temp, [](#__codelineno-1-40) stream=True, [](#__codelineno-1-41) extra_body={ [](#__codelineno-1-42) "repetition_penalty": 1, [](#__codelineno-1-43) "stop_token_ids": [int(id.strip()) for id in stop_token_ids.split(",")] [](#__codelineno-1-44) if stop_token_ids [](#__codelineno-1-45) else [], [](#__codelineno-1-46) }, [](#__codelineno-1-47) ) [](#__codelineno-1-48) [](#__codelineno-1-49) # Collect all chunks and concatenate them into a full message [](#__codelineno-1-50) full_message = "" [](#__codelineno-1-51) for chunk in stream: [](#__codelineno-1-52) full_message += chunk.choices[0].delta.content or "" [](#__codelineno-1-53) [](#__codelineno-1-54) # Return the full message as a single response [](#__codelineno-1-55) return full_message [](#__codelineno-1-56) [](#__codelineno-1-57)[](#__codelineno-1-58)def parse_args(): [](#__codelineno-1-59) parser = argparse.ArgumentParser( [](#__codelineno-1-60) description="Chatbot Interface with Customizable Parameters" [](#__codelineno-1-61) ) [](#__codelineno-1-62) parser.add_argument( [](#__codelineno-1-63) "--model-url", type=str, default="http://localhost:8000/v1", help="Model URL" [](#__codelineno-1-64) ) [](#__codelineno-1-65) parser.add_argument( [](#__codelineno-1-66) "-m", "--model", type=str, required=True, help="Model name for the chatbot" [](#__codelineno-1-67) ) [](#__codelineno-1-68) parser.add_argument( [](#__codelineno-1-69) "--temp", type=float, default=0.8, help="Temperature for text generation" [](#__codelineno-1-70) ) [](#__codelineno-1-71) parser.add_argument( [](#__codelineno-1-72) "--stop-token-ids", type=str, default="", help="Comma-separated stop token IDs" [](#__codelineno-1-73) ) [](#__codelineno-1-74) parser.add_argument("--host", type=str, default=None) [](#__codelineno-1-75) parser.add_argument("--port", type=int, default=8001) [](#__codelineno-1-76) return parser.parse_args() [](#__codelineno-1-77) [](#__codelineno-1-78)[](#__codelineno-1-79)def build_gradio_interface(client, model_name, temp, stop_token_ids): [](#__codelineno-1-80) def chat_predict(message, history): [](#__codelineno-1-81) return predict(message, history, client, model_name, temp, stop_token_ids) [](#__codelineno-1-82) [](#__codelineno-1-83) return gr.ChatInterface( [](#__codelineno-1-84) fn=chat_predict, [](#__codelineno-1-85) title="Chatbot Interface", [](#__codelineno-1-86) description="A simple chatbot powered by vLLM", [](#__codelineno-1-87) ) [](#__codelineno-1-88) [](#__codelineno-1-89)[](#__codelineno-1-90)def main(): [](#__codelineno-1-91) # Parse the arguments [](#__codelineno-1-92) args = parse_args() [](#__codelineno-1-93) [](#__codelineno-1-94) # Set OpenAI's API key and API base to use vLLM's API server [](#__codelineno-1-95) openai_api_key = "EMPTY" [](#__codelineno-1-96) openai_api_base = args.model_url [](#__codelineno-1-97) [](#__codelineno-1-98) # Create an OpenAI client [](#__codelineno-1-99) client = OpenAI(api_key=openai_api_key, base_url=openai_api_base) [](#__codelineno-1-100) [](#__codelineno-1-101) # Define the Gradio chatbot interface using the predict function [](#__codelineno-1-102) gradio_interface = build_gradio_interface( [](#__codelineno-1-103) client, args.model, args.temp, args.stop_token_ids [](#__codelineno-1-104) ) [](#__codelineno-1-105) [](#__codelineno-1-106) gradio_interface.queue().launch( [](#__codelineno-1-107) server_name=args.host, server_port=args.port, share=True [](#__codelineno-1-108) ) [](#__codelineno-1-109) [](#__codelineno-1-110)[](#__codelineno-1-111)if __name__ == "__main__": [](#__codelineno-1-112) main()`` ## Gradio Webserver[¶](#gradio-webserver "Permanent link") ``[](#__codelineno-2-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-2-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-2-3)"""Example for starting a Gradio Webserver [](#__codelineno-2-4)Start vLLM API server: [](#__codelineno-2-5) python -m vllm.entrypoints.api_server \ [](#__codelineno-2-6) --model meta-llama/Llama-2-7b-chat-hf [](#__codelineno-2-7)[](#__codelineno-2-8)Start Webserver: [](#__codelineno-2-9) python examples/applications/chatbot/gradio_webserver.py [](#__codelineno-2-10)[](#__codelineno-2-11)Note that `pip install --upgrade gradio` is needed to run this example. [](#__codelineno-2-12)More details: https://github.com/gradio-app/gradio [](#__codelineno-2-13)[](#__codelineno-2-14)If your antivirus software blocks the download of frpc for gradio, [](#__codelineno-2-15)you can install it manually by following these steps: [](#__codelineno-2-16)[](#__codelineno-2-17)1. Download this file: https://cdn-media.huggingface.co/frpc-gradio-0.3/frpc_linux_amd64 [](#__codelineno-2-18)2. Rename the downloaded file to: frpc_linux_amd64_v0.3 [](#__codelineno-2-19)3. Move the file to this location: /home/user/.cache/huggingface/gradio/frpc [](#__codelineno-2-20)""" [](#__codelineno-2-21)[](#__codelineno-2-22)import argparse [](#__codelineno-2-23)import json [](#__codelineno-2-24)[](#__codelineno-2-25)import gradio as gr [](#__codelineno-2-26)import requests [](#__codelineno-2-27) [](#__codelineno-2-28)[](#__codelineno-2-29)def http_bot(prompt): [](#__codelineno-2-30) headers = {"User-Agent": "vLLM Client"} [](#__codelineno-2-31) pload = { [](#__codelineno-2-32) "prompt": prompt, [](#__codelineno-2-33) "stream": True, [](#__codelineno-2-34) "max_tokens": 128, [](#__codelineno-2-35) } [](#__codelineno-2-36) response = requests.post(args.model_url, headers=headers, json=pload, stream=True) [](#__codelineno-2-37) [](#__codelineno-2-38) for chunk in response.iter_lines( [](#__codelineno-2-39) chunk_size=8192, decode_unicode=False, delimiter=b"\n" [](#__codelineno-2-40) ): [](#__codelineno-2-41) if chunk: [](#__codelineno-2-42) data = json.loads(chunk.decode("utf-8")) [](#__codelineno-2-43) output = data["text"][0] [](#__codelineno-2-44) yield output [](#__codelineno-2-45) [](#__codelineno-2-46)[](#__codelineno-2-47)def build_demo(): [](#__codelineno-2-48) with gr.Blocks() as demo: [](#__codelineno-2-49) gr.Markdown("# vLLM text completion demo\n") [](#__codelineno-2-50) inputbox = gr.Textbox(label="Input", placeholder="Enter text and press ENTER") [](#__codelineno-2-51) outputbox = gr.Textbox( [](#__codelineno-2-52) label="Output", placeholder="Generated result from the model" [](#__codelineno-2-53) ) [](#__codelineno-2-54) inputbox.submit(http_bot, [inputbox], [outputbox]) [](#__codelineno-2-55) return demo [](#__codelineno-2-56) [](#__codelineno-2-57)[](#__codelineno-2-58)def parse_args(): [](#__codelineno-2-59) parser = argparse.ArgumentParser() [](#__codelineno-2-60) parser.add_argument("--host", type=str, default=None) [](#__codelineno-2-61) parser.add_argument("--port", type=int, default=8001) [](#__codelineno-2-62) parser.add_argument( [](#__codelineno-2-63) "--model-url", type=str, default="http://localhost:8000/generate" [](#__codelineno-2-64) ) [](#__codelineno-2-65) return parser.parse_args() [](#__codelineno-2-66) [](#__codelineno-2-67)[](#__codelineno-2-68)def main(args): [](#__codelineno-2-69) demo = build_demo() [](#__codelineno-2-70) demo.queue().launch(server_name=args.host, server_port=args.port, share=True) [](#__codelineno-2-71) [](#__codelineno-2-72)[](#__codelineno-2-73)if __name__ == "__main__": [](#__codelineno-2-74) args = parse_args() [](#__codelineno-2-75) main(args)`` ## Streamlit OpenAI Chatbot Webserver[¶](#streamlit-openai-chatbot-webserver "Permanent link") `[](#__codelineno-3-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-3-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-3-3)""" [](#__codelineno-3-4)vLLM Chat Assistant - A Streamlit Web Interface [](#__codelineno-3-5)[](#__codelineno-3-6)A streamlined chat interface that quickly integrates [](#__codelineno-3-7)with vLLM API server. [](#__codelineno-3-8)[](#__codelineno-3-9)Features: [](#__codelineno-3-10)- Multiple chat sessions management [](#__codelineno-3-11)- Streaming response display [](#__codelineno-3-12)- Configurable API endpoint [](#__codelineno-3-13)- Real-time chat history [](#__codelineno-3-14)- Reasoning Display: Optional thinking process visualization [](#__codelineno-3-15)[](#__codelineno-3-16)Requirements: [](#__codelineno-3-17) pip install streamlit openai [](#__codelineno-3-18)[](#__codelineno-3-19)Usage: [](#__codelineno-3-20) # Start the app with default settings [](#__codelineno-3-21) streamlit run streamlit_openai_chatbot_webserver.py [](#__codelineno-3-22) [](#__codelineno-3-23) # Start with custom vLLM API endpoint [](#__codelineno-3-24) VLLM_API_BASE="http://your-server:8000/v1" \ [](#__codelineno-3-25) streamlit run streamlit_openai_chatbot_webserver.py [](#__codelineno-3-26) [](#__codelineno-3-27) # Enable debug mode [](#__codelineno-3-28) streamlit run streamlit_openai_chatbot_webserver.py \ [](#__codelineno-3-29) --logger.level=debug [](#__codelineno-3-30)""" [](#__codelineno-3-31)[](#__codelineno-3-32)import os [](#__codelineno-3-33)from datetime import datetime [](#__codelineno-3-34)[](#__codelineno-3-35)import streamlit as st [](#__codelineno-3-36)from openai import OpenAI [](#__codelineno-3-37)[](#__codelineno-3-38)# Get command line arguments from environment variables [](#__codelineno-3-39)openai_api_key = os.getenv("VLLM_API_KEY", "EMPTY") [](#__codelineno-3-40)openai_api_base = os.getenv("VLLM_API_BASE", "http://localhost:8000/v1") [](#__codelineno-3-41)[](#__codelineno-3-42)# Initialize session states for managing chat sessions [](#__codelineno-3-43)if "sessions" not in st.session_state: [](#__codelineno-3-44) st.session_state.sessions = {} [](#__codelineno-3-45)[](#__codelineno-3-46)if "current_session" not in st.session_state: [](#__codelineno-3-47) st.session_state.current_session = None [](#__codelineno-3-48)[](#__codelineno-3-49)if "messages" not in st.session_state: [](#__codelineno-3-50) st.session_state.messages = [] [](#__codelineno-3-51)[](#__codelineno-3-52)if "active_session" not in st.session_state: [](#__codelineno-3-53) st.session_state.active_session = None [](#__codelineno-3-54)[](#__codelineno-3-55)# Add new session state for reasoning [](#__codelineno-3-56)if "show_reasoning" not in st.session_state: [](#__codelineno-3-57) st.session_state.show_reasoning = {} [](#__codelineno-3-58)[](#__codelineno-3-59)# Initialize session state for API base URL [](#__codelineno-3-60)if "api_base_url" not in st.session_state: [](#__codelineno-3-61) st.session_state.api_base_url = openai_api_base [](#__codelineno-3-62) [](#__codelineno-3-63)[](#__codelineno-3-64)def create_new_chat_session(): [](#__codelineno-3-65) """Create a new chat session with timestamp as unique identifier. [](#__codelineno-3-66) [](#__codelineno-3-67) This function initializes a new chat session by: [](#__codelineno-3-68) 1. Generating a timestamp-based session ID [](#__codelineno-3-69) 2. Creating an empty message list for the new session [](#__codelineno-3-70) 3. Setting the new session as both current and active session [](#__codelineno-3-71) 4. Resetting the messages list for the new session [](#__codelineno-3-72) [](#__codelineno-3-73) Returns: [](#__codelineno-3-74) None [](#__codelineno-3-75) [](#__codelineno-3-76) Session State Updates: [](#__codelineno-3-77) - sessions: Adds new empty message list with timestamp key [](#__codelineno-3-78) - current_session: Sets to new session ID [](#__codelineno-3-79) - active_session: Sets to new session ID [](#__codelineno-3-80) - messages: Resets to empty list [](#__codelineno-3-81) """ [](#__codelineno-3-82) session_id = datetime.now().strftime("%Y-%m-%d %H:%M:%S") [](#__codelineno-3-83) st.session_state.sessions[session_id] = [] [](#__codelineno-3-84) st.session_state.current_session = session_id [](#__codelineno-3-85) st.session_state.active_session = session_id [](#__codelineno-3-86) st.session_state.messages = [] [](#__codelineno-3-87) [](#__codelineno-3-88)[](#__codelineno-3-89)def switch_to_chat_session(session_id): [](#__codelineno-3-90) """Switch the active chat context to a different session. [](#__codelineno-3-91) [](#__codelineno-3-92) Args: [](#__codelineno-3-93) session_id (str): The timestamp ID of the session to switch to [](#__codelineno-3-94) [](#__codelineno-3-95) This function handles chat session switching by: [](#__codelineno-3-96) 1. Setting the specified session as current [](#__codelineno-3-97) 2. Updating the active session marker [](#__codelineno-3-98) 3. Loading the messages history from the specified session [](#__codelineno-3-99) [](#__codelineno-3-100) Session State Updates: [](#__codelineno-3-101) - current_session: Updated to specified session_id [](#__codelineno-3-102) - active_session: Updated to specified session_id [](#__codelineno-3-103) - messages: Loaded from sessions[session_id] [](#__codelineno-3-104) """ [](#__codelineno-3-105) st.session_state.current_session = session_id [](#__codelineno-3-106) st.session_state.active_session = session_id [](#__codelineno-3-107) st.session_state.messages = st.session_state.sessions[session_id] [](#__codelineno-3-108) [](#__codelineno-3-109)[](#__codelineno-3-110)def get_llm_response(messages, model, reason, content_ph=None, reasoning_ph=None): [](#__codelineno-3-111) """Generate and stream LLM response with optional reasoning process. [](#__codelineno-3-112) [](#__codelineno-3-113) Args: [](#__codelineno-3-114) messages (list): List of conversation message dicts with 'role' and 'content' [](#__codelineno-3-115) model (str): The model identifier to use for generation [](#__codelineno-3-116) reason (bool): Whether to enable and display reasoning process [](#__codelineno-3-117) content_ph (streamlit.empty): Placeholder for streaming response content [](#__codelineno-3-118) reasoning_ph (streamlit.empty): Placeholder for streaming reasoning process [](#__codelineno-3-119) [](#__codelineno-3-120) Returns: [](#__codelineno-3-121) tuple: (str, str) [](#__codelineno-3-122) - First string contains the complete response text [](#__codelineno-3-123) - Second string contains the complete reasoning text (if enabled) [](#__codelineno-3-124) [](#__codelineno-3-125) Features: [](#__codelineno-3-126) - Streams both reasoning and response text in real-time [](#__codelineno-3-127) - Handles model API errors gracefully [](#__codelineno-3-128) - Supports live updating of thinking process [](#__codelineno-3-129) - Maintains separate content and reasoning displays [](#__codelineno-3-130) [](#__codelineno-3-131) Raises: [](#__codelineno-3-132) Exception: Wrapped in error message if API call fails [](#__codelineno-3-133) [](#__codelineno-3-134) Note: [](#__codelineno-3-135) The function uses streamlit placeholders for live updates. [](#__codelineno-3-136) When reason=True, the reasoning process appears above the response. [](#__codelineno-3-137) """ [](#__codelineno-3-138) full_text = "" [](#__codelineno-3-139) think_text = "" [](#__codelineno-3-140) live_think = None [](#__codelineno-3-141) # Build request parameters [](#__codelineno-3-142) params = {"model": model, "messages": messages, "stream": True} [](#__codelineno-3-143) if reason: [](#__codelineno-3-144) params["extra_body"] = {"chat_template_kwargs": {"enable_thinking": True}} [](#__codelineno-3-145) [](#__codelineno-3-146) try: [](#__codelineno-3-147) response = client.chat.completions.create(**params) [](#__codelineno-3-148) if isinstance(response, str): [](#__codelineno-3-149) if content_ph: [](#__codelineno-3-150) content_ph.markdown(response) [](#__codelineno-3-151) return response, "" [](#__codelineno-3-152) [](#__codelineno-3-153) # Prepare reasoning expander above content [](#__codelineno-3-154) if reason and reasoning_ph: [](#__codelineno-3-155) exp = reasoning_ph.expander("💭 Thinking Process (live)", expanded=True) [](#__codelineno-3-156) live_think = exp.empty() [](#__codelineno-3-157) [](#__codelineno-3-158) # Stream chunks [](#__codelineno-3-159) for chunk in response: [](#__codelineno-3-160) delta = chunk.choices[0].delta [](#__codelineno-3-161) # Stream reasoning first [](#__codelineno-3-162) if reason and hasattr(delta, "reasoning") and live_think: [](#__codelineno-3-163) rc = delta.reasoning [](#__codelineno-3-164) if rc: [](#__codelineno-3-165) think_text += rc [](#__codelineno-3-166) live_think.markdown(think_text + "▌") [](#__codelineno-3-167) # Then stream content [](#__codelineno-3-168) if hasattr(delta, "content") and delta.content and content_ph: [](#__codelineno-3-169) full_text += delta.content [](#__codelineno-3-170) content_ph.markdown(full_text + "▌") [](#__codelineno-3-171) [](#__codelineno-3-172) # Finalize displays: reasoning remains above, content below [](#__codelineno-3-173) if reason and live_think: [](#__codelineno-3-174) live_think.markdown(think_text) [](#__codelineno-3-175) if content_ph: [](#__codelineno-3-176) content_ph.markdown(full_text) [](#__codelineno-3-177) [](#__codelineno-3-178) return full_text, think_text [](#__codelineno-3-179) except Exception as e: [](#__codelineno-3-180) st.error(f"Error details: {str(e)}") [](#__codelineno-3-181) return f"Error: {str(e)}", "" [](#__codelineno-3-182) [](#__codelineno-3-183)[](#__codelineno-3-184)# Sidebar - API Settings first [](#__codelineno-3-185)st.sidebar.title("API Settings") [](#__codelineno-3-186)new_api_base = st.sidebar.text_input( [](#__codelineno-3-187) "API Base URL:", value=st.session_state.api_base_url [](#__codelineno-3-188)) [](#__codelineno-3-189)if new_api_base != st.session_state.api_base_url: [](#__codelineno-3-190) st.session_state.api_base_url = new_api_base [](#__codelineno-3-191) st.rerun() [](#__codelineno-3-192)[](#__codelineno-3-193)st.sidebar.divider() [](#__codelineno-3-194)[](#__codelineno-3-195)# Sidebar - Session Management [](#__codelineno-3-196)st.sidebar.title("Chat Sessions") [](#__codelineno-3-197)if st.sidebar.button("New Session"): [](#__codelineno-3-198) create_new_chat_session() [](#__codelineno-3-199) [](#__codelineno-3-200)[](#__codelineno-3-201)# Display all sessions in reverse chronological order [](#__codelineno-3-202)for session_id in sorted(st.session_state.sessions.keys(), reverse=True): [](#__codelineno-3-203) # Mark the active session with a pinned button [](#__codelineno-3-204) if session_id == st.session_state.active_session: [](#__codelineno-3-205) st.sidebar.button( [](#__codelineno-3-206) f"📍 {session_id}", [](#__codelineno-3-207) key=session_id, [](#__codelineno-3-208) type="primary", [](#__codelineno-3-209) on_click=switch_to_chat_session, [](#__codelineno-3-210) args=(session_id,), [](#__codelineno-3-211) ) [](#__codelineno-3-212) else: [](#__codelineno-3-213) st.sidebar.button( [](#__codelineno-3-214) f"Session {session_id}", [](#__codelineno-3-215) key=session_id, [](#__codelineno-3-216) on_click=switch_to_chat_session, [](#__codelineno-3-217) args=(session_id,), [](#__codelineno-3-218) ) [](#__codelineno-3-219)[](#__codelineno-3-220)# Main interface [](#__codelineno-3-221)st.title("vLLM Chat Assistant") [](#__codelineno-3-222)[](#__codelineno-3-223)# Initialize OpenAI client with API settings [](#__codelineno-3-224)client = OpenAI(api_key=openai_api_key, base_url=st.session_state.api_base_url) [](#__codelineno-3-225)[](#__codelineno-3-226)# Get and display current model id [](#__codelineno-3-227)models = client.models.list() [](#__codelineno-3-228)model = models.data[0].id [](#__codelineno-3-229)st.markdown(f"**Model**: {model}") [](#__codelineno-3-230)[](#__codelineno-3-231)# Initialize first session if none exists [](#__codelineno-3-232)if st.session_state.current_session is None: [](#__codelineno-3-233) create_new_chat_session() [](#__codelineno-3-234) st.session_state.active_session = st.session_state.current_session [](#__codelineno-3-235)[](#__codelineno-3-236)# Update the chat history display section [](#__codelineno-3-237)for idx, msg in enumerate(st.session_state.messages): [](#__codelineno-3-238) # Render user messages normally [](#__codelineno-3-239) if msg["role"] == "user": [](#__codelineno-3-240) with st.chat_message("user"): [](#__codelineno-3-241) st.write(msg["content"]) [](#__codelineno-3-242) # Render assistant messages with reasoning above [](#__codelineno-3-243) else: [](#__codelineno-3-244) # If reasoning exists for this assistant message, show it above the content [](#__codelineno-3-245) if idx in st.session_state.show_reasoning: [](#__codelineno-3-246) with st.expander("💭 Thinking Process", expanded=False): [](#__codelineno-3-247) st.markdown(st.session_state.show_reasoning[idx]) [](#__codelineno-3-248) with st.chat_message("assistant"): [](#__codelineno-3-249) st.write(msg["content"]) [](#__codelineno-3-250) [](#__codelineno-3-251)[](#__codelineno-3-252)# Setup & Cache reasoning support check [](#__codelineno-3-253)@st.cache_data(show_spinner=False) [](#__codelineno-3-254)def server_supports_reasoning(): [](#__codelineno-3-255) """Check if the current model supports reasoning capability. [](#__codelineno-3-256) [](#__codelineno-3-257) Returns: [](#__codelineno-3-258) bool: True if the model supports reasoning, False otherwise [](#__codelineno-3-259) """ [](#__codelineno-3-260) resp = client.chat.completions.create( [](#__codelineno-3-261) model=model, [](#__codelineno-3-262) messages=[{"role": "user", "content": "Hi"}], [](#__codelineno-3-263) stream=False, [](#__codelineno-3-264) ) [](#__codelineno-3-265) return hasattr(resp.choices[0].message, "reasoning") and bool( [](#__codelineno-3-266) resp.choices[0].message.reasoning [](#__codelineno-3-267) ) [](#__codelineno-3-268) [](#__codelineno-3-269)[](#__codelineno-3-270)# Check support [](#__codelineno-3-271)supports_reasoning = server_supports_reasoning() [](#__codelineno-3-272)[](#__codelineno-3-273)# Add reasoning toggle in sidebar if supported [](#__codelineno-3-274)reason = False # Default to False [](#__codelineno-3-275)if supports_reasoning: [](#__codelineno-3-276) reason = st.sidebar.checkbox("Enable Reasoning", value=False) [](#__codelineno-3-277)else: [](#__codelineno-3-278) st.sidebar.markdown( [](#__codelineno-3-279) "Reasoning unavailable for this model.", [](#__codelineno-3-280) unsafe_allow_html=True, [](#__codelineno-3-281) ) [](#__codelineno-3-282) # reason remains False [](#__codelineno-3-283)[](#__codelineno-3-284)# Update the input handling section [](#__codelineno-3-285)if prompt := st.chat_input("Type your message here..."): [](#__codelineno-3-286) # Save and display user message [](#__codelineno-3-287) st.session_state.messages.append({"role": "user", "content": prompt}) [](#__codelineno-3-288) st.session_state.sessions[st.session_state.current_session] = ( [](#__codelineno-3-289) st.session_state.messages [](#__codelineno-3-290) ) [](#__codelineno-3-291) with st.chat_message("user"): [](#__codelineno-3-292) st.write(prompt) [](#__codelineno-3-293) [](#__codelineno-3-294) # Prepare LLM messages [](#__codelineno-3-295) msgs = [ [](#__codelineno-3-296) {"role": m["role"], "content": m["content"]} for m in st.session_state.messages [](#__codelineno-3-297) ] [](#__codelineno-3-298) [](#__codelineno-3-299) # Stream assistant response [](#__codelineno-3-300) with st.chat_message("assistant"): [](#__codelineno-3-301) # Placeholders: reasoning above, content below [](#__codelineno-3-302) reason_ph = st.empty() [](#__codelineno-3-303) content_ph = st.empty() [](#__codelineno-3-304) full, think = get_llm_response(msgs, model, reason, content_ph, reason_ph) [](#__codelineno-3-305) # Determine index for this new assistant message [](#__codelineno-3-306) message_index = len(st.session_state.messages) [](#__codelineno-3-307) # Save assistant reply [](#__codelineno-3-308) st.session_state.messages.append({"role": "assistant", "content": full}) [](#__codelineno-3-309) # Persist reasoning in session state if any [](#__codelineno-3-310) if reason and think: [](#__codelineno-3-311) st.session_state.show_reasoning[message_index] = think` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/applications/rag.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/applications/rag](https://github.com/vllm-project/vllm/tree/main/examples/applications/rag). ## Retrieval Augmented Generation With Langchain[¶](#retrieval-augmented-generation-with-langchain "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Retrieval Augmented Generation (RAG) Implementation with Langchain [](#__codelineno-0-5)================================================================== [](#__codelineno-0-6)[](#__codelineno-0-7)This script demonstrates a RAG implementation using LangChain, Milvus [](#__codelineno-0-8)and vLLM. RAG enhances LLM responses by retrieving relevant context [](#__codelineno-0-9)from a document collection. [](#__codelineno-0-10)[](#__codelineno-0-11)Features: [](#__codelineno-0-12)- Web content loading and chunking [](#__codelineno-0-13)- Vector storage with Milvus [](#__codelineno-0-14)- Embedding generation with vLLM [](#__codelineno-0-15)- Question answering with context [](#__codelineno-0-16)[](#__codelineno-0-17)Prerequisites: [](#__codelineno-0-18)1. Install dependencies: [](#__codelineno-0-19) pip install -U vllm \ [](#__codelineno-0-20) langchain_milvus langchain_openai \ [](#__codelineno-0-21) langchain_community beautifulsoup4 \ [](#__codelineno-0-22) langchain-text-splitters [](#__codelineno-0-23)[](#__codelineno-0-24)2. Start services: [](#__codelineno-0-25) # Start embedding service (port 8000) [](#__codelineno-0-26) vllm serve ssmits/Qwen2-7B-Instruct-embed-base [](#__codelineno-0-27) [](#__codelineno-0-28) # Start chat service (port 8001) [](#__codelineno-0-29) vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001 [](#__codelineno-0-30)[](#__codelineno-0-31)Usage: [](#__codelineno-0-32) python retrieval_augmented_generation_with_langchain.py [](#__codelineno-0-33)[](#__codelineno-0-34)Notes: [](#__codelineno-0-35) - Ensure both vLLM services are running before executing [](#__codelineno-0-36) - Default ports: 8000 (embedding), 8001 (chat) [](#__codelineno-0-37) - First run may take time to download models [](#__codelineno-0-38)""" [](#__codelineno-0-39)[](#__codelineno-0-40)import argparse [](#__codelineno-0-41)from argparse import Namespace [](#__codelineno-0-42)from typing import Any [](#__codelineno-0-43)[](#__codelineno-0-44)from langchain_community.document_loaders import WebBaseLoader [](#__codelineno-0-45)from langchain_core.documents import Document [](#__codelineno-0-46)from langchain_core.output_parsers import StrOutputParser [](#__codelineno-0-47)from langchain_core.prompts import PromptTemplate [](#__codelineno-0-48)from langchain_core.runnables import RunnablePassthrough [](#__codelineno-0-49)from langchain_milvus import Milvus [](#__codelineno-0-50)from langchain_openai import ChatOpenAI, OpenAIEmbeddings [](#__codelineno-0-51)from langchain_text_splitters import RecursiveCharacterTextSplitter [](#__codelineno-0-52) [](#__codelineno-0-53)[](#__codelineno-0-54)def load_and_split_documents(config: dict[str, Any]): [](#__codelineno-0-55) """ [](#__codelineno-0-56) Load and split documents from web URL [](#__codelineno-0-57) """ [](#__codelineno-0-58) try: [](#__codelineno-0-59) loader = WebBaseLoader(web_paths=(config["url"],)) [](#__codelineno-0-60) docs = loader.load() [](#__codelineno-0-61) [](#__codelineno-0-62) text_splitter = RecursiveCharacterTextSplitter( [](#__codelineno-0-63) chunk_size=config["chunk_size"], [](#__codelineno-0-64) chunk_overlap=config["chunk_overlap"], [](#__codelineno-0-65) ) [](#__codelineno-0-66) return text_splitter.split_documents(docs) [](#__codelineno-0-67) except Exception as e: [](#__codelineno-0-68) print(f"Error loading document from {config['url']}: {str(e)}") [](#__codelineno-0-69) raise [](#__codelineno-0-70) [](#__codelineno-0-71)[](#__codelineno-0-72)def init_vectorstore(config: dict[str, Any], documents: list[Document]): [](#__codelineno-0-73) """ [](#__codelineno-0-74) Initialize vector store with documents [](#__codelineno-0-75) """ [](#__codelineno-0-76) return Milvus.from_documents( [](#__codelineno-0-77) documents=documents, [](#__codelineno-0-78) embedding=OpenAIEmbeddings( [](#__codelineno-0-79) model=config["embedding_model"], [](#__codelineno-0-80) openai_api_key=config["vllm_api_key"], [](#__codelineno-0-81) openai_api_base=config["vllm_embedding_endpoint"], [](#__codelineno-0-82) ), [](#__codelineno-0-83) connection_args={"uri": config["uri"]}, [](#__codelineno-0-84) drop_old=True, [](#__codelineno-0-85) ) [](#__codelineno-0-86) [](#__codelineno-0-87)[](#__codelineno-0-88)def init_llm(config: dict[str, Any]): [](#__codelineno-0-89) """ [](#__codelineno-0-90) Initialize llm [](#__codelineno-0-91) """ [](#__codelineno-0-92) return ChatOpenAI( [](#__codelineno-0-93) model=config["chat_model"], [](#__codelineno-0-94) openai_api_key=config["vllm_api_key"], [](#__codelineno-0-95) openai_api_base=config["vllm_chat_endpoint"], [](#__codelineno-0-96) ) [](#__codelineno-0-97) [](#__codelineno-0-98)[](#__codelineno-0-99)def get_qa_prompt(): [](#__codelineno-0-100) """ [](#__codelineno-0-101) Get question answering prompt template [](#__codelineno-0-102) """ [](#__codelineno-0-103) template = """You are an assistant for question-answering tasks. [](#__codelineno-0-104)Use the following pieces of retrieved context to answer the question. [](#__codelineno-0-105)If you don't know the answer, just say that you don't know. [](#__codelineno-0-106)Use three sentences maximum and keep the answer concise. [](#__codelineno-0-107)Question: {question} [](#__codelineno-0-108)Context: {context} [](#__codelineno-0-109)Answer: [](#__codelineno-0-110)""" [](#__codelineno-0-111) return PromptTemplate.from_template(template) [](#__codelineno-0-112) [](#__codelineno-0-113)[](#__codelineno-0-114)def format_docs(docs: list[Document]): [](#__codelineno-0-115) """ [](#__codelineno-0-116) Format documents for prompt [](#__codelineno-0-117) """ [](#__codelineno-0-118) return "\n\n".join(doc.page_content for doc in docs) [](#__codelineno-0-119) [](#__codelineno-0-120)[](#__codelineno-0-121)def create_qa_chain(retriever: Any, llm: ChatOpenAI, prompt: PromptTemplate): [](#__codelineno-0-122) """ [](#__codelineno-0-123) Set up question answering chain [](#__codelineno-0-124) """ [](#__codelineno-0-125) return ( [](#__codelineno-0-126) { [](#__codelineno-0-127) "context": retriever | format_docs, [](#__codelineno-0-128) "question": RunnablePassthrough(), [](#__codelineno-0-129) } [](#__codelineno-0-130) | prompt [](#__codelineno-0-131) | llm [](#__codelineno-0-132) | StrOutputParser() [](#__codelineno-0-133) ) [](#__codelineno-0-134) [](#__codelineno-0-135)[](#__codelineno-0-136)def get_parser() -> argparse.ArgumentParser: [](#__codelineno-0-137) """ [](#__codelineno-0-138) Parse command line arguments [](#__codelineno-0-139) """ [](#__codelineno-0-140) parser = argparse.ArgumentParser(description="RAG with vLLM and langchain") [](#__codelineno-0-141) [](#__codelineno-0-142) # Add command line arguments [](#__codelineno-0-143) parser.add_argument( [](#__codelineno-0-144) "--vllm-api-key", default="EMPTY", help="API key for vLLM compatible services" [](#__codelineno-0-145) ) [](#__codelineno-0-146) parser.add_argument( [](#__codelineno-0-147) "--vllm-embedding-endpoint", [](#__codelineno-0-148) default="http://localhost:8000/v1", [](#__codelineno-0-149) help="Base URL for embedding service", [](#__codelineno-0-150) ) [](#__codelineno-0-151) parser.add_argument( [](#__codelineno-0-152) "--vllm-chat-endpoint", [](#__codelineno-0-153) default="http://localhost:8001/v1", [](#__codelineno-0-154) help="Base URL for chat service", [](#__codelineno-0-155) ) [](#__codelineno-0-156) parser.add_argument("--uri", default="./milvus.db", help="URI for Milvus database") [](#__codelineno-0-157) parser.add_argument( [](#__codelineno-0-158) "--url", [](#__codelineno-0-159) default=("https://docs.vllm.ai/en/latest/getting_started/quickstart.html"), [](#__codelineno-0-160) help="URL of the document to process", [](#__codelineno-0-161) ) [](#__codelineno-0-162) parser.add_argument( [](#__codelineno-0-163) "--embedding-model", [](#__codelineno-0-164) default="ssmits/Qwen2-7B-Instruct-embed-base", [](#__codelineno-0-165) help="Model name for embeddings", [](#__codelineno-0-166) ) [](#__codelineno-0-167) parser.add_argument( [](#__codelineno-0-168) "--chat-model", default="qwen/Qwen1.5-0.5B-Chat", help="Model name for chat" [](#__codelineno-0-169) ) [](#__codelineno-0-170) parser.add_argument( [](#__codelineno-0-171) "-i", "--interactive", action="store_true", help="Enable interactive Q&A mode" [](#__codelineno-0-172) ) [](#__codelineno-0-173) parser.add_argument( [](#__codelineno-0-174) "-k", "--top-k", type=int, default=3, help="Number of top results to retrieve" [](#__codelineno-0-175) ) [](#__codelineno-0-176) parser.add_argument( [](#__codelineno-0-177) "-c", [](#__codelineno-0-178) "--chunk-size", [](#__codelineno-0-179) type=int, [](#__codelineno-0-180) default=1000, [](#__codelineno-0-181) help="Chunk size for document splitting", [](#__codelineno-0-182) ) [](#__codelineno-0-183) parser.add_argument( [](#__codelineno-0-184) "-o", [](#__codelineno-0-185) "--chunk-overlap", [](#__codelineno-0-186) type=int, [](#__codelineno-0-187) default=200, [](#__codelineno-0-188) help="Chunk overlap for document splitting", [](#__codelineno-0-189) ) [](#__codelineno-0-190) [](#__codelineno-0-191) return parser [](#__codelineno-0-192) [](#__codelineno-0-193)[](#__codelineno-0-194)def init_config(args: Namespace): [](#__codelineno-0-195) """ [](#__codelineno-0-196) Initialize configuration settings from command line arguments [](#__codelineno-0-197) """ [](#__codelineno-0-198) [](#__codelineno-0-199) return { [](#__codelineno-0-200) "vllm_api_key": args.vllm_api_key, [](#__codelineno-0-201) "vllm_embedding_endpoint": args.vllm_embedding_endpoint, [](#__codelineno-0-202) "vllm_chat_endpoint": args.vllm_chat_endpoint, [](#__codelineno-0-203) "uri": args.uri, [](#__codelineno-0-204) "embedding_model": args.embedding_model, [](#__codelineno-0-205) "chat_model": args.chat_model, [](#__codelineno-0-206) "url": args.url, [](#__codelineno-0-207) "chunk_size": args.chunk_size, [](#__codelineno-0-208) "chunk_overlap": args.chunk_overlap, [](#__codelineno-0-209) "top_k": args.top_k, [](#__codelineno-0-210) } [](#__codelineno-0-211) [](#__codelineno-0-212)[](#__codelineno-0-213)def main(): [](#__codelineno-0-214) # Parse command line arguments [](#__codelineno-0-215) args = get_parser().parse_args() [](#__codelineno-0-216) [](#__codelineno-0-217) # Initialize configuration [](#__codelineno-0-218) config = init_config(args) [](#__codelineno-0-219) [](#__codelineno-0-220) # Load and split documents [](#__codelineno-0-221) documents = load_and_split_documents(config) [](#__codelineno-0-222) [](#__codelineno-0-223) # Initialize vector store and retriever [](#__codelineno-0-224) vectorstore = init_vectorstore(config, documents) [](#__codelineno-0-225) retriever = vectorstore.as_retriever(search_kwargs={"k": config["top_k"]}) [](#__codelineno-0-226) [](#__codelineno-0-227) # Initialize llm and prompt [](#__codelineno-0-228) llm = init_llm(config) [](#__codelineno-0-229) prompt = get_qa_prompt() [](#__codelineno-0-230) [](#__codelineno-0-231) # Set up QA chain [](#__codelineno-0-232) qa_chain = create_qa_chain(retriever, llm, prompt) [](#__codelineno-0-233) [](#__codelineno-0-234) # Interactive mode [](#__codelineno-0-235) if args.interactive: [](#__codelineno-0-236) print("\nWelcome to Interactive Q&A System!") [](#__codelineno-0-237) print("Enter 'q' or 'quit' to exit.") [](#__codelineno-0-238) [](#__codelineno-0-239) while True: [](#__codelineno-0-240) question = input("\nPlease enter your question: ") [](#__codelineno-0-241) if question.lower() in ["q", "quit"]: [](#__codelineno-0-242) print("\nThank you for using! Goodbye!") [](#__codelineno-0-243) break [](#__codelineno-0-244) [](#__codelineno-0-245) output = qa_chain.invoke(question) [](#__codelineno-0-246) print(output) [](#__codelineno-0-247) else: [](#__codelineno-0-248) # Default single question mode [](#__codelineno-0-249) question = "How to install vLLM?" [](#__codelineno-0-250) output = qa_chain.invoke(question) [](#__codelineno-0-251) print("-" * 50) [](#__codelineno-0-252) print(output) [](#__codelineno-0-253) print("-" * 50) [](#__codelineno-0-254) [](#__codelineno-0-255)[](#__codelineno-0-256)if __name__ == "__main__": [](#__codelineno-0-257) main()` ## Retrieval Augmented Generation With Llamaindex[¶](#retrieval-augmented-generation-with-llamaindex "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)RAG (Retrieval Augmented Generation) Implementation with LlamaIndex [](#__codelineno-1-5)================================================================ [](#__codelineno-1-6)[](#__codelineno-1-7)This script demonstrates a RAG system using: [](#__codelineno-1-8)- LlamaIndex: For document indexing and retrieval [](#__codelineno-1-9)- Milvus: As vector store backend [](#__codelineno-1-10)- vLLM: For embedding and text generation [](#__codelineno-1-11)[](#__codelineno-1-12)Features: [](#__codelineno-1-13)1. Document Loading & Processing [](#__codelineno-1-14)2. Embedding & Storage [](#__codelineno-1-15)3. Query Processing [](#__codelineno-1-16)[](#__codelineno-1-17)Requirements: [](#__codelineno-1-18)1. Install dependencies: [](#__codelineno-1-19)pip install llama-index llama-index-readers-web \ [](#__codelineno-1-20) llama-index-llms-openai-like \ [](#__codelineno-1-21) llama-index-embeddings-openai-like \ [](#__codelineno-1-22) llama-index-vector-stores-milvus \ [](#__codelineno-1-23)[](#__codelineno-1-24)2. Start services: [](#__codelineno-1-25) # Start embedding service (port 8000) [](#__codelineno-1-26) vllm serve ssmits/Qwen2-7B-Instruct-embed-base [](#__codelineno-1-27) [](#__codelineno-1-28) # Start chat service (port 8001) [](#__codelineno-1-29) vllm serve qwen/Qwen1.5-0.5B-Chat --port 8001 [](#__codelineno-1-30)[](#__codelineno-1-31)Usage: [](#__codelineno-1-32) python retrieval_augmented_generation_with_llamaindex.py [](#__codelineno-1-33)[](#__codelineno-1-34)Notes: [](#__codelineno-1-35) - Ensure both vLLM services are running before executing [](#__codelineno-1-36) - Default ports: 8000 (embedding), 8001 (chat) [](#__codelineno-1-37) - First run may take time to download models [](#__codelineno-1-38)""" [](#__codelineno-1-39)[](#__codelineno-1-40)import argparse [](#__codelineno-1-41)from argparse import Namespace [](#__codelineno-1-42)from typing import Any [](#__codelineno-1-43)[](#__codelineno-1-44)from llama_index.core import Settings, StorageContext, VectorStoreIndex [](#__codelineno-1-45)from llama_index.core.node_parser import SentenceSplitter [](#__codelineno-1-46)from llama_index.embeddings.openai_like import OpenAILikeEmbedding [](#__codelineno-1-47)from llama_index.llms.openai_like import OpenAILike [](#__codelineno-1-48)from llama_index.readers.web import SimpleWebPageReader [](#__codelineno-1-49)from llama_index.vector_stores.milvus import MilvusVectorStore [](#__codelineno-1-50) [](#__codelineno-1-51)[](#__codelineno-1-52)def init_config(args: Namespace): [](#__codelineno-1-53) """Initialize configuration with command line arguments""" [](#__codelineno-1-54) return { [](#__codelineno-1-55) "url": args.url, [](#__codelineno-1-56) "embedding_model": args.embedding_model, [](#__codelineno-1-57) "chat_model": args.chat_model, [](#__codelineno-1-58) "vllm_api_key": args.vllm_api_key, [](#__codelineno-1-59) "embedding_endpoint": args.embedding_endpoint, [](#__codelineno-1-60) "chat_endpoint": args.chat_endpoint, [](#__codelineno-1-61) "db_path": args.db_path, [](#__codelineno-1-62) "chunk_size": args.chunk_size, [](#__codelineno-1-63) "chunk_overlap": args.chunk_overlap, [](#__codelineno-1-64) "top_k": args.top_k, [](#__codelineno-1-65) } [](#__codelineno-1-66) [](#__codelineno-1-67)[](#__codelineno-1-68)def load_documents(url: str) -> list: [](#__codelineno-1-69) """Load and process web documents""" [](#__codelineno-1-70) return SimpleWebPageReader(html_to_text=True).load_data([url]) [](#__codelineno-1-71) [](#__codelineno-1-72)[](#__codelineno-1-73)def setup_models(config: dict[str, Any]): [](#__codelineno-1-74) """Configure embedding and chat models""" [](#__codelineno-1-75) Settings.embed_model = OpenAILikeEmbedding( [](#__codelineno-1-76) api_base=config["embedding_endpoint"], [](#__codelineno-1-77) api_key=config["vllm_api_key"], [](#__codelineno-1-78) model_name=config["embedding_model"], [](#__codelineno-1-79) ) [](#__codelineno-1-80) [](#__codelineno-1-81) Settings.llm = OpenAILike( [](#__codelineno-1-82) model=config["chat_model"], [](#__codelineno-1-83) api_key=config["vllm_api_key"], [](#__codelineno-1-84) api_base=config["chat_endpoint"], [](#__codelineno-1-85) context_window=128000, [](#__codelineno-1-86) is_chat_model=True, [](#__codelineno-1-87) is_function_calling_model=False, [](#__codelineno-1-88) ) [](#__codelineno-1-89) [](#__codelineno-1-90) Settings.transformations = [ [](#__codelineno-1-91) SentenceSplitter( [](#__codelineno-1-92) chunk_size=config["chunk_size"], [](#__codelineno-1-93) chunk_overlap=config["chunk_overlap"], [](#__codelineno-1-94) ) [](#__codelineno-1-95) ] [](#__codelineno-1-96) [](#__codelineno-1-97)[](#__codelineno-1-98)def setup_vector_store(db_path: str) -> MilvusVectorStore: [](#__codelineno-1-99) """Initialize vector store""" [](#__codelineno-1-100) sample_emb = Settings.embed_model.get_text_embedding("test") [](#__codelineno-1-101) print(f"Embedding dimension: {len(sample_emb)}") [](#__codelineno-1-102) return MilvusVectorStore(uri=db_path, dim=len(sample_emb), overwrite=True) [](#__codelineno-1-103) [](#__codelineno-1-104)[](#__codelineno-1-105)def create_index(documents: list, vector_store: MilvusVectorStore): [](#__codelineno-1-106) """Create document index""" [](#__codelineno-1-107) storage_context = StorageContext.from_defaults(vector_store=vector_store) [](#__codelineno-1-108) return VectorStoreIndex.from_documents( [](#__codelineno-1-109) documents, [](#__codelineno-1-110) storage_context=storage_context, [](#__codelineno-1-111) ) [](#__codelineno-1-112) [](#__codelineno-1-113)[](#__codelineno-1-114)def query_document(index: VectorStoreIndex, question: str, top_k: int): [](#__codelineno-1-115) """Query document with given question""" [](#__codelineno-1-116) query_engine = index.as_query_engine(similarity_top_k=top_k) [](#__codelineno-1-117) return query_engine.query(question) [](#__codelineno-1-118) [](#__codelineno-1-119)[](#__codelineno-1-120)def get_parser() -> argparse.ArgumentParser: [](#__codelineno-1-121) """Parse command line arguments""" [](#__codelineno-1-122) parser = argparse.ArgumentParser(description="RAG with vLLM and LlamaIndex") [](#__codelineno-1-123) [](#__codelineno-1-124) # Add command line arguments [](#__codelineno-1-125) parser.add_argument( [](#__codelineno-1-126) "--url", [](#__codelineno-1-127) default=("https://docs.vllm.ai/en/latest/getting_started/quickstart.html"), [](#__codelineno-1-128) help="URL of the document to process", [](#__codelineno-1-129) ) [](#__codelineno-1-130) parser.add_argument( [](#__codelineno-1-131) "--embedding-model", [](#__codelineno-1-132) default="ssmits/Qwen2-7B-Instruct-embed-base", [](#__codelineno-1-133) help="Model name for embeddings", [](#__codelineno-1-134) ) [](#__codelineno-1-135) parser.add_argument( [](#__codelineno-1-136) "--chat-model", default="qwen/Qwen1.5-0.5B-Chat", help="Model name for chat" [](#__codelineno-1-137) ) [](#__codelineno-1-138) parser.add_argument( [](#__codelineno-1-139) "--vllm-api-key", default="EMPTY", help="API key for vLLM compatible services" [](#__codelineno-1-140) ) [](#__codelineno-1-141) parser.add_argument( [](#__codelineno-1-142) "--embedding-endpoint", [](#__codelineno-1-143) default="http://localhost:8000/v1", [](#__codelineno-1-144) help="Base URL for embedding service", [](#__codelineno-1-145) ) [](#__codelineno-1-146) parser.add_argument( [](#__codelineno-1-147) "--chat-endpoint", [](#__codelineno-1-148) default="http://localhost:8001/v1", [](#__codelineno-1-149) help="Base URL for chat service", [](#__codelineno-1-150) ) [](#__codelineno-1-151) parser.add_argument( [](#__codelineno-1-152) "--db-path", default="./milvus_demo.db", help="Path to Milvus database" [](#__codelineno-1-153) ) [](#__codelineno-1-154) parser.add_argument( [](#__codelineno-1-155) "-i", "--interactive", action="store_true", help="Enable interactive Q&A mode" [](#__codelineno-1-156) ) [](#__codelineno-1-157) parser.add_argument( [](#__codelineno-1-158) "-c", [](#__codelineno-1-159) "--chunk-size", [](#__codelineno-1-160) type=int, [](#__codelineno-1-161) default=1000, [](#__codelineno-1-162) help="Chunk size for document splitting", [](#__codelineno-1-163) ) [](#__codelineno-1-164) parser.add_argument( [](#__codelineno-1-165) "-o", [](#__codelineno-1-166) "--chunk-overlap", [](#__codelineno-1-167) type=int, [](#__codelineno-1-168) default=200, [](#__codelineno-1-169) help="Chunk overlap for document splitting", [](#__codelineno-1-170) ) [](#__codelineno-1-171) parser.add_argument( [](#__codelineno-1-172) "-k", "--top-k", type=int, default=3, help="Number of top results to retrieve" [](#__codelineno-1-173) ) [](#__codelineno-1-174) [](#__codelineno-1-175) return parser [](#__codelineno-1-176) [](#__codelineno-1-177)[](#__codelineno-1-178)def main(): [](#__codelineno-1-179) # Parse command line arguments [](#__codelineno-1-180) args = get_parser().parse_args() [](#__codelineno-1-181) [](#__codelineno-1-182) # Initialize configuration [](#__codelineno-1-183) config = init_config(args) [](#__codelineno-1-184) [](#__codelineno-1-185) # Load documents [](#__codelineno-1-186) documents = load_documents(config["url"]) [](#__codelineno-1-187) [](#__codelineno-1-188) # Setup models [](#__codelineno-1-189) setup_models(config) [](#__codelineno-1-190) [](#__codelineno-1-191) # Setup vector store [](#__codelineno-1-192) vector_store = setup_vector_store(config["db_path"]) [](#__codelineno-1-193) [](#__codelineno-1-194) # Create index [](#__codelineno-1-195) index = create_index(documents, vector_store) [](#__codelineno-1-196) [](#__codelineno-1-197) if args.interactive: [](#__codelineno-1-198) print("\nEntering interactive mode. Type 'quit' to exit.") [](#__codelineno-1-199) while True: [](#__codelineno-1-200) # Get user question [](#__codelineno-1-201) question = input("\nEnter your question: ") [](#__codelineno-1-202) [](#__codelineno-1-203) # Check for exit command [](#__codelineno-1-204) if question.lower() in ["quit", "exit", "q"]: [](#__codelineno-1-205) print("Exiting interactive mode...") [](#__codelineno-1-206) break [](#__codelineno-1-207) [](#__codelineno-1-208) # Get and print response [](#__codelineno-1-209) print("\n" + "-" * 50) [](#__codelineno-1-210) print("Response:\n") [](#__codelineno-1-211) response = query_document(index, question, config["top_k"]) [](#__codelineno-1-212) print(response) [](#__codelineno-1-213) print("-" * 50) [](#__codelineno-1-214) else: [](#__codelineno-1-215) # Single query mode [](#__codelineno-1-216) question = "How to install vLLM?" [](#__codelineno-1-217) response = query_document(index, question, config["top_k"]) [](#__codelineno-1-218) print("-" * 50) [](#__codelineno-1-219) print("Response:\n") [](#__codelineno-1-220) print(response) [](#__codelineno-1-221) print("-" * 50) [](#__codelineno-1-222) [](#__codelineno-1-223)[](#__codelineno-1-224)if __name__ == "__main__": [](#__codelineno-1-225) main()` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/basic/offline_inference.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/basic/offline\_inference](https://github.com/vllm-project/vllm/tree/main/examples/basic/offline_inference). The [`LLM`](https://docs.vllm.ai/en/api/vllm/entrypoints/llm/#vllm.entrypoints.llm.LLM " LLM") class provides the primary Python interface for doing offline inference, which is interacting with a model without using a separate model inference server. ## Usage[¶](#usage "Permanent link") The first script in this example shows the most basic usage of vLLM. If you are new to Python and vLLM, you should start here. `[](#__codelineno-0-1)python examples/basic/offline_inference/basic.py` The rest of the scripts include an [argument parser](https://docs.python.org/3/library/argparse.html), which you can use to pass any arguments that are compatible with [`LLM`](https://docs.vllm.ai/en/latest/api/offline_inference/llm.html). Try running the script with `--help` for a list of all available arguments. `[](#__codelineno-1-1)python examples/basic/offline_inference/classify.py` `[](#__codelineno-2-1)python examples/basic/offline_inference/embed.py` `[](#__codelineno-3-1)python examples/basic/offline_inference/score.py` The chat and generate scripts also accept the [sampling parameters](https://docs.vllm.ai/en/latest/api/inference_params.html#sampling-parameters): `max_tokens`, `temperature`, `top_p` and `top_k`. `[](#__codelineno-4-1)python examples/basic/offline_inference/chat.py` `[](#__codelineno-5-1)python examples/basic/offline_inference/generate.py` ## Features[¶](#features "Permanent link") In the scripts that support passing arguments, you can experiment with the following features. ### Default generation config[¶](#default-generation-config "Permanent link") The `--generation-config` argument specifies where the generation config will be loaded from when calling `LLM.get_default_sampling_params()`. If set to ‘auto’, the generation config will be loaded from model path. If set to a folder path, the generation config will be loaded from the specified folder path. If it is not provided, vLLM defaults will be used. > If max\_new\_tokens is specified in generation config, then it sets a server-wide limit on the number of output tokens for all requests. Try it yourself with the following argument: `[](#__codelineno-6-1)--generation-config auto` ### Quantization[¶](#quantization "Permanent link") #### GGUF[¶](#gguf "Permanent link") vLLM supports models that are quantized using GGUF. Try one yourself using the `repo_id:quant_type` format to load directly from HuggingFace: `[](#__codelineno-7-1)--model unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B` ### CPU offload[¶](#cpu-offload "Permanent link") The `--cpu-offload-gb` argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight, which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model is loaded from CPU memory to GPU memory on the fly in each model forward pass. Try it yourself with the following arguments: `[](#__codelineno-8-1)--model meta-llama/Llama-2-13b-chat-hf --cpu-offload-gb 10` ## Example materials[¶](#example-materials "Permanent link") basic.py `[](#__codelineno-9-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-9-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-9-3)[](#__codelineno-9-4)from vllm import LLM, SamplingParams [](#__codelineno-9-5)[](#__codelineno-9-6)# Sample prompts. [](#__codelineno-9-7)prompts = [ [](#__codelineno-9-8) "Hello, my name is", [](#__codelineno-9-9) "The president of the United States is", [](#__codelineno-9-10) "The capital of France is", [](#__codelineno-9-11) "The future of AI is", [](#__codelineno-9-12)] [](#__codelineno-9-13)# Create a sampling params object. [](#__codelineno-9-14)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-9-15) [](#__codelineno-9-16)[](#__codelineno-9-17)def main(): [](#__codelineno-9-18) # Create an LLM. [](#__codelineno-9-19) llm = LLM(model="facebook/opt-125m") [](#__codelineno-9-20) # Generate texts from the prompts. [](#__codelineno-9-21) # The output is a list of RequestOutput objects [](#__codelineno-9-22) # that contain the prompt, generated text, and other information. [](#__codelineno-9-23) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-9-24) # Print the outputs. [](#__codelineno-9-25) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-9-26) for output in outputs: [](#__codelineno-9-27) prompt = output.prompt [](#__codelineno-9-28) generated_text = output.outputs[0].text [](#__codelineno-9-29) print(f"Prompt: {prompt!r}") [](#__codelineno-9-30) print(f"Output: {generated_text!r}") [](#__codelineno-9-31) print("-" * 60) [](#__codelineno-9-32) [](#__codelineno-9-33)[](#__codelineno-9-34)if __name__ == "__main__": [](#__codelineno-9-35) main()` chat.py `[](#__codelineno-10-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-10-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-10-3)[](#__codelineno-10-4)from vllm import LLM, EngineArgs [](#__codelineno-10-5)from vllm.outputs import RequestOutput [](#__codelineno-10-6)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-10-7) [](#__codelineno-10-8)[](#__codelineno-10-9)def create_parser(): [](#__codelineno-10-10) parser = FlexibleArgumentParser() [](#__codelineno-10-11) # Add engine args [](#__codelineno-10-12) EngineArgs.add_cli_args(parser) [](#__codelineno-10-13) parser.set_defaults(model="meta-llama/Llama-3.2-1B-Instruct") [](#__codelineno-10-14) # Add sampling params [](#__codelineno-10-15) sampling_group = parser.add_argument_group("Sampling parameters") [](#__codelineno-10-16) sampling_group.add_argument("--max-tokens", type=int) [](#__codelineno-10-17) sampling_group.add_argument("--temperature", type=float) [](#__codelineno-10-18) sampling_group.add_argument("--top-p", type=float) [](#__codelineno-10-19) sampling_group.add_argument("--top-k", type=int) [](#__codelineno-10-20) # Add example params [](#__codelineno-10-21) parser.add_argument("--chat-template-path", type=str) [](#__codelineno-10-22) [](#__codelineno-10-23) return parser [](#__codelineno-10-24) [](#__codelineno-10-25)[](#__codelineno-10-26)def main(args: dict): [](#__codelineno-10-27) # Pop arguments not used by LLM [](#__codelineno-10-28) max_tokens = args.pop("max_tokens") [](#__codelineno-10-29) temperature = args.pop("temperature") [](#__codelineno-10-30) top_p = args.pop("top_p") [](#__codelineno-10-31) top_k = args.pop("top_k") [](#__codelineno-10-32) chat_template_path = args.pop("chat_template_path") [](#__codelineno-10-33) [](#__codelineno-10-34) # Create an LLM [](#__codelineno-10-35) llm = LLM(**args) [](#__codelineno-10-36) [](#__codelineno-10-37) # Create sampling params object [](#__codelineno-10-38) sampling_params = llm.get_default_sampling_params() [](#__codelineno-10-39) if max_tokens is not None: [](#__codelineno-10-40) sampling_params.max_tokens = max_tokens [](#__codelineno-10-41) if temperature is not None: [](#__codelineno-10-42) sampling_params.temperature = temperature [](#__codelineno-10-43) if top_p is not None: [](#__codelineno-10-44) sampling_params.top_p = top_p [](#__codelineno-10-45) if top_k is not None: [](#__codelineno-10-46) sampling_params.top_k = top_k [](#__codelineno-10-47) [](#__codelineno-10-48) def print_outputs(outputs: list[RequestOutput], prompts: list): [](#__codelineno-10-49) assert len(outputs) == len(prompts) [](#__codelineno-10-50) print("\nGenerated Outputs:\n" + "-" * 80) [](#__codelineno-10-51) for i, output in enumerate(outputs): [](#__codelineno-10-52) generated_text = output.outputs[0].text [](#__codelineno-10-53) print(f"Prompt: {prompts[i]!r}\n") [](#__codelineno-10-54) print(f"Generated text: {generated_text!r}") [](#__codelineno-10-55) print("-" * 80) [](#__codelineno-10-56) [](#__codelineno-10-57) print("=" * 80) [](#__codelineno-10-58) [](#__codelineno-10-59) # In this script, we demonstrate how to pass input to the chat method: [](#__codelineno-10-60) conversation = [ [](#__codelineno-10-61) {"role": "system", "content": "You are a helpful assistant"}, [](#__codelineno-10-62) {"role": "user", "content": "Hello"}, [](#__codelineno-10-63) {"role": "assistant", "content": "Hello! How can I assist you today?"}, [](#__codelineno-10-64) { [](#__codelineno-10-65) "role": "user", [](#__codelineno-10-66) "content": "Write an essay about the importance of higher education.", [](#__codelineno-10-67) }, [](#__codelineno-10-68) ] [](#__codelineno-10-69) outputs = llm.chat(conversation, sampling_params, use_tqdm=False) [](#__codelineno-10-70) print_outputs( [](#__codelineno-10-71) outputs, [](#__codelineno-10-72) [ [](#__codelineno-10-73) conversation, [](#__codelineno-10-74) ], [](#__codelineno-10-75) ) [](#__codelineno-10-76) [](#__codelineno-10-77) # You can run batch inference with llm.chat API [](#__codelineno-10-78) conversations = [conversation for _ in range(10)] [](#__codelineno-10-79) [](#__codelineno-10-80) # We turn on tqdm progress bar to verify it's indeed running batch inference [](#__codelineno-10-81) outputs = llm.chat(conversations, sampling_params, use_tqdm=True) [](#__codelineno-10-82) print_outputs(outputs, conversations) [](#__codelineno-10-83) [](#__codelineno-10-84) # A chat template can be optionally supplied. [](#__codelineno-10-85) # If not, the model will use its default chat template. [](#__codelineno-10-86) if chat_template_path is not None: [](#__codelineno-10-87) with open(chat_template_path) as f: [](#__codelineno-10-88) chat_template = f.read() [](#__codelineno-10-89) [](#__codelineno-10-90) outputs = llm.chat( [](#__codelineno-10-91) conversations, [](#__codelineno-10-92) sampling_params, [](#__codelineno-10-93) use_tqdm=False, [](#__codelineno-10-94) chat_template=chat_template, [](#__codelineno-10-95) ) [](#__codelineno-10-96) print_outputs(outputs, conversations) [](#__codelineno-10-97) [](#__codelineno-10-98)[](#__codelineno-10-99)if __name__ == "__main__": [](#__codelineno-10-100) parser = create_parser() [](#__codelineno-10-101) args: dict = vars(parser.parse_args()) [](#__codelineno-10-102) main(args)` classify.py `[](#__codelineno-11-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-11-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-11-3)[](#__codelineno-11-4)from argparse import Namespace [](#__codelineno-11-5)[](#__codelineno-11-6)from vllm import LLM, EngineArgs [](#__codelineno-11-7)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-11-8) [](#__codelineno-11-9)[](#__codelineno-11-10)def parse_args(): [](#__codelineno-11-11) parser = FlexibleArgumentParser() [](#__codelineno-11-12) parser = EngineArgs.add_cli_args(parser) [](#__codelineno-11-13) # Set example specific arguments [](#__codelineno-11-14) parser.set_defaults( [](#__codelineno-11-15) model="jason9693/Qwen2.5-1.5B-apeach", [](#__codelineno-11-16) runner="pooling", [](#__codelineno-11-17) enforce_eager=True, [](#__codelineno-11-18) ) [](#__codelineno-11-19) return parser.parse_args() [](#__codelineno-11-20) [](#__codelineno-11-21)[](#__codelineno-11-22)def main(args: Namespace): [](#__codelineno-11-23) # Sample prompts. [](#__codelineno-11-24) prompts = [ [](#__codelineno-11-25) "Hello, my name is", [](#__codelineno-11-26) "The president of the United States is", [](#__codelineno-11-27) "The capital of France is", [](#__codelineno-11-28) "The future of AI is", [](#__codelineno-11-29) ] [](#__codelineno-11-30) [](#__codelineno-11-31) # Create an LLM. [](#__codelineno-11-32) # You should pass runner="pooling" for classification models [](#__codelineno-11-33) llm = LLM(**vars(args)) [](#__codelineno-11-34) [](#__codelineno-11-35) # Generate logits. The output is a list of ClassificationRequestOutputs. [](#__codelineno-11-36) outputs = llm.classify(prompts) [](#__codelineno-11-37) [](#__codelineno-11-38) # Print the outputs. [](#__codelineno-11-39) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-11-40) for prompt, output in zip(prompts, outputs): [](#__codelineno-11-41) probs = output.outputs.probs [](#__codelineno-11-42) probs_trimmed = (str(probs[:16])[:-1] + ", ...]") if len(probs) > 16 else probs [](#__codelineno-11-43) print( [](#__codelineno-11-44) f"Prompt: {prompt!r} \n" [](#__codelineno-11-45) f"Class Probabilities: {probs_trimmed} (size={len(probs)})" [](#__codelineno-11-46) ) [](#__codelineno-11-47) print("-" * 60) [](#__codelineno-11-48) [](#__codelineno-11-49)[](#__codelineno-11-50)if __name__ == "__main__": [](#__codelineno-11-51) args = parse_args() [](#__codelineno-11-52) main(args)` embed.py `[](#__codelineno-12-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-12-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-12-3)[](#__codelineno-12-4)from argparse import Namespace [](#__codelineno-12-5)[](#__codelineno-12-6)from vllm import LLM, EngineArgs [](#__codelineno-12-7)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-12-8)from vllm.utils.print_utils import print_embeddings [](#__codelineno-12-9) [](#__codelineno-12-10)[](#__codelineno-12-11)def parse_args(): [](#__codelineno-12-12) parser = FlexibleArgumentParser() [](#__codelineno-12-13) parser = EngineArgs.add_cli_args(parser) [](#__codelineno-12-14) # Set example specific arguments [](#__codelineno-12-15) parser.set_defaults( [](#__codelineno-12-16) model="intfloat/e5-small", [](#__codelineno-12-17) runner="pooling", [](#__codelineno-12-18) enforce_eager=True, [](#__codelineno-12-19) ) [](#__codelineno-12-20) return parser.parse_args() [](#__codelineno-12-21) [](#__codelineno-12-22)[](#__codelineno-12-23)def main(args: Namespace): [](#__codelineno-12-24) # Sample prompts. [](#__codelineno-12-25) prompts = [ [](#__codelineno-12-26) "Hello, my name is", [](#__codelineno-12-27) "The president of the United States is", [](#__codelineno-12-28) "The capital of France is", [](#__codelineno-12-29) "The future of AI is", [](#__codelineno-12-30) ] [](#__codelineno-12-31) [](#__codelineno-12-32) # Create an LLM. [](#__codelineno-12-33) # You should pass runner="pooling" for embedding models [](#__codelineno-12-34) llm = LLM(**vars(args)) [](#__codelineno-12-35) [](#__codelineno-12-36) # Generate embedding. The output is a list of EmbeddingRequestOutputs. [](#__codelineno-12-37) outputs = llm.embed(prompts) [](#__codelineno-12-38) [](#__codelineno-12-39) # Print the outputs. [](#__codelineno-12-40) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-12-41) for prompt, output in zip(prompts, outputs): [](#__codelineno-12-42) embeds = output.outputs.embedding [](#__codelineno-12-43) print(f"Prompt: {prompt!r}") [](#__codelineno-12-44) print_embeddings(embeds) [](#__codelineno-12-45) print("-" * 60) [](#__codelineno-12-46) [](#__codelineno-12-47)[](#__codelineno-12-48)if __name__ == "__main__": [](#__codelineno-12-49) args = parse_args() [](#__codelineno-12-50) main(args)` generate.py `[](#__codelineno-13-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-13-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-13-3)[](#__codelineno-13-4)from vllm import LLM, EngineArgs [](#__codelineno-13-5)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-13-6) [](#__codelineno-13-7)[](#__codelineno-13-8)def create_parser(): [](#__codelineno-13-9) parser = FlexibleArgumentParser() [](#__codelineno-13-10) # Add engine args [](#__codelineno-13-11) EngineArgs.add_cli_args(parser) [](#__codelineno-13-12) parser.set_defaults(model="meta-llama/Llama-3.2-1B-Instruct") [](#__codelineno-13-13) # Add sampling params [](#__codelineno-13-14) sampling_group = parser.add_argument_group("Sampling parameters") [](#__codelineno-13-15) sampling_group.add_argument("--max-tokens", type=int) [](#__codelineno-13-16) sampling_group.add_argument("--temperature", type=float) [](#__codelineno-13-17) sampling_group.add_argument("--top-p", type=float) [](#__codelineno-13-18) sampling_group.add_argument("--top-k", type=int) [](#__codelineno-13-19) [](#__codelineno-13-20) return parser [](#__codelineno-13-21) [](#__codelineno-13-22)[](#__codelineno-13-23)def main(args: dict): [](#__codelineno-13-24) # Pop arguments not used by LLM [](#__codelineno-13-25) max_tokens = args.pop("max_tokens") [](#__codelineno-13-26) temperature = args.pop("temperature") [](#__codelineno-13-27) top_p = args.pop("top_p") [](#__codelineno-13-28) top_k = args.pop("top_k") [](#__codelineno-13-29) [](#__codelineno-13-30) # Create an LLM [](#__codelineno-13-31) llm = LLM(**args) [](#__codelineno-13-32) [](#__codelineno-13-33) # Create a sampling params object [](#__codelineno-13-34) sampling_params = llm.get_default_sampling_params() [](#__codelineno-13-35) if max_tokens is not None: [](#__codelineno-13-36) sampling_params.max_tokens = max_tokens [](#__codelineno-13-37) if temperature is not None: [](#__codelineno-13-38) sampling_params.temperature = temperature [](#__codelineno-13-39) if top_p is not None: [](#__codelineno-13-40) sampling_params.top_p = top_p [](#__codelineno-13-41) if top_k is not None: [](#__codelineno-13-42) sampling_params.top_k = top_k [](#__codelineno-13-43) [](#__codelineno-13-44) # Generate texts from the prompts. The output is a list of RequestOutput [](#__codelineno-13-45) # objects that contain the prompt, generated text, and other information. [](#__codelineno-13-46) prompts = [ [](#__codelineno-13-47) "Hello, my name is", [](#__codelineno-13-48) "The president of the United States is", [](#__codelineno-13-49) "The capital of France is", [](#__codelineno-13-50) "The future of AI is", [](#__codelineno-13-51) ] [](#__codelineno-13-52) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-13-53) # Print the outputs. [](#__codelineno-13-54) print("-" * 50) [](#__codelineno-13-55) for output in outputs: [](#__codelineno-13-56) prompt = output.prompt [](#__codelineno-13-57) generated_text = output.outputs[0].text [](#__codelineno-13-58) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-13-59) print("-" * 50) [](#__codelineno-13-60) [](#__codelineno-13-61)[](#__codelineno-13-62)if __name__ == "__main__": [](#__codelineno-13-63) parser = create_parser() [](#__codelineno-13-64) args: dict = vars(parser.parse_args()) [](#__codelineno-13-65) main(args)` score.py `[](#__codelineno-14-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-14-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-14-3)[](#__codelineno-14-4)from argparse import Namespace [](#__codelineno-14-5)[](#__codelineno-14-6)from vllm import LLM, EngineArgs [](#__codelineno-14-7)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-14-8) [](#__codelineno-14-9)[](#__codelineno-14-10)def parse_args(): [](#__codelineno-14-11) parser = FlexibleArgumentParser() [](#__codelineno-14-12) parser = EngineArgs.add_cli_args(parser) [](#__codelineno-14-13) # Set example specific arguments [](#__codelineno-14-14) parser.set_defaults( [](#__codelineno-14-15) model="BAAI/bge-reranker-v2-m3", [](#__codelineno-14-16) runner="pooling", [](#__codelineno-14-17) enforce_eager=True, [](#__codelineno-14-18) ) [](#__codelineno-14-19) return parser.parse_args() [](#__codelineno-14-20) [](#__codelineno-14-21)[](#__codelineno-14-22)def main(args: Namespace): [](#__codelineno-14-23) # Sample prompts. [](#__codelineno-14-24) query = "What is the capital of France?" [](#__codelineno-14-25) documents = [ [](#__codelineno-14-26) "The capital of Brazil is Brasilia.", [](#__codelineno-14-27) "The capital of France is Paris.", [](#__codelineno-14-28) ] [](#__codelineno-14-29) [](#__codelineno-14-30) # Create an LLM. [](#__codelineno-14-31) # You should pass runner="pooling" for cross-encoder models [](#__codelineno-14-32) llm = LLM(**vars(args)) [](#__codelineno-14-33) [](#__codelineno-14-34) # Generate scores. The output is a list of ScoringRequestOutputs. [](#__codelineno-14-35) outputs = llm.score(query, documents) [](#__codelineno-14-36) [](#__codelineno-14-37) # Print the outputs. [](#__codelineno-14-38) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-14-39) for document, output in zip(documents, outputs): [](#__codelineno-14-40) score = output.outputs.score [](#__codelineno-14-41) print(f"Pair: {[query, document]!r} \nScore: {score}") [](#__codelineno-14-42) print("-" * 60) [](#__codelineno-14-43) [](#__codelineno-14-44)[](#__codelineno-14-45)if __name__ == "__main__": [](#__codelineno-14-46) args = parse_args() [](#__codelineno-14-47) main(args)` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/basic/online_serving.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/basic/online\_serving](https://github.com/vllm-project/vllm/tree/main/examples/basic/online_serving). ## OpenAI Chat Completion Client[¶](#openai-chat-completion-client "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)"""Example Python client for OpenAI Chat Completion using vLLM API server [](#__codelineno-0-4)NOTE: start a supported chat completion model server with `vllm serve`, e.g. [](#__codelineno-0-5) vllm serve meta-llama/Llama-2-7b-chat-hf [](#__codelineno-0-6)""" [](#__codelineno-0-7)[](#__codelineno-0-8)import argparse [](#__codelineno-0-9)[](#__codelineno-0-10)from openai import OpenAI [](#__codelineno-0-11)[](#__codelineno-0-12)# Modify OpenAI's API key and API base to use vLLM's API server. [](#__codelineno-0-13)openai_api_key = "EMPTY" [](#__codelineno-0-14)openai_api_base = "http://localhost:8000/v1" [](#__codelineno-0-15)[](#__codelineno-0-16)messages = [ [](#__codelineno-0-17) {"role": "system", "content": "You are a helpful assistant."}, [](#__codelineno-0-18) {"role": "user", "content": "Who won the world series in 2020?"}, [](#__codelineno-0-19) { [](#__codelineno-0-20) "role": "assistant", [](#__codelineno-0-21) "content": "The Los Angeles Dodgers won the World Series in 2020.", [](#__codelineno-0-22) }, [](#__codelineno-0-23) {"role": "user", "content": "Where was it played?"}, [](#__codelineno-0-24)] [](#__codelineno-0-25) [](#__codelineno-0-26)[](#__codelineno-0-27)def parse_args(): [](#__codelineno-0-28) parser = argparse.ArgumentParser(description="Client for vLLM API server") [](#__codelineno-0-29) parser.add_argument( [](#__codelineno-0-30) "--stream", action="store_true", help="Enable streaming response" [](#__codelineno-0-31) ) [](#__codelineno-0-32) return parser.parse_args() [](#__codelineno-0-33) [](#__codelineno-0-34)[](#__codelineno-0-35)def main(args): [](#__codelineno-0-36) client = OpenAI( [](#__codelineno-0-37) # defaults to os.environ.get("OPENAI_API_KEY") [](#__codelineno-0-38) api_key=openai_api_key, [](#__codelineno-0-39) base_url=openai_api_base, [](#__codelineno-0-40) ) [](#__codelineno-0-41) [](#__codelineno-0-42) models = client.models.list() [](#__codelineno-0-43) model = models.data[0].id [](#__codelineno-0-44) [](#__codelineno-0-45) # Chat Completion API [](#__codelineno-0-46) chat_completion = client.chat.completions.create( [](#__codelineno-0-47) messages=messages, [](#__codelineno-0-48) model=model, [](#__codelineno-0-49) stream=args.stream, [](#__codelineno-0-50) ) [](#__codelineno-0-51) [](#__codelineno-0-52) print("-" * 50) [](#__codelineno-0-53) print("Chat completion results:") [](#__codelineno-0-54) if args.stream: [](#__codelineno-0-55) for c in chat_completion: [](#__codelineno-0-56) print(c) [](#__codelineno-0-57) else: [](#__codelineno-0-58) print(chat_completion) [](#__codelineno-0-59) print("-" * 50) [](#__codelineno-0-60) [](#__codelineno-0-61)[](#__codelineno-0-62)if __name__ == "__main__": [](#__codelineno-0-63) args = parse_args() [](#__codelineno-0-64) main(args)`` ## OpenAI Completion Client[¶](#openai-completion-client "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)[](#__codelineno-1-4)import argparse [](#__codelineno-1-5)[](#__codelineno-1-6)from openai import OpenAI [](#__codelineno-1-7)[](#__codelineno-1-8)# Modify OpenAI's API key and API base to use vLLM's API server. [](#__codelineno-1-9)openai_api_key = "EMPTY" [](#__codelineno-1-10)openai_api_base = "http://localhost:8000/v1" [](#__codelineno-1-11) [](#__codelineno-1-12)[](#__codelineno-1-13)def parse_args(): [](#__codelineno-1-14) parser = argparse.ArgumentParser(description="Client for vLLM API server") [](#__codelineno-1-15) parser.add_argument( [](#__codelineno-1-16) "--stream", action="store_true", help="Enable streaming response" [](#__codelineno-1-17) ) [](#__codelineno-1-18) return parser.parse_args() [](#__codelineno-1-19) [](#__codelineno-1-20)[](#__codelineno-1-21)def main(args): [](#__codelineno-1-22) client = OpenAI( [](#__codelineno-1-23) # defaults to os.environ.get("OPENAI_API_KEY") [](#__codelineno-1-24) api_key=openai_api_key, [](#__codelineno-1-25) base_url=openai_api_base, [](#__codelineno-1-26) ) [](#__codelineno-1-27) [](#__codelineno-1-28) models = client.models.list() [](#__codelineno-1-29) model = models.data[0].id [](#__codelineno-1-30) [](#__codelineno-1-31) # Completion API [](#__codelineno-1-32) completion = client.completions.create( [](#__codelineno-1-33) model=model, [](#__codelineno-1-34) prompt="A robot may not injure a human being", [](#__codelineno-1-35) echo=False, [](#__codelineno-1-36) n=2, [](#__codelineno-1-37) stream=args.stream, [](#__codelineno-1-38) logprobs=3, [](#__codelineno-1-39) ) [](#__codelineno-1-40) [](#__codelineno-1-41) print("-" * 50) [](#__codelineno-1-42) print("Completion results:") [](#__codelineno-1-43) if args.stream: [](#__codelineno-1-44) for c in completion: [](#__codelineno-1-45) print(c) [](#__codelineno-1-46) else: [](#__codelineno-1-47) print(completion) [](#__codelineno-1-48) print("-" * 50) [](#__codelineno-1-49) [](#__codelineno-1-50)[](#__codelineno-1-51)if __name__ == "__main__": [](#__codelineno-1-52) args = parse_args() [](#__codelineno-1-53) main(args)` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) 5. [Deployment](https://docs.vllm.ai/en/latest/examples/deployment/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/deployment/async_llm_streaming.md "Edit this page") Source [https://github.com/vllm-project/vllm/blob/main/examples/deployment/async\_llm\_streaming.py](https://github.com/vllm-project/vllm/blob/main/examples/deployment/async_llm_streaming.py). `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Simple example demonstrating streaming offline inference with AsyncLLM (V1 engine). [](#__codelineno-0-5)[](#__codelineno-0-6)This script shows the core functionality of vLLM's AsyncLLM engine for streaming [](#__codelineno-0-7)token-by-token output in offline inference scenarios. It demonstrates DELTA mode [](#__codelineno-0-8)streaming where you receive new tokens as they are generated. [](#__codelineno-0-9)[](#__codelineno-0-10)Usage: [](#__codelineno-0-11) python examples/deployment/async_llm_streaming.py [](#__codelineno-0-12)""" [](#__codelineno-0-13)[](#__codelineno-0-14)import asyncio [](#__codelineno-0-15)[](#__codelineno-0-16)from vllm import SamplingParams [](#__codelineno-0-17)from vllm.engine.arg_utils import AsyncEngineArgs [](#__codelineno-0-18)from vllm.sampling_params import RequestOutputKind [](#__codelineno-0-19)from vllm.v1.engine.async_llm import AsyncLLM [](#__codelineno-0-20) [](#__codelineno-0-21)[](#__codelineno-0-22)async def stream_response(engine: AsyncLLM, prompt: str, request_id: str) -> None: [](#__codelineno-0-23) """ [](#__codelineno-0-24) Stream response from AsyncLLM and display tokens as they arrive. [](#__codelineno-0-25) [](#__codelineno-0-26) This function demonstrates the core streaming pattern: [](#__codelineno-0-27) 1. Create SamplingParams with DELTA output kind [](#__codelineno-0-28) 2. Call engine.generate() and iterate over the async generator [](#__codelineno-0-29) 3. Print new tokens as they arrive [](#__codelineno-0-30) 4. Handle the finished flag to know when generation is complete [](#__codelineno-0-31) """ [](#__codelineno-0-32) print(f"\n🚀 Prompt: {prompt!r}") [](#__codelineno-0-33) print("💬 Response: ", end="", flush=True) [](#__codelineno-0-34) [](#__codelineno-0-35) # Configure sampling parameters for streaming [](#__codelineno-0-36) sampling_params = SamplingParams( [](#__codelineno-0-37) max_tokens=100, [](#__codelineno-0-38) temperature=0.8, [](#__codelineno-0-39) top_p=0.95, [](#__codelineno-0-40) seed=42, # For reproducible results [](#__codelineno-0-41) output_kind=RequestOutputKind.DELTA, # Get only new tokens each iteration [](#__codelineno-0-42) ) [](#__codelineno-0-43) [](#__codelineno-0-44) try: [](#__codelineno-0-45) # Stream tokens from AsyncLLM [](#__codelineno-0-46) async for output in engine.generate( [](#__codelineno-0-47) request_id=request_id, prompt=prompt, sampling_params=sampling_params [](#__codelineno-0-48) ): [](#__codelineno-0-49) # Process each completion in the output [](#__codelineno-0-50) for completion in output.outputs: [](#__codelineno-0-51) # In DELTA mode, we get only new tokens generated since last iteration [](#__codelineno-0-52) new_text = completion.text [](#__codelineno-0-53) if new_text: [](#__codelineno-0-54) print(new_text, end="", flush=True) [](#__codelineno-0-55) [](#__codelineno-0-56) # Check if generation is finished [](#__codelineno-0-57) if output.finished: [](#__codelineno-0-58) print("\n✅ Generation complete!") [](#__codelineno-0-59) break [](#__codelineno-0-60) [](#__codelineno-0-61) except Exception as e: [](#__codelineno-0-62) print(f"\n❌ Error during streaming: {e}") [](#__codelineno-0-63) raise [](#__codelineno-0-64) [](#__codelineno-0-65)[](#__codelineno-0-66)async def main(): [](#__codelineno-0-67) print("🔧 Initializing AsyncLLM...") [](#__codelineno-0-68) [](#__codelineno-0-69) # Create AsyncLLM engine with simple configuration [](#__codelineno-0-70) engine_args = AsyncEngineArgs( [](#__codelineno-0-71) model="meta-llama/Llama-3.2-1B-Instruct", [](#__codelineno-0-72) enforce_eager=True, # Faster startup for examples [](#__codelineno-0-73) ) [](#__codelineno-0-74) engine = AsyncLLM.from_engine_args(engine_args) [](#__codelineno-0-75) [](#__codelineno-0-76) try: [](#__codelineno-0-77) # Example prompts to demonstrate streaming [](#__codelineno-0-78) prompts = [ [](#__codelineno-0-79) "The future of artificial intelligence is", [](#__codelineno-0-80) "In a galaxy far, far away", [](#__codelineno-0-81) "The key to happiness is", [](#__codelineno-0-82) ] [](#__codelineno-0-83) [](#__codelineno-0-84) print(f"🎯 Running {len(prompts)} streaming examples...") [](#__codelineno-0-85) [](#__codelineno-0-86) # Process each prompt [](#__codelineno-0-87) for i, prompt in enumerate(prompts, 1): [](#__codelineno-0-88) print(f"\n{'=' * 60}") [](#__codelineno-0-89) print(f"Example {i}/{len(prompts)}") [](#__codelineno-0-90) print(f"{'=' * 60}") [](#__codelineno-0-91) [](#__codelineno-0-92) request_id = f"stream-example-{i}" [](#__codelineno-0-93) await stream_response(engine, prompt, request_id) [](#__codelineno-0-94) [](#__codelineno-0-95) # Brief pause between examples [](#__codelineno-0-96) if i < len(prompts): [](#__codelineno-0-97) await asyncio.sleep(0.5) [](#__codelineno-0-98) [](#__codelineno-0-99) print("\n🎉 All streaming examples completed!") [](#__codelineno-0-100) [](#__codelineno-0-101) finally: [](#__codelineno-0-102) # Always clean up the engine [](#__codelineno-0-103) print("🔧 Shutting down engine...") [](#__codelineno-0-104) engine.shutdown() [](#__codelineno-0-105) [](#__codelineno-0-106)[](#__codelineno-0-107)if __name__ == "__main__": [](#__codelineno-0-108) try: [](#__codelineno-0-109) asyncio.run(main()) [](#__codelineno-0-110) except KeyboardInterrupt: [](#__codelineno-0-111) print("\n🛑 Interrupted by user")` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/deployment/chart-helm.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/deployment/chart-helm](https://github.com/vllm-project/vllm/tree/main/examples/deployment/chart-helm). This directory contains a Helm chart for deploying the vllm application. The chart includes configurations for deployment, autoscaling, resource management, and more. ## Files[¶](#files "Permanent link") - Chart.yaml: Defines the chart metadata including name, version, and maintainers. - ct.yaml: Configuration for chart testing. - lintconf.yaml: Linting rules for YAML files. - values.schema.json: JSON schema for validating values.yaml. - values.yaml: Default values for the Helm chart. - templates/\_helpers.tpl: Helper templates for defining common configurations. - templates/configmap.yaml: Template for creating ConfigMaps. - templates/custom-objects.yaml: Template for custom Kubernetes objects. - templates/deployment.yaml: Template for creating Deployments. - templates/hpa.yaml: Template for Horizontal Pod Autoscaler. - templates/job.yaml: Template for Kubernetes Jobs. - templates/poddisruptionbudget.yaml: Template for Pod Disruption Budget. - templates/pvc.yaml: Template for Persistent Volume Claims. - templates/secrets.yaml: Template for Kubernetes Secrets. - templates/service.yaml: Template for creating Services. ## Running Tests[¶](#running-tests "Permanent link") This chart includes unit tests using [helm-unittest](https://github.com/helm-unittest/helm-unittest). Install the plugin and run tests: `[](#__codelineno-0-1)# Install plugin [](#__codelineno-0-2)helm plugin install https://github.com/helm-unittest/helm-unittest [](#__codelineno-0-3)[](#__codelineno-0-4)# Run tests [](#__codelineno-0-5)helm unittest .` ## Example materials[¶](#example-materials "Permanent link") .helmignore `[](#__codelineno-1-1)*.png [](#__codelineno-1-2).git/ [](#__codelineno-1-3)ct.yaml [](#__codelineno-1-4)lintconf.yaml [](#__codelineno-1-5)values.schema.json [](#__codelineno-1-6)/workflows` Chart.yaml `[](#__codelineno-2-1)apiVersion: v2 [](#__codelineno-2-2)name: chart-vllm [](#__codelineno-2-3)description: Chart vllm [](#__codelineno-2-4)[](#__codelineno-2-5)# A chart can be either an 'application' or a 'library' chart. [](#__codelineno-2-6)# [](#__codelineno-2-7)# Application charts are a collection of templates that can be packaged into versioned archives [](#__codelineno-2-8)# to be deployed. [](#__codelineno-2-9)# [](#__codelineno-2-10)# Library charts provide useful utilities or functions for the chart developer. They're included as [](#__codelineno-2-11)# a dependency of application charts to inject those utilities and functions into the rendering [](#__codelineno-2-12)# pipeline. Library charts do not define any templates and therefore cannot be deployed. [](#__codelineno-2-13)type: application [](#__codelineno-2-14)[](#__codelineno-2-15)# This is the chart version. This version number should be incremented each time you make changes [](#__codelineno-2-16)# to the chart and its templates, including the app version. [](#__codelineno-2-17)# Versions are expected to follow Semantic Versioning (https://semver.org/) [](#__codelineno-2-18)version: 0.0.1 [](#__codelineno-2-19)[](#__codelineno-2-20)maintainers: [](#__codelineno-2-21) - name: mfournioux` ct.yaml `[](#__codelineno-3-1)chart-dirs: [](#__codelineno-3-2) - charts [](#__codelineno-3-3)validate-maintainers: false` lintconf.yaml `[](#__codelineno-4-1)--- [](#__codelineno-4-2)rules: [](#__codelineno-4-3) braces: [](#__codelineno-4-4) min-spaces-inside: 0 [](#__codelineno-4-5) max-spaces-inside: 0 [](#__codelineno-4-6) min-spaces-inside-empty: -1 [](#__codelineno-4-7) max-spaces-inside-empty: -1 [](#__codelineno-4-8) brackets: [](#__codelineno-4-9) min-spaces-inside: 0 [](#__codelineno-4-10) max-spaces-inside: 0 [](#__codelineno-4-11) min-spaces-inside-empty: -1 [](#__codelineno-4-12) max-spaces-inside-empty: -1 [](#__codelineno-4-13) colons: [](#__codelineno-4-14) max-spaces-before: 0 [](#__codelineno-4-15) max-spaces-after: 1 [](#__codelineno-4-16) commas: [](#__codelineno-4-17) max-spaces-before: 0 [](#__codelineno-4-18) min-spaces-after: 1 [](#__codelineno-4-19) max-spaces-after: 1 [](#__codelineno-4-20) comments: [](#__codelineno-4-21) require-starting-space: true [](#__codelineno-4-22) min-spaces-from-content: 2 [](#__codelineno-4-23) document-end: disable [](#__codelineno-4-24) document-start: disable # No --- to start a file [](#__codelineno-4-25) empty-lines: [](#__codelineno-4-26) max: 2 [](#__codelineno-4-27) max-start: 0 [](#__codelineno-4-28) max-end: 0 [](#__codelineno-4-29) hyphens: [](#__codelineno-4-30) max-spaces-after: 1 [](#__codelineno-4-31) indentation: [](#__codelineno-4-32) spaces: consistent [](#__codelineno-4-33) indent-sequences: whatever # - list indentation will handle both indentation and without [](#__codelineno-4-34) check-multi-line-strings: false [](#__codelineno-4-35) key-duplicates: enable [](#__codelineno-4-36) line-length: disable # Lines can be any length [](#__codelineno-4-37) new-line-at-end-of-file: disable [](#__codelineno-4-38) new-lines: [](#__codelineno-4-39) type: unix [](#__codelineno-4-40) trailing-spaces: enable [](#__codelineno-4-41) truthy: [](#__codelineno-4-42) level: warning` templates/\_helpers.tpl `[](#__codelineno-5-1){{/* [](#__codelineno-5-2)Define ports for the pods [](#__codelineno-5-3)*/}} [](#__codelineno-5-4){{- define "chart.container-port" -}} [](#__codelineno-5-5){{- default "8000" .Values.containerPort }} [](#__codelineno-5-6){{- end }} [](#__codelineno-5-7)[](#__codelineno-5-8){{/* [](#__codelineno-5-9)Define service name [](#__codelineno-5-10)*/}} [](#__codelineno-5-11){{- define "chart.service-name" -}} [](#__codelineno-5-12){{- if .Values.serviceName }} [](#__codelineno-5-13){{- .Values.serviceName | lower | trim }} [](#__codelineno-5-14){{- else }} [](#__codelineno-5-15)"{{ .Release.Name }}-service" [](#__codelineno-5-16){{- end }} [](#__codelineno-5-17){{- end }} [](#__codelineno-5-18)[](#__codelineno-5-19){{/* [](#__codelineno-5-20)Define service port [](#__codelineno-5-21)*/}} [](#__codelineno-5-22){{- define "chart.service-port" -}} [](#__codelineno-5-23){{- if .Values.servicePort }} [](#__codelineno-5-24){{- .Values.servicePort }} [](#__codelineno-5-25){{- else }} [](#__codelineno-5-26){{- include "chart.container-port" . }} [](#__codelineno-5-27){{- end }} [](#__codelineno-5-28){{- end }} [](#__codelineno-5-29)[](#__codelineno-5-30){{/* [](#__codelineno-5-31)Define service port name [](#__codelineno-5-32)*/}} [](#__codelineno-5-33){{- define "chart.service-port-name" -}} [](#__codelineno-5-34)"service-port" [](#__codelineno-5-35){{- end }} [](#__codelineno-5-36)[](#__codelineno-5-37){{/* [](#__codelineno-5-38)Define container port name [](#__codelineno-5-39)*/}} [](#__codelineno-5-40){{- define "chart.container-port-name" -}} [](#__codelineno-5-41)"container-port" [](#__codelineno-5-42){{- end }} [](#__codelineno-5-43)[](#__codelineno-5-44){{/* [](#__codelineno-5-45)Define deployment strategy [](#__codelineno-5-46)*/}} [](#__codelineno-5-47){{- define "chart.strategy" -}} [](#__codelineno-5-48)strategy: [](#__codelineno-5-49){{- if not .Values.deploymentStrategy }} [](#__codelineno-5-50) rollingUpdate: [](#__codelineno-5-51) maxSurge: 100% [](#__codelineno-5-52) maxUnavailable: 0 [](#__codelineno-5-53){{- else }} [](#__codelineno-5-54){{ toYaml .Values.deploymentStrategy | indent 2 }} [](#__codelineno-5-55){{- end }} [](#__codelineno-5-56){{- end }} [](#__codelineno-5-57)[](#__codelineno-5-58){{/* [](#__codelineno-5-59)Define additional ports [](#__codelineno-5-60)*/}} [](#__codelineno-5-61){{- define "chart.extraPorts" }} [](#__codelineno-5-62){{- with .Values.extraPorts }} [](#__codelineno-5-63){{ toYaml . }} [](#__codelineno-5-64){{- end }} [](#__codelineno-5-65){{- end }} [](#__codelineno-5-66)[](#__codelineno-5-67){{/* [](#__codelineno-5-68)Define chart external ConfigMaps and Secrets [](#__codelineno-5-69)*/}} [](#__codelineno-5-70){{- define "chart.externalConfigs" -}} [](#__codelineno-5-71){{- with .Values.externalConfigs -}} [](#__codelineno-5-72){{ toYaml . }} [](#__codelineno-5-73){{- end }} [](#__codelineno-5-74){{- end }} [](#__codelineno-5-75) [](#__codelineno-5-76)[](#__codelineno-5-77){{/* [](#__codelineno-5-78)Define liveness et readiness probes [](#__codelineno-5-79)*/}} [](#__codelineno-5-80){{- define "chart.probes" -}} [](#__codelineno-5-81){{- if .Values.readinessProbe }} [](#__codelineno-5-82)readinessProbe: [](#__codelineno-5-83){{- with .Values.readinessProbe }} [](#__codelineno-5-84){{- toYaml . | nindent 2 }} [](#__codelineno-5-85){{- end }} [](#__codelineno-5-86){{- end }} [](#__codelineno-5-87){{- if .Values.livenessProbe }} [](#__codelineno-5-88)livenessProbe: [](#__codelineno-5-89){{- with .Values.livenessProbe }} [](#__codelineno-5-90){{- toYaml . | nindent 2 }} [](#__codelineno-5-91){{- end }} [](#__codelineno-5-92){{- end }} [](#__codelineno-5-93){{- end }} [](#__codelineno-5-94)[](#__codelineno-5-95){{/* [](#__codelineno-5-96)Define resources [](#__codelineno-5-97)*/}} [](#__codelineno-5-98){{- define "chart.resources" -}} [](#__codelineno-5-99)requests: [](#__codelineno-5-100) memory: {{ required "Value 'resources.requests.memory' must be defined !" .Values.resources.requests.memory | quote }} [](#__codelineno-5-101) cpu: {{ required "Value 'resources.requests.cpu' must be defined !" .Values.resources.requests.cpu | quote }} [](#__codelineno-5-102) {{- if and (gt (int (index .Values.resources.requests "nvidia.com/gpu")) 0) (gt (int (index .Values.resources.limits "nvidia.com/gpu")) 0) }} [](#__codelineno-5-103) nvidia.com/gpu: {{ required "Value 'resources.requests.nvidia.com/gpu' must be defined !" (index .Values.resources.requests "nvidia.com/gpu") | quote }} [](#__codelineno-5-104) {{- end }} [](#__codelineno-5-105)limits: [](#__codelineno-5-106) memory: {{ required "Value 'resources.limits.memory' must be defined !" .Values.resources.limits.memory | quote }} [](#__codelineno-5-107) cpu: {{ required "Value 'resources.limits.cpu' must be defined !" .Values.resources.limits.cpu | quote }} [](#__codelineno-5-108) {{- if and (gt (int (index .Values.resources.requests "nvidia.com/gpu")) 0) (gt (int (index .Values.resources.limits "nvidia.com/gpu")) 0) }} [](#__codelineno-5-109) nvidia.com/gpu: {{ required "Value 'resources.limits.nvidia.com/gpu' must be defined !" (index .Values.resources.limits "nvidia.com/gpu") | quote }} [](#__codelineno-5-110) {{- end }} [](#__codelineno-5-111){{- end }} [](#__codelineno-5-112) [](#__codelineno-5-113)[](#__codelineno-5-114){{/* [](#__codelineno-5-115)Define User used for the main container [](#__codelineno-5-116)*/}} [](#__codelineno-5-117){{- define "chart.user" }} [](#__codelineno-5-118){{- if .Values.image.runAsUser }} [](#__codelineno-5-119)runAsUser: [](#__codelineno-5-120){{- with .Values.runAsUser }} [](#__codelineno-5-121){{- toYaml . | nindent 2 }} [](#__codelineno-5-122){{- end }} [](#__codelineno-5-123){{- end }} [](#__codelineno-5-124){{- end }} [](#__codelineno-5-125) [](#__codelineno-5-126)[](#__codelineno-5-127){{- define "chart.extraInitEnv" -}} [](#__codelineno-5-128)- name: S3_ENDPOINT_URL [](#__codelineno-5-129) valueFrom: [](#__codelineno-5-130) secretKeyRef: [](#__codelineno-5-131) name: {{ .Release.Name }}-secrets [](#__codelineno-5-132) key: s3endpoint [](#__codelineno-5-133)- name: S3_BUCKET_NAME [](#__codelineno-5-134) valueFrom: [](#__codelineno-5-135) secretKeyRef: [](#__codelineno-5-136) name: {{ .Release.Name }}-secrets [](#__codelineno-5-137) key: s3bucketname [](#__codelineno-5-138)- name: AWS_ACCESS_KEY_ID [](#__codelineno-5-139) valueFrom: [](#__codelineno-5-140) secretKeyRef: [](#__codelineno-5-141) name: {{ .Release.Name }}-secrets [](#__codelineno-5-142) key: s3accesskeyid [](#__codelineno-5-143)- name: AWS_SECRET_ACCESS_KEY [](#__codelineno-5-144) valueFrom: [](#__codelineno-5-145) secretKeyRef: [](#__codelineno-5-146) name: {{ .Release.Name }}-secrets [](#__codelineno-5-147) key: s3accesskey [](#__codelineno-5-148){{- if .Values.extraInit.s3modelpath }} [](#__codelineno-5-149)- name: S3_PATH [](#__codelineno-5-150) value: "{{ .Values.extraInit.s3modelpath }}" [](#__codelineno-5-151){{- end }} [](#__codelineno-5-152){{- if hasKey .Values.extraInit "awsEc2MetadataDisabled" }} [](#__codelineno-5-153)- name: AWS_EC2_METADATA_DISABLED [](#__codelineno-5-154) value: "{{ .Values.extraInit.awsEc2MetadataDisabled }}" [](#__codelineno-5-155){{- end }} [](#__codelineno-5-156){{- end }} [](#__codelineno-5-157)[](#__codelineno-5-158){{/* [](#__codelineno-5-159) Define chart labels [](#__codelineno-5-160)*/}} [](#__codelineno-5-161){{- define "chart.labels" -}} [](#__codelineno-5-162){{- with .Values.labels -}} [](#__codelineno-5-163){{ toYaml . }} [](#__codelineno-5-164){{- end }} [](#__codelineno-5-165){{- end }}` templates/configmap.yaml `[](#__codelineno-6-1){{- if .Values.configs -}} [](#__codelineno-6-2)apiVersion: v1 [](#__codelineno-6-3)kind: ConfigMap [](#__codelineno-6-4)metadata: [](#__codelineno-6-5) name: "{{ .Release.Name }}-configs" [](#__codelineno-6-6) namespace: {{ .Release.Namespace }} [](#__codelineno-6-7)data: [](#__codelineno-6-8) {{- with .Values.configs }} [](#__codelineno-6-9) {{- toYaml . | nindent 2 }} [](#__codelineno-6-10) {{- end }} [](#__codelineno-6-11){{- end -}}` templates/custom-objects.yaml `[](#__codelineno-7-1){{- if .Values.customObjects }} [](#__codelineno-7-2){{- range .Values.customObjects }} [](#__codelineno-7-3){{- tpl (. | toYaml) $ }} [](#__codelineno-7-4)--- [](#__codelineno-7-5){{- end }} [](#__codelineno-7-6){{- end }}` templates/deployment.yaml `[](#__codelineno-8-1)apiVersion: apps/v1 [](#__codelineno-8-2)kind: Deployment [](#__codelineno-8-3)metadata: [](#__codelineno-8-4) name: "{{ .Release.Name }}-deployment-vllm" [](#__codelineno-8-5) namespace: {{ .Release.Namespace }} [](#__codelineno-8-6) labels: [](#__codelineno-8-7) {{- include "chart.labels" . | nindent 4 }} [](#__codelineno-8-8)spec: [](#__codelineno-8-9) replicas: {{ .Values.replicaCount }} [](#__codelineno-8-10) {{- include "chart.strategy" . | nindent 2 }} [](#__codelineno-8-11) selector: [](#__codelineno-8-12) matchLabels: [](#__codelineno-8-13) environment: "test" [](#__codelineno-8-14) release: "test" [](#__codelineno-8-15) progressDeadlineSeconds: 1200 [](#__codelineno-8-16) template: [](#__codelineno-8-17) metadata: [](#__codelineno-8-18) labels: [](#__codelineno-8-19) environment: "test" [](#__codelineno-8-20) release: "test" [](#__codelineno-8-21) spec: [](#__codelineno-8-22) containers: [](#__codelineno-8-23) - name: "vllm" [](#__codelineno-8-24) image: "{{ required "Required value 'image.repository' must be defined !" .Values.image.repository }}:{{ required "Required value 'image.tag' must be defined !" .Values.image.tag }}" [](#__codelineno-8-25) {{- if .Values.image.command }} [](#__codelineno-8-26) command : [](#__codelineno-8-27) {{- with .Values.image.command }} [](#__codelineno-8-28) {{- toYaml . | nindent 10 }} [](#__codelineno-8-29) {{- end }} [](#__codelineno-8-30) {{- end }} [](#__codelineno-8-31) securityContext: [](#__codelineno-8-32) {{- if .Values.image.securityContext }} [](#__codelineno-8-33) {{- with .Values.image.securityContext }} [](#__codelineno-8-34) {{- toYaml . | nindent 12 }} [](#__codelineno-8-35) {{- end }} [](#__codelineno-8-36) {{- else }} [](#__codelineno-8-37) runAsNonRoot: false [](#__codelineno-8-38) {{- include "chart.user" . | indent 12 }} [](#__codelineno-8-39) {{- end }} [](#__codelineno-8-40) imagePullPolicy: IfNotPresent [](#__codelineno-8-41) {{- if .Values.image.env }} [](#__codelineno-8-42) env : [](#__codelineno-8-43) {{- with .Values.image.env }} [](#__codelineno-8-44) {{- toYaml . | nindent 10 }} [](#__codelineno-8-45) {{- end }} [](#__codelineno-8-46) {{- else }} [](#__codelineno-8-47) env: [] [](#__codelineno-8-48) {{- end }} [](#__codelineno-8-49) {{- if or .Values.externalConfigs .Values.configs .Values.secrets }} [](#__codelineno-8-50) envFrom: [](#__codelineno-8-51) {{- if .Values.configs }} [](#__codelineno-8-52) - configMapRef: [](#__codelineno-8-53) name: "{{ .Release.Name }}-configs" [](#__codelineno-8-54) {{- end }} [](#__codelineno-8-55) {{- if .Values.secrets}} [](#__codelineno-8-56) - secretRef: [](#__codelineno-8-57) name: "{{ .Release.Name }}-secrets" [](#__codelineno-8-58) {{- end }} [](#__codelineno-8-59) {{- include "chart.externalConfigs" . | nindent 12 }} [](#__codelineno-8-60) {{- end }} [](#__codelineno-8-61) ports: [](#__codelineno-8-62) - name: {{ include "chart.container-port-name" . }} [](#__codelineno-8-63) containerPort: {{ include "chart.container-port" . }} [](#__codelineno-8-64) {{- include "chart.extraPorts" . | nindent 12 }} [](#__codelineno-8-65) {{- include "chart.probes" . | indent 10 }} [](#__codelineno-8-66) resources: {{- include "chart.resources" . | nindent 12 }} [](#__codelineno-8-67) volumeMounts: [](#__codelineno-8-68) - name: {{ .Release.Name }}-storage [](#__codelineno-8-69) mountPath: /data [](#__codelineno-8-70) [](#__codelineno-8-71) {{- with .Values.extraContainers }} [](#__codelineno-8-72) {{ toYaml . | nindent 8 }} [](#__codelineno-8-73) {{- end }} [](#__codelineno-8-74) [](#__codelineno-8-75) {{- if and .Values.extraInit (or .Values.extraInit.modelDownload.enabled .Values.extraInit.initContainers) }} [](#__codelineno-8-76) initContainers: [](#__codelineno-8-77) {{- if .Values.extraInit.modelDownload.enabled }} [](#__codelineno-8-78) - name: wait-download-model [](#__codelineno-8-79) image: {{ .Values.extraInit.modelDownload.image.repository }}:{{ .Values.extraInit.modelDownload.image.tag }} [](#__codelineno-8-80) imagePullPolicy: {{ .Values.extraInit.modelDownload.image.pullPolicy }} [](#__codelineno-8-81) command: {{ .Values.extraInit.modelDownload.waitContainer.command | toJson }} [](#__codelineno-8-82) args: [](#__codelineno-8-83) {{- toYaml .Values.extraInit.modelDownload.waitContainer.args | nindent 10 }} [](#__codelineno-8-84) env: [](#__codelineno-8-85) {{- if .Values.extraInit.modelDownload.waitContainer.env }} [](#__codelineno-8-86) {{- toYaml .Values.extraInit.modelDownload.waitContainer.env | nindent 10 }} [](#__codelineno-8-87) {{- else }} [](#__codelineno-8-88) {{- include "chart.extraInitEnv" . | nindent 10 }} [](#__codelineno-8-89) {{- end }} [](#__codelineno-8-90) resources: [](#__codelineno-8-91) requests: [](#__codelineno-8-92) cpu: 200m [](#__codelineno-8-93) memory: 1Gi [](#__codelineno-8-94) limits: [](#__codelineno-8-95) cpu: 500m [](#__codelineno-8-96) memory: 2Gi [](#__codelineno-8-97) volumeMounts: [](#__codelineno-8-98) - name: {{ .Release.Name }}-storage [](#__codelineno-8-99) mountPath: /data [](#__codelineno-8-100) {{- end }} [](#__codelineno-8-101) {{- with .Values.extraInit.initContainers }} [](#__codelineno-8-102) {{- toYaml . | nindent 6 }} [](#__codelineno-8-103) {{- end }} [](#__codelineno-8-104) {{- end }} [](#__codelineno-8-105) volumes: [](#__codelineno-8-106) - name: {{ .Release.Name }}-storage [](#__codelineno-8-107) persistentVolumeClaim: [](#__codelineno-8-108) claimName: {{ .Release.Name }}-storage-claim [](#__codelineno-8-109) [](#__codelineno-8-110) {{- with .Values.nodeSelector }} [](#__codelineno-8-111) nodeSelector: [](#__codelineno-8-112) {{- toYaml . | nindent 8 }} [](#__codelineno-8-113) {{- end }} [](#__codelineno-8-114) {{- with .Values.tolerations }} [](#__codelineno-8-115) tolerations: [](#__codelineno-8-116) {{- toYaml . | nindent 8 }} [](#__codelineno-8-117) {{- end }} [](#__codelineno-8-118) {{- if and (gt (int (index .Values.resources.requests "nvidia.com/gpu")) 0) (gt (int (index .Values.resources.limits "nvidia.com/gpu")) 0) }} [](#__codelineno-8-119) runtimeClassName: nvidia [](#__codelineno-8-120) affinity: [](#__codelineno-8-121) nodeAffinity: [](#__codelineno-8-122) requiredDuringSchedulingIgnoredDuringExecution: [](#__codelineno-8-123) nodeSelectorTerms: [](#__codelineno-8-124) - matchExpressions: [](#__codelineno-8-125) - key: nvidia.com/gpu.product [](#__codelineno-8-126) operator: In [](#__codelineno-8-127) {{- with .Values.gpuModels }} [](#__codelineno-8-128) values: [](#__codelineno-8-129) {{- toYaml . | nindent 20 }} [](#__codelineno-8-130) {{- end }} [](#__codelineno-8-131) {{- end }}` templates/hpa.yaml `[](#__codelineno-9-1){{- if .Values.autoscaling.enabled }} [](#__codelineno-9-2)apiVersion: autoscaling/v2 [](#__codelineno-9-3)kind: HorizontalPodAutoscaler [](#__codelineno-9-4)metadata: [](#__codelineno-9-5) name: "{{ .Release.Name }}-hpa" [](#__codelineno-9-6) namespace: {{ .Release.Namespace }} [](#__codelineno-9-7)spec: [](#__codelineno-9-8) scaleTargetRef: [](#__codelineno-9-9) apiVersion: apps/v1 [](#__codelineno-9-10) kind: Deployment [](#__codelineno-9-11) name: vllm [](#__codelineno-9-12) minReplicas: {{ .Values.autoscaling.minReplicas }} [](#__codelineno-9-13) maxReplicas: {{ .Values.autoscaling.maxReplicas }} [](#__codelineno-9-14) metrics: [](#__codelineno-9-15) {{- if .Values.autoscaling.targetCPUUtilizationPercentage }} [](#__codelineno-9-16) - type: Resource [](#__codelineno-9-17) resource: [](#__codelineno-9-18) name: cpu [](#__codelineno-9-19) target: [](#__codelineno-9-20) type: Utilization [](#__codelineno-9-21) averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }} [](#__codelineno-9-22) {{- end }} [](#__codelineno-9-23) {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }} [](#__codelineno-9-24) - type: Resource [](#__codelineno-9-25) resource: [](#__codelineno-9-26) name: memory [](#__codelineno-9-27) target: [](#__codelineno-9-28) type: Utilization [](#__codelineno-9-29) averageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }} [](#__codelineno-9-30) {{- end }} [](#__codelineno-9-31){{- end }}` templates/job.yaml `[](#__codelineno-10-1){{- if and .Values.extraInit .Values.extraInit.modelDownload.enabled }} [](#__codelineno-10-2)apiVersion: batch/v1 [](#__codelineno-10-3)kind: Job [](#__codelineno-10-4)metadata: [](#__codelineno-10-5) name: "{{ .Release.Name }}-init-vllm" [](#__codelineno-10-6) namespace: {{ .Release.Namespace }} [](#__codelineno-10-7)spec: [](#__codelineno-10-8) ttlSecondsAfterFinished: 100 [](#__codelineno-10-9) template: [](#__codelineno-10-10) metadata: [](#__codelineno-10-11) name: init-vllm [](#__codelineno-10-12) spec: [](#__codelineno-10-13) containers: [](#__codelineno-10-14) - name: job-download-model [](#__codelineno-10-15) image: {{ .Values.extraInit.modelDownload.image.repository }}:{{ .Values.extraInit.modelDownload.image.tag }} [](#__codelineno-10-16) imagePullPolicy: {{ .Values.extraInit.modelDownload.image.pullPolicy }} [](#__codelineno-10-17) command: {{ .Values.extraInit.modelDownload.downloadJob.command | toJson }} [](#__codelineno-10-18) args: [](#__codelineno-10-19) {{- toYaml .Values.extraInit.modelDownload.downloadJob.args | nindent 8 }} [](#__codelineno-10-20) env: [](#__codelineno-10-21) {{- if .Values.extraInit.modelDownload.downloadJob.env }} [](#__codelineno-10-22) {{- toYaml .Values.extraInit.modelDownload.downloadJob.env | nindent 8 }} [](#__codelineno-10-23) {{- else }} [](#__codelineno-10-24) {{- include "chart.extraInitEnv" . | nindent 8 }} [](#__codelineno-10-25) {{- end }} [](#__codelineno-10-26) volumeMounts: [](#__codelineno-10-27) - name: {{ .Release.Name }}-storage [](#__codelineno-10-28) mountPath: /data [](#__codelineno-10-29) resources: [](#__codelineno-10-30) requests: [](#__codelineno-10-31) cpu: 200m [](#__codelineno-10-32) memory: 1Gi [](#__codelineno-10-33) limits: [](#__codelineno-10-34) cpu: 500m [](#__codelineno-10-35) memory: 2Gi [](#__codelineno-10-36) restartPolicy: OnFailure [](#__codelineno-10-37) volumes: [](#__codelineno-10-38) - name: {{ .Release.Name }}-storage [](#__codelineno-10-39) persistentVolumeClaim: [](#__codelineno-10-40) claimName: "{{ .Release.Name }}-storage-claim" [](#__codelineno-10-41){{- end }}` templates/poddisruptionbudget.yaml `[](#__codelineno-11-1)apiVersion: policy/v1 [](#__codelineno-11-2)kind: PodDisruptionBudget [](#__codelineno-11-3)metadata: [](#__codelineno-11-4) name: "{{ .Release.Name }}-pdb" [](#__codelineno-11-5) namespace: {{ .Release.Namespace }} [](#__codelineno-11-6)spec: [](#__codelineno-11-7) maxUnavailable: {{ default 1 .Values.maxUnavailablePodDisruptionBudget }}` templates/pvc.yaml `[](#__codelineno-12-1){{- if .Values.extraInit }} [](#__codelineno-12-2)apiVersion: v1 [](#__codelineno-12-3)kind: PersistentVolumeClaim [](#__codelineno-12-4)metadata: [](#__codelineno-12-5) name: "{{ .Release.Name }}-storage-claim" [](#__codelineno-12-6) namespace: {{ .Release.Namespace }} [](#__codelineno-12-7)spec: [](#__codelineno-12-8) accessModes: [](#__codelineno-12-9) - ReadWriteOnce [](#__codelineno-12-10) resources: [](#__codelineno-12-11) requests: [](#__codelineno-12-12) storage: {{ .Values.extraInit.pvcStorage }} [](#__codelineno-12-13){{- end }}` templates/secrets.yaml `[](#__codelineno-13-1)apiVersion: v1 [](#__codelineno-13-2)kind: Secret [](#__codelineno-13-3)metadata: [](#__codelineno-13-4) name: "{{ .Release.Name }}-secrets" [](#__codelineno-13-5) namespace: {{ .Release.Namespace }} [](#__codelineno-13-6)type: Opaque [](#__codelineno-13-7)data: [](#__codelineno-13-8) {{- range $key, $val := .Values.secrets }} [](#__codelineno-13-9) {{ $key }}: {{ $val | b64enc | quote }} [](#__codelineno-13-10) {{- end }}` templates/service.yaml `[](#__codelineno-14-1)apiVersion: v1 [](#__codelineno-14-2)kind: Service [](#__codelineno-14-3)metadata: [](#__codelineno-14-4) name: "{{ .Release.Name }}-service" [](#__codelineno-14-5) namespace: {{ .Release.Namespace }} [](#__codelineno-14-6)spec: [](#__codelineno-14-7) type: ClusterIP [](#__codelineno-14-8) ports: [](#__codelineno-14-9) - name: {{ include "chart.service-port-name" . }} [](#__codelineno-14-10) port: {{ include "chart.service-port" . }} [](#__codelineno-14-11) targetPort: {{ include "chart.container-port-name" . }} [](#__codelineno-14-12) protocol: TCP [](#__codelineno-14-13) selector: [](#__codelineno-14-14) {{- include "chart.labels" . | nindent 4 }}` tests/deployment\_test.yaml `[](#__codelineno-15-1)suite: test deployment [](#__codelineno-15-2)templates: [](#__codelineno-15-3) - deployment.yaml [](#__codelineno-15-4)tests: [](#__codelineno-15-5) - it: should create wait-download-model init container when modelDownload is enabled [](#__codelineno-15-6) set: [](#__codelineno-15-7) extraInit: [](#__codelineno-15-8) modelDownload: [](#__codelineno-15-9) enabled: true [](#__codelineno-15-10) image: [](#__codelineno-15-11) repository: "amazon/aws-cli" [](#__codelineno-15-12) tag: "2.6.4" [](#__codelineno-15-13) pullPolicy: "IfNotPresent" [](#__codelineno-15-14) waitContainer: [](#__codelineno-15-15) command: [ "/bin/bash" ] [](#__codelineno-15-16) args: [](#__codelineno-15-17) - "-eucx" [](#__codelineno-15-18) - "while aws --endpoint-url $S3_ENDPOINT_URL s3 sync --dryrun s3://$S3_BUCKET_NAME/$S3_PATH /data | grep -q download; do sleep 10; done" [](#__codelineno-15-19) downloadJob: [](#__codelineno-15-20) command: [ "/bin/bash" ] [](#__codelineno-15-21) args: [](#__codelineno-15-22) - "-eucx" [](#__codelineno-15-23) - "aws --endpoint-url $S3_ENDPOINT_URL s3 sync s3://$S3_BUCKET_NAME/$S3_PATH /data" [](#__codelineno-15-24) initContainers: [ ] [](#__codelineno-15-25) pvcStorage: "1Gi" [](#__codelineno-15-26) s3modelpath: "relative_s3_model_path/opt-125m" [](#__codelineno-15-27) awsEc2MetadataDisabled: true [](#__codelineno-15-28) asserts: [](#__codelineno-15-29) - hasDocuments: [](#__codelineno-15-30) count: 1 [](#__codelineno-15-31) - isKind: [](#__codelineno-15-32) of: Deployment [](#__codelineno-15-33) - isNotEmpty: [](#__codelineno-15-34) path: spec.template.spec.initContainers [](#__codelineno-15-35) - equal: [](#__codelineno-15-36) path: spec.template.spec.initContainers[0].name [](#__codelineno-15-37) value: wait-download-model [](#__codelineno-15-38) - equal: [](#__codelineno-15-39) path: spec.template.spec.initContainers[0].image [](#__codelineno-15-40) value: amazon/aws-cli:2.6.4 [](#__codelineno-15-41) - equal: [](#__codelineno-15-42) path: spec.template.spec.initContainers[0].imagePullPolicy [](#__codelineno-15-43) value: IfNotPresent [](#__codelineno-15-44) [](#__codelineno-15-45) - it: should only create custom init containers when modelDownload is disabled [](#__codelineno-15-46) set: [](#__codelineno-15-47) extraInit: [](#__codelineno-15-48) modelDownload: [](#__codelineno-15-49) enabled: false [](#__codelineno-15-50) image: [](#__codelineno-15-51) repository: "amazon/aws-cli" [](#__codelineno-15-52) tag: "2.6.4" [](#__codelineno-15-53) pullPolicy: "IfNotPresent" [](#__codelineno-15-54) waitContainer: [](#__codelineno-15-55) command: [ "/bin/bash" ] [](#__codelineno-15-56) args: [ "-c", "echo test" ] [](#__codelineno-15-57) downloadJob: [](#__codelineno-15-58) command: [ "/bin/bash" ] [](#__codelineno-15-59) args: [ "-c", "echo test" ] [](#__codelineno-15-60) initContainers: [](#__codelineno-15-61) - name: llm-d-routing-proxy [](#__codelineno-15-62) image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.2.0 [](#__codelineno-15-63) imagePullPolicy: IfNotPresent [](#__codelineno-15-64) ports: [](#__codelineno-15-65) - containerPort: 8080 [](#__codelineno-15-66) name: proxy [](#__codelineno-15-67) pvcStorage: "10Gi" [](#__codelineno-15-68) asserts: [](#__codelineno-15-69) - hasDocuments: [](#__codelineno-15-70) count: 1 [](#__codelineno-15-71) - isKind: [](#__codelineno-15-72) of: Deployment [](#__codelineno-15-73) - lengthEqual: [](#__codelineno-15-74) path: spec.template.spec.initContainers [](#__codelineno-15-75) count: 1 [](#__codelineno-15-76) - equal: [](#__codelineno-15-77) path: spec.template.spec.initContainers[0].name [](#__codelineno-15-78) value: llm-d-routing-proxy [](#__codelineno-15-79) - equal: [](#__codelineno-15-80) path: spec.template.spec.initContainers[0].image [](#__codelineno-15-81) value: ghcr.io/llm-d/llm-d-routing-sidecar:v0.2.0 [](#__codelineno-15-82) - equal: [](#__codelineno-15-83) path: spec.template.spec.initContainers[0].ports[0].containerPort [](#__codelineno-15-84) value: 8080 [](#__codelineno-15-85) [](#__codelineno-15-86) - it: should create both wait-download-model and custom init containers when both are enabled [](#__codelineno-15-87) set: [](#__codelineno-15-88) extraInit: [](#__codelineno-15-89) modelDownload: [](#__codelineno-15-90) enabled: true [](#__codelineno-15-91) image: [](#__codelineno-15-92) repository: "amazon/aws-cli" [](#__codelineno-15-93) tag: "2.6.4" [](#__codelineno-15-94) pullPolicy: "IfNotPresent" [](#__codelineno-15-95) waitContainer: [](#__codelineno-15-96) command: [ "/bin/bash" ] [](#__codelineno-15-97) args: [](#__codelineno-15-98) - "-eucx" [](#__codelineno-15-99) - "while aws --endpoint-url $S3_ENDPOINT_URL s3 sync --dryrun s3://$S3_BUCKET_NAME/$S3_PATH /data | grep -q download; do sleep 10; done" [](#__codelineno-15-100) downloadJob: [](#__codelineno-15-101) command: [ "/bin/bash" ] [](#__codelineno-15-102) args: [](#__codelineno-15-103) - "-eucx" [](#__codelineno-15-104) - "aws --endpoint-url $S3_ENDPOINT_URL s3 sync s3://$S3_BUCKET_NAME/$S3_PATH /data" [](#__codelineno-15-105) initContainers: [](#__codelineno-15-106) - name: llm-d-routing-proxy [](#__codelineno-15-107) image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.2.0 [](#__codelineno-15-108) imagePullPolicy: IfNotPresent [](#__codelineno-15-109) ports: [](#__codelineno-15-110) - containerPort: 8080 [](#__codelineno-15-111) name: proxy [](#__codelineno-15-112) pvcStorage: "10Gi" [](#__codelineno-15-113) asserts: [](#__codelineno-15-114) - hasDocuments: [](#__codelineno-15-115) count: 1 [](#__codelineno-15-116) - isKind: [](#__codelineno-15-117) of: Deployment [](#__codelineno-15-118) - lengthEqual: [](#__codelineno-15-119) path: spec.template.spec.initContainers [](#__codelineno-15-120) count: 2 [](#__codelineno-15-121) - equal: [](#__codelineno-15-122) path: spec.template.spec.initContainers[0].name [](#__codelineno-15-123) value: wait-download-model [](#__codelineno-15-124) - equal: [](#__codelineno-15-125) path: spec.template.spec.initContainers[0].image [](#__codelineno-15-126) value: amazon/aws-cli:2.6.4 [](#__codelineno-15-127) - equal: [](#__codelineno-15-128) path: spec.template.spec.initContainers[1].name [](#__codelineno-15-129) value: llm-d-routing-proxy [](#__codelineno-15-130) - equal: [](#__codelineno-15-131) path: spec.template.spec.initContainers[1].image [](#__codelineno-15-132) value: ghcr.io/llm-d/llm-d-routing-sidecar:v0.2.0 [](#__codelineno-15-133) - equal: [](#__codelineno-15-134) path: spec.template.spec.initContainers[1].ports[0].containerPort [](#__codelineno-15-135) value: 8080` tests/job\_test.yaml `[](#__codelineno-16-1)suite: test job [](#__codelineno-16-2)templates: [](#__codelineno-16-3) - job.yaml [](#__codelineno-16-4)tests: [](#__codelineno-16-5) - it: should create job when modelDownload is enabled [](#__codelineno-16-6) set: [](#__codelineno-16-7) extraInit: [](#__codelineno-16-8) modelDownload: [](#__codelineno-16-9) enabled: true [](#__codelineno-16-10) image: [](#__codelineno-16-11) repository: "amazon/aws-cli" [](#__codelineno-16-12) tag: "2.6.4" [](#__codelineno-16-13) pullPolicy: "IfNotPresent" [](#__codelineno-16-14) waitContainer: [](#__codelineno-16-15) command: [ "/bin/bash" ] [](#__codelineno-16-16) args: [ "-c", "wait" ] [](#__codelineno-16-17) downloadJob: [](#__codelineno-16-18) command: [ "/bin/bash" ] [](#__codelineno-16-19) args: [](#__codelineno-16-20) - "-eucx" [](#__codelineno-16-21) - "aws --endpoint-url $S3_ENDPOINT_URL s3 sync s3://$S3_BUCKET_NAME/$S3_PATH /data" [](#__codelineno-16-22) pvcStorage: "1Gi" [](#__codelineno-16-23) s3modelpath: "relative_s3_model_path/opt-125m" [](#__codelineno-16-24) awsEc2MetadataDisabled: true [](#__codelineno-16-25) asserts: [](#__codelineno-16-26) - hasDocuments: [](#__codelineno-16-27) count: 1 [](#__codelineno-16-28) - isKind: [](#__codelineno-16-29) of: Job [](#__codelineno-16-30) - equal: [](#__codelineno-16-31) path: spec.template.spec.containers[0].name [](#__codelineno-16-32) value: job-download-model [](#__codelineno-16-33) - equal: [](#__codelineno-16-34) path: spec.template.spec.containers[0].image [](#__codelineno-16-35) value: amazon/aws-cli:2.6.4 [](#__codelineno-16-36) - equal: [](#__codelineno-16-37) path: spec.template.spec.restartPolicy [](#__codelineno-16-38) value: OnFailure [](#__codelineno-16-39) [](#__codelineno-16-40) - it: should not create job when modelDownload is disabled [](#__codelineno-16-41) set: [](#__codelineno-16-42) extraInit: [](#__codelineno-16-43) modelDownload: [](#__codelineno-16-44) enabled: false [](#__codelineno-16-45) image: [](#__codelineno-16-46) repository: "amazon/aws-cli" [](#__codelineno-16-47) tag: "2.6.4" [](#__codelineno-16-48) pullPolicy: "IfNotPresent" [](#__codelineno-16-49) waitContainer: [](#__codelineno-16-50) command: [ "/bin/bash" ] [](#__codelineno-16-51) args: [ "-c", "wait" ] [](#__codelineno-16-52) downloadJob: [](#__codelineno-16-53) command: [ "/bin/bash" ] [](#__codelineno-16-54) args: [ "-c", "download" ] [](#__codelineno-16-55) initContainers: [](#__codelineno-16-56) - name: llm-d-routing-proxy [](#__codelineno-16-57) image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.2.0 [](#__codelineno-16-58) pvcStorage: "10Gi" [](#__codelineno-16-59) asserts: [](#__codelineno-16-60) - hasDocuments: [](#__codelineno-16-61) count: 0` tests/pvc\_test.yaml `[](#__codelineno-17-1)suite: test pvc [](#__codelineno-17-2)templates: [](#__codelineno-17-3) - pvc.yaml [](#__codelineno-17-4)tests: [](#__codelineno-17-5) # Test Case: PVC Created When extraInit Defined [](#__codelineno-17-6) - it: should create pvc when extraInit is defined [](#__codelineno-17-7) set: [](#__codelineno-17-8) extraInit: [](#__codelineno-17-9) modelDownload: [](#__codelineno-17-10) enabled: true [](#__codelineno-17-11) image: [](#__codelineno-17-12) repository: "amazon/aws-cli" [](#__codelineno-17-13) tag: "2.6.4" [](#__codelineno-17-14) pullPolicy: "IfNotPresent" [](#__codelineno-17-15) waitContainer: [](#__codelineno-17-16) command: ["/bin/bash"] [](#__codelineno-17-17) args: ["-c", "wait"] [](#__codelineno-17-18) downloadJob: [](#__codelineno-17-19) command: ["/bin/bash"] [](#__codelineno-17-20) args: ["-c", "download"] [](#__codelineno-17-21) pvcStorage: "10Gi" [](#__codelineno-17-22) asserts: [](#__codelineno-17-23) - hasDocuments: [](#__codelineno-17-24) count: 1 [](#__codelineno-17-25) - isKind: [](#__codelineno-17-26) of: PersistentVolumeClaim [](#__codelineno-17-27) - equal: [](#__codelineno-17-28) path: spec.accessModes[0] [](#__codelineno-17-29) value: ReadWriteOnce [](#__codelineno-17-30) - equal: [](#__codelineno-17-31) path: spec.resources.requests.storage [](#__codelineno-17-32) value: 10Gi` values.schema.json `[](#__codelineno-18-1){ [](#__codelineno-18-2) "$schema": "http://json-schema.org/schema#", [](#__codelineno-18-3) "type": "object", [](#__codelineno-18-4) "properties": { [](#__codelineno-18-5) "image": { [](#__codelineno-18-6) "type": "object", [](#__codelineno-18-7) "properties": { [](#__codelineno-18-8) "repository": { [](#__codelineno-18-9) "type": "string" [](#__codelineno-18-10) }, [](#__codelineno-18-11) "tag": { [](#__codelineno-18-12) "type": "string" [](#__codelineno-18-13) }, [](#__codelineno-18-14) "command": { [](#__codelineno-18-15) "type": "array", [](#__codelineno-18-16) "items": { [](#__codelineno-18-17) "type": "string" [](#__codelineno-18-18) } [](#__codelineno-18-19) } [](#__codelineno-18-20) }, [](#__codelineno-18-21) "required": [ [](#__codelineno-18-22) "command", [](#__codelineno-18-23) "repository", [](#__codelineno-18-24) "tag" [](#__codelineno-18-25) ] [](#__codelineno-18-26) }, [](#__codelineno-18-27) "containerPort": { [](#__codelineno-18-28) "type": "integer" [](#__codelineno-18-29) }, [](#__codelineno-18-30) "serviceName": { [](#__codelineno-18-31) "type": "null" [](#__codelineno-18-32) }, [](#__codelineno-18-33) "servicePort": { [](#__codelineno-18-34) "type": "integer" [](#__codelineno-18-35) }, [](#__codelineno-18-36) "extraPorts": { [](#__codelineno-18-37) "type": "array" [](#__codelineno-18-38) }, [](#__codelineno-18-39) "replicaCount": { [](#__codelineno-18-40) "type": "integer" [](#__codelineno-18-41) }, [](#__codelineno-18-42) "deploymentStrategy": { [](#__codelineno-18-43) "type": "object" [](#__codelineno-18-44) }, [](#__codelineno-18-45) "resources": { [](#__codelineno-18-46) "type": "object", [](#__codelineno-18-47) "properties": { [](#__codelineno-18-48) "requests": { [](#__codelineno-18-49) "type": "object", [](#__codelineno-18-50) "properties": { [](#__codelineno-18-51) "cpu": { [](#__codelineno-18-52) "type": "integer" [](#__codelineno-18-53) }, [](#__codelineno-18-54) "memory": { [](#__codelineno-18-55) "type": "string" [](#__codelineno-18-56) }, [](#__codelineno-18-57) "nvidia.com/gpu": { [](#__codelineno-18-58) "type": "integer" [](#__codelineno-18-59) } [](#__codelineno-18-60) }, [](#__codelineno-18-61) "required": [ [](#__codelineno-18-62) "cpu", [](#__codelineno-18-63) "memory", [](#__codelineno-18-64) "nvidia.com/gpu" [](#__codelineno-18-65) ] [](#__codelineno-18-66) }, [](#__codelineno-18-67) "limits": { [](#__codelineno-18-68) "type": "object", [](#__codelineno-18-69) "properties": { [](#__codelineno-18-70) "cpu": { [](#__codelineno-18-71) "type": "integer" [](#__codelineno-18-72) }, [](#__codelineno-18-73) "memory": { [](#__codelineno-18-74) "type": "string" [](#__codelineno-18-75) }, [](#__codelineno-18-76) "nvidia.com/gpu": { [](#__codelineno-18-77) "type": "integer" [](#__codelineno-18-78) } [](#__codelineno-18-79) }, [](#__codelineno-18-80) "required": [ [](#__codelineno-18-81) "cpu", [](#__codelineno-18-82) "memory", [](#__codelineno-18-83) "nvidia.com/gpu" [](#__codelineno-18-84) ] [](#__codelineno-18-85) } [](#__codelineno-18-86) }, [](#__codelineno-18-87) "required": [ [](#__codelineno-18-88) "limits", [](#__codelineno-18-89) "requests" [](#__codelineno-18-90) ] [](#__codelineno-18-91) }, [](#__codelineno-18-92) "gpuModels": { [](#__codelineno-18-93) "type": "array", [](#__codelineno-18-94) "items": { [](#__codelineno-18-95) "type": "string" [](#__codelineno-18-96) } [](#__codelineno-18-97) }, [](#__codelineno-18-98) "autoscaling": { [](#__codelineno-18-99) "type": "object", [](#__codelineno-18-100) "properties": { [](#__codelineno-18-101) "enabled": { [](#__codelineno-18-102) "type": "boolean" [](#__codelineno-18-103) }, [](#__codelineno-18-104) "minReplicas": { [](#__codelineno-18-105) "type": "integer" [](#__codelineno-18-106) }, [](#__codelineno-18-107) "maxReplicas": { [](#__codelineno-18-108) "type": "integer" [](#__codelineno-18-109) }, [](#__codelineno-18-110) "targetCPUUtilizationPercentage": { [](#__codelineno-18-111) "type": "integer" [](#__codelineno-18-112) } [](#__codelineno-18-113) }, [](#__codelineno-18-114) "required": [ [](#__codelineno-18-115) "enabled", [](#__codelineno-18-116) "maxReplicas", [](#__codelineno-18-117) "minReplicas", [](#__codelineno-18-118) "targetCPUUtilizationPercentage" [](#__codelineno-18-119) ] [](#__codelineno-18-120) }, [](#__codelineno-18-121) "configs": { [](#__codelineno-18-122) "type": "object" [](#__codelineno-18-123) }, [](#__codelineno-18-124) "secrets": { [](#__codelineno-18-125) "type": "object" [](#__codelineno-18-126) }, [](#__codelineno-18-127) "externalConfigs": { [](#__codelineno-18-128) "type": "array" [](#__codelineno-18-129) }, [](#__codelineno-18-130) "customObjects": { [](#__codelineno-18-131) "type": "array" [](#__codelineno-18-132) }, [](#__codelineno-18-133) "maxUnavailablePodDisruptionBudget": { [](#__codelineno-18-134) "type": "string" [](#__codelineno-18-135) }, [](#__codelineno-18-136) "extraInit": { [](#__codelineno-18-137) "type": "object", [](#__codelineno-18-138) "properties": { [](#__codelineno-18-139) "modelDownload": { [](#__codelineno-18-140) "type": "object", [](#__codelineno-18-141) "properties": { [](#__codelineno-18-142) "enabled": { [](#__codelineno-18-143) "type": "boolean" [](#__codelineno-18-144) }, [](#__codelineno-18-145) "image": { [](#__codelineno-18-146) "type": "object", [](#__codelineno-18-147) "properties": { [](#__codelineno-18-148) "repository": { [](#__codelineno-18-149) "type": "string" [](#__codelineno-18-150) }, [](#__codelineno-18-151) "tag": { [](#__codelineno-18-152) "type": "string" [](#__codelineno-18-153) }, [](#__codelineno-18-154) "pullPolicy": { [](#__codelineno-18-155) "type": "string" [](#__codelineno-18-156) } [](#__codelineno-18-157) }, [](#__codelineno-18-158) "required": ["repository", "tag", "pullPolicy"] [](#__codelineno-18-159) }, [](#__codelineno-18-160) "waitContainer": { [](#__codelineno-18-161) "type": "object", [](#__codelineno-18-162) "properties": { [](#__codelineno-18-163) "command": { [](#__codelineno-18-164) "type": "array", [](#__codelineno-18-165) "items": {"type": "string"} [](#__codelineno-18-166) }, [](#__codelineno-18-167) "args": { [](#__codelineno-18-168) "type": "array", [](#__codelineno-18-169) "items": {"type": "string"} [](#__codelineno-18-170) }, [](#__codelineno-18-171) "env": { [](#__codelineno-18-172) "type": "array", [](#__codelineno-18-173) "items": {"type": "object"} [](#__codelineno-18-174) } [](#__codelineno-18-175) }, [](#__codelineno-18-176) "required": ["command", "args"] [](#__codelineno-18-177) }, [](#__codelineno-18-178) "downloadJob": { [](#__codelineno-18-179) "type": "object", [](#__codelineno-18-180) "properties": { [](#__codelineno-18-181) "command": { [](#__codelineno-18-182) "type": "array", [](#__codelineno-18-183) "items": {"type": "string"} [](#__codelineno-18-184) }, [](#__codelineno-18-185) "args": { [](#__codelineno-18-186) "type": "array", [](#__codelineno-18-187) "items": {"type": "string"} [](#__codelineno-18-188) }, [](#__codelineno-18-189) "env": { [](#__codelineno-18-190) "type": "array", [](#__codelineno-18-191) "items": {"type": "object"} [](#__codelineno-18-192) } [](#__codelineno-18-193) }, [](#__codelineno-18-194) "required": ["command", "args"] [](#__codelineno-18-195) } [](#__codelineno-18-196) }, [](#__codelineno-18-197) "required": ["enabled", "image", "waitContainer", "downloadJob"] [](#__codelineno-18-198) }, [](#__codelineno-18-199) "initContainers": { [](#__codelineno-18-200) "type": "array", [](#__codelineno-18-201) "items": {"type": "object"} [](#__codelineno-18-202) }, [](#__codelineno-18-203) "s3modelpath": { [](#__codelineno-18-204) "type": "string" [](#__codelineno-18-205) }, [](#__codelineno-18-206) "pvcStorage": { [](#__codelineno-18-207) "type": "string" [](#__codelineno-18-208) }, [](#__codelineno-18-209) "awsEc2MetadataDisabled": { [](#__codelineno-18-210) "type": "boolean" [](#__codelineno-18-211) } [](#__codelineno-18-212) }, [](#__codelineno-18-213) "required": [ [](#__codelineno-18-214) "modelDownload", [](#__codelineno-18-215) "initContainers", [](#__codelineno-18-216) "pvcStorage" [](#__codelineno-18-217) ] [](#__codelineno-18-218) }, [](#__codelineno-18-219) "extraContainers": { [](#__codelineno-18-220) "type": "array" [](#__codelineno-18-221) }, [](#__codelineno-18-222) "readinessProbe": { [](#__codelineno-18-223) "type": "object", [](#__codelineno-18-224) "properties": { [](#__codelineno-18-225) "initialDelaySeconds": { [](#__codelineno-18-226) "type": "integer" [](#__codelineno-18-227) }, [](#__codelineno-18-228) "periodSeconds": { [](#__codelineno-18-229) "type": "integer" [](#__codelineno-18-230) }, [](#__codelineno-18-231) "failureThreshold": { [](#__codelineno-18-232) "type": "integer" [](#__codelineno-18-233) }, [](#__codelineno-18-234) "httpGet": { [](#__codelineno-18-235) "type": "object", [](#__codelineno-18-236) "properties": { [](#__codelineno-18-237) "path": { [](#__codelineno-18-238) "type": "string" [](#__codelineno-18-239) }, [](#__codelineno-18-240) "port": { [](#__codelineno-18-241) "type": "integer" [](#__codelineno-18-242) } [](#__codelineno-18-243) }, [](#__codelineno-18-244) "required": [ [](#__codelineno-18-245) "path", [](#__codelineno-18-246) "port" [](#__codelineno-18-247) ] [](#__codelineno-18-248) } [](#__codelineno-18-249) }, [](#__codelineno-18-250) "required": [ [](#__codelineno-18-251) "failureThreshold", [](#__codelineno-18-252) "httpGet", [](#__codelineno-18-253) "initialDelaySeconds", [](#__codelineno-18-254) "periodSeconds" [](#__codelineno-18-255) ] [](#__codelineno-18-256) }, [](#__codelineno-18-257) "livenessProbe": { [](#__codelineno-18-258) "type": "object", [](#__codelineno-18-259) "properties": { [](#__codelineno-18-260) "initialDelaySeconds": { [](#__codelineno-18-261) "type": "integer" [](#__codelineno-18-262) }, [](#__codelineno-18-263) "failureThreshold": { [](#__codelineno-18-264) "type": "integer" [](#__codelineno-18-265) }, [](#__codelineno-18-266) "periodSeconds": { [](#__codelineno-18-267) "type": "integer" [](#__codelineno-18-268) }, [](#__codelineno-18-269) "httpGet": { [](#__codelineno-18-270) "type": "object", [](#__codelineno-18-271) "properties": { [](#__codelineno-18-272) "path": { [](#__codelineno-18-273) "type": "string" [](#__codelineno-18-274) }, [](#__codelineno-18-275) "port": { [](#__codelineno-18-276) "type": "integer" [](#__codelineno-18-277) } [](#__codelineno-18-278) }, [](#__codelineno-18-279) "required": [ [](#__codelineno-18-280) "path", [](#__codelineno-18-281) "port" [](#__codelineno-18-282) ] [](#__codelineno-18-283) } [](#__codelineno-18-284) }, [](#__codelineno-18-285) "required": [ [](#__codelineno-18-286) "failureThreshold", [](#__codelineno-18-287) "httpGet", [](#__codelineno-18-288) "initialDelaySeconds", [](#__codelineno-18-289) "periodSeconds" [](#__codelineno-18-290) ] [](#__codelineno-18-291) }, [](#__codelineno-18-292) "labels": { [](#__codelineno-18-293) "type": "object", [](#__codelineno-18-294) "properties": { [](#__codelineno-18-295) "environment": { [](#__codelineno-18-296) "type": "string" [](#__codelineno-18-297) }, [](#__codelineno-18-298) "release": { [](#__codelineno-18-299) "type": "string" [](#__codelineno-18-300) } [](#__codelineno-18-301) }, [](#__codelineno-18-302) "required": [ [](#__codelineno-18-303) "environment", [](#__codelineno-18-304) "release" [](#__codelineno-18-305) ] [](#__codelineno-18-306) } [](#__codelineno-18-307) }, [](#__codelineno-18-308) "required": [ [](#__codelineno-18-309) "autoscaling", [](#__codelineno-18-310) "configs", [](#__codelineno-18-311) "containerPort", [](#__codelineno-18-312) "customObjects", [](#__codelineno-18-313) "deploymentStrategy", [](#__codelineno-18-314) "externalConfigs", [](#__codelineno-18-315) "extraContainers", [](#__codelineno-18-316) "extraInit", [](#__codelineno-18-317) "extraPorts", [](#__codelineno-18-318) "gpuModels", [](#__codelineno-18-319) "image", [](#__codelineno-18-320) "labels", [](#__codelineno-18-321) "livenessProbe", [](#__codelineno-18-322) "maxUnavailablePodDisruptionBudget", [](#__codelineno-18-323) "readinessProbe", [](#__codelineno-18-324) "replicaCount", [](#__codelineno-18-325) "resources", [](#__codelineno-18-326) "secrets", [](#__codelineno-18-327) "servicePort" [](#__codelineno-18-328) ] [](#__codelineno-18-329)}` values.yaml `[](#__codelineno-19-1)# -- Default values for chart vllm [](#__codelineno-19-2)# -- Declare variables to be passed into your templates. [](#__codelineno-19-3)[](#__codelineno-19-4)# -- Image configuration [](#__codelineno-19-5)image: [](#__codelineno-19-6) # -- Image repository [](#__codelineno-19-7) repository: "vllm/vllm-openai" [](#__codelineno-19-8) # -- Image tag [](#__codelineno-19-9) tag: "latest" [](#__codelineno-19-10) # -- Container launch command [](#__codelineno-19-11) command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--enforce-eager", "--dtype", "bfloat16", "--block-size", "16", "--host", "0.0.0.0", "--port", "8000"] [](#__codelineno-19-12)[](#__codelineno-19-13)# -- Container port [](#__codelineno-19-14)containerPort: 8000 [](#__codelineno-19-15)# -- Service name [](#__codelineno-19-16)serviceName: [](#__codelineno-19-17)# -- Service port [](#__codelineno-19-18)servicePort: 80 [](#__codelineno-19-19)# -- Additional ports configuration [](#__codelineno-19-20)extraPorts: [] [](#__codelineno-19-21)[](#__codelineno-19-22)# -- Number of replicas [](#__codelineno-19-23)replicaCount: 1 [](#__codelineno-19-24)[](#__codelineno-19-25)# -- Deployment strategy configuration [](#__codelineno-19-26)deploymentStrategy: {} [](#__codelineno-19-27)[](#__codelineno-19-28)# -- Resource configuration [](#__codelineno-19-29)resources: [](#__codelineno-19-30) requests: [](#__codelineno-19-31) # -- Number of CPUs [](#__codelineno-19-32) cpu: 4 [](#__codelineno-19-33) # -- CPU memory configuration [](#__codelineno-19-34) memory: 16Gi [](#__codelineno-19-35) # -- Number of gpus used [](#__codelineno-19-36) nvidia.com/gpu: 1 [](#__codelineno-19-37) limits: [](#__codelineno-19-38) # -- Number of CPUs [](#__codelineno-19-39) cpu: 4 [](#__codelineno-19-40) # -- CPU memory configuration [](#__codelineno-19-41) memory: 16Gi [](#__codelineno-19-42) # -- Number of gpus used [](#__codelineno-19-43) nvidia.com/gpu: 1 [](#__codelineno-19-44)[](#__codelineno-19-45)# -- Type of gpu used [](#__codelineno-19-46)gpuModels: [](#__codelineno-19-47) - "TYPE_GPU_USED" [](#__codelineno-19-48)[](#__codelineno-19-49)# -- Autoscaling configuration [](#__codelineno-19-50)autoscaling: [](#__codelineno-19-51) # -- Enable autoscaling [](#__codelineno-19-52) enabled: false [](#__codelineno-19-53) # -- Minimum replicas [](#__codelineno-19-54) minReplicas: 1 [](#__codelineno-19-55) # -- Maximum replicas [](#__codelineno-19-56) maxReplicas: 100 [](#__codelineno-19-57) # -- Target CPU utilization for autoscaling [](#__codelineno-19-58) targetCPUUtilizationPercentage: 80 [](#__codelineno-19-59) # targetMemoryUtilizationPercentage: 80 [](#__codelineno-19-60)[](#__codelineno-19-61)# -- Configmap [](#__codelineno-19-62)configs: {} [](#__codelineno-19-63)[](#__codelineno-19-64)# -- Secrets configuration [](#__codelineno-19-65)secrets: {} [](#__codelineno-19-66)[](#__codelineno-19-67)# -- External configuration [](#__codelineno-19-68)externalConfigs: [] [](#__codelineno-19-69)[](#__codelineno-19-70)# -- Custom Objects configuration [](#__codelineno-19-71)customObjects: [] [](#__codelineno-19-72)[](#__codelineno-19-73)# -- Disruption Budget Configuration [](#__codelineno-19-74)maxUnavailablePodDisruptionBudget: "" [](#__codelineno-19-75)[](#__codelineno-19-76)# -- Additional configuration for the init container [](#__codelineno-19-77)extraInit: [](#__codelineno-19-78) # -- Model download functionality (optional) [](#__codelineno-19-79) modelDownload: [](#__codelineno-19-80) # -- Enable model download job and wait container [](#__codelineno-19-81) enabled: true [](#__codelineno-19-82) # -- Image configuration for model download operations [](#__codelineno-19-83) image: [](#__codelineno-19-84) # -- Image repository [](#__codelineno-19-85) repository: "amazon/aws-cli" [](#__codelineno-19-86) # -- Image tag [](#__codelineno-19-87) tag: "2.6.4" [](#__codelineno-19-88) # -- Image pull policy [](#__codelineno-19-89) pullPolicy: "IfNotPresent" [](#__codelineno-19-90) # -- Wait container configuration (init container that waits for model to be ready) [](#__codelineno-19-91) waitContainer: [](#__codelineno-19-92) # -- Command to execute [](#__codelineno-19-93) command: ["/bin/bash"] [](#__codelineno-19-94) # -- Arguments for the wait container [](#__codelineno-19-95) args: [](#__codelineno-19-96) - "-eucx" [](#__codelineno-19-97) - "while aws --endpoint-url $S3_ENDPOINT_URL s3 sync --dryrun s3://$S3_BUCKET_NAME/$S3_PATH /data | grep -q download; do sleep 10; done" [](#__codelineno-19-98) # -- Environment variables (optional, overrides S3 defaults entirely if specified) [](#__codelineno-19-99) # env: [](#__codelineno-19-100) # - name: HUGGING_FACE_HUB_TOKEN [](#__codelineno-19-101) # value: "your-token" [](#__codelineno-19-102) # - name: MODEL_ID [](#__codelineno-19-103) # value: "meta-llama/Llama-2-7b" [](#__codelineno-19-104) # -- Download job configuration (job that actually downloads the model) [](#__codelineno-19-105) downloadJob: [](#__codelineno-19-106) # -- Command to execute [](#__codelineno-19-107) command: ["/bin/bash"] [](#__codelineno-19-108) # -- Arguments for the download job [](#__codelineno-19-109) args: [](#__codelineno-19-110) - "-eucx" [](#__codelineno-19-111) - "aws --endpoint-url $S3_ENDPOINT_URL s3 sync s3://$S3_BUCKET_NAME/$S3_PATH /data" [](#__codelineno-19-112) # -- Environment variables (optional, overrides S3 defaults entirely if specified) [](#__codelineno-19-113) # env: [](#__codelineno-19-114) # - name: HUGGING_FACE_HUB_TOKEN [](#__codelineno-19-115) # value: "your-token" [](#__codelineno-19-116) # - name: MODEL_ID [](#__codelineno-19-117) # value: "meta-llama/Llama-2-7b" [](#__codelineno-19-118) [](#__codelineno-19-119) # -- Custom init containers (appended after wait-download-model if modelDownload is enabled) [](#__codelineno-19-120) initContainers: [] [](#__codelineno-19-121) # Example for llm-d sidecar: [](#__codelineno-19-122) # initContainers: [](#__codelineno-19-123) # - name: llm-d-routing-proxy [](#__codelineno-19-124) # image: ghcr.io/llm-d/llm-d-routing-sidecar:v0.2.0 [](#__codelineno-19-125) # imagePullPolicy: IfNotPresent [](#__codelineno-19-126) # ports: [](#__codelineno-19-127) # - containerPort: 8080 [](#__codelineno-19-128) # name: proxy [](#__codelineno-19-129) # securityContext: [](#__codelineno-19-130) # runAsUser: 1000 [](#__codelineno-19-131) [](#__codelineno-19-132) # -- Path of the model on the s3 which hosts model weights and config files [](#__codelineno-19-133) s3modelpath: "relative_s3_model_path/opt-125m" [](#__codelineno-19-134) # -- Storage size for the PVC [](#__codelineno-19-135) pvcStorage: "1Gi" [](#__codelineno-19-136) # -- Disable AWS EC2 metadata service [](#__codelineno-19-137) awsEc2MetadataDisabled: true [](#__codelineno-19-138)[](#__codelineno-19-139)# -- Additional containers configuration [](#__codelineno-19-140)extraContainers: [] [](#__codelineno-19-141)[](#__codelineno-19-142)# -- Readiness probe configuration [](#__codelineno-19-143)readinessProbe: [](#__codelineno-19-144) # -- Number of seconds after the container has started before readiness probe is initiated [](#__codelineno-19-145) initialDelaySeconds: 5 [](#__codelineno-19-146) # -- How often (in seconds) to perform the readiness probe [](#__codelineno-19-147) periodSeconds: 5 [](#__codelineno-19-148) # -- Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not ready [](#__codelineno-19-149) failureThreshold: 3 [](#__codelineno-19-150) # -- Configuration of the Kubelet http request on the server [](#__codelineno-19-151) httpGet: [](#__codelineno-19-152) # -- Path to access on the HTTP server [](#__codelineno-19-153) path: /health [](#__codelineno-19-154) # -- Name or number of the port to access on the container, on which the server is listening [](#__codelineno-19-155) port: 8000 [](#__codelineno-19-156)[](#__codelineno-19-157)# -- Liveness probe configuration [](#__codelineno-19-158)livenessProbe: [](#__codelineno-19-159) # -- Number of seconds after the container has started before liveness probe is initiated [](#__codelineno-19-160) initialDelaySeconds: 15 [](#__codelineno-19-161) # -- Number of times after which if a probe fails in a row, Kubernetes considers that the overall check has failed: the container is not alive [](#__codelineno-19-162) failureThreshold: 3 [](#__codelineno-19-163) # -- How often (in seconds) to perform the liveness probe [](#__codelineno-19-164) periodSeconds: 10 [](#__codelineno-19-165) # -- Configuration of the Kubelet http request on the server [](#__codelineno-19-166) httpGet: [](#__codelineno-19-167) # -- Path to access on the HTTP server [](#__codelineno-19-168) path: /health [](#__codelineno-19-169) # -- Name or number of the port to access on the container, on which the server is listening [](#__codelineno-19-170) port: 8000 [](#__codelineno-19-171)[](#__codelineno-19-172)labels: [](#__codelineno-19-173) environment: "test" [](#__codelineno-19-174) release: "test"` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) 5. [Deployment](https://docs.vllm.ai/en/latest/examples/async_llm_streaming/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/deployment/llm_engine_example.md "Edit this page") Source [https://github.com/vllm-project/vllm/blob/main/examples/deployment/llm\_engine\_example.py](https://github.com/vllm-project/vllm/blob/main/examples/deployment/llm_engine_example.py). ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This file demonstrates using the `LLMEngine` [](#__codelineno-0-5)for processing prompts with various sampling parameters. [](#__codelineno-0-6)""" [](#__codelineno-0-7)[](#__codelineno-0-8)import argparse [](#__codelineno-0-9)[](#__codelineno-0-10)from vllm import EngineArgs, LLMEngine, RequestOutput, SamplingParams [](#__codelineno-0-11)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-0-12) [](#__codelineno-0-13)[](#__codelineno-0-14)def create_test_prompts() -> list[tuple[str, SamplingParams]]: [](#__codelineno-0-15) """Create a list of test prompts with their sampling parameters.""" [](#__codelineno-0-16) return [ [](#__codelineno-0-17) ( [](#__codelineno-0-18) "A robot may not injure a human being", [](#__codelineno-0-19) SamplingParams(temperature=0.0, logprobs=1, prompt_logprobs=1), [](#__codelineno-0-20) ), [](#__codelineno-0-21) ( [](#__codelineno-0-22) "To be or not to be,", [](#__codelineno-0-23) SamplingParams(temperature=0.8, top_k=5, presence_penalty=0.2), [](#__codelineno-0-24) ), [](#__codelineno-0-25) ( [](#__codelineno-0-26) "What is the meaning of life?", [](#__codelineno-0-27) SamplingParams(n=2, temperature=0.8, top_p=0.95, frequency_penalty=0.1), [](#__codelineno-0-28) ), [](#__codelineno-0-29) ] [](#__codelineno-0-30) [](#__codelineno-0-31)[](#__codelineno-0-32)def process_requests(engine: LLMEngine, test_prompts: list[tuple[str, SamplingParams]]): [](#__codelineno-0-33) """Continuously process a list of prompts and handle the outputs.""" [](#__codelineno-0-34) request_id = 0 [](#__codelineno-0-35) [](#__codelineno-0-36) print("-" * 50) [](#__codelineno-0-37) while test_prompts or engine.has_unfinished_requests(): [](#__codelineno-0-38) if test_prompts: [](#__codelineno-0-39) prompt, sampling_params = test_prompts.pop(0) [](#__codelineno-0-40) engine.add_request(str(request_id), prompt, sampling_params) [](#__codelineno-0-41) request_id += 1 [](#__codelineno-0-42) [](#__codelineno-0-43) request_outputs: list[RequestOutput] = engine.step() [](#__codelineno-0-44) [](#__codelineno-0-45) for request_output in request_outputs: [](#__codelineno-0-46) if request_output.finished: [](#__codelineno-0-47) print(request_output) [](#__codelineno-0-48) print("-" * 50) [](#__codelineno-0-49) [](#__codelineno-0-50)[](#__codelineno-0-51)def initialize_engine(args: argparse.Namespace) -> LLMEngine: [](#__codelineno-0-52) """Initialize the LLMEngine from the command line arguments.""" [](#__codelineno-0-53) engine_args = EngineArgs.from_cli_args(args) [](#__codelineno-0-54) return LLMEngine.from_engine_args(engine_args) [](#__codelineno-0-55) [](#__codelineno-0-56)[](#__codelineno-0-57)def parse_args(): [](#__codelineno-0-58) parser = FlexibleArgumentParser( [](#__codelineno-0-59) description="Demo on using the LLMEngine class directly" [](#__codelineno-0-60) ) [](#__codelineno-0-61) parser = EngineArgs.add_cli_args(parser) [](#__codelineno-0-62) return parser.parse_args() [](#__codelineno-0-63) [](#__codelineno-0-64)[](#__codelineno-0-65)def main(args: argparse.Namespace): [](#__codelineno-0-66) """Main function that sets up and runs the prompt processing.""" [](#__codelineno-0-67) engine = initialize_engine(args) [](#__codelineno-0-68) test_prompts = create_test_prompts() [](#__codelineno-0-69) process_requests(engine, test_prompts) [](#__codelineno-0-70) [](#__codelineno-0-71)[](#__codelineno-0-72)if __name__ == "__main__": [](#__codelineno-0-73) args = parse_args() [](#__codelineno-0-74) main(args)`` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) 5. [Deployment](https://docs.vllm.ai/en/latest/examples/async_llm_streaming/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/deployment/sagemaker-entrypoint.md "Edit this page") Source [https://github.com/vllm-project/vllm/blob/main/examples/deployment/sagemaker-entrypoint.sh](https://github.com/vllm-project/vllm/blob/main/examples/deployment/sagemaker-entrypoint.sh). `[](#__codelineno-0-1)#!/bin/bash [](#__codelineno-0-2)[](#__codelineno-0-3)# Define the prefix for environment variables to look for [](#__codelineno-0-4)PREFIX="SM_VLLM_" [](#__codelineno-0-5)ARG_PREFIX="--" [](#__codelineno-0-6)[](#__codelineno-0-7)# Initialize an array for storing the arguments [](#__codelineno-0-8)# port 8080 required by sagemaker, https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html#your-algorithms-inference-code-container-response [](#__codelineno-0-9)ARGS=(--port 8080) [](#__codelineno-0-10)[](#__codelineno-0-11)# Loop through all environment variables [](#__codelineno-0-12)while IFS='=' read -r key value; do [](#__codelineno-0-13) # Remove the prefix from the key, convert to lowercase, and replace underscores with dashes [](#__codelineno-0-14) arg_name=$(echo "${key#"${PREFIX}"}" | tr '[:upper:]' '[:lower:]' | tr '_' '-') [](#__codelineno-0-15) [](#__codelineno-0-16) # Add the argument name and value to the ARGS array [](#__codelineno-0-17) ARGS+=("${ARG_PREFIX}${arg_name}") [](#__codelineno-0-18) if [ -n "$value" ]; then [](#__codelineno-0-19) ARGS+=("$value") [](#__codelineno-0-20) fi [](#__codelineno-0-21)done < <(env | grep "^${PREFIX}") [](#__codelineno-0-22)[](#__codelineno-0-23)# Pass the collected arguments to the main entrypoint [](#__codelineno-0-24)exec standard-supervisor vllm serve "${ARGS[@]}"` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/disaggregated_encoder.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/disaggregated\_encoder](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/disaggregated_encoder). These example scripts that demonstrate the disaggregated encoder (EPD) features of vLLM. For a detailed explanation of the EPD features, please refer to the [Disaggregated Encoder Feature Documentation](https://github.com/vllm-project/vllm/tree/main/docs/features/disagg_encoder.md). ## Files[¶](#files "Permanent link") - `disagg_epd_proxy.py` - Proxy script that demonstrates the XeYpZd setup (X encode instances, Y prefill instances, Z decode instances). Currently stable for the 1e1p1d configuration. - `disagg_1e1p1d_example.sh` - Sets up the 1e1p1d configuration, runs the VisionArena benchmark, and processes a single request with a local image. - `disagg_1e1pd_example.sh` - Sets up the 1e1pd configuration, runs the VisionArena benchmark, and processes a single request with a local image. ### Custom Configuration[¶](#custom-configuration "Permanent link") `[](#__codelineno-0-1)# Use specific GPUs [](#__codelineno-0-2)GPU_E=0 GPU_PD=1 GPU_P=1 GPU_D=2 bash disagg_1e1p1d_example.sh [](#__codelineno-0-3)[](#__codelineno-0-4)# Use specific ports [](#__codelineno-0-5)ENDPOINT_PORT=10001 bash disagg_1e1p1d_example.sh [](#__codelineno-0-6)[](#__codelineno-0-7)# Use specific model [](#__codelineno-0-8)MODEL="Qwen/Qwen2.5-VL-3B-Instruct" bash disagg_1e1p1d_example.sh [](#__codelineno-0-9)[](#__codelineno-0-10)# Use specific storage path [](#__codelineno-0-11)EC_SHARED_STORAGE_PATH="/tmp/my_ec_cache" bash disagg_1e1p1d_example.sh [](#__codelineno-0-12)[](#__codelineno-0-13)# Run on XPU; scripts switch from CUDA_VISIBLE_DEVICES to ZE_AFFINITY_MASK [](#__codelineno-0-14)DEVICE_PLATFORM=xpu GPU_E=0 GPU_PD=1 bash disagg_1e1pd_example.sh` `DEVICE_PLATFORM` defaults to `cuda`. Set `DEVICE_PLATFORM=xpu` when running these examples on Intel GPUs so the scripts use `ZE_AFFINITY_MASK` instead of `CUDA_VISIBLE_DEVICES` for device selection. ## Encoder Instances[¶](#encoder-instances "Permanent link") Encoder engines should be launched with the following flags: - `--enforce-eager` **(required)** – The current EPD implementation is only compatible with encoder instances running in this mode. - `--no-enable-prefix-caching` **(required)** – Encoder instances do not consume KV cache; prefix caching is disabled to avoid conflicts with other features. - `--max-num-batched-tokens=` **(default: 2048)** – This flag controls the token scheduling budget per decoding step and is irrelevant to encoder-only instances. **Set it to a very high value (effectively unlimited) to bypass scheduler limitations.** The actual token budget is managed by the encoder cache manager. - `--mm-encoder-only` **(Optional)** - If possible, skips the language model during initialization to reduce device memory usage. To support local image inputs (from your `MEDIA_PATH` directory), add the following flag to the encoder instance: `[](#__codelineno-1-1)--allowed-local-media-path $MEDIA_PATH` The vllm instances and `disagg_encoder_proxy` supports local URIs with `{"url": "file://'"$MEDIA_PATH_FILENAME"'}` as multimodal inputs. Each URI is passed unchanged from the `disagg_encoder_proxy` to the encoder instance so that the encoder can load the media locally. ## EC connector and KV transfer[¶](#ec-connector-and-kv-transfer "Permanent link") The `ECExampleonnector` is used to store the encoder cache on local disk and facilitate transfer. To enable the encoder disaggregation feature, add the following configuration: `[](#__codelineno-2-1)# Add to encoder instance: [](#__codelineno-2-2)--ec-transfer-config '{ [](#__codelineno-2-3) "ec_connector": "ECExampleConnector", [](#__codelineno-2-4) "ec_role": "ec_producer", [](#__codelineno-2-5) "ec_connector_extra_config": { [](#__codelineno-2-6) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-2-7) } [](#__codelineno-2-8)}' [](#__codelineno-2-9)[](#__codelineno-2-10)# Add to prefill/prefill+decode instance: [](#__codelineno-2-11)--ec-transfer-config '{ [](#__codelineno-2-12) "ec_connector": "ECExampleConnector", [](#__codelineno-2-13) "ec_role": "ec_consumer", [](#__codelineno-2-14) "ec_connector_extra_config": { [](#__codelineno-2-15) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-2-16) } [](#__codelineno-2-17)}'` `$EC_SHARED_STORAGE_PATH` is the path where the EC connector temporarily stores the cache. If you enable prefill instance (`--prefill-servers-urls` not disabled), you will need --kv-transfer-config to facilitate the PD disaggregation. Currently, we use the `NixlConnector` for this purpose. Refer to `tests/v1/kv_connector/nixl_integration` for more example codes on PD disaggregation with Nixl. `[](#__codelineno-3-1)# Add to prefill instance: [](#__codelineno-3-2)--kv-transfer-config '{ [](#__codelineno-3-3) "kv_connector": "NixlConnector", [](#__codelineno-3-4) "kv_role": "kv_producer" [](#__codelineno-3-5)}' [](#__codelineno-3-6)[](#__codelineno-3-7)# Add to decode instance: [](#__codelineno-3-8)--kv-transfer-config '{ [](#__codelineno-3-9) "kv_connector": "NixlConnector", [](#__codelineno-3-10) "kv_role": "kv_consumer" [](#__codelineno-3-11)}'` ## Proxy Instance Flags (`disagg_epd_proxy.py`)[¶](#proxy-instance-flags-disagg_epd_proxypy "Permanent link") Flag Description `--encode-servers-urls` Comma-separated list of encoder endpoints. Every multimodal item extracted from the request is fanned out to one of these URLs in a round-robin fashion. `--prefill-servers-urls` Comma-separated list of prefill endpoints. Set to `disable`, `none`, or `""` to skip the dedicated prefill phase and run E+PD (encoder + combined prefill/decode). `--decode-servers-urls` Comma-separated list of decode endpoints. Non-stream and stream paths both round-robin over this list. `--host`, `--port` Bind address for the proxy itself (defaults: `0.0.0.0:8000`). Example usage: For E + PD setup: `[](#__codelineno-4-1)$ python disagg_encoder_proxy.py \ [](#__codelineno-4-2) --encode-servers-urls "http://e1:8001,http://e2:8002" \ [](#__codelineno-4-3) --prefill-servers-urls "disable" \ [](#__codelineno-4-4) --decode-servers-urls "http://pd1:8003,http://pd2:8004"` For E + P + D setup: `[](#__codelineno-5-1)$ python disagg_encoder_proxy.py \ [](#__codelineno-5-2) --encode-servers-urls "http://e1:8001,http://e2:8001" \ [](#__codelineno-5-3) --prefill-servers-urls "http://p1:8003,http://p2:8004" \ [](#__codelineno-5-4) --decode-servers-urls "http://d1:8005,http://d2:8006"` ## Example materials[¶](#example-materials "Permanent link") disagg\_1e1p1d\_example.sh `[](#__codelineno-6-1)#!/bin/bash [](#__codelineno-6-2)set -euo pipefail [](#__codelineno-6-3)[](#__codelineno-6-4)declare -a PIDS=() [](#__codelineno-6-5)[](#__codelineno-6-6)############################################################################### [](#__codelineno-6-7)# Configuration -- override via env before running [](#__codelineno-6-8)############################################################################### [](#__codelineno-6-9)MODEL="${MODEL:-Qwen/Qwen2.5-VL-3B-Instruct}" [](#__codelineno-6-10)LOG_PATH="${LOG_PATH:-./logs}" [](#__codelineno-6-11)mkdir -p "$LOG_PATH" [](#__codelineno-6-12)[](#__codelineno-6-13)ENCODE_PORT="${ENCODE_PORT:-19534}" [](#__codelineno-6-14)PREFILL_PORT="${PREFILL_PORT:-19535}" [](#__codelineno-6-15)DECODE_PORT="${DECODE_PORT:-19536}" [](#__codelineno-6-16)PROXY_PORT="${PROXY_PORT:-10001}" [](#__codelineno-6-17)[](#__codelineno-6-18)GPU_E="${GPU_E:-2}" [](#__codelineno-6-19)GPU_P="${GPU_P:-2}" [](#__codelineno-6-20)GPU_D="${GPU_D:-3}" [](#__codelineno-6-21)[](#__codelineno-6-22)# Device platform and affinity env name. [](#__codelineno-6-23)# DEVICE_PLATFORM supports: cuda, xpu [](#__codelineno-6-24)DEVICE_PLATFORM="${DEVICE_PLATFORM:-cuda}" [](#__codelineno-6-25)if [[ -z "${DEVICE_AFFINITY_ENV:-}" ]]; then [](#__codelineno-6-26) if [[ "${DEVICE_PLATFORM,,}" == "xpu" ]]; then [](#__codelineno-6-27) DEVICE_AFFINITY_ENV="ZE_AFFINITY_MASK" [](#__codelineno-6-28) else [](#__codelineno-6-29) DEVICE_AFFINITY_ENV="CUDA_VISIBLE_DEVICES" [](#__codelineno-6-30) fi [](#__codelineno-6-31)fi [](#__codelineno-6-32)[](#__codelineno-6-33)EC_SHARED_STORAGE_PATH="${EC_SHARED_STORAGE_PATH:-/tmp/ec_cache}" [](#__codelineno-6-34)TIMEOUT_SECONDS="${TIMEOUT_SECONDS:-12000}" # wait_for_server timeout [](#__codelineno-6-35)[](#__codelineno-6-36)NUM_PROMPTS="${NUM_PROMPTS:-100}" # number of prompts to send in benchmark [](#__codelineno-6-37)[](#__codelineno-6-38)# Serve args [](#__codelineno-6-39)GPU_MEMORY_UTILIZATION_E="${GPU_MEMORY_UTILIZATION_E:-0.01}" [](#__codelineno-6-40)GPU_MEMORY_UTILIZATION_P="${GPU_MEMORY_UTILIZATION_P:-0.7}" [](#__codelineno-6-41)GPU_MEMORY_UTILIZATION_D="${GPU_MEMORY_UTILIZATION_D:-0.7}" [](#__codelineno-6-42)MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" [](#__codelineno-6-43)MAX_MODEL_LEN="${MAX_MODEL_LEN:-32768}" [](#__codelineno-6-44)[](#__codelineno-6-45)export UCX_TLS=all [](#__codelineno-6-46)export UCX_NET_DEVICES=all [](#__codelineno-6-47)[](#__codelineno-6-48)############################################################################### [](#__codelineno-6-49)# Helpers [](#__codelineno-6-50)############################################################################### [](#__codelineno-6-51)# Find the git repository root directory [](#__codelineno-6-52)GIT_ROOT=$(git rev-parse --show-toplevel) [](#__codelineno-6-53)[](#__codelineno-6-54)START_TIME=$(date +"%Y%m%d_%H%M%S") [](#__codelineno-6-55)ENC_LOG=$LOG_PATH/encoder_${START_TIME}.log [](#__codelineno-6-56)P_LOG=$LOG_PATH/p_${START_TIME}.log [](#__codelineno-6-57)D_LOG=$LOG_PATH/d_${START_TIME}.log [](#__codelineno-6-58)PROXY_LOG=$LOG_PATH/proxy_${START_TIME}.log [](#__codelineno-6-59)[](#__codelineno-6-60)wait_for_server() { [](#__codelineno-6-61) local port=$1 [](#__codelineno-6-62) timeout "$TIMEOUT_SECONDS" bash -c " [](#__codelineno-6-63) until curl -s localhost:$port/v1/chat/completions > /dev/null; do [](#__codelineno-6-64) sleep 1 [](#__codelineno-6-65) done" && return 0 || return 1 [](#__codelineno-6-66)} [](#__codelineno-6-67)[](#__codelineno-6-68)# Cleanup function [](#__codelineno-6-69)cleanup() { [](#__codelineno-6-70) echo "Stopping everything…" [](#__codelineno-6-71) trap - INT TERM USR1 # prevent re-entrancy [](#__codelineno-6-72) [](#__codelineno-6-73) # Kill all tracked PIDs [](#__codelineno-6-74) for pid in "${PIDS[@]}"; do [](#__codelineno-6-75) if kill -0 "$pid" 2>/dev/null; then [](#__codelineno-6-76) echo "Killing process $pid" [](#__codelineno-6-77) kill "$pid" 2>/dev/null [](#__codelineno-6-78) fi [](#__codelineno-6-79) done [](#__codelineno-6-80) [](#__codelineno-6-81) # Wait a moment for graceful shutdown [](#__codelineno-6-82) sleep 2 [](#__codelineno-6-83) [](#__codelineno-6-84) # Force kill any remaining processes [](#__codelineno-6-85) for pid in "${PIDS[@]}"; do [](#__codelineno-6-86) if kill -0 "$pid" 2>/dev/null; then [](#__codelineno-6-87) echo "Force killing process $pid" [](#__codelineno-6-88) kill -9 "$pid" 2>/dev/null [](#__codelineno-6-89) fi [](#__codelineno-6-90) done [](#__codelineno-6-91) [](#__codelineno-6-92) # Kill the entire process group as backup [](#__codelineno-6-93) kill -- -$$ 2>/dev/null [](#__codelineno-6-94) [](#__codelineno-6-95) echo "All processes stopped." [](#__codelineno-6-96) exit 0 [](#__codelineno-6-97)} [](#__codelineno-6-98)[](#__codelineno-6-99)trap cleanup INT [](#__codelineno-6-100)trap cleanup USR1 [](#__codelineno-6-101)trap cleanup TERM [](#__codelineno-6-102)[](#__codelineno-6-103)# clear previous cache [](#__codelineno-6-104)echo "remove previous ec cache folder" [](#__codelineno-6-105)rm -rf "$EC_SHARED_STORAGE_PATH" [](#__codelineno-6-106)[](#__codelineno-6-107)echo "make ec cache folder" [](#__codelineno-6-108)mkdir -p "$EC_SHARED_STORAGE_PATH" [](#__codelineno-6-109)[](#__codelineno-6-110)############################################################################### [](#__codelineno-6-111)# Encoder worker [](#__codelineno-6-112)############################################################################### [](#__codelineno-6-113)env "$DEVICE_AFFINITY_ENV=$GPU_E" vllm serve "$MODEL" \ [](#__codelineno-6-114) --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION_E" \ [](#__codelineno-6-115) --port "$ENCODE_PORT" \ [](#__codelineno-6-116) --enforce-eager \ [](#__codelineno-6-117) --enable-request-id-headers \ [](#__codelineno-6-118) --no-enable-prefix-caching \ [](#__codelineno-6-119) --max-num-batched-tokens 114688 \ [](#__codelineno-6-120) --max-num-seqs "$MAX_NUM_SEQS" \ [](#__codelineno-6-121) --allowed-local-media-path "${GIT_ROOT}"/tests/v1/ec_connector/integration \ [](#__codelineno-6-122) --ec-transfer-config '{ [](#__codelineno-6-123) "ec_connector": "ECExampleConnector", [](#__codelineno-6-124) "ec_role": "ec_producer", [](#__codelineno-6-125) "ec_connector_extra_config": { [](#__codelineno-6-126) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-6-127) } [](#__codelineno-6-128) }' \ [](#__codelineno-6-129) >"${ENC_LOG}" 2>&1 & [](#__codelineno-6-130)[](#__codelineno-6-131)PIDS+=($!) [](#__codelineno-6-132)[](#__codelineno-6-133)############################################################################### [](#__codelineno-6-134)# Prefill worker [](#__codelineno-6-135)############################################################################### [](#__codelineno-6-136)env "$DEVICE_AFFINITY_ENV=$GPU_P" \ [](#__codelineno-6-137)UCX_NET_DEVICES=all \ [](#__codelineno-6-138)VLLM_NIXL_SIDE_CHANNEL_PORT=5559 \ [](#__codelineno-6-139)vllm serve "$MODEL" \ [](#__codelineno-6-140) --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION_P" \ [](#__codelineno-6-141) --port "$PREFILL_PORT" \ [](#__codelineno-6-142) --enforce-eager \ [](#__codelineno-6-143) --enable-request-id-headers \ [](#__codelineno-6-144) --max-num-seqs "$MAX_NUM_SEQS" \ [](#__codelineno-6-145) --max-model-len "$MAX_MODEL_LEN" \ [](#__codelineno-6-146) --allowed-local-media-path "${GIT_ROOT}"/tests/v1/ec_connector/integration \ [](#__codelineno-6-147) --ec-transfer-config '{ [](#__codelineno-6-148) "ec_connector": "ECExampleConnector", [](#__codelineno-6-149) "ec_role": "ec_consumer", [](#__codelineno-6-150) "ec_connector_extra_config": { [](#__codelineno-6-151) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-6-152) } [](#__codelineno-6-153) }' \ [](#__codelineno-6-154) --kv-transfer-config '{ [](#__codelineno-6-155) "kv_connector": "NixlConnector", [](#__codelineno-6-156) "kv_role": "kv_producer" [](#__codelineno-6-157) }' \ [](#__codelineno-6-158) >"${P_LOG}" 2>&1 & [](#__codelineno-6-159)[](#__codelineno-6-160)PIDS+=($!) [](#__codelineno-6-161)[](#__codelineno-6-162)############################################################################### [](#__codelineno-6-163)# Decode worker [](#__codelineno-6-164)############################################################################### [](#__codelineno-6-165)env "$DEVICE_AFFINITY_ENV=$GPU_D" \ [](#__codelineno-6-166)UCX_NET_DEVICES=all \ [](#__codelineno-6-167)VLLM_NIXL_SIDE_CHANNEL_PORT=6000 \ [](#__codelineno-6-168)vllm serve "$MODEL" \ [](#__codelineno-6-169) --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION_D" \ [](#__codelineno-6-170) --port "$DECODE_PORT" \ [](#__codelineno-6-171) --enforce-eager \ [](#__codelineno-6-172) --enable-request-id-headers \ [](#__codelineno-6-173) --max-num-seqs "$MAX_NUM_SEQS" \ [](#__codelineno-6-174) --max-model-len "$MAX_MODEL_LEN" \ [](#__codelineno-6-175) --allowed-local-media-path "${GIT_ROOT}"/tests/v1/ec_connector/integration \ [](#__codelineno-6-176) --kv-transfer-config '{ [](#__codelineno-6-177) "kv_connector": "NixlConnector", [](#__codelineno-6-178) "kv_role": "kv_consumer" [](#__codelineno-6-179) }' \ [](#__codelineno-6-180) >"${D_LOG}" 2>&1 & [](#__codelineno-6-181)[](#__codelineno-6-182)PIDS+=($!) [](#__codelineno-6-183)[](#__codelineno-6-184)# Wait for workers [](#__codelineno-6-185)wait_for_server "$ENCODE_PORT" [](#__codelineno-6-186)wait_for_server "$PREFILL_PORT" [](#__codelineno-6-187)wait_for_server "$DECODE_PORT" [](#__codelineno-6-188)[](#__codelineno-6-189)############################################################################### [](#__codelineno-6-190)# Proxy [](#__codelineno-6-191)############################################################################### [](#__codelineno-6-192)python disagg_epd_proxy.py \ [](#__codelineno-6-193) --host "0.0.0.0" \ [](#__codelineno-6-194) --port "$PROXY_PORT" \ [](#__codelineno-6-195) --encode-servers-urls "http://localhost:$ENCODE_PORT" \ [](#__codelineno-6-196) --prefill-servers-urls "http://localhost:$PREFILL_PORT" \ [](#__codelineno-6-197) --decode-servers-urls "http://localhost:$DECODE_PORT" \ [](#__codelineno-6-198) >"${PROXY_LOG}" 2>&1 & [](#__codelineno-6-199)[](#__codelineno-6-200)PIDS+=($!) [](#__codelineno-6-201)[](#__codelineno-6-202)wait_for_server "$PROXY_PORT" [](#__codelineno-6-203)echo "All services are up!" [](#__codelineno-6-204)[](#__codelineno-6-205)############################################################################### [](#__codelineno-6-206)# Benchmark [](#__codelineno-6-207)############################################################################### [](#__codelineno-6-208)echo "Running benchmark (stream)..." [](#__codelineno-6-209)vllm bench serve \ [](#__codelineno-6-210) --model "$MODEL" \ [](#__codelineno-6-211) --backend openai-chat \ [](#__codelineno-6-212) --endpoint /v1/chat/completions \ [](#__codelineno-6-213) --dataset-name hf \ [](#__codelineno-6-214) --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-6-215) --seed 0 \ [](#__codelineno-6-216) --num-prompts "$NUM_PROMPTS" \ [](#__codelineno-6-217) --port "$PROXY_PORT" [](#__codelineno-6-218)[](#__codelineno-6-219)PIDS+=($!) [](#__codelineno-6-220)[](#__codelineno-6-221)############################################################################### [](#__codelineno-6-222)# Single request with local image [](#__codelineno-6-223)############################################################################### [](#__codelineno-6-224)echo "Running single request with local image (non-stream)..." [](#__codelineno-6-225)curl http://127.0.0.1:"${PROXY_PORT}"/v1/chat/completions \ [](#__codelineno-6-226) -H "Content-Type: application/json" \ [](#__codelineno-6-227) -d '{ [](#__codelineno-6-228) "model": "'"${MODEL}"'", [](#__codelineno-6-229) "messages": [ [](#__codelineno-6-230) {"role": "system", "content": "You are a helpful assistant."}, [](#__codelineno-6-231) {"role": "user", "content": [ [](#__codelineno-6-232) {"type": "image_url", "image_url": {"url": "file://'"${GIT_ROOT}"'/tests/v1/ec_connector/integration/hato.jpg"}}, [](#__codelineno-6-233) {"type": "text", "text": "What is in this image?"} [](#__codelineno-6-234) ]} [](#__codelineno-6-235) ] [](#__codelineno-6-236) }' [](#__codelineno-6-237) [](#__codelineno-6-238)[](#__codelineno-6-239)# cleanup [](#__codelineno-6-240)echo "cleanup..." [](#__codelineno-6-241)cleanup` disagg\_1e1pd\_example.sh `[](#__codelineno-7-1)#!/bin/bash [](#__codelineno-7-2)set -euo pipefail [](#__codelineno-7-3)[](#__codelineno-7-4)declare -a PIDS=() [](#__codelineno-7-5)[](#__codelineno-7-6)############################################################################### [](#__codelineno-7-7)# Configuration -- override via env before running [](#__codelineno-7-8)############################################################################### [](#__codelineno-7-9)MODEL="${MODEL:-Qwen/Qwen2.5-VL-3B-Instruct}" [](#__codelineno-7-10)LOG_PATH="${LOG_PATH:-./logs}" [](#__codelineno-7-11)mkdir -p "$LOG_PATH" [](#__codelineno-7-12)[](#__codelineno-7-13)ENCODE_PORT="${ENCODE_PORT:-19534}" [](#__codelineno-7-14)PREFILL_DECODE_PORT="${PREFILL_DECODE_PORT:-19535}" [](#__codelineno-7-15)PROXY_PORT="${PROXY_PORT:-10001}" [](#__codelineno-7-16)[](#__codelineno-7-17)GPU_E="${GPU_E:-0}" [](#__codelineno-7-18)GPU_PD="${GPU_PD:-1}" [](#__codelineno-7-19)[](#__codelineno-7-20)# Device platform and affinity env name. [](#__codelineno-7-21)# DEVICE_PLATFORM supports: cuda, xpu [](#__codelineno-7-22)DEVICE_PLATFORM="${DEVICE_PLATFORM:-cuda}" [](#__codelineno-7-23)if [[ -z "${DEVICE_AFFINITY_ENV:-}" ]]; then [](#__codelineno-7-24) if [[ "${DEVICE_PLATFORM,,}" == "xpu" ]]; then [](#__codelineno-7-25) DEVICE_AFFINITY_ENV="ZE_AFFINITY_MASK" [](#__codelineno-7-26) else [](#__codelineno-7-27) DEVICE_AFFINITY_ENV="CUDA_VISIBLE_DEVICES" [](#__codelineno-7-28) fi [](#__codelineno-7-29)fi [](#__codelineno-7-30)[](#__codelineno-7-31)EC_SHARED_STORAGE_PATH="${EC_SHARED_STORAGE_PATH:-/tmp/ec_cache}" [](#__codelineno-7-32)TIMEOUT_SECONDS="${TIMEOUT_SECONDS:-12000}" # wait_for_server timeout [](#__codelineno-7-33)[](#__codelineno-7-34)NUM_PROMPTS="${NUM_PROMPTS:-100}" # number of prompts to send in benchmark [](#__codelineno-7-35)[](#__codelineno-7-36)# Serve args [](#__codelineno-7-37)GPU_MEMORY_UTILIZATION_E="${GPU_MEMORY_UTILIZATION_E:-0.01}" [](#__codelineno-7-38)GPU_MEMORY_UTILIZATION_PD="${GPU_MEMORY_UTILIZATION_PD:-0.7}" [](#__codelineno-7-39)MAX_NUM_SEQS="${MAX_NUM_SEQS:-128}" [](#__codelineno-7-40)MAX_MODEL_LEN="${MAX_MODEL_LEN:-32768}" [](#__codelineno-7-41)[](#__codelineno-7-42)############################################################################### [](#__codelineno-7-43)# Helpers [](#__codelineno-7-44)############################################################################### [](#__codelineno-7-45)# Find the git repository root directory [](#__codelineno-7-46)GIT_ROOT=$(git rev-parse --show-toplevel) [](#__codelineno-7-47)[](#__codelineno-7-48)START_TIME=$(date +"%Y%m%d_%H%M%S") [](#__codelineno-7-49)ENC_LOG=$LOG_PATH/encoder_${START_TIME}.log [](#__codelineno-7-50)PD_LOG=$LOG_PATH/pd_${START_TIME}.log [](#__codelineno-7-51)PROXY_LOG=$LOG_PATH/proxy_${START_TIME}.log [](#__codelineno-7-52)[](#__codelineno-7-53)wait_for_server() { [](#__codelineno-7-54) local port=$1 [](#__codelineno-7-55) timeout "$TIMEOUT_SECONDS" bash -c " [](#__codelineno-7-56) until curl -s localhost:$port/v1/chat/completions > /dev/null; do [](#__codelineno-7-57) sleep 1 [](#__codelineno-7-58) done" && return 0 || return 1 [](#__codelineno-7-59)} [](#__codelineno-7-60)[](#__codelineno-7-61)# Cleanup function [](#__codelineno-7-62)cleanup() { [](#__codelineno-7-63) echo "Stopping everything…" [](#__codelineno-7-64) trap - INT TERM USR1 # prevent re-entrancy [](#__codelineno-7-65) [](#__codelineno-7-66) # Kill all tracked PIDs [](#__codelineno-7-67) for pid in "${PIDS[@]}"; do [](#__codelineno-7-68) if kill -0 "$pid" 2>/dev/null; then [](#__codelineno-7-69) echo "Killing process $pid" [](#__codelineno-7-70) kill "$pid" 2>/dev/null [](#__codelineno-7-71) fi [](#__codelineno-7-72) done [](#__codelineno-7-73) [](#__codelineno-7-74) # Wait a moment for graceful shutdown [](#__codelineno-7-75) sleep 2 [](#__codelineno-7-76) [](#__codelineno-7-77) # Force kill any remaining processes [](#__codelineno-7-78) for pid in "${PIDS[@]}"; do [](#__codelineno-7-79) if kill -0 "$pid" 2>/dev/null; then [](#__codelineno-7-80) echo "Force killing process $pid" [](#__codelineno-7-81) kill -9 "$pid" 2>/dev/null [](#__codelineno-7-82) fi [](#__codelineno-7-83) done [](#__codelineno-7-84) [](#__codelineno-7-85) # Kill the entire process group as backup [](#__codelineno-7-86) kill -- -$$ 2>/dev/null [](#__codelineno-7-87) [](#__codelineno-7-88) echo "All processes stopped." [](#__codelineno-7-89) exit 0 [](#__codelineno-7-90)} [](#__codelineno-7-91)[](#__codelineno-7-92)trap cleanup INT [](#__codelineno-7-93)trap cleanup USR1 [](#__codelineno-7-94)trap cleanup TERM [](#__codelineno-7-95)[](#__codelineno-7-96)# clear previous cache [](#__codelineno-7-97)echo "remove previous ec cache folder" [](#__codelineno-7-98)rm -rf "$EC_SHARED_STORAGE_PATH" [](#__codelineno-7-99)[](#__codelineno-7-100)echo "make ec cache folder" [](#__codelineno-7-101)mkdir -p "$EC_SHARED_STORAGE_PATH" [](#__codelineno-7-102)[](#__codelineno-7-103)############################################################################### [](#__codelineno-7-104)# Encoder worker [](#__codelineno-7-105)############################################################################### [](#__codelineno-7-106)env "$DEVICE_AFFINITY_ENV=$GPU_E" vllm serve "$MODEL" \ [](#__codelineno-7-107) --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION_E" \ [](#__codelineno-7-108) --port "$ENCODE_PORT" \ [](#__codelineno-7-109) --enforce-eager \ [](#__codelineno-7-110) --enable-request-id-headers \ [](#__codelineno-7-111) --no-enable-prefix-caching \ [](#__codelineno-7-112) --max-num-batched-tokens 114688 \ [](#__codelineno-7-113) --max-num-seqs "$MAX_NUM_SEQS" \ [](#__codelineno-7-114) --allowed-local-media-path "${GIT_ROOT}"/tests/v1/ec_connector/integration \ [](#__codelineno-7-115) --ec-transfer-config '{ [](#__codelineno-7-116) "ec_connector": "ECExampleConnector", [](#__codelineno-7-117) "ec_role": "ec_producer", [](#__codelineno-7-118) "ec_connector_extra_config": { [](#__codelineno-7-119) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-7-120) } [](#__codelineno-7-121) }' \ [](#__codelineno-7-122) >"${ENC_LOG}" 2>&1 & [](#__codelineno-7-123)[](#__codelineno-7-124)PIDS+=($!) [](#__codelineno-7-125)[](#__codelineno-7-126)############################################################################### [](#__codelineno-7-127)# Prefill+Decode worker [](#__codelineno-7-128)############################################################################### [](#__codelineno-7-129)env "$DEVICE_AFFINITY_ENV=$GPU_PD" vllm serve "$MODEL" \ [](#__codelineno-7-130) --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION_PD" \ [](#__codelineno-7-131) --port "$PREFILL_DECODE_PORT" \ [](#__codelineno-7-132) --enforce-eager \ [](#__codelineno-7-133) --enable-request-id-headers \ [](#__codelineno-7-134) --max-num-seqs "$MAX_NUM_SEQS" \ [](#__codelineno-7-135) --max-model-len "$MAX_MODEL_LEN" \ [](#__codelineno-7-136) --allowed-local-media-path "${GIT_ROOT}"/tests/v1/ec_connector/integration \ [](#__codelineno-7-137) --ec-transfer-config '{ [](#__codelineno-7-138) "ec_connector": "ECExampleConnector", [](#__codelineno-7-139) "ec_role": "ec_consumer", [](#__codelineno-7-140) "ec_connector_extra_config": { [](#__codelineno-7-141) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-7-142) } [](#__codelineno-7-143) }' \ [](#__codelineno-7-144) >"${PD_LOG}" 2>&1 & [](#__codelineno-7-145)[](#__codelineno-7-146)PIDS+=($!) [](#__codelineno-7-147)[](#__codelineno-7-148)# Wait for workers [](#__codelineno-7-149)wait_for_server "$ENCODE_PORT" [](#__codelineno-7-150)wait_for_server "$PREFILL_DECODE_PORT" [](#__codelineno-7-151)[](#__codelineno-7-152)############################################################################### [](#__codelineno-7-153)# Proxy [](#__codelineno-7-154)############################################################################### [](#__codelineno-7-155)python disagg_epd_proxy.py \ [](#__codelineno-7-156) --host "0.0.0.0" \ [](#__codelineno-7-157) --port "$PROXY_PORT" \ [](#__codelineno-7-158) --encode-servers-urls "http://localhost:$ENCODE_PORT" \ [](#__codelineno-7-159) --prefill-servers-urls "disable" \ [](#__codelineno-7-160) --decode-servers-urls "http://localhost:$PREFILL_DECODE_PORT" \ [](#__codelineno-7-161) >"${PROXY_LOG}" 2>&1 & [](#__codelineno-7-162)[](#__codelineno-7-163)PIDS+=($!) [](#__codelineno-7-164)[](#__codelineno-7-165)wait_for_server "$PROXY_PORT" [](#__codelineno-7-166)echo "All services are up!" [](#__codelineno-7-167)[](#__codelineno-7-168)############################################################################### [](#__codelineno-7-169)# Benchmark [](#__codelineno-7-170)############################################################################### [](#__codelineno-7-171)echo "Running benchmark (stream)..." [](#__codelineno-7-172)vllm bench serve \ [](#__codelineno-7-173) --model "$MODEL" \ [](#__codelineno-7-174) --backend openai-chat \ [](#__codelineno-7-175) --endpoint /v1/chat/completions \ [](#__codelineno-7-176) --dataset-name hf \ [](#__codelineno-7-177) --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-7-178) --seed 0 \ [](#__codelineno-7-179) --num-prompts "$NUM_PROMPTS" \ [](#__codelineno-7-180) --port "$PROXY_PORT" [](#__codelineno-7-181)[](#__codelineno-7-182)PIDS+=($!) [](#__codelineno-7-183)[](#__codelineno-7-184)############################################################################### [](#__codelineno-7-185)# Single request with local image [](#__codelineno-7-186)############################################################################### [](#__codelineno-7-187)echo "Running single request with local image (non-stream)..." [](#__codelineno-7-188)curl http://127.0.0.1:"${PROXY_PORT}"/v1/chat/completions \ [](#__codelineno-7-189) -H "Content-Type: application/json" \ [](#__codelineno-7-190) -d '{ [](#__codelineno-7-191) "model": "'"${MODEL}"'", [](#__codelineno-7-192) "messages": [ [](#__codelineno-7-193) {"role": "system", "content": "You are a helpful assistant."}, [](#__codelineno-7-194) {"role": "user", "content": [ [](#__codelineno-7-195) {"type": "image_url", "image_url": {"url": "file://'"${GIT_ROOT}"'/tests/v1/ec_connector/integration/hato.jpg"}}, [](#__codelineno-7-196) {"type": "text", "text": "What is in this image?"} [](#__codelineno-7-197) ]} [](#__codelineno-7-198) ] [](#__codelineno-7-199) }' [](#__codelineno-7-200) [](#__codelineno-7-201)[](#__codelineno-7-202)# cleanup [](#__codelineno-7-203)echo "cleanup..." [](#__codelineno-7-204)cleanup` disagg\_epd\_proxy.py ``[](#__codelineno-8-1)#!/usr/bin/env python3 [](#__codelineno-8-2)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-8-3)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-8-4)""" [](#__codelineno-8-5)disagg_encoder_proxy.py [](#__codelineno-8-6)[](#__codelineno-8-7)Proxy that routes OpenAI-compatible “/v1/chat/completions” requests to two [](#__codelineno-8-8)clusters: [](#__codelineno-8-9) • encode (multimodal feature extraction) [](#__codelineno-8-10) • decode (language-model inference) [](#__codelineno-8-11)[](#__codelineno-8-12)For MM input we: [](#__codelineno-8-13) 1. Extract *every* image/audio item. [](#__codelineno-8-14) 2. Fire N concurrent requests to the encoder cluster [](#__codelineno-8-15) (one request per item, with **all text removed**). [](#__codelineno-8-16) 3. Wait for all of them to succeed. [](#__codelineno-8-17) 4. Forward the *original* request to a decode server. [](#__codelineno-8-18)""" [](#__codelineno-8-19)[](#__codelineno-8-20)from __future__ import annotations [](#__codelineno-8-21)[](#__codelineno-8-22)import argparse [](#__codelineno-8-23)import asyncio [](#__codelineno-8-24)import logging [](#__codelineno-8-25)import os [](#__codelineno-8-26)import random [](#__codelineno-8-27)import uuid [](#__codelineno-8-28)from collections.abc import AsyncIterator [](#__codelineno-8-29)[](#__codelineno-8-30)import aiohttp [](#__codelineno-8-31)import uvicorn [](#__codelineno-8-32)from fastapi import FastAPI, HTTPException, Request [](#__codelineno-8-33)from fastapi.responses import JSONResponse, StreamingResponse [](#__codelineno-8-34)[](#__codelineno-8-35)############################################################################### [](#__codelineno-8-36)# FastAPI app & global state [](#__codelineno-8-37)############################################################################### [](#__codelineno-8-38)[](#__codelineno-8-39)logging.basicConfig( [](#__codelineno-8-40) level=logging.DEBUG, format="%(asctime)s %(levelname)s: %(message)s" [](#__codelineno-8-41)) [](#__codelineno-8-42)logger = logging.getLogger("proxy") [](#__codelineno-8-43)[](#__codelineno-8-44)app = FastAPI() [](#__codelineno-8-45)encode_session: aiohttp.ClientSession | None = None [](#__codelineno-8-46)prefill_session: aiohttp.ClientSession | None = None [](#__codelineno-8-47)decode_session: aiohttp.ClientSession | None = None [](#__codelineno-8-48)[](#__codelineno-8-49)############################################################################### [](#__codelineno-8-50)# Utils [](#__codelineno-8-51)############################################################################### [](#__codelineno-8-52) [](#__codelineno-8-53)[](#__codelineno-8-54)MM_TYPES = {"image_url", "audio_url", "input_audio"} [](#__codelineno-8-55) [](#__codelineno-8-56)[](#__codelineno-8-57)def extract_mm_items(request_data: dict) -> list[dict]: [](#__codelineno-8-58) """ [](#__codelineno-8-59) Return *all* image/audio items that appear anywhere in `messages`. [](#__codelineno-8-60) [](#__codelineno-8-61) Each returned dict looks like: [](#__codelineno-8-62) { "type": "image_url", "image_url": {...} } [](#__codelineno-8-63) """ [](#__codelineno-8-64) items: list[dict] = [] [](#__codelineno-8-65) for msg in request_data.get("messages", []): [](#__codelineno-8-66) content = msg.get("content") [](#__codelineno-8-67) if not isinstance(content, list): [](#__codelineno-8-68) continue [](#__codelineno-8-69) [](#__codelineno-8-70) for item in content: [](#__codelineno-8-71) if item.get("type") in MM_TYPES: [](#__codelineno-8-72) items.append(item) [](#__codelineno-8-73) return items [](#__codelineno-8-74) [](#__codelineno-8-75)[](#__codelineno-8-76)async def fanout_encoder_primer( [](#__codelineno-8-77) orig_request: dict, [](#__codelineno-8-78) e_urls: list[str], [](#__codelineno-8-79) req_id: str, [](#__codelineno-8-80)) -> None: [](#__codelineno-8-81) """ [](#__codelineno-8-82) 1. Build one request *per MM item* with all text removed. [](#__codelineno-8-83) 2. Send them concurrently to the encode cluster. [](#__codelineno-8-84) 3. Raise if any of them fails. [](#__codelineno-8-85) """ [](#__codelineno-8-86) logger.info("[%s] Processing multimodal items...", req_id) [](#__codelineno-8-87) [](#__codelineno-8-88) mm_items = extract_mm_items(orig_request) [](#__codelineno-8-89) if not mm_items: [](#__codelineno-8-90) logger.info("[%s] No multimodal items, skipping encoder", req_id) [](#__codelineno-8-91) return # nothing to do [](#__codelineno-8-92) [](#__codelineno-8-93) logger.info("[%s] got %d multimodal items...", req_id, len(mm_items)) [](#__codelineno-8-94) [](#__codelineno-8-95) tasks = [] [](#__codelineno-8-96) [](#__codelineno-8-97) # Round-robin over encode servers to distribute load a bit [](#__codelineno-8-98) url_cycle = (e_urls[i % len(e_urls)] for i in range(len(mm_items))) [](#__codelineno-8-99) [](#__codelineno-8-100) for idx, (item, target_url) in enumerate(zip(mm_items, url_cycle)): [](#__codelineno-8-101) # Derive a *child* request id: :: [](#__codelineno-8-102) child_req_id = f"{req_id}:{idx}:{uuid.uuid4().hex[:6]}" [](#__codelineno-8-103) headers = {"x-request-id": child_req_id} [](#__codelineno-8-104) [](#__codelineno-8-105) encoder_req = { [](#__codelineno-8-106) # You *may* need to keep additional fields [](#__codelineno-8-107) "model": orig_request.get("model"), [](#__codelineno-8-108) "messages": [ [](#__codelineno-8-109) {"role": "user", "content": [item]}, [](#__codelineno-8-110) ], [](#__codelineno-8-111) # Only need 1 token so the server actually runs the encoder path [](#__codelineno-8-112) "max_tokens": 1, [](#__codelineno-8-113) "stream": False, [](#__codelineno-8-114) } [](#__codelineno-8-115) tasks.append( [](#__codelineno-8-116) encode_session.post( [](#__codelineno-8-117) f"{target_url}/v1/chat/completions", [](#__codelineno-8-118) json=encoder_req, [](#__codelineno-8-119) headers=headers, [](#__codelineno-8-120) ) [](#__codelineno-8-121) ) [](#__codelineno-8-122) [](#__codelineno-8-123) results = await asyncio.gather(*tasks, return_exceptions=True) [](#__codelineno-8-124) [](#__codelineno-8-125) # Fail fast if any sub-request failed [](#__codelineno-8-126) for idx, r in enumerate(results): [](#__codelineno-8-127) if isinstance(r, Exception): [](#__codelineno-8-128) logger.error( [](#__codelineno-8-129) "[%s] Encoder request #%d raised exception: %s", [](#__codelineno-8-130) req_id, [](#__codelineno-8-131) idx, [](#__codelineno-8-132) r, [](#__codelineno-8-133) exc_info=r, [](#__codelineno-8-134) ) [](#__codelineno-8-135) raise HTTPException( [](#__codelineno-8-136) status_code=502, detail=f"Encoder request failed: {str(r)}" [](#__codelineno-8-137) ) [](#__codelineno-8-138) if r.status != 200: [](#__codelineno-8-139) try: [](#__codelineno-8-140) detail = await r.text() [](#__codelineno-8-141) except Exception: [](#__codelineno-8-142) detail = "" [](#__codelineno-8-143) logger.error( [](#__codelineno-8-144) "[%s] Encoder request #%d returned status %s: %s", [](#__codelineno-8-145) req_id, [](#__codelineno-8-146) idx, [](#__codelineno-8-147) r.status, [](#__codelineno-8-148) detail, [](#__codelineno-8-149) ) [](#__codelineno-8-150) raise HTTPException( [](#__codelineno-8-151) status_code=r.status, [](#__codelineno-8-152) detail=f"Encoder request failed: {detail}", [](#__codelineno-8-153) ) [](#__codelineno-8-154) [](#__codelineno-8-155) logger.info( [](#__codelineno-8-156) "[%s] All %d encoder requests completed successfully", req_id, len(mm_items) [](#__codelineno-8-157) ) [](#__codelineno-8-158) [](#__codelineno-8-159)[](#__codelineno-8-160)async def maybe_prefill( [](#__codelineno-8-161) req_data: dict, [](#__codelineno-8-162) p_url: str, [](#__codelineno-8-163) req_id: str, [](#__codelineno-8-164)) -> dict: [](#__codelineno-8-165) """ [](#__codelineno-8-166) - Do prefill-only task if p_url exist; [](#__codelineno-8-167) - Return modified request data with kv transfer params (for nixl connector) [](#__codelineno-8-168) - Else, skip and return the original request data for decode [](#__codelineno-8-169) """ [](#__codelineno-8-170) if p_url: [](#__codelineno-8-171) logger.info("[%s] Processing through prefill: %s", req_id, p_url) [](#__codelineno-8-172) [](#__codelineno-8-173) prefill_response = await process_prefill_stage(req_data, p_url, req_id) [](#__codelineno-8-174) # for nixl connector to facilitate kv transfer... [](#__codelineno-8-175) prefill_response_json = await prefill_response.json() [](#__codelineno-8-176) kv_transfer_params = prefill_response_json.get("kv_transfer_params", {}) [](#__codelineno-8-177) if kv_transfer_params: [](#__codelineno-8-178) req_data["kv_transfer_params"] = kv_transfer_params [](#__codelineno-8-179) [](#__codelineno-8-180) return req_data [](#__codelineno-8-181) else: [](#__codelineno-8-182) return req_data [](#__codelineno-8-183) [](#__codelineno-8-184)[](#__codelineno-8-185)async def process_prefill_stage( [](#__codelineno-8-186) req_data: dict, [](#__codelineno-8-187) p_url: str, [](#__codelineno-8-188) req_id: str, [](#__codelineno-8-189)) -> dict: [](#__codelineno-8-190) """Process request through Prefill stage and return kv_transfer_params""" [](#__codelineno-8-191) logger.info("[%s] Sending prefill request to: %s", req_id, p_url) [](#__codelineno-8-192) [](#__codelineno-8-193) prefill_request = req_data.copy() [](#__codelineno-8-194) prefill_request["kv_transfer_params"] = { [](#__codelineno-8-195) "do_remote_decode": True, [](#__codelineno-8-196) "do_remote_prefill": False, [](#__codelineno-8-197) "remote_engine_id": None, [](#__codelineno-8-198) "remote_block_ids": None, [](#__codelineno-8-199) "remote_host": None, [](#__codelineno-8-200) "remote_port": None, [](#__codelineno-8-201) } [](#__codelineno-8-202) prefill_request["stream"] = False [](#__codelineno-8-203) prefill_request["max_tokens"] = 1 [](#__codelineno-8-204) if "max_completion_tokens" in prefill_request: [](#__codelineno-8-205) prefill_request["max_completion_tokens"] = 1 [](#__codelineno-8-206) if "stream_options" in prefill_request: [](#__codelineno-8-207) del prefill_request["stream_options"] [](#__codelineno-8-208) [](#__codelineno-8-209) headers = {"x-request-id": req_id} [](#__codelineno-8-210) try: [](#__codelineno-8-211) prefill_response = await prefill_session.post( [](#__codelineno-8-212) f"{p_url}/v1/chat/completions", json=prefill_request, headers=headers [](#__codelineno-8-213) ) [](#__codelineno-8-214) prefill_response.raise_for_status() [](#__codelineno-8-215) [](#__codelineno-8-216) if prefill_response.status != 200: [](#__codelineno-8-217) error_text = await prefill_response.text() [](#__codelineno-8-218) logger.error( [](#__codelineno-8-219) "[%s] Prefill request failed with status %d: %s", [](#__codelineno-8-220) req_id, [](#__codelineno-8-221) prefill_response.status, [](#__codelineno-8-222) error_text, [](#__codelineno-8-223) ) [](#__codelineno-8-224) raise HTTPException( [](#__codelineno-8-225) status_code=prefill_response.status, [](#__codelineno-8-226) detail={"error": "Prefill request failed", "message": error_text}, [](#__codelineno-8-227) ) [](#__codelineno-8-228) logger.info("[%s] Prefill request completed successfully", req_id) [](#__codelineno-8-229) [](#__codelineno-8-230) return prefill_response [](#__codelineno-8-231) [](#__codelineno-8-232) except Exception as e: [](#__codelineno-8-233) logger.error("Prefill processing failed: %s", str(e)) [](#__codelineno-8-234) raise HTTPException( [](#__codelineno-8-235) status_code=500, [](#__codelineno-8-236) detail={"error": "Prefill processing error", "message": str(e)}, [](#__codelineno-8-237) ) from e [](#__codelineno-8-238) [](#__codelineno-8-239)[](#__codelineno-8-240)############################################################################### [](#__codelineno-8-241)# Middleware for request/response logging [](#__codelineno-8-242)############################################################################### [](#__codelineno-8-243) [](#__codelineno-8-244)[](#__codelineno-8-245)@app.middleware("http") [](#__codelineno-8-246)async def log_requests(request: Request, call_next): [](#__codelineno-8-247) """Middleware to log all incoming requests and responses""" [](#__codelineno-8-248) req_id = request.headers.get("x-request-id", str(uuid.uuid4())) [](#__codelineno-8-249) [](#__codelineno-8-250) # Log incoming request [](#__codelineno-8-251) logger.info( [](#__codelineno-8-252) ">>> [%s] %s %s from %s", [](#__codelineno-8-253) req_id, [](#__codelineno-8-254) request.method, [](#__codelineno-8-255) request.url.path, [](#__codelineno-8-256) request.client.host if request.client else "unknown", [](#__codelineno-8-257) ) [](#__codelineno-8-258) [](#__codelineno-8-259) try: [](#__codelineno-8-260) # Process request [](#__codelineno-8-261) response = await call_next(request) [](#__codelineno-8-262) [](#__codelineno-8-263) # Log response [](#__codelineno-8-264) logger.info( [](#__codelineno-8-265) "<<< [%s] %s %s completed with status %d", [](#__codelineno-8-266) req_id, [](#__codelineno-8-267) request.method, [](#__codelineno-8-268) request.url.path, [](#__codelineno-8-269) response.status_code, [](#__codelineno-8-270) ) [](#__codelineno-8-271) [](#__codelineno-8-272) return response [](#__codelineno-8-273) except Exception as e: [](#__codelineno-8-274) # Log errors [](#__codelineno-8-275) logger.exception( [](#__codelineno-8-276) "!!! [%s] %s %s failed with error: %s", [](#__codelineno-8-277) req_id, [](#__codelineno-8-278) request.method, [](#__codelineno-8-279) request.url.path, [](#__codelineno-8-280) str(e), [](#__codelineno-8-281) ) [](#__codelineno-8-282) raise [](#__codelineno-8-283) [](#__codelineno-8-284)[](#__codelineno-8-285)############################################################################### [](#__codelineno-8-286)# FastAPI lifecycle [](#__codelineno-8-287)############################################################################### [](#__codelineno-8-288) [](#__codelineno-8-289)[](#__codelineno-8-290)@app.on_event("startup") [](#__codelineno-8-291)async def on_startup() -> None: [](#__codelineno-8-292) global encode_session, prefill_session, decode_session [](#__codelineno-8-293) timeout = aiohttp.ClientTimeout(total=100_000) [](#__codelineno-8-294) connector = aiohttp.TCPConnector(limit=0, force_close=False) [](#__codelineno-8-295) encode_session = aiohttp.ClientSession(timeout=timeout, connector=connector) [](#__codelineno-8-296) if app.state.p_urls: [](#__codelineno-8-297) # only setup if prefill instance(s) exist [](#__codelineno-8-298) prefill_session = aiohttp.ClientSession(timeout=timeout, connector=connector) [](#__codelineno-8-299) decode_session = aiohttp.ClientSession(timeout=timeout, connector=connector) [](#__codelineno-8-300) [](#__codelineno-8-301)[](#__codelineno-8-302)@app.on_event("shutdown") [](#__codelineno-8-303)async def on_shutdown() -> None: [](#__codelineno-8-304) global encode_session, prefill_session, decode_session [](#__codelineno-8-305) if encode_session: [](#__codelineno-8-306) await encode_session.close() [](#__codelineno-8-307) if prefill_session: [](#__codelineno-8-308) await prefill_session.close() [](#__codelineno-8-309) if decode_session: [](#__codelineno-8-310) await decode_session.close() [](#__codelineno-8-311) [](#__codelineno-8-312)[](#__codelineno-8-313)############################################################################### [](#__codelineno-8-314)# Core forwarding [](#__codelineno-8-315)############################################################################### [](#__codelineno-8-316) [](#__codelineno-8-317)[](#__codelineno-8-318)async def forward_non_stream( [](#__codelineno-8-319) req_data: dict, req_id: str, e_urls: list[str], p_url: str, d_url: str [](#__codelineno-8-320)) -> dict: [](#__codelineno-8-321) try: [](#__codelineno-8-322) # Step 1: Process through Encoder instance (if has MM input) [](#__codelineno-8-323) await fanout_encoder_primer(req_data, e_urls, req_id) [](#__codelineno-8-324) [](#__codelineno-8-325) # Step 2: Process through Prefill instance [](#__codelineno-8-326) req_data = await maybe_prefill(req_data, p_url, req_id) [](#__codelineno-8-327) [](#__codelineno-8-328) # Step 3: Process through Decode instance [](#__codelineno-8-329) logger.info("[%s] Forwarding to decode: %s", req_id, d_url) [](#__codelineno-8-330) headers = {"x-request-id": req_id} [](#__codelineno-8-331) [](#__codelineno-8-332) # Non-streaming response [](#__codelineno-8-333) async with decode_session.post( [](#__codelineno-8-334) f"{d_url}/v1/chat/completions", json=req_data, headers=headers [](#__codelineno-8-335) ) as resp: [](#__codelineno-8-336) resp.raise_for_status() [](#__codelineno-8-337) return await resp.json() [](#__codelineno-8-338) [](#__codelineno-8-339) except HTTPException: [](#__codelineno-8-340) raise [](#__codelineno-8-341) except Exception as e: [](#__codelineno-8-342) logger.exception("[%s] Error in forward_non_stream: %s", req_id, str(e)) [](#__codelineno-8-343) raise HTTPException(status_code=500, detail=f"Proxy error: {str(e)}") from e [](#__codelineno-8-344) [](#__codelineno-8-345)[](#__codelineno-8-346)async def forward_stream( [](#__codelineno-8-347) req_data: dict, req_id: str, e_urls: list[str], p_url: str, d_url: str [](#__codelineno-8-348)) -> AsyncIterator[str]: [](#__codelineno-8-349) try: [](#__codelineno-8-350) # Step 1: Process through Encoder instance (if has MM input) [](#__codelineno-8-351) await fanout_encoder_primer(req_data, e_urls, req_id) [](#__codelineno-8-352) [](#__codelineno-8-353) # Step 2: Process through Prefill instance [](#__codelineno-8-354) req_data = await maybe_prefill(req_data, p_url, req_id) [](#__codelineno-8-355) [](#__codelineno-8-356) # Step 3: Process through Decode instance [](#__codelineno-8-357) logger.info("[%s] Starting streaming from decode: %s", req_id, d_url) [](#__codelineno-8-358) headers = {"x-request-id": req_id} [](#__codelineno-8-359) [](#__codelineno-8-360) # Streaming response [](#__codelineno-8-361) async with decode_session.post( [](#__codelineno-8-362) f"{d_url}/v1/chat/completions", [](#__codelineno-8-363) json=req_data, [](#__codelineno-8-364) headers=headers, [](#__codelineno-8-365) ) as resp: [](#__codelineno-8-366) resp.raise_for_status() [](#__codelineno-8-367) async for chunk in resp.content.iter_chunked(1024): [](#__codelineno-8-368) if chunk: [](#__codelineno-8-369) yield chunk.decode("utf-8", errors="ignore") [](#__codelineno-8-370) [](#__codelineno-8-371) logger.info("[%s] Streaming completed", req_id) [](#__codelineno-8-372) [](#__codelineno-8-373) except HTTPException: [](#__codelineno-8-374) logger.exception("[%s] HTTPException in forward_stream", req_id) [](#__codelineno-8-375) raise [](#__codelineno-8-376) except Exception as e: [](#__codelineno-8-377) logger.exception("[%s] Error in forward_stream: %s", req_id, str(e)) [](#__codelineno-8-378) raise HTTPException( [](#__codelineno-8-379) status_code=500, detail=f"Proxy streaming error: {str(e)}" [](#__codelineno-8-380) ) from e [](#__codelineno-8-381) [](#__codelineno-8-382)[](#__codelineno-8-383)############################################################################### [](#__codelineno-8-384)# Public routes [](#__codelineno-8-385)############################################################################### [](#__codelineno-8-386) [](#__codelineno-8-387)[](#__codelineno-8-388)@app.post("/v1/chat/completions") [](#__codelineno-8-389)async def chat_completions(request: Request): [](#__codelineno-8-390) try: [](#__codelineno-8-391) req_data = await request.json() [](#__codelineno-8-392) req_id = request.headers.get("x-request-id", str(uuid.uuid4())) [](#__codelineno-8-393) [](#__codelineno-8-394) e_urls = app.state.e_urls # we want the full list for fan-out [](#__codelineno-8-395) p_url = random.choice(app.state.p_urls) if app.state.p_urls else None [](#__codelineno-8-396) d_url = random.choice(app.state.d_urls) [](#__codelineno-8-397) [](#__codelineno-8-398) is_streaming = req_data.get("stream", False) [](#__codelineno-8-399) [](#__codelineno-8-400) if is_streaming: [](#__codelineno-8-401) return StreamingResponse( [](#__codelineno-8-402) forward_stream(req_data, req_id, e_urls, p_url, d_url), [](#__codelineno-8-403) media_type="text/event-stream", [](#__codelineno-8-404) ) [](#__codelineno-8-405) result = await forward_non_stream(req_data, req_id, e_urls, p_url, d_url) [](#__codelineno-8-406) return JSONResponse(content=result) [](#__codelineno-8-407) [](#__codelineno-8-408) except HTTPException: [](#__codelineno-8-409) raise [](#__codelineno-8-410) except Exception as e: [](#__codelineno-8-411) logger.exception("Error in chat_completions endpoint: %s", str(e)) [](#__codelineno-8-412) raise HTTPException( [](#__codelineno-8-413) status_code=500, detail=f"Request processing error: {str(e)}" [](#__codelineno-8-414) ) from e [](#__codelineno-8-415) [](#__codelineno-8-416)[](#__codelineno-8-417)@app.get("/v1/models") [](#__codelineno-8-418)async def list_models(): [](#__codelineno-8-419) async with decode_session.get(f"{app.state.d_urls[0]}/v1/models") as resp: [](#__codelineno-8-420) resp.raise_for_status() [](#__codelineno-8-421) return await resp.json() [](#__codelineno-8-422) [](#__codelineno-8-423)[](#__codelineno-8-424)@app.get("/health") [](#__codelineno-8-425)async def health_check(): [](#__codelineno-8-426) async def healthy(urls): [](#__codelineno-8-427) if not urls: [](#__codelineno-8-428) return "empty" [](#__codelineno-8-429) for u in urls: [](#__codelineno-8-430) try: [](#__codelineno-8-431) async with encode_session.get(f"{u}/health") as resp: [](#__codelineno-8-432) resp.raise_for_status() [](#__codelineno-8-433) except Exception: [](#__codelineno-8-434) return "unhealthy" [](#__codelineno-8-435) return "healthy" [](#__codelineno-8-436) [](#__codelineno-8-437) e_status, p_status, d_status = await asyncio.gather( [](#__codelineno-8-438) healthy(app.state.e_urls), healthy(app.state.p_urls), healthy(app.state.d_urls) [](#__codelineno-8-439) ) [](#__codelineno-8-440) [](#__codelineno-8-441) overall_healthy = all( [](#__codelineno-8-442) status != "unhealthy" for status in (e_status, p_status, d_status) [](#__codelineno-8-443) ) [](#__codelineno-8-444) [](#__codelineno-8-445) status_code = 200 if overall_healthy else 503 [](#__codelineno-8-446) [](#__codelineno-8-447) return JSONResponse( [](#__codelineno-8-448) { [](#__codelineno-8-449) "proxy": "healthy", [](#__codelineno-8-450) "encode_cluster": e_status, [](#__codelineno-8-451) "prefill_cluster": p_status, [](#__codelineno-8-452) "decode_cluster": d_status, [](#__codelineno-8-453) }, [](#__codelineno-8-454) status_code=status_code, [](#__codelineno-8-455) ) [](#__codelineno-8-456) [](#__codelineno-8-457)[](#__codelineno-8-458)############################################################################### [](#__codelineno-8-459)# Simple profiler fan-out (unchanged except for sessions) [](#__codelineno-8-460)############################################################################### [](#__codelineno-8-461) [](#__codelineno-8-462)[](#__codelineno-8-463)async def _post_if_available( [](#__codelineno-8-464) session: aiohttp.ClientSession, [](#__codelineno-8-465) url: str, [](#__codelineno-8-466) payload: dict, [](#__codelineno-8-467) headers: dict, [](#__codelineno-8-468)) -> dict | None: [](#__codelineno-8-469) """ [](#__codelineno-8-470) POST `payload` to `url`. [](#__codelineno-8-471) [](#__codelineno-8-472) Returns [](#__codelineno-8-473) ------- [](#__codelineno-8-474) • The decoded JSON body on success (2xx) [](#__codelineno-8-475) • None if the endpoint does not exist (404) [](#__codelineno-8-476) • Raises for anything else. [](#__codelineno-8-477) """ [](#__codelineno-8-478) try: [](#__codelineno-8-479) resp = await session.post(url, json=payload, headers=headers) [](#__codelineno-8-480) if resp.status == 404: # profiling disabled on that server [](#__codelineno-8-481) logger.warning("Profiling endpoint missing on %s", url) [](#__codelineno-8-482) return None [](#__codelineno-8-483) resp.raise_for_status() [](#__codelineno-8-484) return await resp.json(content_type=None) [](#__codelineno-8-485) except aiohttp.ClientResponseError as exc: [](#__codelineno-8-486) # Pass 404 through the branch above, re-raise everything else [](#__codelineno-8-487) if exc.status == 404: [](#__codelineno-8-488) logger.warning("Profiling endpoint missing on %s", url) [](#__codelineno-8-489) return None [](#__codelineno-8-490) raise [](#__codelineno-8-491) except Exception: [](#__codelineno-8-492) # Network errors etc.: propagate [](#__codelineno-8-493) raise [](#__codelineno-8-494) [](#__codelineno-8-495)[](#__codelineno-8-496)async def _profile_cmd(cmd: str, payload: dict, e_url: str, p_url: str, d_url: str): [](#__codelineno-8-497) """ [](#__codelineno-8-498) Fire & forget to both clusters, tolerate 404. [](#__codelineno-8-499) """ [](#__codelineno-8-500) headers = {"Authorization": f"Bearer {os.getenv('OPENAI_API_KEY', '')}"} [](#__codelineno-8-501) [](#__codelineno-8-502) encode_task = _post_if_available( [](#__codelineno-8-503) encode_session, f"{e_url}/{cmd}_profile", payload, headers [](#__codelineno-8-504) ) [](#__codelineno-8-505) prefill_task = ( [](#__codelineno-8-506) _post_if_available(prefill_session, f"{p_url}/{cmd}_profile", payload, headers) [](#__codelineno-8-507) if p_url is not None [](#__codelineno-8-508) else asyncio.sleep(0) [](#__codelineno-8-509) ) [](#__codelineno-8-510) decode_task = _post_if_available( [](#__codelineno-8-511) decode_session, f"{d_url}/{cmd}_profile", payload, headers [](#__codelineno-8-512) ) [](#__codelineno-8-513) [](#__codelineno-8-514) encode_res, prefill_res, decode_res = await asyncio.gather( [](#__codelineno-8-515) encode_task, prefill_task, decode_task [](#__codelineno-8-516) ) [](#__codelineno-8-517) [](#__codelineno-8-518) # If *all* clusters said “I don’t have that route”, surface an error [](#__codelineno-8-519) if encode_res is prefill_res is decode_res is None: [](#__codelineno-8-520) raise HTTPException( [](#__codelineno-8-521) status_code=503, [](#__codelineno-8-522) detail="Profiling endpoints are disabled on all clusters", [](#__codelineno-8-523) ) [](#__codelineno-8-524) [](#__codelineno-8-525) return { [](#__codelineno-8-526) "encode": encode_res, # may be None [](#__codelineno-8-527) "prefill": prefill_res, # may be None [](#__codelineno-8-528) "decode": decode_res, # may be None [](#__codelineno-8-529) } [](#__codelineno-8-530) [](#__codelineno-8-531)[](#__codelineno-8-532)@app.post("/start_profile") [](#__codelineno-8-533)async def start_profile(request: Request): [](#__codelineno-8-534) body = await request.json() [](#__codelineno-8-535) # TODO: handle multi urls properly [](#__codelineno-8-536) e_url = random.choice(app.state.e_urls) [](#__codelineno-8-537) p_url = random.choice(app.state.p_urls) if app.state.p_urls else None [](#__codelineno-8-538) d_url = random.choice(app.state.d_urls) [](#__codelineno-8-539) return await _profile_cmd("start", body, e_url, p_url, d_url) [](#__codelineno-8-540) [](#__codelineno-8-541)[](#__codelineno-8-542)@app.post("/stop_profile") [](#__codelineno-8-543)async def stop_profile(request: Request): [](#__codelineno-8-544) body = await request.json() [](#__codelineno-8-545) # TODO: handle multi urls properly [](#__codelineno-8-546) e_url = random.choice(app.state.e_urls) [](#__codelineno-8-547) p_url = random.choice(app.state.p_urls) if app.state.p_urls else None [](#__codelineno-8-548) d_url = random.choice(app.state.d_urls) [](#__codelineno-8-549) return await _profile_cmd("stop", body, e_url, p_url, d_url) [](#__codelineno-8-550) [](#__codelineno-8-551)[](#__codelineno-8-552)if __name__ == "__main__": [](#__codelineno-8-553) parser = argparse.ArgumentParser() [](#__codelineno-8-554) parser.add_argument("--host", default="0.0.0.0") [](#__codelineno-8-555) parser.add_argument("--port", type=int, default=8000) [](#__codelineno-8-556) parser.add_argument( [](#__codelineno-8-557) "--encode-servers-urls", [](#__codelineno-8-558) required=True, [](#__codelineno-8-559) help='Comma-separated encode URLs ("http://e1:8001,http://e2:8001")', [](#__codelineno-8-560) ) [](#__codelineno-8-561) parser.add_argument( [](#__codelineno-8-562) "--prefill-servers-urls", [](#__codelineno-8-563) required=True, [](#__codelineno-8-564) help=( [](#__codelineno-8-565) 'Comma-separated prefill URLs ("http://p1:8003,http://p2:8004") ', [](#__codelineno-8-566) 'to enable E->P->D, set "disable" or "none" to enable E->PD', [](#__codelineno-8-567) ), [](#__codelineno-8-568) ) [](#__codelineno-8-569) parser.add_argument( [](#__codelineno-8-570) "--decode-servers-urls", [](#__codelineno-8-571) required=True, [](#__codelineno-8-572) help='Comma-separated decode URLs ("http://d1:8005,http://d2:8006")', [](#__codelineno-8-573) ) [](#__codelineno-8-574) [](#__codelineno-8-575) args = parser.parse_args() [](#__codelineno-8-576) app.state.e_urls = [ [](#__codelineno-8-577) u.strip() for u in args.encode_servers_urls.split(",") if u.strip() [](#__codelineno-8-578) ] [](#__codelineno-8-579) app.state.d_urls = [ [](#__codelineno-8-580) u.strip() for u in args.decode_servers_urls.split(",") if u.strip() [](#__codelineno-8-581) ] [](#__codelineno-8-582) # handle prefill instances [](#__codelineno-8-583) if args.prefill_servers_urls.lower() in ("disable", "none", ""): [](#__codelineno-8-584) app.state.p_urls = [] [](#__codelineno-8-585) logger.info( [](#__codelineno-8-586) "Disaggregated prefill phase explicitly disabled by user. Running E + PD..." [](#__codelineno-8-587) ) [](#__codelineno-8-588) else: [](#__codelineno-8-589) app.state.p_urls = [ [](#__codelineno-8-590) u.strip() for u in args.prefill_servers_urls.split(",") if u.strip() [](#__codelineno-8-591) ] [](#__codelineno-8-592) logger.info("Disaggregated prefill phase is enabled. Running E + P + D...") [](#__codelineno-8-593) [](#__codelineno-8-594) logger.info("Proxy listening on %s:%s", args.host, args.port) [](#__codelineno-8-595) logger.info("Encode servers: %s", app.state.e_urls) [](#__codelineno-8-596) logger.info("Prefill instances %s", app.state.p_urls) [](#__codelineno-8-597) logger.info("Decode servers: %s", app.state.d_urls) [](#__codelineno-8-598) [](#__codelineno-8-599) uvicorn.run( [](#__codelineno-8-600) app, [](#__codelineno-8-601) host=args.host, [](#__codelineno-8-602) port=args.port, [](#__codelineno-8-603) log_level="info", [](#__codelineno-8-604) loop="uvloop", [](#__codelineno-8-605) access_log=True, [](#__codelineno-8-606) )`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/disaggregated_prefill.md "Edit this page") Source [https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/disaggregated\_prefill.sh](https://github.com/vllm-project/vllm/blob/main/examples/disaggregated/disaggregated_prefill.sh). `[](#__codelineno-0-1)#!/bin/bash [](#__codelineno-0-2)# This file demonstrates the example usage of disaggregated prefilling [](#__codelineno-0-3)# We will launch 2 vllm instances (1 for prefill and 1 for decode), [](#__codelineno-0-4)# and then transfer the KV cache between them. [](#__codelineno-0-5)[](#__codelineno-0-6)set -xe [](#__codelineno-0-7)[](#__codelineno-0-8)echo "🚧🚧 Warning: The usage of disaggregated prefill is experimental and subject to change 🚧🚧" [](#__codelineno-0-9)sleep 1 [](#__codelineno-0-10)[](#__codelineno-0-11)# meta-llama/Meta-Llama-3.1-8B-Instruct or deepseek-ai/DeepSeek-V2-Lite [](#__codelineno-0-12)MODEL_NAME=${HF_MODEL_NAME:-meta-llama/Meta-Llama-3.1-8B-Instruct} [](#__codelineno-0-13)[](#__codelineno-0-14)# Trap the SIGINT signal (triggered by Ctrl+C) [](#__codelineno-0-15)trap 'cleanup' INT [](#__codelineno-0-16)[](#__codelineno-0-17)# Cleanup function [](#__codelineno-0-18)cleanup() { [](#__codelineno-0-19) echo "Caught Ctrl+C, cleaning up..." [](#__codelineno-0-20) # Cleanup commands [](#__codelineno-0-21) pgrep python | xargs kill -9 [](#__codelineno-0-22) pkill -f python [](#__codelineno-0-23) echo "Cleanup complete. Exiting." [](#__codelineno-0-24) exit 0 [](#__codelineno-0-25)} [](#__codelineno-0-26) [](#__codelineno-0-27)[](#__codelineno-0-28)if [[ -z "${VLLM_HOST_IP:-}" ]]; then [](#__codelineno-0-29) export VLLM_HOST_IP=127.0.0.1 [](#__codelineno-0-30) echo "Using default VLLM_HOST_IP=127.0.0.1 (override by exporting VLLM_HOST_IP before running this script)" [](#__codelineno-0-31)else [](#__codelineno-0-32) echo "Using provided VLLM_HOST_IP=${VLLM_HOST_IP}" [](#__codelineno-0-33)fi [](#__codelineno-0-34) [](#__codelineno-0-35)[](#__codelineno-0-36)# install quart first -- required for disagg prefill proxy serve [](#__codelineno-0-37)if python3 -c "import quart" &> /dev/null; then [](#__codelineno-0-38) echo "Quart is already installed." [](#__codelineno-0-39)else [](#__codelineno-0-40) echo "Quart is not installed. Installing..." [](#__codelineno-0-41) python3 -m pip install quart [](#__codelineno-0-42)fi [](#__codelineno-0-43)[](#__codelineno-0-44)# a function that waits vLLM server to start [](#__codelineno-0-45)wait_for_server() { [](#__codelineno-0-46) local port=$1 [](#__codelineno-0-47) timeout 1200 bash -c " [](#__codelineno-0-48) until curl -i localhost:${port}/v1/models > /dev/null; do [](#__codelineno-0-49) sleep 1 [](#__codelineno-0-50) done" && return 0 || return 1 [](#__codelineno-0-51)} [](#__codelineno-0-52) [](#__codelineno-0-53)[](#__codelineno-0-54)# You can also adjust --kv-ip and --kv-port for distributed inference. [](#__codelineno-0-55)[](#__codelineno-0-56)# prefilling instance, which is the KV producer [](#__codelineno-0-57)CUDA_VISIBLE_DEVICES=0 vllm serve "$MODEL_NAME" \ [](#__codelineno-0-58) --host 0.0.0.0 \ [](#__codelineno-0-59) --port 8100 \ [](#__codelineno-0-60) --max-model-len 100 \ [](#__codelineno-0-61) --gpu-memory-utilization 0.8 \ [](#__codelineno-0-62) --trust-remote-code \ [](#__codelineno-0-63) --kv-transfer-config \ [](#__codelineno-0-64) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":"1e9","kv_port":"14579","kv_connector_extra_config":{"proxy_ip":"'"$VLLM_HOST_IP"'","proxy_port":"30001","http_ip":"'"$VLLM_HOST_IP"'","http_port":"8100","send_type":"PUT_ASYNC"}}' & [](#__codelineno-0-65)[](#__codelineno-0-66)# decoding instance, which is the KV consumer [](#__codelineno-0-67)CUDA_VISIBLE_DEVICES=1 vllm serve "$MODEL_NAME" \ [](#__codelineno-0-68) --host 0.0.0.0 \ [](#__codelineno-0-69) --port 8200 \ [](#__codelineno-0-70) --max-model-len 100 \ [](#__codelineno-0-71) --gpu-memory-utilization 0.8 \ [](#__codelineno-0-72) --trust-remote-code \ [](#__codelineno-0-73) --kv-transfer-config \ [](#__codelineno-0-74) '{"kv_connector":"P2pNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":"1e10","kv_port":"14580","kv_connector_extra_config":{"proxy_ip":"'"$VLLM_HOST_IP"'","proxy_port":"30001","http_ip":"'"$VLLM_HOST_IP"'","http_port":"8200","send_type":"PUT_ASYNC"}}' & [](#__codelineno-0-75)[](#__codelineno-0-76)# wait until prefill and decode instances are ready [](#__codelineno-0-77)wait_for_server 8100 [](#__codelineno-0-78)wait_for_server 8200 [](#__codelineno-0-79)[](#__codelineno-0-80)# launch a proxy server that opens the service at port 8000 [](#__codelineno-0-81)# the workflow of this proxy: [](#__codelineno-0-82)# - send the request to prefill vLLM instance (port 8100), change max_tokens [](#__codelineno-0-83)# to 1 [](#__codelineno-0-84)# - after the prefill vLLM finishes prefill, send the request to decode vLLM [](#__codelineno-0-85)# instance [](#__codelineno-0-86)# NOTE: the usage of this API is subject to change --- in the future we will [](#__codelineno-0-87)# introduce "vllm connect" to connect between prefill and decode instances [](#__codelineno-0-88)python3 ../../benchmarks/disagg_benchmarks/disagg_prefill_proxy_server.py & [](#__codelineno-0-89)sleep 1 [](#__codelineno-0-90)[](#__codelineno-0-91)# serve two example requests [](#__codelineno-0-92)output1=$(curl -X POST -s http://localhost:8000/v1/completions \ [](#__codelineno-0-93)-H "Content-Type: application/json" \ [](#__codelineno-0-94)-d '{ [](#__codelineno-0-95)"model": "'"$MODEL_NAME"'", [](#__codelineno-0-96)"prompt": "San Francisco is a", [](#__codelineno-0-97)"max_tokens": 10, [](#__codelineno-0-98)"temperature": 0 [](#__codelineno-0-99)}') [](#__codelineno-0-100)[](#__codelineno-0-101)output2=$(curl -X POST -s http://localhost:8000/v1/completions \ [](#__codelineno-0-102)-H "Content-Type: application/json" \ [](#__codelineno-0-103)-d '{ [](#__codelineno-0-104)"model": "'"$MODEL_NAME"'", [](#__codelineno-0-105)"prompt": "Santa Clara is a", [](#__codelineno-0-106)"max_tokens": 10, [](#__codelineno-0-107)"temperature": 0 [](#__codelineno-0-108)}') [](#__codelineno-0-109) [](#__codelineno-0-110)[](#__codelineno-0-111)# Cleanup commands [](#__codelineno-0-112)pgrep python | xargs kill -9 [](#__codelineno-0-113)pkill -f python [](#__codelineno-0-114)[](#__codelineno-0-115)echo "" [](#__codelineno-0-116)[](#__codelineno-0-117)sleep 1 [](#__codelineno-0-118)[](#__codelineno-0-119)# Print the outputs of the curl requests [](#__codelineno-0-120)echo "" [](#__codelineno-0-121)echo "Output of first request: $output1" [](#__codelineno-0-122)echo "Output of second request: $output2" [](#__codelineno-0-123)[](#__codelineno-0-124)echo "🎉🎉 Successfully finished 2 test requests! 🎉🎉" [](#__codelineno-0-125)echo ""` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) 5. [Disaggregated](https://docs.vllm.ai/en/latest/examples/disaggregated_encoder/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/example_connector.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/example\_connector](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/example_connector). This example contains scripts that demonstrate disaggregated prefill in the offline setting of vLLM. ## Files[¶](#files "Permanent link") - `run.sh` - A helper script that will run `prefill_example.py` and `decode_example.py` sequentially. - Make sure you are in the `examples/disaggregated/example_connector` directory before running `run.sh`. - `prefill_example.py` - A script which performs prefill only, saving the KV state to the `local_storage` directory and the prompts to `output.txt`. - `decode_example.py` - A script which performs decode only, loading the KV state from the `local_storage` directory and the prompts from `output.txt`. ## Example materials[¶](#example-materials "Permanent link") decode\_example.py `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)[](#__codelineno-0-4)from vllm import LLM, SamplingParams [](#__codelineno-0-5)from vllm.config import KVTransferConfig [](#__codelineno-0-6) [](#__codelineno-0-7)[](#__codelineno-0-8)def read_prompts(): [](#__codelineno-0-9) """Read prompts from output.txt""" [](#__codelineno-0-10) prompts = [] [](#__codelineno-0-11) try: [](#__codelineno-0-12) with open("output.txt") as f: [](#__codelineno-0-13) for line in f: [](#__codelineno-0-14) prompts.append(line.strip()) [](#__codelineno-0-15) print(f"Loaded {len(prompts)} prompts from output.txt") [](#__codelineno-0-16) return prompts [](#__codelineno-0-17) except FileNotFoundError: [](#__codelineno-0-18) print("Error: output.txt file not found") [](#__codelineno-0-19) exit(-1) [](#__codelineno-0-20) [](#__codelineno-0-21)[](#__codelineno-0-22)def main(): [](#__codelineno-0-23) prompts = read_prompts() [](#__codelineno-0-24) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) [](#__codelineno-0-25) [](#__codelineno-0-26) llm = LLM( [](#__codelineno-0-27) model="meta-llama/Llama-3.2-1B-Instruct", [](#__codelineno-0-28) enforce_eager=True, [](#__codelineno-0-29) gpu_memory_utilization=0.8, [](#__codelineno-0-30) max_num_batched_tokens=64, [](#__codelineno-0-31) max_num_seqs=16, [](#__codelineno-0-32) kv_transfer_config=KVTransferConfig( [](#__codelineno-0-33) kv_connector="ExampleConnector", [](#__codelineno-0-34) kv_role="kv_both", [](#__codelineno-0-35) kv_connector_extra_config={"shared_storage_path": "local_storage"}, [](#__codelineno-0-36) ), [](#__codelineno-0-37) ) # , max_model_len=2048, max_num_batched_tokens=2048) [](#__codelineno-0-38) [](#__codelineno-0-39) # 1ST generation (prefill instance) [](#__codelineno-0-40) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-41) [](#__codelineno-0-42) print("-" * 30) [](#__codelineno-0-43) for output in outputs: [](#__codelineno-0-44) prompt = output.prompt [](#__codelineno-0-45) generated_text = output.outputs[0].text [](#__codelineno-0-46) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-0-47) print("-" * 30) [](#__codelineno-0-48) [](#__codelineno-0-49)[](#__codelineno-0-50)if __name__ == "__main__": [](#__codelineno-0-51) main()` prefill\_example.py `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)[](#__codelineno-1-4)from vllm import LLM, SamplingParams [](#__codelineno-1-5)from vllm.config import KVTransferConfig [](#__codelineno-1-6) [](#__codelineno-1-7)[](#__codelineno-1-8)def read_prompts(): [](#__codelineno-1-9) context = "Hi " * 1000 [](#__codelineno-1-10) context2 = "Hey " * 500 [](#__codelineno-1-11) return [ [](#__codelineno-1-12) context + "Hello, my name is", [](#__codelineno-1-13) context + "The capital of France is", [](#__codelineno-1-14) context2 + "Your name is", [](#__codelineno-1-15) context2 + "The capital of China is", [](#__codelineno-1-16) ] [](#__codelineno-1-17) [](#__codelineno-1-18)[](#__codelineno-1-19)def main(): [](#__codelineno-1-20) prompts = read_prompts() [](#__codelineno-1-21) [](#__codelineno-1-22) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1) [](#__codelineno-1-23) [](#__codelineno-1-24) llm = LLM( [](#__codelineno-1-25) model="meta-llama/Llama-3.2-1B-Instruct", [](#__codelineno-1-26) enforce_eager=True, [](#__codelineno-1-27) gpu_memory_utilization=0.8, [](#__codelineno-1-28) kv_transfer_config=KVTransferConfig( [](#__codelineno-1-29) kv_connector="ExampleConnector", [](#__codelineno-1-30) kv_role="kv_both", [](#__codelineno-1-31) kv_connector_extra_config={"shared_storage_path": "local_storage"}, [](#__codelineno-1-32) ), [](#__codelineno-1-33) ) # , max_model_len=2048, max_num_batched_tokens=2048) [](#__codelineno-1-34) [](#__codelineno-1-35) # 1ST generation (prefill instance) [](#__codelineno-1-36) outputs = llm.generate( [](#__codelineno-1-37) prompts, [](#__codelineno-1-38) sampling_params, [](#__codelineno-1-39) ) [](#__codelineno-1-40) [](#__codelineno-1-41) new_prompts = [] [](#__codelineno-1-42) print("-" * 30) [](#__codelineno-1-43) for output in outputs: [](#__codelineno-1-44) prompt = output.prompt [](#__codelineno-1-45) generated_text = output.outputs[0].text [](#__codelineno-1-46) new_prompts.append(prompt + generated_text) [](#__codelineno-1-47) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-1-48) print("-" * 30) [](#__codelineno-1-49) [](#__codelineno-1-50) # Write new_prompts to output.txt [](#__codelineno-1-51) with open("output.txt", "w") as f: [](#__codelineno-1-52) for prompt in new_prompts: [](#__codelineno-1-53) f.write(prompt + "\n") [](#__codelineno-1-54) print(f"Saved {len(new_prompts)} prompts to output.txt") [](#__codelineno-1-55) [](#__codelineno-1-56)[](#__codelineno-1-57)if __name__ == "__main__": [](#__codelineno-1-58) main()` run.sh `[](#__codelineno-2-1)rm -rf local_storage/ [](#__codelineno-2-2)[](#__codelineno-2-3)if [ -f "output.txt" ]; then [](#__codelineno-2-4) rm output.txt [](#__codelineno-2-5)fi [](#__codelineno-2-6)[](#__codelineno-2-7)# The directory of current script [](#__codelineno-2-8)SCRIPT_DIR=$(dirname "$(readlink -f "$0")") [](#__codelineno-2-9)[](#__codelineno-2-10)VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 "$SCRIPT_DIR/prefill_example.py" [](#__codelineno-2-11)VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 "$SCRIPT_DIR/decode_example.py"` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/disaggregated_serving.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/disaggregated\_serving](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/disaggregated_serving). This example contains scripts that demonstrate the disaggregated serving features of vLLM. ## Files[¶](#files "Permanent link") - `disagg_proxy_demo.py` - Demonstrates XpYd (X prefill instances, Y decode instances). - `kv_events.sh` - Demonstrates KV cache event publishing. - `mooncake_connector` - A proxy demo for MooncakeConnector. ## Example materials[¶](#example-materials "Permanent link") disagg\_proxy\_demo.py `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This file provides a disaggregated prefilling proxy demo to demonstrate an [](#__codelineno-0-5)example usage of XpYd disaggregated prefilling. [](#__codelineno-0-6)We can launch multiple vllm instances (2 for prefill and 2 for decode), and [](#__codelineno-0-7)launch this proxy demo through: [](#__codelineno-0-8) python3 examples/disaggregated/disaggregated_serving/disagg_proxy_demo.py \ [](#__codelineno-0-9) --model $model_name \ [](#__codelineno-0-10) --prefill localhost:8100 localhost:8101 \ [](#__codelineno-0-11) --decode localhost:8200 localhost:8201 \ [](#__codelineno-0-12) --port 8000 [](#__codelineno-0-13)[](#__codelineno-0-14)Note: This demo will be removed once the PDController implemented in PR 15343 [](#__codelineno-0-15)(https://github.com/vllm-project/vllm/pull/15343) supports XpYd. [](#__codelineno-0-16)""" [](#__codelineno-0-17)[](#__codelineno-0-18)import argparse [](#__codelineno-0-19)import ipaddress [](#__codelineno-0-20)import itertools [](#__codelineno-0-21)import json [](#__codelineno-0-22)import logging [](#__codelineno-0-23)import os [](#__codelineno-0-24)import sys [](#__codelineno-0-25)from abc import ABC, abstractmethod [](#__codelineno-0-26)from collections.abc import Callable [](#__codelineno-0-27)[](#__codelineno-0-28)import aiohttp [](#__codelineno-0-29)import requests [](#__codelineno-0-30)import uvicorn [](#__codelineno-0-31)from fastapi import APIRouter, Depends, FastAPI, Header, HTTPException, Request, status [](#__codelineno-0-32)from fastapi.responses import JSONResponse, StreamingResponse [](#__codelineno-0-33)[](#__codelineno-0-34)AIOHTTP_TIMEOUT = aiohttp.ClientTimeout(total=6 * 60 * 60) [](#__codelineno-0-35)logger = logging.getLogger() [](#__codelineno-0-36)logging.basicConfig(level=logging.INFO) [](#__codelineno-0-37) [](#__codelineno-0-38)[](#__codelineno-0-39)class SchedulingPolicy(ABC): [](#__codelineno-0-40) @abstractmethod [](#__codelineno-0-41) def schedule(self, cycler: itertools.cycle): [](#__codelineno-0-42) raise NotImplementedError("Scheduling Proxy is not set.") [](#__codelineno-0-43) [](#__codelineno-0-44)[](#__codelineno-0-45)class Proxy: [](#__codelineno-0-46) def __init__( [](#__codelineno-0-47) self, [](#__codelineno-0-48) prefill_instances: list[str], [](#__codelineno-0-49) decode_instances: list[str], [](#__codelineno-0-50) model: str, [](#__codelineno-0-51) scheduling_policy: SchedulingPolicy, [](#__codelineno-0-52) custom_create_completion: Callable[[Request], StreamingResponse] | None = None, [](#__codelineno-0-53) custom_create_chat_completion: Callable[[Request], StreamingResponse] [](#__codelineno-0-54) | None = None, [](#__codelineno-0-55) ): [](#__codelineno-0-56) self.prefill_instances = prefill_instances [](#__codelineno-0-57) self.decode_instances = decode_instances [](#__codelineno-0-58) self.prefill_cycler = itertools.cycle(prefill_instances) [](#__codelineno-0-59) self.decode_cycler = itertools.cycle(decode_instances) [](#__codelineno-0-60) self.model = model [](#__codelineno-0-61) self.scheduling_policy = scheduling_policy [](#__codelineno-0-62) self.custom_create_completion = custom_create_completion [](#__codelineno-0-63) self.custom_create_chat_completion = custom_create_chat_completion [](#__codelineno-0-64) self.router = APIRouter() [](#__codelineno-0-65) self.setup_routes() [](#__codelineno-0-66) [](#__codelineno-0-67) def setup_routes(self): [](#__codelineno-0-68) self.router.post( [](#__codelineno-0-69) "/v1/completions", dependencies=[Depends(self.validate_json_request)] [](#__codelineno-0-70) )( [](#__codelineno-0-71) self.custom_create_completion [](#__codelineno-0-72) if self.custom_create_completion [](#__codelineno-0-73) else self.create_completion [](#__codelineno-0-74) ) [](#__codelineno-0-75) self.router.post( [](#__codelineno-0-76) "/v1/chat/completions", dependencies=[Depends(self.validate_json_request)] [](#__codelineno-0-77) )( [](#__codelineno-0-78) self.custom_create_chat_completion [](#__codelineno-0-79) if self.custom_create_chat_completion [](#__codelineno-0-80) else self.create_chat_completion [](#__codelineno-0-81) ) [](#__codelineno-0-82) self.router.get("/status", response_class=JSONResponse)(self.get_status) [](#__codelineno-0-83) self.router.post( [](#__codelineno-0-84) "/instances/add", dependencies=[Depends(self.api_key_authenticate)] [](#__codelineno-0-85) )(self.add_instance_endpoint) [](#__codelineno-0-86) [](#__codelineno-0-87) async def validate_json_request(self, raw_request: Request): [](#__codelineno-0-88) content_type = raw_request.headers.get("content-type", "").lower() [](#__codelineno-0-89) if content_type != "application/json": [](#__codelineno-0-90) raise HTTPException( [](#__codelineno-0-91) status_code=415, [](#__codelineno-0-92) detail="Unsupported Media Type: Only 'application/json' is allowed", [](#__codelineno-0-93) ) [](#__codelineno-0-94) [](#__codelineno-0-95) def api_key_authenticate(self, x_api_key: str = Header(...)): [](#__codelineno-0-96) expected_api_key = os.environ.get("ADMIN_API_KEY") [](#__codelineno-0-97) if not expected_api_key: [](#__codelineno-0-98) logger.error("ADMIN_API_KEY is not set in the environment.") [](#__codelineno-0-99) raise HTTPException( [](#__codelineno-0-100) status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, [](#__codelineno-0-101) detail="Server configuration error.", [](#__codelineno-0-102) ) [](#__codelineno-0-103) if x_api_key != expected_api_key: [](#__codelineno-0-104) logger.warning("Unauthorized access attempt with API Key: %s", x_api_key) [](#__codelineno-0-105) raise HTTPException( [](#__codelineno-0-106) status_code=status.HTTP_403_FORBIDDEN, [](#__codelineno-0-107) detail="Forbidden: Invalid API Key.", [](#__codelineno-0-108) ) [](#__codelineno-0-109) [](#__codelineno-0-110) async def validate_instance(self, instance: str) -> bool: [](#__codelineno-0-111) url = f"http://{instance}/v1/models" [](#__codelineno-0-112) try: [](#__codelineno-0-113) async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as client: [](#__codelineno-0-114) logger.info("Verifying %s ...", instance) [](#__codelineno-0-115) async with client.get(url) as response: [](#__codelineno-0-116) if response.status == 200: [](#__codelineno-0-117) data = await response.json() [](#__codelineno-0-118) if "data" in data and len(data["data"]) > 0: [](#__codelineno-0-119) model_cur = data["data"][0].get("id", "") [](#__codelineno-0-120) if model_cur == self.model: [](#__codelineno-0-121) logger.info("Instance: %s could be added.", instance) [](#__codelineno-0-122) return True [](#__codelineno-0-123) else: [](#__codelineno-0-124) logger.warning( [](#__codelineno-0-125) "Mismatch model %s : %s != %s", [](#__codelineno-0-126) instance, [](#__codelineno-0-127) model_cur, [](#__codelineno-0-128) self.model, [](#__codelineno-0-129) ) [](#__codelineno-0-130) return False [](#__codelineno-0-131) else: [](#__codelineno-0-132) return False [](#__codelineno-0-133) else: [](#__codelineno-0-134) return False [](#__codelineno-0-135) except aiohttp.ClientError as e: [](#__codelineno-0-136) logger.error(str(e)) [](#__codelineno-0-137) return False [](#__codelineno-0-138) except Exception as e: [](#__codelineno-0-139) logger.error(str(e)) [](#__codelineno-0-140) return False [](#__codelineno-0-141) [](#__codelineno-0-142) async def add_instance_endpoint(self, request: Request): [](#__codelineno-0-143) try: [](#__codelineno-0-144) data = await request.json() [](#__codelineno-0-145) logger.warning(str(data)) [](#__codelineno-0-146) instance_type = data.get("type") [](#__codelineno-0-147) instance = data.get("instance") [](#__codelineno-0-148) if instance_type not in ["prefill", "decode"]: [](#__codelineno-0-149) raise HTTPException(status_code=400, detail="Invalid instance type.") [](#__codelineno-0-150) if not instance or ":" not in instance: [](#__codelineno-0-151) raise HTTPException(status_code=400, detail="Invalid instance format.") [](#__codelineno-0-152) host, port_str = instance.split(":") [](#__codelineno-0-153) try: [](#__codelineno-0-154) if host != "localhost": [](#__codelineno-0-155) ipaddress.ip_address(host) [](#__codelineno-0-156) port = int(port_str) [](#__codelineno-0-157) if not (0 < port < 65536): [](#__codelineno-0-158) raise HTTPException(status_code=400, detail="Invalid port number.") [](#__codelineno-0-159) except Exception as e: [](#__codelineno-0-160) raise HTTPException( [](#__codelineno-0-161) status_code=400, detail="Invalid instance address." [](#__codelineno-0-162) ) from e [](#__codelineno-0-163) [](#__codelineno-0-164) is_valid = await self.validate_instance(instance) [](#__codelineno-0-165) if not is_valid: [](#__codelineno-0-166) raise HTTPException( [](#__codelineno-0-167) status_code=400, detail="Instance validation failed." [](#__codelineno-0-168) ) [](#__codelineno-0-169) [](#__codelineno-0-170) if instance_type == "prefill": [](#__codelineno-0-171) if instance not in self.prefill_instances: [](#__codelineno-0-172) self.prefill_instances.append(instance) [](#__codelineno-0-173) self.prefill_cycler = itertools.cycle(self.prefill_instances) [](#__codelineno-0-174) else: [](#__codelineno-0-175) raise HTTPException( [](#__codelineno-0-176) status_code=400, detail="Instance already exists." [](#__codelineno-0-177) ) [](#__codelineno-0-178) else: [](#__codelineno-0-179) if instance not in self.decode_instances: [](#__codelineno-0-180) self.decode_instances.append(instance) [](#__codelineno-0-181) self.decode_cycler = itertools.cycle(self.decode_instances) [](#__codelineno-0-182) else: [](#__codelineno-0-183) raise HTTPException( [](#__codelineno-0-184) status_code=400, detail="Instance already exists." [](#__codelineno-0-185) ) [](#__codelineno-0-186) [](#__codelineno-0-187) return JSONResponse( [](#__codelineno-0-188) content={"message": f"Added {instance} to {instance_type}_instances."} [](#__codelineno-0-189) ) [](#__codelineno-0-190) except HTTPException as http_exc: [](#__codelineno-0-191) raise http_exc [](#__codelineno-0-192) except Exception as e: [](#__codelineno-0-193) logger.error("Error in add_instance_endpoint: %s", str(e)) [](#__codelineno-0-194) raise HTTPException(status_code=500, detail=str(e)) from e [](#__codelineno-0-195) [](#__codelineno-0-196) async def forward_request(self, url, data, use_chunked=True): [](#__codelineno-0-197) async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session: [](#__codelineno-0-198) headers = {"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"} [](#__codelineno-0-199) try: [](#__codelineno-0-200) async with session.post( [](#__codelineno-0-201) url=url, json=data, headers=headers [](#__codelineno-0-202) ) as response: [](#__codelineno-0-203) if 200 <= response.status < 300 or 400 <= response.status < 500: [](#__codelineno-0-204) if use_chunked: [](#__codelineno-0-205) async for chunk_bytes in response.content.iter_chunked( [](#__codelineno-0-206) 1024 [](#__codelineno-0-207) ): [](#__codelineno-0-208) yield chunk_bytes [](#__codelineno-0-209) else: [](#__codelineno-0-210) content = await response.read() [](#__codelineno-0-211) yield content [](#__codelineno-0-212) else: [](#__codelineno-0-213) error_content = await response.text() [](#__codelineno-0-214) try: [](#__codelineno-0-215) error_content = json.loads(error_content) [](#__codelineno-0-216) except json.JSONDecodeError: [](#__codelineno-0-217) error_content = error_content [](#__codelineno-0-218) logger.error( [](#__codelineno-0-219) "Request failed with status %s: %s", [](#__codelineno-0-220) response.status, [](#__codelineno-0-221) error_content, [](#__codelineno-0-222) ) [](#__codelineno-0-223) raise HTTPException( [](#__codelineno-0-224) status_code=response.status, [](#__codelineno-0-225) detail=f"Request failed with status {response.status}: " [](#__codelineno-0-226) f"{error_content}", [](#__codelineno-0-227) ) [](#__codelineno-0-228) except aiohttp.ClientError as e: [](#__codelineno-0-229) logger.error("ClientError occurred: %s", str(e)) [](#__codelineno-0-230) raise HTTPException( [](#__codelineno-0-231) status_code=502, [](#__codelineno-0-232) detail="Bad Gateway: Error communicating with upstream server.", [](#__codelineno-0-233) ) from e [](#__codelineno-0-234) except Exception as e: [](#__codelineno-0-235) logger.error("Unexpected error: %s", str(e)) [](#__codelineno-0-236) raise HTTPException(status_code=500, detail=str(e)) from e [](#__codelineno-0-237) [](#__codelineno-0-238) def schedule(self, cycler: itertools.cycle) -> str: [](#__codelineno-0-239) return self.scheduling_policy.schedule(cycler) [](#__codelineno-0-240) [](#__codelineno-0-241) async def get_status(self): [](#__codelineno-0-242) status = { [](#__codelineno-0-243) "prefill_node_count": len(self.prefill_instances), [](#__codelineno-0-244) "decode_node_count": len(self.decode_instances), [](#__codelineno-0-245) "prefill_nodes": self.prefill_instances, [](#__codelineno-0-246) "decode_nodes": self.decode_instances, [](#__codelineno-0-247) } [](#__codelineno-0-248) return status [](#__codelineno-0-249) [](#__codelineno-0-250) async def create_completion(self, raw_request: Request): [](#__codelineno-0-251) try: [](#__codelineno-0-252) request = await raw_request.json() [](#__codelineno-0-253) [](#__codelineno-0-254) kv_prepare_request = request.copy() [](#__codelineno-0-255) kv_prepare_request["max_tokens"] = 1 [](#__codelineno-0-256) [](#__codelineno-0-257) prefill_instance = self.schedule(self.prefill_cycler) [](#__codelineno-0-258) try: [](#__codelineno-0-259) async for _ in self.forward_request( [](#__codelineno-0-260) f"http://{prefill_instance}/v1/completions", kv_prepare_request [](#__codelineno-0-261) ): [](#__codelineno-0-262) continue [](#__codelineno-0-263) except HTTPException as http_exc: [](#__codelineno-0-264) self.remove_instance_endpoint("prefill", prefill_instance) [](#__codelineno-0-265) raise http_exc [](#__codelineno-0-266) [](#__codelineno-0-267) # Perform kv recv and decoding stage [](#__codelineno-0-268) decode_instance = self.schedule(self.decode_cycler) [](#__codelineno-0-269) [](#__codelineno-0-270) try: [](#__codelineno-0-271) generator = self.forward_request( [](#__codelineno-0-272) f"http://{decode_instance}/v1/completions", request [](#__codelineno-0-273) ) [](#__codelineno-0-274) except HTTPException as http_exc: [](#__codelineno-0-275) self.remove_instance_endpoint("decode", decode_instance) [](#__codelineno-0-276) raise http_exc [](#__codelineno-0-277) response = StreamingResponse(generator) [](#__codelineno-0-278) return response [](#__codelineno-0-279) except Exception: [](#__codelineno-0-280) import sys [](#__codelineno-0-281) [](#__codelineno-0-282) exc_info = sys.exc_info() [](#__codelineno-0-283) print("Error occurred in disagg proxy server") [](#__codelineno-0-284) print(exc_info) [](#__codelineno-0-285) [](#__codelineno-0-286) async def create_chat_completion(self, raw_request: Request): [](#__codelineno-0-287) try: [](#__codelineno-0-288) request = await raw_request.json() [](#__codelineno-0-289) [](#__codelineno-0-290) # add params to request [](#__codelineno-0-291) kv_prepare_request = request.copy() [](#__codelineno-0-292) kv_prepare_request["max_tokens"] = 1 [](#__codelineno-0-293) if "max_completion_tokens" in kv_prepare_request: [](#__codelineno-0-294) kv_prepare_request["max_completion_tokens"] = 1 [](#__codelineno-0-295) [](#__codelineno-0-296) # prefill stage [](#__codelineno-0-297) prefill_instance = self.schedule(self.prefill_cycler) [](#__codelineno-0-298) try: [](#__codelineno-0-299) async for _ in self.forward_request( [](#__codelineno-0-300) f"http://{prefill_instance}/v1/chat/completions", kv_prepare_request [](#__codelineno-0-301) ): [](#__codelineno-0-302) continue [](#__codelineno-0-303) except HTTPException as http_exc: [](#__codelineno-0-304) self.remove_instance_endpoint("prefill", prefill_instance) [](#__codelineno-0-305) raise http_exc [](#__codelineno-0-306) # Perform kv recv and decoding stage [](#__codelineno-0-307) decode_instance = self.schedule(self.decode_cycler) [](#__codelineno-0-308) [](#__codelineno-0-309) try: [](#__codelineno-0-310) generator = self.forward_request( [](#__codelineno-0-311) "http://" + decode_instance + "/v1/chat/completions", request [](#__codelineno-0-312) ) [](#__codelineno-0-313) except HTTPException as http_exc: [](#__codelineno-0-314) self.remove_instance_endpoint("decode", decode_instance) [](#__codelineno-0-315) raise http_exc [](#__codelineno-0-316) response = StreamingResponse(content=generator) [](#__codelineno-0-317) return response [](#__codelineno-0-318) except Exception: [](#__codelineno-0-319) exc_info = sys.exc_info() [](#__codelineno-0-320) error_messages = [str(e) for e in exc_info if e] [](#__codelineno-0-321) print("Error occurred in disagg proxy server") [](#__codelineno-0-322) print(error_messages) [](#__codelineno-0-323) return StreamingResponse( [](#__codelineno-0-324) content=iter(error_messages), media_type="text/event-stream" [](#__codelineno-0-325) ) [](#__codelineno-0-326) [](#__codelineno-0-327) def remove_instance_endpoint(self, instance_type, instance): [](#__codelineno-0-328) if instance_type == "decode" and instance in self.decode_instances: [](#__codelineno-0-329) self.decode_instances.remove(instance) [](#__codelineno-0-330) self.decode_cycler = itertools.cycle(self.decode_instances) [](#__codelineno-0-331) if instance_type == "prefill" and instance in self.prefill_instances: [](#__codelineno-0-332) self.prefill_instances.remove(instance) [](#__codelineno-0-333) self.prefill_cycler = itertools.cycle(self.prefill_instances) [](#__codelineno-0-334) [](#__codelineno-0-335)[](#__codelineno-0-336)class RoundRobinSchedulingPolicy(SchedulingPolicy): [](#__codelineno-0-337) def __init__(self): [](#__codelineno-0-338) super().__init__() [](#__codelineno-0-339) [](#__codelineno-0-340) def schedule(self, cycler: itertools.cycle) -> str: [](#__codelineno-0-341) return next(cycler) [](#__codelineno-0-342) [](#__codelineno-0-343)[](#__codelineno-0-344)class ProxyServer: [](#__codelineno-0-345) def __init__( [](#__codelineno-0-346) self, [](#__codelineno-0-347) args: argparse.Namespace, [](#__codelineno-0-348) scheduling_policy: SchedulingPolicy | None = None, [](#__codelineno-0-349) create_completion: Callable[[Request], StreamingResponse] | None = None, [](#__codelineno-0-350) create_chat_completion: Callable[[Request], StreamingResponse] | None = None, [](#__codelineno-0-351) ): [](#__codelineno-0-352) self.validate_parsed_serve_args(args) [](#__codelineno-0-353) self.port = args.port [](#__codelineno-0-354) self.proxy_instance = Proxy( [](#__codelineno-0-355) prefill_instances=[] if args.prefill is None else args.prefill, [](#__codelineno-0-356) decode_instances=[] if args.decode is None else args.decode, [](#__codelineno-0-357) model=args.model, [](#__codelineno-0-358) scheduling_policy=( [](#__codelineno-0-359) scheduling_policy [](#__codelineno-0-360) if scheduling_policy is not None [](#__codelineno-0-361) else RoundRobinSchedulingPolicy() [](#__codelineno-0-362) ), [](#__codelineno-0-363) custom_create_completion=create_completion, [](#__codelineno-0-364) custom_create_chat_completion=create_chat_completion, [](#__codelineno-0-365) ) [](#__codelineno-0-366) [](#__codelineno-0-367) def validate_parsed_serve_args(self, args: argparse.Namespace): [](#__codelineno-0-368) if not args.prefill: [](#__codelineno-0-369) raise ValueError("Please specify at least one prefill node.") [](#__codelineno-0-370) if not args.decode: [](#__codelineno-0-371) raise ValueError("Please specify at least one decode node.") [](#__codelineno-0-372) self.validate_instances(args.prefill) [](#__codelineno-0-373) self.validate_instances(args.decode) [](#__codelineno-0-374) self.verify_model_config(args.prefill, args.model) [](#__codelineno-0-375) self.verify_model_config(args.decode, args.model) [](#__codelineno-0-376) [](#__codelineno-0-377) def validate_instances(self, instances: list): [](#__codelineno-0-378) for instance in instances: [](#__codelineno-0-379) if len(instance.split(":")) != 2: [](#__codelineno-0-380) raise ValueError(f"Invalid instance format: {instance}") [](#__codelineno-0-381) host, port = instance.split(":") [](#__codelineno-0-382) try: [](#__codelineno-0-383) if host != "localhost": [](#__codelineno-0-384) ipaddress.ip_address(host) [](#__codelineno-0-385) port = int(port) [](#__codelineno-0-386) if not (0 < port < 65536): [](#__codelineno-0-387) raise ValueError(f"Invalid port number in instance: {instance}") [](#__codelineno-0-388) except Exception as e: [](#__codelineno-0-389) raise ValueError(f"Invalid instance {instance}: {str(e)}") from e [](#__codelineno-0-390) [](#__codelineno-0-391) def verify_model_config(self, instances: list, model: str) -> None: [](#__codelineno-0-392) model_suffix = model.split("/")[-1] [](#__codelineno-0-393) for instance in instances: [](#__codelineno-0-394) try: [](#__codelineno-0-395) response = requests.get(f"http://{instance}/v1/models") [](#__codelineno-0-396) if response.status_code == 200: [](#__codelineno-0-397) model_cur = response.json()["data"][0]["id"] [](#__codelineno-0-398) model_cur_suffix = model_cur.split("/")[-1] [](#__codelineno-0-399) if model_cur_suffix != model_suffix: [](#__codelineno-0-400) raise ValueError( [](#__codelineno-0-401) f"{instance} serves a different model: " [](#__codelineno-0-402) f"{model_cur} != {model}" [](#__codelineno-0-403) ) [](#__codelineno-0-404) else: [](#__codelineno-0-405) raise ValueError(f"Cannot get model id from {instance}!") [](#__codelineno-0-406) except requests.RequestException as e: [](#__codelineno-0-407) raise ValueError( [](#__codelineno-0-408) f"Error communicating with {instance}: {str(e)}" [](#__codelineno-0-409) ) from e [](#__codelineno-0-410) [](#__codelineno-0-411) def run_server(self): [](#__codelineno-0-412) app = FastAPI() [](#__codelineno-0-413) app.include_router(self.proxy_instance.router) [](#__codelineno-0-414) config = uvicorn.Config(app, port=self.port, loop="uvloop") [](#__codelineno-0-415) server = uvicorn.Server(config) [](#__codelineno-0-416) server.run() [](#__codelineno-0-417) [](#__codelineno-0-418)[](#__codelineno-0-419)def parse_args(): [](#__codelineno-0-420) # Todo: allow more config [](#__codelineno-0-421) parser = argparse.ArgumentParser("vLLM disaggregated proxy server.") [](#__codelineno-0-422) parser.add_argument("--model", "-m", type=str, required=True, help="Model name") [](#__codelineno-0-423) [](#__codelineno-0-424) parser.add_argument( [](#__codelineno-0-425) "--prefill", [](#__codelineno-0-426) "-p", [](#__codelineno-0-427) type=str, [](#__codelineno-0-428) nargs="+", [](#__codelineno-0-429) help="List of prefill node URLs (host:port)", [](#__codelineno-0-430) ) [](#__codelineno-0-431) [](#__codelineno-0-432) parser.add_argument( [](#__codelineno-0-433) "--decode", [](#__codelineno-0-434) "-d", [](#__codelineno-0-435) type=str, [](#__codelineno-0-436) nargs="+", [](#__codelineno-0-437) help="List of decode node URLs (host:port)", [](#__codelineno-0-438) ) [](#__codelineno-0-439) [](#__codelineno-0-440) parser.add_argument( [](#__codelineno-0-441) "--port", [](#__codelineno-0-442) type=int, [](#__codelineno-0-443) default=8000, [](#__codelineno-0-444) help="Server port number", [](#__codelineno-0-445) ) [](#__codelineno-0-446) return parser.parse_args() [](#__codelineno-0-447) [](#__codelineno-0-448)[](#__codelineno-0-449)if __name__ == "__main__": [](#__codelineno-0-450) args = parse_args() [](#__codelineno-0-451) proxy_server = ProxyServer(args=args) [](#__codelineno-0-452) proxy_server.run_server()` disagg\_proxy\_multiturn.py `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)Disaggregated Prefill/Decode Proxy with Bidirectional KV Transfer [](#__codelineno-1-5)[](#__codelineno-1-6)This proxy sits between clients and a vLLM Prefill/Decode (P/D) deployment, [](#__codelineno-1-7)routing multi-turn chat requests so that each turn reuses KV cache blocks [](#__codelineno-1-8)from the previous turn's Decode node via bidirectional KV transfer. [](#__codelineno-1-9)[](#__codelineno-1-10)Architecture: [](#__codelineno-1-11) Client ──► Proxy ──► Prefill (P) ──► Decode (D) [](#__codelineno-1-12) │ │ │ [](#__codelineno-1-13) │ kv_transfer_params flow: │ [](#__codelineno-1-14) │ D finish ──► proxy caches │ [](#__codelineno-1-15) │ next turn ──► proxy sends │ [](#__codelineno-1-16) │ cached D blocks to P ──► │ [](#__codelineno-1-17) │ P reads D blocks (bidir) │ [](#__codelineno-1-18) │ P sends its blocks to D │ [](#__codelineno-1-19)[](#__codelineno-1-20)Per-request flow: [](#__codelineno-1-21) 1. Client sends chat/completions request to proxy. [](#__codelineno-1-22) 2. Proxy looks up cached D block info from the previous turn [](#__codelineno-1-23) (keyed by conversation_id). [](#__codelineno-1-24) 3. If cache hit, proxy attaches D's block info to the request [](#__codelineno-1-25) so P can read D's KV blocks instead of recomputing. [](#__codelineno-1-26) 4. Proxy sends request to P (max_tokens=1, non-streaming). [](#__codelineno-1-27) 5. P returns kv_transfer_params with its own block info. [](#__codelineno-1-28) 6. Proxy forwards request + P's block info to D (streaming). [](#__codelineno-1-29) 7. D streams the response. The final chunk includes D's [](#__codelineno-1-30) kv_transfer_params, which the proxy caches for the next turn. [](#__codelineno-1-31) 8. Proxy returns D's response to the client. [](#__codelineno-1-32)[](#__codelineno-1-33)Conversation isolation: [](#__codelineno-1-34) Each request must include a ``conversation_id`` field (top-level in [](#__codelineno-1-35) the JSON body) to scope the KV cache across turns. Without it, the [](#__codelineno-1-36) proxy cannot link turns and falls back to no-cache behavior. [](#__codelineno-1-37)[](#__codelineno-1-38)Usage: [](#__codelineno-1-39) python disagg_proxy_multiturn.py \\ [](#__codelineno-1-40) --host 0.0.0.0 --port 8000 \\ [](#__codelineno-1-41) --prefiller-host 10.0.0.1 --prefiller-port 8100 \\ [](#__codelineno-1-42) --decoder-host 10.0.0.2 --decoder-port 8200 [](#__codelineno-1-43)[](#__codelineno-1-44)Dependencies: [](#__codelineno-1-45) pip install fastapi uvicorn httpx [](#__codelineno-1-46)""" [](#__codelineno-1-47)[](#__codelineno-1-48)from __future__ import annotations [](#__codelineno-1-49)[](#__codelineno-1-50)import argparse [](#__codelineno-1-51)import itertools [](#__codelineno-1-52)import json [](#__codelineno-1-53)import logging [](#__codelineno-1-54)import os [](#__codelineno-1-55)import time [](#__codelineno-1-56)import uuid [](#__codelineno-1-57)from contextlib import asynccontextmanager [](#__codelineno-1-58)from dataclasses import dataclass, field [](#__codelineno-1-59)from typing import Any [](#__codelineno-1-60)[](#__codelineno-1-61)import httpx [](#__codelineno-1-62)from fastapi import FastAPI, Request [](#__codelineno-1-63)from fastapi.responses import JSONResponse, StreamingResponse [](#__codelineno-1-64)[](#__codelineno-1-65)# Logging [](#__codelineno-1-66)logging.basicConfig( [](#__codelineno-1-67) format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", [](#__codelineno-1-68) datefmt="%Y-%m-%d %H:%M:%S", [](#__codelineno-1-69) level=logging.INFO, [](#__codelineno-1-70)) [](#__codelineno-1-71)logger = logging.getLogger("disagg_proxy") [](#__codelineno-1-72) [](#__codelineno-1-73)[](#__codelineno-1-74)# Data structures [](#__codelineno-1-75)@dataclass [](#__codelineno-1-76)class CachedKVEntry: [](#__codelineno-1-77) """KV transfer parameters cached from D's response for one turn.""" [](#__codelineno-1-78) [](#__codelineno-1-79) kv_transfer_params: dict[str, Any] [](#__codelineno-1-80) timestamp: float = field(default_factory=time.time) [](#__codelineno-1-81) [](#__codelineno-1-82)[](#__codelineno-1-83)class ConversationKVCache: [](#__codelineno-1-84) """Per-conversation KV block cache. [](#__codelineno-1-85) [](#__codelineno-1-86) Each conversation is identified by a ``conversation_id`` supplied by [](#__codelineno-1-87) the client. After D finishes a turn, its ``kv_transfer_params`` are [](#__codelineno-1-88) stored here. On the next turn, the proxy retrieves them so P can [](#__codelineno-1-89) read D's blocks via bidirectional KV transfer. [](#__codelineno-1-90) """ [](#__codelineno-1-91) [](#__codelineno-1-92) def __init__(self, ttl_seconds: float = 600.0) -> None: [](#__codelineno-1-93) self._store: dict[str, CachedKVEntry] = {} [](#__codelineno-1-94) self._ttl = ttl_seconds [](#__codelineno-1-95) [](#__codelineno-1-96) def get(self, conversation_id: str) -> dict[str, Any] | None: [](#__codelineno-1-97) """Retrieve and consume cached KV params for a conversation. [](#__codelineno-1-98) [](#__codelineno-1-99) Returns a *copy* of the kv_transfer_params dict, or None. [](#__codelineno-1-100) The entry is removed after retrieval (single-use). [](#__codelineno-1-101) """ [](#__codelineno-1-102) entry = self._store.pop(conversation_id, None) [](#__codelineno-1-103) if entry is None: [](#__codelineno-1-104) return None [](#__codelineno-1-105) age = time.time() - entry.timestamp [](#__codelineno-1-106) if age > self._ttl: [](#__codelineno-1-107) logger.info( [](#__codelineno-1-108) "conv=%s: stale cache entry (age=%.1fs > ttl=%.1fs), discarding", [](#__codelineno-1-109) conversation_id, [](#__codelineno-1-110) age, [](#__codelineno-1-111) self._ttl, [](#__codelineno-1-112) ) [](#__codelineno-1-113) return None [](#__codelineno-1-114) logger.info( [](#__codelineno-1-115) "conv=%s: cache HIT (age=%.1fs)", [](#__codelineno-1-116) conversation_id, [](#__codelineno-1-117) age, [](#__codelineno-1-118) ) [](#__codelineno-1-119) return dict(entry.kv_transfer_params) [](#__codelineno-1-120) [](#__codelineno-1-121) def put(self, conversation_id: str, kv_params: dict[str, Any]) -> None: [](#__codelineno-1-122) """Store D's kv_transfer_params for a conversation.""" [](#__codelineno-1-123) self._store[conversation_id] = CachedKVEntry( [](#__codelineno-1-124) kv_transfer_params=dict(kv_params), # defensive copy [](#__codelineno-1-125) ) [](#__codelineno-1-126) logger.info( [](#__codelineno-1-127) "conv=%s: cached D blocks (remote_request_id=%s, blocks=%d)", [](#__codelineno-1-128) conversation_id, [](#__codelineno-1-129) kv_params.get("remote_request_id", "?"), [](#__codelineno-1-130) len(kv_params.get("remote_block_ids", [[]])[0]) [](#__codelineno-1-131) if kv_params.get("remote_block_ids") [](#__codelineno-1-132) else 0, [](#__codelineno-1-133) ) [](#__codelineno-1-134) [](#__codelineno-1-135) def evict_stale(self) -> int: [](#__codelineno-1-136) """Remove entries older than TTL. Returns count of evicted entries.""" [](#__codelineno-1-137) now = time.time() [](#__codelineno-1-138) stale = [ [](#__codelineno-1-139) cid [](#__codelineno-1-140) for cid, entry in self._store.items() [](#__codelineno-1-141) if now - entry.timestamp > self._ttl [](#__codelineno-1-142) ] [](#__codelineno-1-143) for cid in stale: [](#__codelineno-1-144) del self._store[cid] [](#__codelineno-1-145) return len(stale) [](#__codelineno-1-146) [](#__codelineno-1-147) @property [](#__codelineno-1-148) def size(self) -> int: [](#__codelineno-1-149) return len(self._store) [](#__codelineno-1-150) [](#__codelineno-1-151)[](#__codelineno-1-152)# Global state [](#__codelineno-1-153)kv_cache = ConversationKVCache( [](#__codelineno-1-154) ttl_seconds=450.0 [](#__codelineno-1-155)) # Must be < VLLM_NIXL_ABORT_REQUEST_TIMEOUT (480s) [](#__codelineno-1-156) [](#__codelineno-1-157)[](#__codelineno-1-158)# Service client helpers [](#__codelineno-1-159)@dataclass [](#__codelineno-1-160)class ServiceClient: [](#__codelineno-1-161) """Wrapper around an httpx.AsyncClient for a P or D instance.""" [](#__codelineno-1-162) [](#__codelineno-1-163) client: httpx.AsyncClient [](#__codelineno-1-164) host: str [](#__codelineno-1-165) port: int [](#__codelineno-1-166) id: int [](#__codelineno-1-167) [](#__codelineno-1-168)[](#__codelineno-1-169)def _make_headers(request_id: str) -> dict[str, str]: [](#__codelineno-1-170) """Build HTTP headers for upstream requests.""" [](#__codelineno-1-171) headers = {"X-Request-Id": request_id} [](#__codelineno-1-172) api_key = os.environ.get("OPENAI_API_KEY") [](#__codelineno-1-173) if api_key: [](#__codelineno-1-174) headers["Authorization"] = f"Bearer {api_key}" [](#__codelineno-1-175) return headers [](#__codelineno-1-176) [](#__codelineno-1-177)[](#__codelineno-1-178)async def _send_to_prefill( [](#__codelineno-1-179) client: ServiceClient, [](#__codelineno-1-180) endpoint: str, [](#__codelineno-1-181) req_data: dict[str, Any], [](#__codelineno-1-182) request_id: str, [](#__codelineno-1-183)) -> dict[str, Any]: [](#__codelineno-1-184) """Send a non-streaming prefill request (max_tokens=1). [](#__codelineno-1-185) [](#__codelineno-1-186) Returns the JSON response from P, which includes kv_transfer_params. [](#__codelineno-1-187) """ [](#__codelineno-1-188) payload = req_data.copy() [](#__codelineno-1-189) payload["stream"] = False [](#__codelineno-1-190) payload["max_tokens"] = 1 [](#__codelineno-1-191) payload.pop("max_completion_tokens", None) [](#__codelineno-1-192) payload.pop("min_tokens", None) [](#__codelineno-1-193) payload.pop("stream_options", None) [](#__codelineno-1-194) [](#__codelineno-1-195) resp = await client.client.post( [](#__codelineno-1-196) endpoint, [](#__codelineno-1-197) json=payload, [](#__codelineno-1-198) headers=_make_headers(request_id), [](#__codelineno-1-199) ) [](#__codelineno-1-200) resp.raise_for_status() [](#__codelineno-1-201) return resp.json() [](#__codelineno-1-202) [](#__codelineno-1-203)[](#__codelineno-1-204)async def _stream_from_decode( [](#__codelineno-1-205) client: ServiceClient, [](#__codelineno-1-206) endpoint: str, [](#__codelineno-1-207) req_data: dict[str, Any], [](#__codelineno-1-208) request_id: str, [](#__codelineno-1-209) conversation_id: str, [](#__codelineno-1-210)) -> tuple[str, str | None, dict[str, Any] | None, str, str | None, int | None]: [](#__codelineno-1-211) """Stream response from D, capturing text and kv_transfer_params. [](#__codelineno-1-212) [](#__codelineno-1-213) Returns (collected_text, finish_reason, kv_params, response_id, created). [](#__codelineno-1-214) Also stores kv_params in the conversation cache. [](#__codelineno-1-215) """ [](#__codelineno-1-216) payload = req_data.copy() [](#__codelineno-1-217) payload["stream"] = True [](#__codelineno-1-218) [](#__codelineno-1-219) collected_text = "" [](#__codelineno-1-220) finish_reason: str | None = None [](#__codelineno-1-221) response_id: str | None = None [](#__codelineno-1-222) model_name: str | None = None [](#__codelineno-1-223) created: int | None = None [](#__codelineno-1-224) captured_kv: dict[str, Any] | None = None [](#__codelineno-1-225) [](#__codelineno-1-226) async with client.client.stream( [](#__codelineno-1-227) "POST", [](#__codelineno-1-228) endpoint, [](#__codelineno-1-229) json=payload, [](#__codelineno-1-230) headers=_make_headers(request_id), [](#__codelineno-1-231) ) as resp: [](#__codelineno-1-232) resp.raise_for_status() [](#__codelineno-1-233) async for line in resp.aiter_lines(): [](#__codelineno-1-234) if not line or not line.startswith("data: "): [](#__codelineno-1-235) continue [](#__codelineno-1-236) if line == "data: [DONE]": [](#__codelineno-1-237) break [](#__codelineno-1-238) try: [](#__codelineno-1-239) chunk = json.loads(line[6:]) [](#__codelineno-1-240) except json.JSONDecodeError: [](#__codelineno-1-241) continue [](#__codelineno-1-242) [](#__codelineno-1-243) if response_id is None: [](#__codelineno-1-244) response_id = chunk.get("id") [](#__codelineno-1-245) model_name = chunk.get("model") [](#__codelineno-1-246) created = chunk.get("created") [](#__codelineno-1-247) [](#__codelineno-1-248) for choice in chunk.get("choices", []): [](#__codelineno-1-249) collected_text += choice.get("text", "") [](#__codelineno-1-250) delta = choice.get("delta", {}) [](#__codelineno-1-251) collected_text += delta.get("content", "") [](#__codelineno-1-252) if choice.get("finish_reason"): [](#__codelineno-1-253) finish_reason = choice["finish_reason"] [](#__codelineno-1-254) [](#__codelineno-1-255) kv_params = chunk.get("kv_transfer_params") [](#__codelineno-1-256) if kv_params: [](#__codelineno-1-257) kv_params["remote_host"] = client.host [](#__codelineno-1-258) captured_kv = kv_params [](#__codelineno-1-259) if conversation_id: [](#__codelineno-1-260) kv_cache.put(conversation_id, kv_params) [](#__codelineno-1-261) [](#__codelineno-1-262) return ( [](#__codelineno-1-263) collected_text, [](#__codelineno-1-264) finish_reason, [](#__codelineno-1-265) captured_kv, [](#__codelineno-1-266) response_id or request_id, [](#__codelineno-1-267) model_name, [](#__codelineno-1-268) created, [](#__codelineno-1-269) ) [](#__codelineno-1-270) [](#__codelineno-1-271)[](#__codelineno-1-272)async def _stream_from_decode_sse( [](#__codelineno-1-273) client: ServiceClient, [](#__codelineno-1-274) endpoint: str, [](#__codelineno-1-275) req_data: dict[str, Any], [](#__codelineno-1-276) request_id: str, [](#__codelineno-1-277) conversation_id: str, [](#__codelineno-1-278)): [](#__codelineno-1-279) """Yield SSE chunks from D to the client, capturing kv_transfer_params.""" [](#__codelineno-1-280) payload = req_data.copy() [](#__codelineno-1-281) payload["stream"] = True [](#__codelineno-1-282) [](#__codelineno-1-283) async with client.client.stream( [](#__codelineno-1-284) "POST", [](#__codelineno-1-285) endpoint, [](#__codelineno-1-286) json=payload, [](#__codelineno-1-287) headers=_make_headers(request_id), [](#__codelineno-1-288) ) as resp: [](#__codelineno-1-289) resp.raise_for_status() [](#__codelineno-1-290) async for line in resp.aiter_lines(): [](#__codelineno-1-291) if not line: [](#__codelineno-1-292) yield "\n" [](#__codelineno-1-293) continue [](#__codelineno-1-294) [](#__codelineno-1-295) if line.startswith("data: ") and line != "data: [DONE]": [](#__codelineno-1-296) try: [](#__codelineno-1-297) chunk = json.loads(line[6:]) [](#__codelineno-1-298) kv_params = chunk.get("kv_transfer_params") [](#__codelineno-1-299) if kv_params and conversation_id: [](#__codelineno-1-300) kv_params["remote_host"] = client.host [](#__codelineno-1-301) kv_cache.put(conversation_id, kv_params) [](#__codelineno-1-302) except json.JSONDecodeError: [](#__codelineno-1-303) pass [](#__codelineno-1-304) [](#__codelineno-1-305) yield line + "\n" [](#__codelineno-1-306) [](#__codelineno-1-307)[](#__codelineno-1-308)# FastAPI application [](#__codelineno-1-309)@asynccontextmanager [](#__codelineno-1-310)async def lifespan(app: FastAPI): [](#__codelineno-1-311) """Initialize HTTP clients for P and D instances.""" [](#__codelineno-1-312) app.state.prefill_clients: list[ServiceClient] = [] [](#__codelineno-1-313) app.state.decode_clients: list[ServiceClient] = [] [](#__codelineno-1-314) [](#__codelineno-1-315) for i, (host, port) in enumerate(global_args.prefiller_instances): [](#__codelineno-1-316) app.state.prefill_clients.append( [](#__codelineno-1-317) ServiceClient( [](#__codelineno-1-318) client=httpx.AsyncClient( [](#__codelineno-1-319) timeout=None, [](#__codelineno-1-320) base_url=f"http://{host}:{port}/v1", [](#__codelineno-1-321) ), [](#__codelineno-1-322) host=host, [](#__codelineno-1-323) port=port, [](#__codelineno-1-324) id=i, [](#__codelineno-1-325) ) [](#__codelineno-1-326) ) [](#__codelineno-1-327) [](#__codelineno-1-328) for i, (host, port) in enumerate(global_args.decoder_instances): [](#__codelineno-1-329) app.state.decode_clients.append( [](#__codelineno-1-330) ServiceClient( [](#__codelineno-1-331) client=httpx.AsyncClient( [](#__codelineno-1-332) timeout=None, [](#__codelineno-1-333) base_url=f"http://{host}:{port}/v1", [](#__codelineno-1-334) ), [](#__codelineno-1-335) host=host, [](#__codelineno-1-336) port=port, [](#__codelineno-1-337) id=i, [](#__codelineno-1-338) ) [](#__codelineno-1-339) ) [](#__codelineno-1-340) [](#__codelineno-1-341) app.state.prefill_iter = itertools.cycle(range(len(app.state.prefill_clients))) [](#__codelineno-1-342) app.state.decode_iter = itertools.cycle(range(len(app.state.decode_clients))) [](#__codelineno-1-343) [](#__codelineno-1-344) logger.info( [](#__codelineno-1-345) "Ready: %d prefill, %d decode instances", [](#__codelineno-1-346) len(app.state.prefill_clients), [](#__codelineno-1-347) len(app.state.decode_clients), [](#__codelineno-1-348) ) [](#__codelineno-1-349) yield [](#__codelineno-1-350) [](#__codelineno-1-351) for sc in app.state.prefill_clients + app.state.decode_clients: [](#__codelineno-1-352) await sc.client.aclose() [](#__codelineno-1-353) [](#__codelineno-1-354)[](#__codelineno-1-355)app = FastAPI(title="Disaggregated P/D Proxy (Multi-turn)", lifespan=lifespan) [](#__codelineno-1-356) [](#__codelineno-1-357)[](#__codelineno-1-358)def _next_client(app_state, role: str) -> ServiceClient: [](#__codelineno-1-359) if role == "prefill": [](#__codelineno-1-360) return app_state.prefill_clients[next(app_state.prefill_iter)] [](#__codelineno-1-361) return app_state.decode_clients[next(app_state.decode_iter)] [](#__codelineno-1-362) [](#__codelineno-1-363)[](#__codelineno-1-364)# Request handler [](#__codelineno-1-365)async def _handle_request(api_path: str, request: Request): [](#__codelineno-1-366) """Core request handler for both /v1/chat/completions and /v1/completions.""" [](#__codelineno-1-367) req_data = await request.json() [](#__codelineno-1-368) request_id = str(uuid.uuid4()) [](#__codelineno-1-369) conversation_id: str = req_data.pop("conversation_id", "") [](#__codelineno-1-370) client_wants_stream = req_data.get("stream", False) [](#__codelineno-1-371) [](#__codelineno-1-372) if not conversation_id: [](#__codelineno-1-373) logger.warning( [](#__codelineno-1-374) "[%s] No conversation_id provided — KV cache reuse disabled " [](#__codelineno-1-375) "for this request. Add a 'conversation_id' field to enable " [](#__codelineno-1-376) "cross-turn KV sharing.", [](#__codelineno-1-377) request_id, [](#__codelineno-1-378) ) [](#__codelineno-1-379) [](#__codelineno-1-380) # Step 1: Look up cached D blocks from the previous turn [](#__codelineno-1-381) cached_kv = kv_cache.get(conversation_id) if conversation_id else None [](#__codelineno-1-382) [](#__codelineno-1-383) if cached_kv: [](#__codelineno-1-384) # Tell P to read D's blocks (bidirectional transfer) [](#__codelineno-1-385) cached_kv["do_remote_decode"] = True [](#__codelineno-1-386) cached_kv["do_remote_prefill"] = False [](#__codelineno-1-387) req_data["kv_transfer_params"] = cached_kv [](#__codelineno-1-388) logger.info( [](#__codelineno-1-389) "[%s] conv=%s: sending D's cached blocks to P (remote_request_id=%s)", [](#__codelineno-1-390) request_id, [](#__codelineno-1-391) conversation_id, [](#__codelineno-1-392) cached_kv.get("remote_request_id"), [](#__codelineno-1-393) ) [](#__codelineno-1-394) else: [](#__codelineno-1-395) # No cached blocks — P recomputes from scratch [](#__codelineno-1-396) req_data["kv_transfer_params"] = { [](#__codelineno-1-397) "do_remote_decode": True, [](#__codelineno-1-398) "do_remote_prefill": False, [](#__codelineno-1-399) "remote_engine_id": None, [](#__codelineno-1-400) "remote_block_ids": None, [](#__codelineno-1-401) "remote_host": None, [](#__codelineno-1-402) "remote_port": None, [](#__codelineno-1-403) } [](#__codelineno-1-404) logger.info("[%s] conv=%s: cache MISS", request_id, conversation_id) [](#__codelineno-1-405) [](#__codelineno-1-406) # Step 2: Send to Prefill node (non-streaming, max_tokens=1) [](#__codelineno-1-407) prefill_client = _next_client(request.app.state, "prefill") [](#__codelineno-1-408) t0 = time.time() [](#__codelineno-1-409) prefill_resp = await _send_to_prefill( [](#__codelineno-1-410) prefill_client, [](#__codelineno-1-411) api_path, [](#__codelineno-1-412) req_data, [](#__codelineno-1-413) request_id, [](#__codelineno-1-414) ) [](#__codelineno-1-415) logger.info( [](#__codelineno-1-416) "[%s] Prefill done in %.0fms", [](#__codelineno-1-417) request_id, [](#__codelineno-1-418) (time.time() - t0) * 1000, [](#__codelineno-1-419) ) [](#__codelineno-1-420) [](#__codelineno-1-421) # Attach P's kv_transfer_params for D to read P's blocks [](#__codelineno-1-422) p_kv_params = prefill_resp.get("kv_transfer_params", {}) [](#__codelineno-1-423) if p_kv_params: [](#__codelineno-1-424) p_kv_params["remote_host"] = prefill_client.host [](#__codelineno-1-425) req_data["kv_transfer_params"] = p_kv_params [](#__codelineno-1-426) [](#__codelineno-1-427) # Step 3: Stream from Decode node, capturing kv_transfer_params [](#__codelineno-1-428) decode_client = _next_client(request.app.state, "decode") [](#__codelineno-1-429) [](#__codelineno-1-430) if client_wants_stream: [](#__codelineno-1-431) return StreamingResponse( [](#__codelineno-1-432) _stream_from_decode_sse( [](#__codelineno-1-433) decode_client, [](#__codelineno-1-434) api_path, [](#__codelineno-1-435) req_data, [](#__codelineno-1-436) request_id, [](#__codelineno-1-437) conversation_id, [](#__codelineno-1-438) ), [](#__codelineno-1-439) media_type="text/event-stream", [](#__codelineno-1-440) ) [](#__codelineno-1-441) [](#__codelineno-1-442) text, finish_reason, _, resp_id, model, created = await _stream_from_decode( [](#__codelineno-1-443) decode_client, [](#__codelineno-1-444) api_path, [](#__codelineno-1-445) req_data, [](#__codelineno-1-446) request_id, [](#__codelineno-1-447) conversation_id, [](#__codelineno-1-448) ) [](#__codelineno-1-449) [](#__codelineno-1-450) # Build OpenAI-compatible response [](#__codelineno-1-451) is_chat = "messages" in req_data [](#__codelineno-1-452) if is_chat: [](#__codelineno-1-453) body = { [](#__codelineno-1-454) "id": resp_id, [](#__codelineno-1-455) "object": "chat.completion", [](#__codelineno-1-456) "created": created or int(time.time()), [](#__codelineno-1-457) "model": model or req_data.get("model", ""), [](#__codelineno-1-458) "choices": [ [](#__codelineno-1-459) { [](#__codelineno-1-460) "index": 0, [](#__codelineno-1-461) "message": {"role": "assistant", "content": text}, [](#__codelineno-1-462) "finish_reason": finish_reason, [](#__codelineno-1-463) } [](#__codelineno-1-464) ], [](#__codelineno-1-465) "usage": None, [](#__codelineno-1-466) } [](#__codelineno-1-467) else: [](#__codelineno-1-468) body = { [](#__codelineno-1-469) "id": resp_id, [](#__codelineno-1-470) "object": "text_completion", [](#__codelineno-1-471) "created": created or int(time.time()), [](#__codelineno-1-472) "model": model or req_data.get("model", ""), [](#__codelineno-1-473) "choices": [ [](#__codelineno-1-474) { [](#__codelineno-1-475) "index": 0, [](#__codelineno-1-476) "text": text, [](#__codelineno-1-477) "logprobs": None, [](#__codelineno-1-478) "finish_reason": finish_reason, [](#__codelineno-1-479) } [](#__codelineno-1-480) ], [](#__codelineno-1-481) "usage": None, [](#__codelineno-1-482) } [](#__codelineno-1-483) return JSONResponse(content=body) [](#__codelineno-1-484) [](#__codelineno-1-485)[](#__codelineno-1-486)# Routes [](#__codelineno-1-487)@app.post("/v1/chat/completions") [](#__codelineno-1-488)async def chat_completions(request: Request): [](#__codelineno-1-489) return await _handle_request("/chat/completions", request) [](#__codelineno-1-490) [](#__codelineno-1-491)[](#__codelineno-1-492)@app.post("/v1/completions") [](#__codelineno-1-493)async def completions(request: Request): [](#__codelineno-1-494) return await _handle_request("/completions", request) [](#__codelineno-1-495) [](#__codelineno-1-496)[](#__codelineno-1-497)@app.get("/health") [](#__codelineno-1-498)async def health(): [](#__codelineno-1-499) evicted = kv_cache.evict_stale() [](#__codelineno-1-500) return { [](#__codelineno-1-501) "status": "ok", [](#__codelineno-1-502) "cached_conversations": kv_cache.size, [](#__codelineno-1-503) "evicted_stale": evicted, [](#__codelineno-1-504) } [](#__codelineno-1-505) [](#__codelineno-1-506)[](#__codelineno-1-507)# CLI [](#__codelineno-1-508)def parse_args() -> argparse.Namespace: [](#__codelineno-1-509) p = argparse.ArgumentParser( [](#__codelineno-1-510) description="Disaggregated P/D proxy with bidirectional KV transfer", [](#__codelineno-1-511) ) [](#__codelineno-1-512) p.add_argument("--host", default="0.0.0.0") [](#__codelineno-1-513) p.add_argument("--port", type=int, default=8000) [](#__codelineno-1-514) p.add_argument( [](#__codelineno-1-515) "--prefiller-host", [](#__codelineno-1-516) "--prefiller-hosts", [](#__codelineno-1-517) dest="prefiller_hosts", [](#__codelineno-1-518) nargs="+", [](#__codelineno-1-519) default=["localhost"], [](#__codelineno-1-520) ) [](#__codelineno-1-521) p.add_argument( [](#__codelineno-1-522) "--prefiller-port", [](#__codelineno-1-523) "--prefiller-ports", [](#__codelineno-1-524) dest="prefiller_ports", [](#__codelineno-1-525) type=int, [](#__codelineno-1-526) nargs="+", [](#__codelineno-1-527) default=[8100], [](#__codelineno-1-528) ) [](#__codelineno-1-529) p.add_argument( [](#__codelineno-1-530) "--decoder-host", [](#__codelineno-1-531) "--decoder-hosts", [](#__codelineno-1-532) dest="decoder_hosts", [](#__codelineno-1-533) nargs="+", [](#__codelineno-1-534) default=["localhost"], [](#__codelineno-1-535) ) [](#__codelineno-1-536) p.add_argument( [](#__codelineno-1-537) "--decoder-port", [](#__codelineno-1-538) "--decoder-ports", [](#__codelineno-1-539) dest="decoder_ports", [](#__codelineno-1-540) type=int, [](#__codelineno-1-541) nargs="+", [](#__codelineno-1-542) default=[8200], [](#__codelineno-1-543) ) [](#__codelineno-1-544) args = p.parse_args() [](#__codelineno-1-545) [](#__codelineno-1-546) if len(args.prefiller_hosts) != len(args.prefiller_ports): [](#__codelineno-1-547) p.error("Number of prefiller hosts must match ports") [](#__codelineno-1-548) if len(args.decoder_hosts) != len(args.decoder_ports): [](#__codelineno-1-549) p.error("Number of decoder hosts must match ports") [](#__codelineno-1-550) [](#__codelineno-1-551) args.prefiller_instances = list(zip(args.prefiller_hosts, args.prefiller_ports)) [](#__codelineno-1-552) args.decoder_instances = list(zip(args.decoder_hosts, args.decoder_ports)) [](#__codelineno-1-553) return args [](#__codelineno-1-554) [](#__codelineno-1-555)[](#__codelineno-1-556)if __name__ == "__main__": [](#__codelineno-1-557) global global_args [](#__codelineno-1-558) global_args = parse_args() [](#__codelineno-1-559) [](#__codelineno-1-560) import uvicorn [](#__codelineno-1-561) [](#__codelineno-1-562) uvicorn.run(app, host=global_args.host, port=global_args.port)` example\_mm\_serve.py `[](#__codelineno-2-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-2-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-2-3)"""Disaggregated multimodal serving: render → generate round-trip. [](#__codelineno-2-4)[](#__codelineno-2-5)Demonstrates the two-phase disaggregated flow: [](#__codelineno-2-6) 1. /v1/chat/completions/render – preprocesses a multimodal chat request [](#__codelineno-2-7) into token IDs and serialized tensor features. [](#__codelineno-2-8) 2. /inference/v1/generate – runs inference on the preprocessed tokens. [](#__codelineno-2-9)[](#__codelineno-2-10)The render response is passed *directly* to generate with only [](#__codelineno-2-11)``sampling_params`` added, showing that the two endpoints compose with [](#__codelineno-2-12)zero client-side transformation. [](#__codelineno-2-13)[](#__codelineno-2-14)Launch the server first: [](#__codelineno-2-15) [](#__codelineno-2-16) vllm serve Qwen/Qwen3-VL-2B-Instruct \ [](#__codelineno-2-17) --dtype bfloat16 --max-model-len 4096 --enforce-eager [](#__codelineno-2-18)[](#__codelineno-2-19)Then run this script: [](#__codelineno-2-20) [](#__codelineno-2-21) python example_mm_serve.py [](#__codelineno-2-22)""" [](#__codelineno-2-23)[](#__codelineno-2-24)import io [](#__codelineno-2-25)[](#__codelineno-2-26)import pybase64 as base64 [](#__codelineno-2-27)import requests [](#__codelineno-2-28)from PIL import Image [](#__codelineno-2-29)from transformers import AutoTokenizer [](#__codelineno-2-30)[](#__codelineno-2-31)BASE_URL = "http://localhost:8000" [](#__codelineno-2-32)MODEL_NAME = "Qwen/Qwen3-VL-2B-Instruct" [](#__codelineno-2-33) [](#__codelineno-2-34)[](#__codelineno-2-35)def make_data_url(image: Image.Image) -> str: [](#__codelineno-2-36) """Encode a PIL image as a base64 data URL.""" [](#__codelineno-2-37) buf = io.BytesIO() [](#__codelineno-2-38) image.save(buf, format="PNG") [](#__codelineno-2-39) b64 = base64.b64encode(buf.getvalue()).decode() [](#__codelineno-2-40) return f"data:image/png;base64,{b64}" [](#__codelineno-2-41) [](#__codelineno-2-42)[](#__codelineno-2-43)def main(): [](#__codelineno-2-44) # -- Step 1: Create a test image (solid red) ------------------------- [](#__codelineno-2-45) image = Image.new("RGB", (224, 224), color=(255, 0, 0)) [](#__codelineno-2-46) data_url = make_data_url(image) [](#__codelineno-2-47) print("Created 224x224 red test image") [](#__codelineno-2-48) [](#__codelineno-2-49) # -- Step 2: Render (preprocess) ------------------------------------- [](#__codelineno-2-50) render_payload = { [](#__codelineno-2-51) "model": MODEL_NAME, [](#__codelineno-2-52) "messages": [ [](#__codelineno-2-53) { [](#__codelineno-2-54) "role": "user", [](#__codelineno-2-55) "content": [ [](#__codelineno-2-56) {"type": "image_url", "image_url": {"url": data_url}}, [](#__codelineno-2-57) { [](#__codelineno-2-58) "type": "text", [](#__codelineno-2-59) "text": "What color is this image? Answer in one word.", [](#__codelineno-2-60) }, [](#__codelineno-2-61) ], [](#__codelineno-2-62) } [](#__codelineno-2-63) ], [](#__codelineno-2-64) } [](#__codelineno-2-65) [](#__codelineno-2-66) print("\n--- Render ---") [](#__codelineno-2-67) render_resp = requests.post( [](#__codelineno-2-68) f"{BASE_URL}/v1/chat/completions/render", json=render_payload [](#__codelineno-2-69) ) [](#__codelineno-2-70) render_resp.raise_for_status() [](#__codelineno-2-71) render_data = render_resp.json() [](#__codelineno-2-72) [](#__codelineno-2-73) print(f"Response keys: {list(render_data.keys())}") [](#__codelineno-2-74) print(f"Number of token_ids: {len(render_data['token_ids'])}") [](#__codelineno-2-75) [](#__codelineno-2-76) features = render_data.get("features") [](#__codelineno-2-77) if features and features.get("kwargs_data"): [](#__codelineno-2-78) print(f"kwargs_data modalities: {list(features['kwargs_data'].keys())}") [](#__codelineno-2-79) for modality, items in features["kwargs_data"].items(): [](#__codelineno-2-80) print( [](#__codelineno-2-81) f" {modality}: {len(items)} item(s), " [](#__codelineno-2-82) f"first item type: {type(items[0])} length: {len(items[0])}" [](#__codelineno-2-83) if items [](#__codelineno-2-84) else "First item: (empty)" [](#__codelineno-2-85) ) [](#__codelineno-2-86) else: [](#__codelineno-2-87) print("WARNING: no kwargs_data in render response") [](#__codelineno-2-88) [](#__codelineno-2-89) # -- Step 3: Generate (inference) ------------------------------------ [](#__codelineno-2-90) # Pass the render output directly — only add sampling_params. [](#__codelineno-2-91) generate_payload = render_data [](#__codelineno-2-92) generate_payload["sampling_params"] = { [](#__codelineno-2-93) "max_tokens": 20, [](#__codelineno-2-94) "temperature": 0.0, [](#__codelineno-2-95) } [](#__codelineno-2-96) [](#__codelineno-2-97) print("\n--- Generate ---") [](#__codelineno-2-98) gen_resp = requests.post(f"{BASE_URL}/inference/v1/generate", json=generate_payload) [](#__codelineno-2-99) gen_resp.raise_for_status() [](#__codelineno-2-100) gen_data = gen_resp.json() [](#__codelineno-2-101) [](#__codelineno-2-102) # -- Step 4: Decode & print ------------------------------------------ [](#__codelineno-2-103) output_ids = gen_data["choices"][0]["token_ids"] [](#__codelineno-2-104) tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) [](#__codelineno-2-105) text = tokenizer.decode(output_ids, skip_special_tokens=True) [](#__codelineno-2-106) [](#__codelineno-2-107) print(f"Output token count: {len(output_ids)}") [](#__codelineno-2-108) print(f"Generated text: {text!r}") [](#__codelineno-2-109) [](#__codelineno-2-110) if "red" in text.lower(): [](#__codelineno-2-111) print("\nModel correctly identified the red image.") [](#__codelineno-2-112) else: [](#__codelineno-2-113) print(f"\nWARNING: Expected 'red' in output, got: {text!r}") [](#__codelineno-2-114) [](#__codelineno-2-115)[](#__codelineno-2-116)if __name__ == "__main__": [](#__codelineno-2-117) main()` kv\_events.sh `[](#__codelineno-3-1)#!/bin/bash [](#__codelineno-3-2)# This file demonstrates the KV cache event publishing [](#__codelineno-3-3)# We will launch a vllm instances configured to publish KV cache [](#__codelineno-3-4)# events and launch a simple subscriber to log those events. [](#__codelineno-3-5)[](#__codelineno-3-6)set -xe [](#__codelineno-3-7)[](#__codelineno-3-8)echo "🚧🚧 Warning: The usage of KV cache events is experimental and subject to change 🚧🚧" [](#__codelineno-3-9)sleep 1 [](#__codelineno-3-10)[](#__codelineno-3-11)MODEL_NAME=${HF_MODEL_NAME:-meta-llama/Meta-Llama-3.1-8B-Instruct} [](#__codelineno-3-12)[](#__codelineno-3-13)# Trap the SIGINT signal (triggered by Ctrl+C) [](#__codelineno-3-14)trap 'cleanup' INT [](#__codelineno-3-15)[](#__codelineno-3-16)# Cleanup function [](#__codelineno-3-17)cleanup() { [](#__codelineno-3-18) echo "Caught Ctrl+C, cleaning up..." [](#__codelineno-3-19) # Cleanup commands [](#__codelineno-3-20) pgrep python | xargs kill -9 [](#__codelineno-3-21) pkill -f python [](#__codelineno-3-22) echo "Cleanup complete. Exiting." [](#__codelineno-3-23) exit 0 [](#__codelineno-3-24)} [](#__codelineno-3-25)[](#__codelineno-3-26)export VLLM_HOST_IP=$(hostname -I | awk '{print $1}') [](#__codelineno-3-27)[](#__codelineno-3-28)# a function that waits vLLM server to start [](#__codelineno-3-29)wait_for_server() { [](#__codelineno-3-30) local port=$1 [](#__codelineno-3-31) timeout 1200 bash -c " [](#__codelineno-3-32) until curl -s localhost:${port}/v1/completions > /dev/null; do [](#__codelineno-3-33) sleep 1 [](#__codelineno-3-34) done" && return 0 || return 1 [](#__codelineno-3-35)} [](#__codelineno-3-36)[](#__codelineno-3-37)vllm serve "$MODEL_NAME" \ [](#__codelineno-3-38) --port 8100 \ [](#__codelineno-3-39) --max-model-len 100 \ [](#__codelineno-3-40) --enforce-eager \ [](#__codelineno-3-41) --gpu-memory-utilization 0.8 \ [](#__codelineno-3-42) --trust-remote-code \ [](#__codelineno-3-43) --kv-events-config \ [](#__codelineno-3-44) '{"enable_kv_cache_events": true, "publisher": "zmq", "topic": "kv-events"}' & [](#__codelineno-3-45)[](#__codelineno-3-46)wait_for_server 8100 [](#__codelineno-3-47)[](#__codelineno-3-48)SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" [](#__codelineno-3-49)[](#__codelineno-3-50)python3 "$SCRIPT_DIR/kv_events_subscriber.py" & [](#__codelineno-3-51)sleep 1 [](#__codelineno-3-52)[](#__codelineno-3-53)# serve two example requests [](#__codelineno-3-54)output1=$(curl -X POST -s http://localhost:8100/v1/completions \ [](#__codelineno-3-55)-H "Content-Type: application/json" \ [](#__codelineno-3-56)-d '{ [](#__codelineno-3-57)"model": "'"$MODEL_NAME"'", [](#__codelineno-3-58)"prompt": "Explain quantum computing in simple terms a 5-year-old could understand.", [](#__codelineno-3-59)"max_tokens": 80, [](#__codelineno-3-60)"temperature": 0 [](#__codelineno-3-61)}') [](#__codelineno-3-62)[](#__codelineno-3-63)output2=$(curl -X POST -s http://localhost:8100/v1/completions \ [](#__codelineno-3-64)-H "Content-Type: application/json" \ [](#__codelineno-3-65)-d '{ [](#__codelineno-3-66)"model": "'"$MODEL_NAME"'", [](#__codelineno-3-67)"prompt": "Explain quantum computing in simple terms a 50-year-old could understand.", [](#__codelineno-3-68)"max_tokens": 80, [](#__codelineno-3-69)"temperature": 0 [](#__codelineno-3-70)}') [](#__codelineno-3-71)[](#__codelineno-3-72)# Cleanup commands [](#__codelineno-3-73)pkill -9 -u "$USER" -f python [](#__codelineno-3-74)pkill -9 -u "$USER" -f vllm [](#__codelineno-3-75)[](#__codelineno-3-76)sleep 1 [](#__codelineno-3-77)[](#__codelineno-3-78)echo "Cleaned up" [](#__codelineno-3-79)[](#__codelineno-3-80)# Print the outputs of the curl requests [](#__codelineno-3-81)echo "" [](#__codelineno-3-82)echo "Output of first request: $output1" [](#__codelineno-3-83)echo "Output of second request: $output2" [](#__codelineno-3-84)[](#__codelineno-3-85)echo "🎉🎉 Successfully finished 2 test requests! 🎉🎉" [](#__codelineno-3-86)echo ""` moriio\_toy\_proxy\_server.py `[](#__codelineno-4-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-4-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-4-3)import argparse [](#__codelineno-4-4)import asyncio [](#__codelineno-4-5)import copy [](#__codelineno-4-6)import logging [](#__codelineno-4-7)import os [](#__codelineno-4-8)import socket [](#__codelineno-4-9)import threading [](#__codelineno-4-10)import uuid [](#__codelineno-4-11)from urllib.parse import urlparse [](#__codelineno-4-12)[](#__codelineno-4-13)import aiohttp [](#__codelineno-4-14)import msgpack [](#__codelineno-4-15)import zmq [](#__codelineno-4-16)from quart import Quart, Request, make_response, request [](#__codelineno-4-17)[](#__codelineno-4-18)from vllm.distributed.kv_transfer.kv_connector.v1.moriio.moriio_common import ( [](#__codelineno-4-19) MoRIIOConstants, [](#__codelineno-4-20)) [](#__codelineno-4-21)[](#__codelineno-4-22)logger = logging.getLogger(__name__) [](#__codelineno-4-23)logger.setLevel(logging.DEBUG) [](#__codelineno-4-24)prefill_instances: list[dict] = [] [](#__codelineno-4-25)decode_instances: list[dict] = [] [](#__codelineno-4-26)request_nums = 0 [](#__codelineno-4-27)app = Quart(__name__) [](#__codelineno-4-28) [](#__codelineno-4-29)[](#__codelineno-4-30)TRANSFER_TYPE = None [](#__codelineno-4-31) [](#__codelineno-4-32)[](#__codelineno-4-33)_list_lock = threading.RLock() [](#__codelineno-4-34) [](#__codelineno-4-35)[](#__codelineno-4-36)def _listen_for_register(hostname, port): [](#__codelineno-4-37) context = zmq.Context() [](#__codelineno-4-38) router_socket = context.socket(zmq.ROUTER) [](#__codelineno-4-39) router_socket.bind(f"tcp://{hostname}:{port}") [](#__codelineno-4-40) poller = zmq.Poller() [](#__codelineno-4-41) poller.register(router_socket, zmq.POLLIN) [](#__codelineno-4-42) global prefill_instances [](#__codelineno-4-43) global decode_instances [](#__codelineno-4-44) [](#__codelineno-4-45) while True: [](#__codelineno-4-46) socks = dict(poller.poll()) [](#__codelineno-4-47) if router_socket in socks: [](#__codelineno-4-48) remote_addr, msg = router_socket.recv_multipart() [](#__codelineno-4-49) data = msgpack.loads(msg) [](#__codelineno-4-50) if data.get("type") == "HELLO": [](#__codelineno-4-51) pass [](#__codelineno-4-52) elif data.get("type") in ("P", "D"): [](#__codelineno-4-53) role = data["type"] [](#__codelineno-4-54) required_keys = { [](#__codelineno-4-55) "http_address", [](#__codelineno-4-56) "zmq_address", [](#__codelineno-4-57) "dp_size", [](#__codelineno-4-58) "tp_size", [](#__codelineno-4-59) "transfer_mode", [](#__codelineno-4-60) } [](#__codelineno-4-61) missing = required_keys - data.keys() [](#__codelineno-4-62) if missing: [](#__codelineno-4-63) logger.error( [](#__codelineno-4-64) "Registration message missing required keys %s; skipping", [](#__codelineno-4-65) missing, [](#__codelineno-4-66) ) [](#__codelineno-4-67) continue [](#__codelineno-4-68) # Derive request_address from http_address [](#__codelineno-4-69) # api path suffix is appended at request time [](#__codelineno-4-70) instance = { [](#__codelineno-4-71) "role": role, [](#__codelineno-4-72) "request_address": f"http://{data['http_address']}/v1", [](#__codelineno-4-73) "http_address": data["http_address"], [](#__codelineno-4-74) "zmq_address": data["zmq_address"], [](#__codelineno-4-75) "dp_size": data["dp_size"], [](#__codelineno-4-76) "tp_size": data["tp_size"], [](#__codelineno-4-77) "transfer_mode": data["transfer_mode"], [](#__codelineno-4-78) } [](#__codelineno-4-79) # zmq_address format: "host:IP,handshake:PORT,notify:PORT" [](#__codelineno-4-80) # Stored verbatim; embedded into the request_id by handle_request. [](#__codelineno-4-81) [](#__codelineno-4-82) global TRANSFER_TYPE [](#__codelineno-4-83) transfer_mode = instance["transfer_mode"] [](#__codelineno-4-84) target_list = prefill_instances if role == "P" else decode_instances [](#__codelineno-4-85) with _list_lock: [](#__codelineno-4-86) if TRANSFER_TYPE is None: [](#__codelineno-4-87) TRANSFER_TYPE = transfer_mode [](#__codelineno-4-88) logger.info("SET TRANSFER TYPE TO %s", TRANSFER_TYPE) [](#__codelineno-4-89) elif transfer_mode != TRANSFER_TYPE: [](#__codelineno-4-90) logger.error( [](#__codelineno-4-91) "Mismatched transfer mode: expected %s, got %s;" [](#__codelineno-4-92) " skipping registration of %s", [](#__codelineno-4-93) TRANSFER_TYPE, [](#__codelineno-4-94) transfer_mode, [](#__codelineno-4-95) data["http_address"], [](#__codelineno-4-96) ) [](#__codelineno-4-97) continue [](#__codelineno-4-98) existing_idx = next( [](#__codelineno-4-99) ( [](#__codelineno-4-100) idx [](#__codelineno-4-101) for idx, i in enumerate(target_list) [](#__codelineno-4-102) if i.get("http_address") == data["http_address"] [](#__codelineno-4-103) ), [](#__codelineno-4-104) None, [](#__codelineno-4-105) ) [](#__codelineno-4-106) if existing_idx is not None: [](#__codelineno-4-107) target_list[existing_idx] = instance [](#__codelineno-4-108) logger.info( [](#__codelineno-4-109) "Updated existing %s instance: %s", [](#__codelineno-4-110) "Prefill" if role == "P" else "Decode", [](#__codelineno-4-111) instance, [](#__codelineno-4-112) ) [](#__codelineno-4-113) else: [](#__codelineno-4-114) target_list.append(instance) [](#__codelineno-4-115) logger.info( [](#__codelineno-4-116) "Registered %s instance: %s", [](#__codelineno-4-117) "Prefill" if role == "P" else "Decode", [](#__codelineno-4-118) instance, [](#__codelineno-4-119) ) [](#__codelineno-4-120) else: [](#__codelineno-4-121) logger.warning( [](#__codelineno-4-122) "Received message with unrecognized type %r; ignoring", [](#__codelineno-4-123) data.get("type"), [](#__codelineno-4-124) ) [](#__codelineno-4-125) [](#__codelineno-4-126)[](#__codelineno-4-127)def start_service_discovery(hostname, port): [](#__codelineno-4-128) if not hostname: [](#__codelineno-4-129) hostname = socket.gethostname() [](#__codelineno-4-130) if port == 0: [](#__codelineno-4-131) raise ValueError("Port cannot be 0") [](#__codelineno-4-132) [](#__codelineno-4-133) _listener_thread = threading.Thread( [](#__codelineno-4-134) target=_listen_for_register, args=(hostname, port), daemon=True [](#__codelineno-4-135) ) [](#__codelineno-4-136) _listener_thread.start() [](#__codelineno-4-137) return _listener_thread [](#__codelineno-4-138) [](#__codelineno-4-139)[](#__codelineno-4-140)async def send_request_to_prefill( [](#__codelineno-4-141) endpoint, req_data, request_id, selected_prefill_dp_rank [](#__codelineno-4-142)): [](#__codelineno-4-143) req_data_copy = req_data [](#__codelineno-4-144) [](#__codelineno-4-145) req_data_copy["kv_transfer_params"].update( [](#__codelineno-4-146) { [](#__codelineno-4-147) "do_remote_decode": True, [](#__codelineno-4-148) "do_remote_prefill": False, [](#__codelineno-4-149) "remote_engine_id": None, [](#__codelineno-4-150) "remote_block_ids": None, [](#__codelineno-4-151) } [](#__codelineno-4-152) ) [](#__codelineno-4-153) req_data_copy["stream"] = False [](#__codelineno-4-154) req_data_copy["max_tokens"] = 1 [](#__codelineno-4-155) if "max_completion_tokens" in req_data_copy: [](#__codelineno-4-156) req_data_copy["max_completion_tokens"] = 1 [](#__codelineno-4-157) if "stream_options" in req_data_copy: [](#__codelineno-4-158) del req_data_copy["stream_options"] [](#__codelineno-4-159) async with aiohttp.ClientSession( [](#__codelineno-4-160) timeout=aiohttp.ClientTimeout(total=6 * 6000 * 6000) [](#__codelineno-4-161) ) as session: [](#__codelineno-4-162) headers = { [](#__codelineno-4-163) "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", [](#__codelineno-4-164) "X-Request-Id": request_id, [](#__codelineno-4-165) } [](#__codelineno-4-166) if selected_prefill_dp_rank is not None: [](#__codelineno-4-167) headers["X-data-parallel-rank"] = str(selected_prefill_dp_rank) [](#__codelineno-4-168) async with session.post( [](#__codelineno-4-169) url=endpoint, json=req_data_copy, headers=headers [](#__codelineno-4-170) ) as response: [](#__codelineno-4-171) if response.status == 200: [](#__codelineno-4-172) return await response.json() [](#__codelineno-4-173) [](#__codelineno-4-174) else: [](#__codelineno-4-175) error_message = ( [](#__codelineno-4-176) f"send_request_to_prefill response ={response}," [](#__codelineno-4-177) f"reason={response.reason}, status={response.status}," [](#__codelineno-4-178) f"method={response.method}, url={response.url}," [](#__codelineno-4-179) f"real_url={response.real_url}" [](#__codelineno-4-180) ) [](#__codelineno-4-181) raise RuntimeError(error_message) [](#__codelineno-4-182) [](#__codelineno-4-183)[](#__codelineno-4-184)async def start_decode_request(endpoint, req_data, request_id): [](#__codelineno-4-185) session = aiohttp.ClientSession( [](#__codelineno-4-186) timeout=aiohttp.ClientTimeout(total=6 * 6000 * 6000) [](#__codelineno-4-187) ) [](#__codelineno-4-188) headers = { [](#__codelineno-4-189) "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", [](#__codelineno-4-190) "X-Request-Id": request_id, [](#__codelineno-4-191) } [](#__codelineno-4-192) response = await session.post(url=endpoint, json=req_data, headers=headers) [](#__codelineno-4-193) return session, response [](#__codelineno-4-194) [](#__codelineno-4-195)[](#__codelineno-4-196)async def stream_decode_response(session, response, request_id): [](#__codelineno-4-197) try: [](#__codelineno-4-198) if response.status == 200: [](#__codelineno-4-199) async for chunk_bytes in response.content.iter_chunked(1024): [](#__codelineno-4-200) yield chunk_bytes [](#__codelineno-4-201) else: [](#__codelineno-4-202) error_message = ( [](#__codelineno-4-203) f"stream_decode_response response ={response}," [](#__codelineno-4-204) f"reason={response.reason}, status={response.status}," [](#__codelineno-4-205) f"method={response.method}, url={response.url}," [](#__codelineno-4-206) f"real_url={response.real_url}" [](#__codelineno-4-207) ) [](#__codelineno-4-208) raise RuntimeError(error_message) [](#__codelineno-4-209) finally: [](#__codelineno-4-210) await session.close() [](#__codelineno-4-211) [](#__codelineno-4-212)[](#__codelineno-4-213)def example_round_robin_dp_loader(request_number, dp_size): [](#__codelineno-4-214) return request_nums % dp_size [](#__codelineno-4-215) [](#__codelineno-4-216)[](#__codelineno-4-217)@app.route("/v1/completions", methods=["POST"]) [](#__codelineno-4-218)async def handle_completions_request(): [](#__codelineno-4-219) return await handle_request("/completions", request) [](#__codelineno-4-220) [](#__codelineno-4-221)[](#__codelineno-4-222)@app.route("/v1/chat/completions", methods=["POST"]) [](#__codelineno-4-223)async def handle_chat_completions_request(): [](#__codelineno-4-224) return await handle_request("/chat/completions", request) [](#__codelineno-4-225) [](#__codelineno-4-226)[](#__codelineno-4-227)async def handle_request(api: str, request: Request): [](#__codelineno-4-228) try: [](#__codelineno-4-229) with _list_lock: [](#__codelineno-4-230) global request_nums [](#__codelineno-4-231) request_nums += 1 [](#__codelineno-4-232) [](#__codelineno-4-233) req_data = await request.get_json() [](#__codelineno-4-234) [](#__codelineno-4-235) prefill_instance_endpoint = None [](#__codelineno-4-236) decode_instance_endpoint = None [](#__codelineno-4-237) error_msg = ( [](#__codelineno-4-238) "Service Unavailable: No prefill or decode instances are registered." [](#__codelineno-4-239) ) [](#__codelineno-4-240) if not prefill_instances or not decode_instances: [](#__codelineno-4-241) return await make_response( [](#__codelineno-4-242) ( [](#__codelineno-4-243) error_msg, [](#__codelineno-4-244) 503, [](#__codelineno-4-245) ) [](#__codelineno-4-246) ) [](#__codelineno-4-247) pid = request_nums % len(prefill_instances) [](#__codelineno-4-248) did = request_nums % len(decode_instances) [](#__codelineno-4-249) prefill_instance_endpoint = prefill_instances[pid] [](#__codelineno-4-250) decode_instance_endpoint = decode_instances[did] [](#__codelineno-4-251) [](#__codelineno-4-252) selected_prefill_dp_rank = None [](#__codelineno-4-253) if prefill_instance_endpoint["dp_size"] > 1: [](#__codelineno-4-254) selected_prefill_dp_rank = example_round_robin_dp_loader( [](#__codelineno-4-255) request_nums // len(prefill_instance_endpoint), [](#__codelineno-4-256) prefill_instance_endpoint["dp_size"], [](#__codelineno-4-257) ) [](#__codelineno-4-258) [](#__codelineno-4-259) # Embed both zmq_addresses in the request_id so the connector can parse [](#__codelineno-4-260) # the peer's host/ports from it, similar to P2P-NCCL [](#__codelineno-4-261) uid = str(uuid.uuid4()).replace("-", "") [](#__codelineno-4-262) request_id = ( [](#__codelineno-4-263) f"___prefill_addr_{prefill_instance_endpoint['zmq_address']}" [](#__codelineno-4-264) f"___decode_addr_{decode_instance_endpoint['zmq_address']}" [](#__codelineno-4-265) f"_{uid}" [](#__codelineno-4-266) ) [](#__codelineno-4-267) [](#__codelineno-4-268) transfer_id = f"{MoRIIOConstants.TRANSFER_PREFIX}-{str(uuid.uuid4())}" [](#__codelineno-4-269) [](#__codelineno-4-270) req_data_to_prefill = copy.deepcopy(req_data) [](#__codelineno-4-271) req_data_to_prefill["kv_transfer_params"] = {} [](#__codelineno-4-272) req_data["kv_transfer_params"] = {} [](#__codelineno-4-273) req_data_to_prefill["kv_transfer_params"]["remote_dp_size"] = ( [](#__codelineno-4-274) decode_instance_endpoint["dp_size"] [](#__codelineno-4-275) ) [](#__codelineno-4-276) req_data_to_prefill["kv_transfer_params"]["remote_tp_size"] = ( [](#__codelineno-4-277) decode_instance_endpoint["tp_size"] [](#__codelineno-4-278) ) [](#__codelineno-4-279) req_data_to_prefill["kv_transfer_params"]["transfer_id"] = transfer_id [](#__codelineno-4-280) [](#__codelineno-4-281) prefill_request_url = prefill_instance_endpoint["request_address"] + api [](#__codelineno-4-282) send_prefill_task = asyncio.create_task( [](#__codelineno-4-283) send_request_to_prefill( [](#__codelineno-4-284) prefill_request_url, [](#__codelineno-4-285) req_data_to_prefill, [](#__codelineno-4-286) request_id, [](#__codelineno-4-287) selected_prefill_dp_rank, [](#__codelineno-4-288) ) [](#__codelineno-4-289) ) [](#__codelineno-4-290) [](#__codelineno-4-291) req_data["max_tokens"] -= 1 [](#__codelineno-4-292) [](#__codelineno-4-293) req_data["kv_transfer_params"] = { [](#__codelineno-4-294) "do_remote_decode": False, [](#__codelineno-4-295) "do_remote_prefill": True, [](#__codelineno-4-296) "remote_engine_id": None, [](#__codelineno-4-297) "remote_block_ids": None, [](#__codelineno-4-298) "transfer_id": transfer_id, [](#__codelineno-4-299) } [](#__codelineno-4-300) if TRANSFER_TYPE == "READ": [](#__codelineno-4-301) # In read mode, prefill and decode are executed serially. [](#__codelineno-4-302) prefill_response = await send_prefill_task [](#__codelineno-4-303) prefill_kv = prefill_response["kv_transfer_params"] [](#__codelineno-4-304) req_data["kv_transfer_params"]["remote_engine_id"] = prefill_kv[ [](#__codelineno-4-305) "remote_engine_id" [](#__codelineno-4-306) ] [](#__codelineno-4-307) req_data["kv_transfer_params"]["remote_block_ids"] = prefill_kv[ [](#__codelineno-4-308) "remote_block_ids" [](#__codelineno-4-309) ] [](#__codelineno-4-310) req_data["kv_transfer_params"]["transfer_id"] = prefill_kv["transfer_id"] [](#__codelineno-4-311) [](#__codelineno-4-312) req_data["kv_transfer_params"]["remote_dp_size"] = prefill_instance_endpoint[ [](#__codelineno-4-313) "dp_size" [](#__codelineno-4-314) ] [](#__codelineno-4-315) req_data["kv_transfer_params"]["remote_tp_size"] = prefill_instance_endpoint[ [](#__codelineno-4-316) "tp_size" [](#__codelineno-4-317) ] [](#__codelineno-4-318) [](#__codelineno-4-319) if selected_prefill_dp_rank is not None: [](#__codelineno-4-320) req_data["kv_transfer_params"]["remote_dp_rank"] = selected_prefill_dp_rank [](#__codelineno-4-321) [](#__codelineno-4-322) decode_request_url = decode_instance_endpoint["request_address"] + api [](#__codelineno-4-323) decode_request_task = asyncio.create_task( [](#__codelineno-4-324) start_decode_request(decode_request_url, req_data, request_id) [](#__codelineno-4-325) ) [](#__codelineno-4-326) [](#__codelineno-4-327) session, decode_response = await decode_request_task [](#__codelineno-4-328) stream_generator = stream_decode_response(session, decode_response, request_id) [](#__codelineno-4-329) response = await make_response(stream_generator) [](#__codelineno-4-330) return response [](#__codelineno-4-331) except Exception as e: [](#__codelineno-4-332) logger.exception("An error occurred while handling the request: %s", e) [](#__codelineno-4-333) return await make_response( [](#__codelineno-4-334) ( [](#__codelineno-4-335) f"Internal Server Error: {e!s}", [](#__codelineno-4-336) 500, [](#__codelineno-4-337) ) [](#__codelineno-4-338) ) [](#__codelineno-4-339) [](#__codelineno-4-340)[](#__codelineno-4-341)async def send_profile_cmd(req_data: dict, profiler_cmd: str): [](#__codelineno-4-342) assert profiler_cmd in {"start", "stop"} [](#__codelineno-4-343) [](#__codelineno-4-344) with _list_lock: [](#__codelineno-4-345) p_instances = list(prefill_instances) [](#__codelineno-4-346) d_instances = list(decode_instances) [](#__codelineno-4-347) [](#__codelineno-4-348) if not p_instances and not d_instances: [](#__codelineno-4-349) raise RuntimeError( [](#__codelineno-4-350) "Service Unavailable: No prefill or decode instances are registered." [](#__codelineno-4-351) ) [](#__codelineno-4-352) [](#__codelineno-4-353) headers = { [](#__codelineno-4-354) "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", [](#__codelineno-4-355) } [](#__codelineno-4-356) [](#__codelineno-4-357) tasks = [] [](#__codelineno-4-358) [](#__codelineno-4-359) async with aiohttp.ClientSession( [](#__codelineno-4-360) timeout=aiohttp.ClientTimeout(total=60) [](#__codelineno-4-361) ) as session: [](#__codelineno-4-362) for instances in (p_instances, d_instances): [](#__codelineno-4-363) for inst in instances: [](#__codelineno-4-364) _p = urlparse(inst["request_address"]) [](#__codelineno-4-365) url = f"http://{_p.hostname}:{_p.port}/{profiler_cmd}_profile" [](#__codelineno-4-366) [](#__codelineno-4-367) tasks.append( [](#__codelineno-4-368) session.post( [](#__codelineno-4-369) url, [](#__codelineno-4-370) json=req_data, [](#__codelineno-4-371) headers=headers, [](#__codelineno-4-372) ) [](#__codelineno-4-373) ) [](#__codelineno-4-374) [](#__codelineno-4-375) responses = await asyncio.gather(*tasks, return_exceptions=True) [](#__codelineno-4-376) [](#__codelineno-4-377) for r in responses: [](#__codelineno-4-378) if isinstance(r, Exception): [](#__codelineno-4-379) raise r [](#__codelineno-4-380) if r.status >= 400: [](#__codelineno-4-381) msg = await r.text() [](#__codelineno-4-382) raise RuntimeError(f"{profiler_cmd}_profile failed: {r.status}, {msg}") [](#__codelineno-4-383) [](#__codelineno-4-384) return await responses[0].json() [](#__codelineno-4-385) [](#__codelineno-4-386)[](#__codelineno-4-387)@app.post("/start_profile") [](#__codelineno-4-388)async def start_profile(): [](#__codelineno-4-389) try: [](#__codelineno-4-390) req_data = await request.get_json() [](#__codelineno-4-391) return await send_profile_cmd(req_data, "start") [](#__codelineno-4-392) except Exception as e: [](#__codelineno-4-393) logger.exception("start_profile failed: %s", e) [](#__codelineno-4-394) return await make_response((str(e), 500)) [](#__codelineno-4-395) [](#__codelineno-4-396)[](#__codelineno-4-397)@app.post("/stop_profile") [](#__codelineno-4-398)async def stop_profile(): [](#__codelineno-4-399) try: [](#__codelineno-4-400) req_data = await request.get_json() [](#__codelineno-4-401) return await send_profile_cmd(req_data, "stop") [](#__codelineno-4-402) except Exception as e: [](#__codelineno-4-403) logger.exception("stop_profile failed: %s", e) [](#__codelineno-4-404) return await make_response((str(e), 500)) [](#__codelineno-4-405) [](#__codelineno-4-406)[](#__codelineno-4-407)if __name__ == "__main__": [](#__codelineno-4-408) parser = argparse.ArgumentParser() [](#__codelineno-4-409) parser.add_argument("--port", type=int, default=10001) [](#__codelineno-4-410) args = parser.parse_args() [](#__codelineno-4-411) [](#__codelineno-4-412) t = start_service_discovery("0.0.0.0", 36367) [](#__codelineno-4-413) app.debug = True [](#__codelineno-4-414) app.config["BODY_TIMEOUT"] = 360000 [](#__codelineno-4-415) app.config["RESPONSE_TIMEOUT"] = 360000 [](#__codelineno-4-416) [](#__codelineno-4-417) app.run(host="0.0.0.0", port=args.port) [](#__codelineno-4-418) t.join()` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) 5. [Disaggregated](https://docs.vllm.ai/en/latest/examples/disaggregated_encoder/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/ec_both_encoder.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/ec\_both\_encoder](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/ec_both_encoder). ## Ec Both Encoder[¶](#ec-both-encoder_1 "Permanent link") `[](#__codelineno-0-1)#!/bin/bash [](#__codelineno-0-2)set -euo pipefail [](#__codelineno-0-3)[](#__codelineno-0-4)MODEL="${MODEL:-Qwen/Qwen2.5-VL-3B-Instruct}" [](#__codelineno-0-5)PORT="${PORT:-8000}" [](#__codelineno-0-6)GPU="${GPU:-0}" [](#__codelineno-0-7)NUM_PROMPTS="${NUM_PROMPTS:-200}" [](#__codelineno-0-8)EC_SHARED_STORAGE_PATH="${EC_SHARED_STORAGE_PATH:-/tmp/ec_cache}" [](#__codelineno-0-9)TIMEOUT="${TIMEOUT:-600}" [](#__codelineno-0-10)[](#__codelineno-0-11)SERVER_PID="" [](#__codelineno-0-12)[](#__codelineno-0-13)cleanup() { [](#__codelineno-0-14) echo "Stopping server..." [](#__codelineno-0-15) if [[ -n "$SERVER_PID" ]] && kill -0 "$SERVER_PID" 2>/dev/null; then [](#__codelineno-0-16) kill "$SERVER_PID" 2>/dev/null || true [](#__codelineno-0-17) wait "$SERVER_PID" 2>/dev/null || true [](#__codelineno-0-18) fi [](#__codelineno-0-19) echo "Done." [](#__codelineno-0-20)} [](#__codelineno-0-21)trap cleanup EXIT INT TERM [](#__codelineno-0-22)[](#__codelineno-0-23)wait_for_server() { [](#__codelineno-0-24) local deadline=$((SECONDS + TIMEOUT)) [](#__codelineno-0-25) echo "Waiting for server on port $PORT..." [](#__codelineno-0-26) while (( SECONDS < deadline )); do [](#__codelineno-0-27) if curl -sf "http://localhost:${PORT}/v1/models" > /dev/null 2>&1; then [](#__codelineno-0-28) echo "Server ready." [](#__codelineno-0-29) return 0 [](#__codelineno-0-30) fi [](#__codelineno-0-31) sleep 2 [](#__codelineno-0-32) done [](#__codelineno-0-33) echo "ERROR: Server did not start within ${TIMEOUT}s" [](#__codelineno-0-34) return 1 [](#__codelineno-0-35)} [](#__codelineno-0-36)[](#__codelineno-0-37)rm -rf "$EC_SHARED_STORAGE_PATH" [](#__codelineno-0-38)mkdir -p "$EC_SHARED_STORAGE_PATH" [](#__codelineno-0-39)[](#__codelineno-0-40)############################################################################### [](#__codelineno-0-41)# Start server with ec_both [](#__codelineno-0-42)############################################################################### [](#__codelineno-0-43)CUDA_VISIBLE_DEVICES="$GPU" \ [](#__codelineno-0-44)vllm serve "$MODEL" \ [](#__codelineno-0-45) --port "$PORT" \ [](#__codelineno-0-46) --enforce-eager \ [](#__codelineno-0-47) --ec-transfer-config '{ [](#__codelineno-0-48) "ec_connector": "ECExampleConnector", [](#__codelineno-0-49) "ec_role": "ec_both", [](#__codelineno-0-50) "ec_connector_extra_config": { [](#__codelineno-0-51) "shared_storage_path": "'"$EC_SHARED_STORAGE_PATH"'" [](#__codelineno-0-52) } [](#__codelineno-0-53) }' \ [](#__codelineno-0-54) "$@" & [](#__codelineno-0-55)[](#__codelineno-0-56)SERVER_PID=$! [](#__codelineno-0-57)wait_for_server [](#__codelineno-0-58)[](#__codelineno-0-59)############################################################################### [](#__codelineno-0-60)# Benchmark -- dataset contains duplicate images, exercises cache hits [](#__codelineno-0-61)############################################################################### [](#__codelineno-0-62)echo "Running benchmark ($NUM_PROMPTS prompts)..." [](#__codelineno-0-63)vllm bench serve \ [](#__codelineno-0-64) --model "$MODEL" \ [](#__codelineno-0-65) --backend openai-chat \ [](#__codelineno-0-66) --endpoint /v1/chat/completions \ [](#__codelineno-0-67) --dataset-name hf \ [](#__codelineno-0-68) --dataset-path lmarena-ai/VisionArena-Chat \ [](#__codelineno-0-69) --seed 0 \ [](#__codelineno-0-70) --num-prompts "$NUM_PROMPTS" \ [](#__codelineno-0-71) --port "$PORT" [](#__codelineno-0-72)[](#__codelineno-0-73)echo "Benchmark complete."` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/batch_invariance.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/batch\_invariance](https://github.com/vllm-project/vllm/tree/main/examples/features/batch_invariance). ## Reproducibility Offline[¶](#reproducibility-offline "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Demonstrates how to achieve reproducibility in vLLM. [](#__codelineno-0-5)[](#__codelineno-0-6)Main article: https://docs.vllm.ai/en/latest/usage/reproducibility.html [](#__codelineno-0-7)""" [](#__codelineno-0-8)[](#__codelineno-0-9)import os [](#__codelineno-0-10)import random [](#__codelineno-0-11)[](#__codelineno-0-12)from vllm import LLM, SamplingParams [](#__codelineno-0-13)[](#__codelineno-0-14)# Either: [](#__codelineno-0-15)## Turn off multiprocessing to make the scheduling deterministic, or [](#__codelineno-0-16)os.environ["VLLM_ENABLE_V1_MULTIPROCESSING"] = "0" [](#__codelineno-0-17)## Enable batch invariance to get consistent results regardless of scheduling. [](#__codelineno-0-18)os.environ["VLLM_BATCH_INVARIANT"] = "1" [](#__codelineno-0-19)[](#__codelineno-0-20)prompts = [ [](#__codelineno-0-21) "Hello, my name is", [](#__codelineno-0-22) "The president of the United States is", [](#__codelineno-0-23) "The capital of France is", [](#__codelineno-0-24) "The future of AI is", [](#__codelineno-0-25)] [](#__codelineno-0-26)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-0-27) [](#__codelineno-0-28)[](#__codelineno-0-29)def main(): [](#__codelineno-0-30) llm = LLM(model="facebook/opt-125m") [](#__codelineno-0-31) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-32) print("-" * 50) [](#__codelineno-0-33) for output in outputs: [](#__codelineno-0-34) prompt = output.prompt [](#__codelineno-0-35) generated_text = output.outputs[0].text [](#__codelineno-0-36) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-0-37) print("-" * 50) [](#__codelineno-0-38) [](#__codelineno-0-39) # Try generating random numbers outside vLLM [](#__codelineno-0-40) # The same number is output across runs, meaning that the random state [](#__codelineno-0-41) # in the user code has been updated by vLLM [](#__codelineno-0-42) print(random.randint(0, 100)) [](#__codelineno-0-43) [](#__codelineno-0-44)[](#__codelineno-0-45)if __name__ == "__main__": [](#__codelineno-0-46) main()` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/kv_load_failure_recovery_offline.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/kv\_load\_failure\_recovery\_offline](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/kv_load_failure_recovery_offline). This example builds upon the `example_connector` example in `examples/disaggregated`. It demonstrates vLLM's ability to recover from KV load failures in both synchronous and asynchronous loading modes. The goal is to verify that vLLM correctly identifies invalid KV blocks, reschedules the affected requests, and ensures successful and consistent output. ## Files[¶](#files "Permanent link") - `prefill_example.py` – performs the prefill stage and saves KV data (same as in `example_connector`). - `decode_example.py` – performs the decode stage. Accepts: - `--simulate-failure`: simulates KV load failure using a custom connector. - `--async-load`: enables asynchronous KV loading mode. - `load_recovery_example_connector.py` – defines `LoadRecoveryExampleConnector`, a subclass of `ExampleConnector`, that simulates missing or corrupted external KV blocks by failing to load blocks for the first decode request. - `run.sh` – orchestrates the test: runs the prefill stage, then three decode stages: 1. Normal decode (baseline). 2. Decode with simulated sync KV load failure. 3. Decode with simulated async KV load failure. Finally, it compares the output of the baseline with the recovered outputs to verify correctness. ## How It Works[¶](#how-it-works "Permanent link") - The test dynamically loads `LoadRecoveryExampleConnector` via `KVTransferConfig.kv_connector_module_path`, enabling controlled simulation of load failures without modifying the original connector. - The decode stages that simulate failure are expected to trigger recovery logic in vLLM, resulting in the same output as the baseline decode. - If recovery fails, the script prints a unified diff of the output mismatch and exits with error. ## Usage[¶](#usage "Permanent link") `[](#__codelineno-0-1)./run.sh` ## Example materials[¶](#example-materials "Permanent link") decode\_example.py `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)import argparse [](#__codelineno-1-4)[](#__codelineno-1-5)from vllm import LLM, SamplingParams [](#__codelineno-1-6)from vllm.config import KVTransferConfig [](#__codelineno-1-7) [](#__codelineno-1-8)[](#__codelineno-1-9)def read_prompts(): [](#__codelineno-1-10) """Read prompts from prefill_output.txt""" [](#__codelineno-1-11) prompts = [] [](#__codelineno-1-12) try: [](#__codelineno-1-13) with open("prefill_output.txt") as f: [](#__codelineno-1-14) for line in f: [](#__codelineno-1-15) prompts.append(line.strip()) [](#__codelineno-1-16) print(f"Loaded {len(prompts)} prompts from prefill_output.txt") [](#__codelineno-1-17) return prompts [](#__codelineno-1-18) except FileNotFoundError: [](#__codelineno-1-19) print("Error: prefill_output.txt file not found") [](#__codelineno-1-20) exit(-1) [](#__codelineno-1-21) [](#__codelineno-1-22)[](#__codelineno-1-23)def main(): [](#__codelineno-1-24) prompts = read_prompts() [](#__codelineno-1-25) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) [](#__codelineno-1-26) [](#__codelineno-1-27) parser = argparse.ArgumentParser() [](#__codelineno-1-28) parser.add_argument( [](#__codelineno-1-29) "--simulate-failure", action="store_true", help="Simulate KV load failure." [](#__codelineno-1-30) ) [](#__codelineno-1-31) parser.add_argument( [](#__codelineno-1-32) "--async-load", action="store_true", help="Simulate async KV load" [](#__codelineno-1-33) ) [](#__codelineno-1-34) args = parser.parse_args() [](#__codelineno-1-35) [](#__codelineno-1-36) if args.simulate_failure: [](#__codelineno-1-37) ktc = KVTransferConfig( [](#__codelineno-1-38) kv_connector="LoadRecoveryExampleConnector", [](#__codelineno-1-39) kv_role="kv_both", [](#__codelineno-1-40) kv_connector_extra_config={ [](#__codelineno-1-41) "shared_storage_path": "local_storage", [](#__codelineno-1-42) "async_load": args.async_load, [](#__codelineno-1-43) }, [](#__codelineno-1-44) kv_connector_module_path="load_recovery_example_connector", [](#__codelineno-1-45) kv_load_failure_policy="recompute", [](#__codelineno-1-46) ) [](#__codelineno-1-47) out_file = ( [](#__codelineno-1-48) "async_decode_recovered_output.txt" [](#__codelineno-1-49) if args.async_load [](#__codelineno-1-50) else "sync_decode_recovered_output.txt" [](#__codelineno-1-51) ) [](#__codelineno-1-52) else: [](#__codelineno-1-53) ktc = KVTransferConfig( [](#__codelineno-1-54) kv_connector="ExampleConnector", [](#__codelineno-1-55) kv_role="kv_both", [](#__codelineno-1-56) kv_connector_extra_config={ [](#__codelineno-1-57) "shared_storage_path": "local_storage", [](#__codelineno-1-58) }, [](#__codelineno-1-59) ) [](#__codelineno-1-60) out_file = "decode_output.txt" [](#__codelineno-1-61) [](#__codelineno-1-62) llm = LLM( [](#__codelineno-1-63) model="meta-llama/Llama-3.2-1B-Instruct", [](#__codelineno-1-64) enforce_eager=True, [](#__codelineno-1-65) gpu_memory_utilization=0.8, [](#__codelineno-1-66) max_num_batched_tokens=64, [](#__codelineno-1-67) max_num_seqs=16, [](#__codelineno-1-68) kv_transfer_config=ktc, [](#__codelineno-1-69) ) [](#__codelineno-1-70) [](#__codelineno-1-71) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-1-72) [](#__codelineno-1-73) sep_str = "-" * 30 [](#__codelineno-1-74) with open(out_file, "w", encoding="utf-8") as f: [](#__codelineno-1-75) for output in outputs: [](#__codelineno-1-76) prompt = output.prompt [](#__codelineno-1-77) generated_text = output.outputs[0].text [](#__codelineno-1-78) out_str = f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}" [](#__codelineno-1-79) print(out_str) [](#__codelineno-1-80) print(sep_str) [](#__codelineno-1-81) f.write(out_str) [](#__codelineno-1-82) f.write(sep_str) [](#__codelineno-1-83) [](#__codelineno-1-84)[](#__codelineno-1-85)if __name__ == "__main__": [](#__codelineno-1-86) main()` load\_recovery\_example\_connector.py `[](#__codelineno-2-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-2-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-2-3)# ruff: noqa: E501 [](#__codelineno-2-4)import logging [](#__codelineno-2-5)from dataclasses import dataclass, field [](#__codelineno-2-6)from typing import TYPE_CHECKING [](#__codelineno-2-7)[](#__codelineno-2-8)from vllm.config import VllmConfig [](#__codelineno-2-9)from vllm.distributed.kv_transfer.kv_connector.v1.base import ( [](#__codelineno-2-10) KVConnectorMetadata, [](#__codelineno-2-11) KVConnectorRole, [](#__codelineno-2-12)) [](#__codelineno-2-13)from vllm.distributed.kv_transfer.kv_connector.v1.example_connector import ( [](#__codelineno-2-14) ExampleConnector, [](#__codelineno-2-15) ExampleConnectorMetadata, [](#__codelineno-2-16)) [](#__codelineno-2-17)from vllm.forward_context import ForwardContext [](#__codelineno-2-18)from vllm.v1.core.kv_cache_manager import KVCacheBlocks [](#__codelineno-2-19)from vllm.v1.request import Request [](#__codelineno-2-20)[](#__codelineno-2-21)if TYPE_CHECKING: [](#__codelineno-2-22) from vllm.v1.core.sched.output import SchedulerOutput [](#__codelineno-2-23) from vllm.v1.kv_cache_interface import KVCacheConfig [](#__codelineno-2-24)[](#__codelineno-2-25)logger = logging.getLogger() [](#__codelineno-2-26)logging.basicConfig(level=logging.INFO) [](#__codelineno-2-27) [](#__codelineno-2-28)[](#__codelineno-2-29)@dataclass [](#__codelineno-2-30)class LoadRecoveryExampleConnectorMetadata(ExampleConnectorMetadata): [](#__codelineno-2-31) req_to_block_ids: dict[str, set[int]] = field(default_factory=dict) [](#__codelineno-2-32) [](#__codelineno-2-33) @classmethod [](#__codelineno-2-34) def from_base(cls, base: ExampleConnectorMetadata): [](#__codelineno-2-35) return cls(requests=base.requests) [](#__codelineno-2-36) [](#__codelineno-2-37)[](#__codelineno-2-38)class LoadRecoveryExampleConnector(ExampleConnector): [](#__codelineno-2-39) def __init__( [](#__codelineno-2-40) self, [](#__codelineno-2-41) vllm_config: "VllmConfig", [](#__codelineno-2-42) role: KVConnectorRole, [](#__codelineno-2-43) kv_cache_config: "KVCacheConfig", [](#__codelineno-2-44) ): [](#__codelineno-2-45) super().__init__( [](#__codelineno-2-46) vllm_config=vllm_config, [](#__codelineno-2-47) role=role, [](#__codelineno-2-48) kv_cache_config=kv_cache_config, [](#__codelineno-2-49) ) [](#__codelineno-2-50) self._async_load = vllm_config.kv_transfer_config.get_from_extra_config( [](#__codelineno-2-51) "async_load", False [](#__codelineno-2-52) ) [](#__codelineno-2-53) self._invalid_block_ids: set = None [](#__codelineno-2-54) self._seen_requests: set = set() [](#__codelineno-2-55) self._req_to_block_ids: dict[str, list[int]] = dict() [](#__codelineno-2-56) [](#__codelineno-2-57) def bind_connector_metadata(self, connector_metadata: KVConnectorMetadata) -> None: [](#__codelineno-2-58) assert isinstance(connector_metadata, LoadRecoveryExampleConnectorMetadata) [](#__codelineno-2-59) index, failed_request = next( [](#__codelineno-2-60) ( [](#__codelineno-2-61) (i, x) [](#__codelineno-2-62) for i, x in enumerate(connector_metadata.requests) [](#__codelineno-2-63) if not x.is_store [](#__codelineno-2-64) ), [](#__codelineno-2-65) (None, None), [](#__codelineno-2-66) ) [](#__codelineno-2-67) if index is not None: [](#__codelineno-2-68) del connector_metadata.requests[index] [](#__codelineno-2-69) self._invalid_block_ids = set( [](#__codelineno-2-70) ( [](#__codelineno-2-71) failed_request.slot_mapping[:: self._block_size] // self._block_size [](#__codelineno-2-72) ).tolist() [](#__codelineno-2-73) ) [](#__codelineno-2-74) logger.info( [](#__codelineno-2-75) "Simulating failure to load all KV blocks for the " [](#__codelineno-2-76) "first load request. Total blocks: %d", [](#__codelineno-2-77) len(self._invalid_block_ids), [](#__codelineno-2-78) ) [](#__codelineno-2-79) super().bind_connector_metadata(connector_metadata) [](#__codelineno-2-80) [](#__codelineno-2-81) def clear_connector_metadata(self) -> None: [](#__codelineno-2-82) self._invalid_block_ids = None [](#__codelineno-2-83) super().clear_connector_metadata() [](#__codelineno-2-84) [](#__codelineno-2-85) def start_load_kv(self, forward_context: ForwardContext, **kwargs) -> None: [](#__codelineno-2-86) if self._async_load and forward_context.attn_metadata is None: [](#__codelineno-2-87) # Bypass sanity check in super().start_load_kv [](#__codelineno-2-88) forward_context.attn_metadata = "None" [](#__codelineno-2-89) [](#__codelineno-2-90) super().start_load_kv(forward_context, **kwargs) [](#__codelineno-2-91) [](#__codelineno-2-92) def get_finished( [](#__codelineno-2-93) self, finished_req_ids: set[str] [](#__codelineno-2-94) ) -> tuple[set[str] | None, set[str] | None]: [](#__codelineno-2-95) if self._async_load: [](#__codelineno-2-96) meta = self._get_connector_metadata() [](#__codelineno-2-97) assert isinstance(meta, LoadRecoveryExampleConnectorMetadata) [](#__codelineno-2-98) if meta.req_to_block_ids: [](#__codelineno-2-99) return None, set(meta.req_to_block_ids) [](#__codelineno-2-100) [](#__codelineno-2-101) return None, None [](#__codelineno-2-102) [](#__codelineno-2-103) def get_block_ids_with_load_errors(self) -> set[int]: [](#__codelineno-2-104) return self._invalid_block_ids [](#__codelineno-2-105) [](#__codelineno-2-106) def get_num_new_matched_tokens( [](#__codelineno-2-107) self, [](#__codelineno-2-108) request: Request, [](#__codelineno-2-109) num_computed_tokens: int, [](#__codelineno-2-110) ) -> tuple[int, bool]: [](#__codelineno-2-111) if request.request_id in self._seen_requests: [](#__codelineno-2-112) return 0, False [](#__codelineno-2-113) [](#__codelineno-2-114) self._seen_requests.add(request.request_id) [](#__codelineno-2-115) [](#__codelineno-2-116) num_tokens, _ = super().get_num_new_matched_tokens(request, num_computed_tokens) [](#__codelineno-2-117) return num_tokens, self._async_load and num_tokens > 0 [](#__codelineno-2-118) [](#__codelineno-2-119) def update_state_after_alloc( [](#__codelineno-2-120) self, request: Request, blocks: KVCacheBlocks, num_external_tokens: int [](#__codelineno-2-121) ): [](#__codelineno-2-122) """ [](#__codelineno-2-123) Update KVConnector state after block allocation. [](#__codelineno-2-124) [](#__codelineno-2-125) If blocks were allocated, add to _requests_need_load, [](#__codelineno-2-126) such that we load the KVs in the next forward pass. [](#__codelineno-2-127) """ [](#__codelineno-2-128) super().update_state_after_alloc(request, blocks, num_external_tokens) [](#__codelineno-2-129) [](#__codelineno-2-130) if num_external_tokens > 0: [](#__codelineno-2-131) self._req_to_block_ids[request.request_id] = blocks.get_block_ids()[0] [](#__codelineno-2-132) [](#__codelineno-2-133) def build_connector_meta( [](#__codelineno-2-134) self, [](#__codelineno-2-135) scheduler_output: "SchedulerOutput", [](#__codelineno-2-136) ) -> KVConnectorMetadata: [](#__codelineno-2-137) if not self._async_load: [](#__codelineno-2-138) base = super().build_connector_meta(scheduler_output) [](#__codelineno-2-139) meta = LoadRecoveryExampleConnectorMetadata.from_base(base) [](#__codelineno-2-140) else: [](#__codelineno-2-141) meta = LoadRecoveryExampleConnectorMetadata() [](#__codelineno-2-142) if self._requests_need_load: [](#__codelineno-2-143) for req_id, request in self._requests_need_load.items(): [](#__codelineno-2-144) meta.add_request( [](#__codelineno-2-145) token_ids=request.prompt_token_ids, [](#__codelineno-2-146) block_ids=self._req_to_block_ids[req_id], [](#__codelineno-2-147) block_size=self._block_size, [](#__codelineno-2-148) is_store=False, [](#__codelineno-2-149) mm_hashes=[], [](#__codelineno-2-150) ) [](#__codelineno-2-151) # Clear state [](#__codelineno-2-152) self._requests_need_load.clear() [](#__codelineno-2-153) meta.req_to_block_ids = self._req_to_block_ids [](#__codelineno-2-154) self._req_to_block_ids = dict() [](#__codelineno-2-155) return meta` prefill\_example.py `[](#__codelineno-3-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-3-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-3-3)[](#__codelineno-3-4)from vllm import LLM, SamplingParams [](#__codelineno-3-5)from vllm.config import KVTransferConfig [](#__codelineno-3-6) [](#__codelineno-3-7)[](#__codelineno-3-8)def read_prompts(): [](#__codelineno-3-9) context = "Hi " * 1000 [](#__codelineno-3-10) context2 = "Hey " * 500 [](#__codelineno-3-11) return [ [](#__codelineno-3-12) context + "Hello, my name is", [](#__codelineno-3-13) context + "The capital of France is", [](#__codelineno-3-14) context2 + "Your name is", [](#__codelineno-3-15) context2 + "The capital of China is", [](#__codelineno-3-16) ] [](#__codelineno-3-17) [](#__codelineno-3-18)[](#__codelineno-3-19)def main(): [](#__codelineno-3-20) prompts = read_prompts() [](#__codelineno-3-21) [](#__codelineno-3-22) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1) [](#__codelineno-3-23) [](#__codelineno-3-24) llm = LLM( [](#__codelineno-3-25) model="meta-llama/Llama-3.2-1B-Instruct", [](#__codelineno-3-26) enforce_eager=True, [](#__codelineno-3-27) gpu_memory_utilization=0.8, [](#__codelineno-3-28) kv_transfer_config=KVTransferConfig( [](#__codelineno-3-29) kv_connector="ExampleConnector", [](#__codelineno-3-30) kv_role="kv_both", [](#__codelineno-3-31) kv_connector_extra_config={"shared_storage_path": "local_storage"}, [](#__codelineno-3-32) ), [](#__codelineno-3-33) ) # , max_model_len=2048, max_num_batched_tokens=2048) [](#__codelineno-3-34) [](#__codelineno-3-35) # 1ST generation (prefill instance) [](#__codelineno-3-36) outputs = llm.generate( [](#__codelineno-3-37) prompts, [](#__codelineno-3-38) sampling_params, [](#__codelineno-3-39) ) [](#__codelineno-3-40) [](#__codelineno-3-41) new_prompts = [] [](#__codelineno-3-42) print("-" * 30) [](#__codelineno-3-43) for output in outputs: [](#__codelineno-3-44) prompt = output.prompt [](#__codelineno-3-45) generated_text = output.outputs[0].text [](#__codelineno-3-46) new_prompts.append(prompt + generated_text) [](#__codelineno-3-47) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-3-48) print("-" * 30) [](#__codelineno-3-49) [](#__codelineno-3-50) # Write new_prompts to prefill_output.txt [](#__codelineno-3-51) with open("prefill_output.txt", "w") as f: [](#__codelineno-3-52) for prompt in new_prompts: [](#__codelineno-3-53) f.write(prompt + "\n") [](#__codelineno-3-54) print(f"Saved {len(new_prompts)} prompts to prefill_output.txt") [](#__codelineno-3-55) [](#__codelineno-3-56)[](#__codelineno-3-57)if __name__ == "__main__": [](#__codelineno-3-58) main()` run.sh `[](#__codelineno-4-1)#!/bin/bash [](#__codelineno-4-2)[](#__codelineno-4-3)# Constants [](#__codelineno-4-4)SHARED_STORAGE_DIR="local_storage" [](#__codelineno-4-5)PREFILL_OUTPUT="prefill_output.txt" [](#__codelineno-4-6)DECODE_OUTPUT="decode_output.txt" [](#__codelineno-4-7)SYNC_DECODE_RECOVERED_OUTPUT="sync_decode_recovered_output.txt" [](#__codelineno-4-8)ASYNC_DECODE_RECOVERED_OUTPUT="async_decode_recovered_output.txt" [](#__codelineno-4-9)[](#__codelineno-4-10)# Cleanup [](#__codelineno-4-11)rm -rf "$SHARED_STORAGE_DIR" [](#__codelineno-4-12)rm -f "$PREFILL_OUTPUT" "$DECODE_OUTPUT" "$SYNC_DECODE_RECOVERED_OUTPUT" "$ASYNC_DECODE_RECOVERED_OUTPUT" [](#__codelineno-4-13)[](#__codelineno-4-14)# Run inference examples [](#__codelineno-4-15)VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 prefill_example.py [](#__codelineno-4-16)VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 decode_example.py [](#__codelineno-4-17)VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 decode_example.py --simulate-failure [](#__codelineno-4-18)VLLM_ENABLE_V1_MULTIPROCESSING=0 CUDA_VISIBLE_DEVICES=0 python3 decode_example.py --simulate-failure --async-load [](#__codelineno-4-19)[](#__codelineno-4-20)# Compare outputs [](#__codelineno-4-21)if ! cmp -s "$DECODE_OUTPUT" "$SYNC_DECODE_RECOVERED_OUTPUT"; then [](#__codelineno-4-22) echo "❌ Outputs differ: sync recovery failed." [](#__codelineno-4-23) diff -u "$DECODE_OUTPUT" "$SYNC_DECODE_RECOVERED_OUTPUT" [](#__codelineno-4-24) exit 1 [](#__codelineno-4-25)fi [](#__codelineno-4-26)[](#__codelineno-4-27)if ! cmp -s "$DECODE_OUTPUT" "$ASYNC_DECODE_RECOVERED_OUTPUT"; then [](#__codelineno-4-28) echo "❌ Outputs differ: async recovery failed." [](#__codelineno-4-29) diff -u "$DECODE_OUTPUT" "$ASYNC_DECODE_RECOVERED_OUTPUT" [](#__codelineno-4-30) exit 1 [](#__codelineno-4-31)fi [](#__codelineno-4-32)[](#__codelineno-4-33)echo "✅ Outputs match: recovery successful."` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/p2p_nccl_xpyd.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/p2p\_nccl\_xpyd](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/p2p_nccl_xpyd). ## Disagg Example P2P NCCL Xpyd[¶](#disagg-example-p2p-nccl-xpyd "Permanent link") `[](#__codelineno-0-1)#!/bin/bash [](#__codelineno-0-2)[](#__codelineno-0-3)# ============================================================================= [](#__codelineno-0-4)# vLLM Disaggregated Serving Script - P2P NCCL XpYd Architecture [](#__codelineno-0-5)# ============================================================================= [](#__codelineno-0-6)# This script demonstrates disaggregated prefill and decode serving using [](#__codelineno-0-7)# P2P NCCL communication. The architecture supports various XpYd configurations: [](#__codelineno-0-8)# [](#__codelineno-0-9)# - 1P3D: 1 Prefill server + 3 Decode servers (current default) [](#__codelineno-0-10)# - 3P1D: 3 Prefill servers + 1 Decode server [](#__codelineno-0-11)# - etc. [](#__codelineno-0-12)# [](#__codelineno-0-13)# Configuration can be customized via environment variables: [](#__codelineno-0-14)# MODEL: Model to serve [](#__codelineno-0-15)# PREFILL_GPUS: Comma-separated GPU IDs for prefill servers [](#__codelineno-0-16)# DECODE_GPUS: Comma-separated GPU IDs for decode servers [](#__codelineno-0-17)# PREFILL_PORTS: Comma-separated ports for prefill servers [](#__codelineno-0-18)# DECODE_PORTS: Comma-separated ports for decode servers [](#__codelineno-0-19)# PROXY_PORT: Proxy server port used to setup XpYd connection. [](#__codelineno-0-20)# TIMEOUT_SECONDS: Server startup timeout [](#__codelineno-0-21)# ============================================================================= [](#__codelineno-0-22)[](#__codelineno-0-23)# Configuration - can be overridden via environment variables [](#__codelineno-0-24)MODEL=${MODEL:-meta-llama/Llama-3.1-8B-Instruct} [](#__codelineno-0-25)TIMEOUT_SECONDS=${TIMEOUT_SECONDS:-1200} [](#__codelineno-0-26)PROXY_PORT=${PROXY_PORT:-30001} [](#__codelineno-0-27)[](#__codelineno-0-28)# Default 1P3D configuration (1 Prefill + 3 Decode) [](#__codelineno-0-29)PREFILL_GPUS=${PREFILL_GPUS:-0} [](#__codelineno-0-30)DECODE_GPUS=${DECODE_GPUS:-1,2,3} [](#__codelineno-0-31)PREFILL_PORTS=${PREFILL_PORTS:-20003} [](#__codelineno-0-32)DECODE_PORTS=${DECODE_PORTS:-20005,20007,20009} [](#__codelineno-0-33)[](#__codelineno-0-34)echo "Warning: P2P NCCL disaggregated prefill XpYd support for vLLM v1 is experimental and subject to change." [](#__codelineno-0-35)echo "" [](#__codelineno-0-36)echo "Architecture Configuration:" [](#__codelineno-0-37)echo " Model: $MODEL" [](#__codelineno-0-38)echo " Prefill GPUs: $PREFILL_GPUS, Ports: $PREFILL_PORTS" [](#__codelineno-0-39)echo " Decode GPUs: $DECODE_GPUS, Ports: $DECODE_PORTS" [](#__codelineno-0-40)echo " Proxy Port: $PROXY_PORT" [](#__codelineno-0-41)echo " Timeout: ${TIMEOUT_SECONDS}s" [](#__codelineno-0-42)echo "" [](#__codelineno-0-43)[](#__codelineno-0-44)PIDS=() [](#__codelineno-0-45)[](#__codelineno-0-46)# Switch to the directory of the current script [](#__codelineno-0-47)cd "$(dirname "${BASH_SOURCE[0]}")" [](#__codelineno-0-48)[](#__codelineno-0-49)check_required_files() { [](#__codelineno-0-50) local files=("disagg_proxy_p2p_nccl_xpyd.py") [](#__codelineno-0-51) for file in "${files[@]}"; do [](#__codelineno-0-52) if [[ ! -f "$file" ]]; then [](#__codelineno-0-53) echo "Required file $file not found in $(pwd)" [](#__codelineno-0-54) exit 1 [](#__codelineno-0-55) fi [](#__codelineno-0-56) done [](#__codelineno-0-57)} [](#__codelineno-0-58)[](#__codelineno-0-59)check_hf_token() { [](#__codelineno-0-60) if [ -z "$HF_TOKEN" ]; then [](#__codelineno-0-61) echo "HF_TOKEN is not set. Please set it to your Hugging Face token." [](#__codelineno-0-62) echo "Example: export HF_TOKEN=your_token_here" [](#__codelineno-0-63) exit 1 [](#__codelineno-0-64) fi [](#__codelineno-0-65) if [[ "$HF_TOKEN" != hf_* ]]; then [](#__codelineno-0-66) echo "HF_TOKEN is not a valid Hugging Face token. Please set it to your Hugging Face token." [](#__codelineno-0-67) exit 1 [](#__codelineno-0-68) fi [](#__codelineno-0-69) echo "HF_TOKEN is set and valid." [](#__codelineno-0-70)} [](#__codelineno-0-71)[](#__codelineno-0-72)check_num_gpus() { [](#__codelineno-0-73) # Check if the number of GPUs are >=2 via nvidia-smi [](#__codelineno-0-74) num_gpus=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l) [](#__codelineno-0-75) if [ "$num_gpus" -lt 2 ]; then [](#__codelineno-0-76) echo "You need at least 2 GPUs to run disaggregated prefill." [](#__codelineno-0-77) exit 1 [](#__codelineno-0-78) else [](#__codelineno-0-79) echo "Found $num_gpus GPUs." [](#__codelineno-0-80) fi [](#__codelineno-0-81)} [](#__codelineno-0-82)[](#__codelineno-0-83)ensure_python_library_installed() { [](#__codelineno-0-84) echo "Checking if $1 is installed..." [](#__codelineno-0-85) if ! python3 -c "import $1" > /dev/null 2>&1; then [](#__codelineno-0-86) echo "$1 is not installed. Please install it via pip install $1." [](#__codelineno-0-87) exit 1 [](#__codelineno-0-88) else [](#__codelineno-0-89) echo "$1 is installed." [](#__codelineno-0-90) fi [](#__codelineno-0-91)} [](#__codelineno-0-92)[](#__codelineno-0-93)cleanup() { [](#__codelineno-0-94) echo "Stopping everything…" [](#__codelineno-0-95) trap - INT TERM # prevent re-entrancy [](#__codelineno-0-96) pkill -9 -f "disagg_proxy_p2p_nccl_xpyd.py" [](#__codelineno-0-97) kill -- -$$ # negative PID == "this whole process-group" [](#__codelineno-0-98) wait # reap children so we don't leave zombies [](#__codelineno-0-99) exit 0 [](#__codelineno-0-100)} [](#__codelineno-0-101)[](#__codelineno-0-102)wait_for_server() { [](#__codelineno-0-103) local port=$1 [](#__codelineno-0-104) local timeout_seconds=$TIMEOUT_SECONDS [](#__codelineno-0-105) local start_time=$(date +%s) [](#__codelineno-0-106) [](#__codelineno-0-107) echo "Waiting for server on port $port..." [](#__codelineno-0-108) [](#__codelineno-0-109) while true; do [](#__codelineno-0-110) if curl -s "localhost:${port}/v1/completions" > /dev/null; then [](#__codelineno-0-111) echo "Server on port $port is ready." [](#__codelineno-0-112) return 0 [](#__codelineno-0-113) fi [](#__codelineno-0-114) [](#__codelineno-0-115) local now=$(date +%s) [](#__codelineno-0-116) if (( now - start_time >= timeout_seconds )); then [](#__codelineno-0-117) echo "Timeout waiting for server on port $port" [](#__codelineno-0-118) return 1 [](#__codelineno-0-119) fi [](#__codelineno-0-120) [](#__codelineno-0-121) sleep 1 [](#__codelineno-0-122) done [](#__codelineno-0-123)} [](#__codelineno-0-124)[](#__codelineno-0-125)main() { [](#__codelineno-0-126) check_required_files [](#__codelineno-0-127) check_hf_token [](#__codelineno-0-128) check_num_gpus [](#__codelineno-0-129) ensure_python_library_installed pandas [](#__codelineno-0-130) ensure_python_library_installed datasets [](#__codelineno-0-131) ensure_python_library_installed vllm [](#__codelineno-0-132) ensure_python_library_installed quart [](#__codelineno-0-133) [](#__codelineno-0-134) trap cleanup INT [](#__codelineno-0-135) trap cleanup USR1 [](#__codelineno-0-136) trap cleanup TERM [](#__codelineno-0-137) [](#__codelineno-0-138) echo "Launching disaggregated serving components..." [](#__codelineno-0-139) echo "Please check the log files for detailed output:" [](#__codelineno-0-140) echo " - prefill*.log: Prefill server logs" [](#__codelineno-0-141) echo " - decode*.log: Decode server logs" [](#__codelineno-0-142) echo " - proxy.log: Proxy server log" [](#__codelineno-0-143) [](#__codelineno-0-144) # ============================================================================= [](#__codelineno-0-145) # Launch Proxy Server [](#__codelineno-0-146) # ============================================================================= [](#__codelineno-0-147) echo "" [](#__codelineno-0-148) echo "Starting proxy server on port $PROXY_PORT..." [](#__codelineno-0-149) python3 disagg_proxy_p2p_nccl_xpyd.py & [](#__codelineno-0-150) PIDS+=($!) [](#__codelineno-0-151) [](#__codelineno-0-152) # Parse GPU and port arrays [](#__codelineno-0-153) IFS=',' read -ra PREFILL_GPU_ARRAY <<< "$PREFILL_GPUS" [](#__codelineno-0-154) IFS=',' read -ra DECODE_GPU_ARRAY <<< "$DECODE_GPUS" [](#__codelineno-0-155) IFS=',' read -ra PREFILL_PORT_ARRAY <<< "$PREFILL_PORTS" [](#__codelineno-0-156) IFS=',' read -ra DECODE_PORT_ARRAY <<< "$DECODE_PORTS" [](#__codelineno-0-157) [](#__codelineno-0-158) # ============================================================================= [](#__codelineno-0-159) # Launch Prefill Servers (X Producers) [](#__codelineno-0-160) # ============================================================================= [](#__codelineno-0-161) echo "" [](#__codelineno-0-162) echo "Starting ${#PREFILL_GPU_ARRAY[@]} prefill server(s)..." [](#__codelineno-0-163) for i in "${!PREFILL_GPU_ARRAY[@]}"; do [](#__codelineno-0-164) local gpu_id=${PREFILL_GPU_ARRAY[$i]} [](#__codelineno-0-165) local port=${PREFILL_PORT_ARRAY[$i]} [](#__codelineno-0-166) local kv_port=$((21001 + i)) [](#__codelineno-0-167) [](#__codelineno-0-168) echo " Prefill server $((i+1)): GPU $gpu_id, Port $port, KV Port $kv_port" [](#__codelineno-0-169) CUDA_VISIBLE_DEVICES=$gpu_id vllm serve "$MODEL" \ [](#__codelineno-0-170) --enforce-eager \ [](#__codelineno-0-171) --host 0.0.0.0 \ [](#__codelineno-0-172) --port "$port" \ [](#__codelineno-0-173) --tensor-parallel-size 1 \ [](#__codelineno-0-174) --seed 1024 \ [](#__codelineno-0-175) --dtype float16 \ [](#__codelineno-0-176) --max-model-len 10000 \ [](#__codelineno-0-177) --max-num-batched-tokens 10000 \ [](#__codelineno-0-178) --max-num-seqs 256 \ [](#__codelineno-0-179) --trust-remote-code \ [](#__codelineno-0-180) --gpu-memory-utilization 0.9 \ [](#__codelineno-0-181) --kv-transfer-config \ [](#__codelineno-0-182) "{\"kv_connector\":\"P2pNcclConnector\",\"kv_role\":\"kv_producer\",\"kv_buffer_size\":\"1e1\",\"kv_port\":\"$kv_port\",\"kv_connector_extra_config\":{\"proxy_ip\":\"0.0.0.0\",\"proxy_port\":\"$PROXY_PORT\",\"http_port\":\"$port\",\"send_type\":\"PUT_ASYNC\",\"nccl_num_channels\":\"16\"}}" > prefill$((i+1)).log 2>&1 & [](#__codelineno-0-183) PIDS+=($!) [](#__codelineno-0-184) done [](#__codelineno-0-185) [](#__codelineno-0-186) # ============================================================================= [](#__codelineno-0-187) # Launch Decode Servers (Y Decoders) [](#__codelineno-0-188) # ============================================================================= [](#__codelineno-0-189) echo "" [](#__codelineno-0-190) echo "Starting ${#DECODE_GPU_ARRAY[@]} decode server(s)..." [](#__codelineno-0-191) for i in "${!DECODE_GPU_ARRAY[@]}"; do [](#__codelineno-0-192) local gpu_id=${DECODE_GPU_ARRAY[$i]} [](#__codelineno-0-193) local port=${DECODE_PORT_ARRAY[$i]} [](#__codelineno-0-194) local kv_port=$((22001 + i)) [](#__codelineno-0-195) [](#__codelineno-0-196) echo " Decode server $((i+1)): GPU $gpu_id, Port $port, KV Port $kv_port" [](#__codelineno-0-197) CUDA_VISIBLE_DEVICES=$gpu_id vllm serve "$MODEL" \ [](#__codelineno-0-198) --enforce-eager \ [](#__codelineno-0-199) --host 0.0.0.0 \ [](#__codelineno-0-200) --port "$port" \ [](#__codelineno-0-201) --tensor-parallel-size 1 \ [](#__codelineno-0-202) --seed 1024 \ [](#__codelineno-0-203) --dtype float16 \ [](#__codelineno-0-204) --max-model-len 10000 \ [](#__codelineno-0-205) --max-num-batched-tokens 10000 \ [](#__codelineno-0-206) --max-num-seqs 256 \ [](#__codelineno-0-207) --trust-remote-code \ [](#__codelineno-0-208) --gpu-memory-utilization 0.7 \ [](#__codelineno-0-209) --kv-transfer-config \ [](#__codelineno-0-210) "{\"kv_connector\":\"P2pNcclConnector\",\"kv_role\":\"kv_consumer\",\"kv_buffer_size\":\"8e9\",\"kv_port\":\"$kv_port\",\"kv_connector_extra_config\":{\"proxy_ip\":\"0.0.0.0\",\"proxy_port\":\"$PROXY_PORT\",\"http_port\":\"$port\",\"send_type\":\"PUT_ASYNC\",\"nccl_num_channels\":\"16\"}}" > decode$((i+1)).log 2>&1 & [](#__codelineno-0-211) PIDS+=($!) [](#__codelineno-0-212) done [](#__codelineno-0-213) [](#__codelineno-0-214) # ============================================================================= [](#__codelineno-0-215) # Wait for All Servers to Start [](#__codelineno-0-216) # ============================================================================= [](#__codelineno-0-217) echo "" [](#__codelineno-0-218) echo "Waiting for all servers to start..." [](#__codelineno-0-219) for port in "${PREFILL_PORT_ARRAY[@]}" "${DECODE_PORT_ARRAY[@]}"; do [](#__codelineno-0-220) if ! wait_for_server "$port"; then [](#__codelineno-0-221) echo "Failed to start server on port $port" [](#__codelineno-0-222) cleanup [](#__codelineno-0-223) # shellcheck disable=SC2317 [](#__codelineno-0-224) exit 1 [](#__codelineno-0-225) fi [](#__codelineno-0-226) done [](#__codelineno-0-227) [](#__codelineno-0-228) echo "" [](#__codelineno-0-229) echo "All servers are up. Starting benchmark..." [](#__codelineno-0-230) [](#__codelineno-0-231) # ============================================================================= [](#__codelineno-0-232) # Run Benchmark [](#__codelineno-0-233) # ============================================================================= [](#__codelineno-0-234) cd ../../../benchmarks/ [](#__codelineno-0-235) vllm bench serve --port 10001 --seed "$(date +%s)" \ [](#__codelineno-0-236) --model "$MODEL" \ [](#__codelineno-0-237) --dataset-name random --random-input-len 7500 --random-output-len 200 \ [](#__codelineno-0-238) --num-prompts 200 --burstiness 100 --request-rate 2 | tee benchmark.log [](#__codelineno-0-239) [](#__codelineno-0-240) echo "Benchmarking done. Cleaning up..." [](#__codelineno-0-241) [](#__codelineno-0-242) cleanup [](#__codelineno-0-243)} [](#__codelineno-0-244)[](#__codelineno-0-245)main` ## Disagg Proxy P2P NCCL Xpyd[¶](#disagg-proxy-p2p-nccl-xpyd "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)[](#__codelineno-1-4)import os [](#__codelineno-1-5)import socket [](#__codelineno-1-6)import threading [](#__codelineno-1-7)import time [](#__codelineno-1-8)import uuid [](#__codelineno-1-9)from typing import Any [](#__codelineno-1-10)[](#__codelineno-1-11)import aiohttp [](#__codelineno-1-12)import msgpack [](#__codelineno-1-13)import zmq [](#__codelineno-1-14)from quart import Quart, make_response, request [](#__codelineno-1-15)[](#__codelineno-1-16)count = 0 [](#__codelineno-1-17)prefill_instances: dict[str, Any] = {} # http_address: (zmq_address, stamp) [](#__codelineno-1-18)decode_instances: dict[str, Any] = {} # http_address: (zmq_address, stamp) [](#__codelineno-1-19)[](#__codelineno-1-20)prefill_cv = threading.Condition() [](#__codelineno-1-21)decode_cv = threading.Condition() [](#__codelineno-1-22)[](#__codelineno-1-23)DEFAULT_PING_SECONDS = 5 [](#__codelineno-1-24) [](#__codelineno-1-25)[](#__codelineno-1-26)def _remove_oldest_instances(instances: dict[str, Any]) -> None: [](#__codelineno-1-27) oldest_key = next(iter(instances), None) [](#__codelineno-1-28) while oldest_key is not None: [](#__codelineno-1-29) value = instances[oldest_key] [](#__codelineno-1-30) if value[1] > time.time(): [](#__codelineno-1-31) break [](#__codelineno-1-32) print(f"🔴Remove [HTTP:{oldest_key}, ZMQ:{value[0]}, stamp:{value[1]}]") [](#__codelineno-1-33) instances.pop(oldest_key, None) [](#__codelineno-1-34) oldest_key = next(iter(instances), None) [](#__codelineno-1-35) [](#__codelineno-1-36)[](#__codelineno-1-37)def _listen_for_register(poller, router_socket): [](#__codelineno-1-38) while True: [](#__codelineno-1-39) socks = dict(poller.poll()) [](#__codelineno-1-40) if router_socket in socks: [](#__codelineno-1-41) remote_address, message = router_socket.recv_multipart() [](#__codelineno-1-42) # data: {"type": "P", "http_address": "ip:port", [](#__codelineno-1-43) # "zmq_address": "ip:port"} [](#__codelineno-1-44) data = msgpack.loads(message) [](#__codelineno-1-45) if data["type"] == "P": [](#__codelineno-1-46) global prefill_instances [](#__codelineno-1-47) global prefill_cv [](#__codelineno-1-48) with prefill_cv: [](#__codelineno-1-49) node = prefill_instances.get(data["http_address"], None) [](#__codelineno-1-50) prefill_instances[data["http_address"]] = ( [](#__codelineno-1-51) data["zmq_address"], [](#__codelineno-1-52) time.time() + DEFAULT_PING_SECONDS, [](#__codelineno-1-53) ) [](#__codelineno-1-54) _remove_oldest_instances(prefill_instances) [](#__codelineno-1-55) [](#__codelineno-1-56) elif data["type"] == "D": [](#__codelineno-1-57) global decode_instances [](#__codelineno-1-58) global decode_cv [](#__codelineno-1-59) with decode_cv: [](#__codelineno-1-60) node = decode_instances.get(data["http_address"], None) [](#__codelineno-1-61) decode_instances[data["http_address"]] = ( [](#__codelineno-1-62) data["zmq_address"], [](#__codelineno-1-63) time.time() + DEFAULT_PING_SECONDS, [](#__codelineno-1-64) ) [](#__codelineno-1-65) _remove_oldest_instances(decode_instances) [](#__codelineno-1-66) else: [](#__codelineno-1-67) print( [](#__codelineno-1-68) "Unexpected, Received message from %s, data: %s", [](#__codelineno-1-69) remote_address, [](#__codelineno-1-70) data, [](#__codelineno-1-71) ) [](#__codelineno-1-72) return [](#__codelineno-1-73) [](#__codelineno-1-74) if node is None: [](#__codelineno-1-75) print(f"🔵Add [HTTP:{data['http_address']}, ZMQ:{data['zmq_address']}]") [](#__codelineno-1-76) [](#__codelineno-1-77)[](#__codelineno-1-78)def start_service_discovery(hostname, port): [](#__codelineno-1-79) if not hostname: [](#__codelineno-1-80) hostname = socket.gethostname() [](#__codelineno-1-81) if port == 0: [](#__codelineno-1-82) raise ValueError("Port cannot be 0") [](#__codelineno-1-83) [](#__codelineno-1-84) context = zmq.Context() [](#__codelineno-1-85) router_socket = context.socket(zmq.ROUTER) [](#__codelineno-1-86) router_socket.bind(f"tcp://{hostname}:{port}") [](#__codelineno-1-87) [](#__codelineno-1-88) poller = zmq.Poller() [](#__codelineno-1-89) poller.register(router_socket, zmq.POLLIN) [](#__codelineno-1-90) [](#__codelineno-1-91) _listener_thread = threading.Thread( [](#__codelineno-1-92) target=_listen_for_register, args=[poller, router_socket], daemon=True [](#__codelineno-1-93) ) [](#__codelineno-1-94) _listener_thread.start() [](#__codelineno-1-95) return _listener_thread [](#__codelineno-1-96) [](#__codelineno-1-97)[](#__codelineno-1-98)AIOHTTP_TIMEOUT = aiohttp.ClientTimeout(total=6 * 60 * 60) [](#__codelineno-1-99)[](#__codelineno-1-100)app = Quart(__name__) [](#__codelineno-1-101) [](#__codelineno-1-102)[](#__codelineno-1-103)def random_uuid() -> str: [](#__codelineno-1-104) return str(uuid.uuid4().hex) [](#__codelineno-1-105) [](#__codelineno-1-106)[](#__codelineno-1-107)async def forward_request(url, data, request_id): [](#__codelineno-1-108) async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session: [](#__codelineno-1-109) headers = { [](#__codelineno-1-110) "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", [](#__codelineno-1-111) "X-Request-Id": request_id, [](#__codelineno-1-112) } [](#__codelineno-1-113) async with session.post(url=url, json=data, headers=headers) as response: [](#__codelineno-1-114) if response.status == 200: [](#__codelineno-1-115) if True: [](#__codelineno-1-116) async for chunk_bytes in response.content.iter_chunked(1024): [](#__codelineno-1-117) yield chunk_bytes [](#__codelineno-1-118) else: [](#__codelineno-1-119) content = await response.read() [](#__codelineno-1-120) yield content [](#__codelineno-1-121) [](#__codelineno-1-122)[](#__codelineno-1-123)@app.route("/v1/completions", methods=["POST"]) [](#__codelineno-1-124)@app.route("/v1/chat/completions", methods=["POST"]) [](#__codelineno-1-125)async def handle_request(): [](#__codelineno-1-126) try: [](#__codelineno-1-127) original_request_data = await request.get_json() [](#__codelineno-1-128) [](#__codelineno-1-129) prefill_request = original_request_data.copy() [](#__codelineno-1-130) # change max_tokens = 1 to let it only do prefill [](#__codelineno-1-131) prefill_request["max_tokens"] = 1 [](#__codelineno-1-132) if "max_completion_tokens" in prefill_request: [](#__codelineno-1-133) prefill_request["max_completion_tokens"] = 1 [](#__codelineno-1-134) [](#__codelineno-1-135) global count [](#__codelineno-1-136) global prefill_instances [](#__codelineno-1-137) global prefill_cv [](#__codelineno-1-138) with prefill_cv: [](#__codelineno-1-139) prefill_list = list(prefill_instances.items()) [](#__codelineno-1-140) prefill_addr, prefill_zmq_addr = prefill_list[count % len(prefill_list)] [](#__codelineno-1-141) prefill_zmq_addr = prefill_zmq_addr[0] [](#__codelineno-1-142) [](#__codelineno-1-143) global decode_instances [](#__codelineno-1-144) global decode_cv [](#__codelineno-1-145) with decode_cv: [](#__codelineno-1-146) decode_list = list(decode_instances.items()) [](#__codelineno-1-147) decode_addr, decode_zmq_addr = decode_list[count % len(decode_list)] [](#__codelineno-1-148) decode_zmq_addr = decode_zmq_addr[0] [](#__codelineno-1-149) [](#__codelineno-1-150) print( [](#__codelineno-1-151) f"handle_request count: {count}, [HTTP:{prefill_addr}, " [](#__codelineno-1-152) f"ZMQ:{prefill_zmq_addr}] 👉 [HTTP:{decode_addr}, " [](#__codelineno-1-153) f"ZMQ:{decode_zmq_addr}]" [](#__codelineno-1-154) ) [](#__codelineno-1-155) count += 1 [](#__codelineno-1-156) [](#__codelineno-1-157) request_id = ( [](#__codelineno-1-158) f"___prefill_addr_{prefill_zmq_addr}___decode_addr_" [](#__codelineno-1-159) f"{decode_zmq_addr}_{random_uuid()}" [](#__codelineno-1-160) ) [](#__codelineno-1-161) [](#__codelineno-1-162) # finish prefill [](#__codelineno-1-163) async for _ in forward_request( [](#__codelineno-1-164) f"http://{prefill_addr}{request.path}", prefill_request, request_id [](#__codelineno-1-165) ): [](#__codelineno-1-166) continue [](#__codelineno-1-167) [](#__codelineno-1-168) # return decode [](#__codelineno-1-169) generator = forward_request( [](#__codelineno-1-170) f"http://{decode_addr}{request.path}", original_request_data, request_id [](#__codelineno-1-171) ) [](#__codelineno-1-172) response = await make_response(generator) [](#__codelineno-1-173) response.timeout = None [](#__codelineno-1-174) [](#__codelineno-1-175) return response [](#__codelineno-1-176) [](#__codelineno-1-177) except Exception as e: [](#__codelineno-1-178) import sys [](#__codelineno-1-179) import traceback [](#__codelineno-1-180) [](#__codelineno-1-181) exc_info = sys.exc_info() [](#__codelineno-1-182) print("Error occurred in disagg prefill proxy server") [](#__codelineno-1-183) print(e) [](#__codelineno-1-184) print("".join(traceback.format_exception(*exc_info))) [](#__codelineno-1-185) [](#__codelineno-1-186)[](#__codelineno-1-187)if __name__ == "__main__": [](#__codelineno-1-188) t = start_service_discovery("0.0.0.0", 30001) [](#__codelineno-1-189) app.run(host="0.0.0.0", port=10001) [](#__codelineno-1-190) t.join()` --- # page ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This example shows how to use FlexKV with vLLM for prefix caching. [](#__codelineno-0-5)[](#__codelineno-0-6)FlexKV is a distributed KV Store and multi-level cache management system for [](#__codelineno-0-7)ultra-large-scale LLM inference. [](#__codelineno-0-8)[](#__codelineno-0-9)Requirements: [](#__codelineno-0-10) - Install FlexKV (https://github.com/taco-project/FlexKV): [](#__codelineno-0-11) 1. git clone [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection):taco-project/FlexKV.git [](#__codelineno-0-12) 2. cd FlexKV && bash build.sh [](#__codelineno-0-13) - Ensure FlexKV is compatible with your vLLM version. [](#__codelineno-0-14)[](#__codelineno-0-15)Usage: [](#__codelineno-0-16) 1. Run this script: [](#__codelineno-0-17) python examples/disaggregated/flexkv_connector/prefix_caching_flexkv.py \ [](#__codelineno-0-18) --model /path/to/your/model [](#__codelineno-0-19) [](#__codelineno-0-20) 2. Arguments: [](#__codelineno-0-21) --model Path or name of the model (required) [](#__codelineno-0-22) --tp-size Tensor parallel size (default: 1) [](#__codelineno-0-23) --gpu-memory-util GPU memory utilization (default: 0.4) [](#__codelineno-0-24) [](#__codelineno-0-25) 3. The script will: [](#__codelineno-0-26) - Create a FlexKV configuration file. [](#__codelineno-0-27) - Set the FLEXKV_CONFIG_PATH environment variable. [](#__codelineno-0-28) - Run vLLM with FlexKVConnectorV1 enabled. [](#__codelineno-0-29) - Compare results between regular execution, vLLM's default prefix [](#__codelineno-0-30) caching, and FlexKV. [](#__codelineno-0-31)""" [](#__codelineno-0-32)[](#__codelineno-0-33)import argparse [](#__codelineno-0-34)import json [](#__codelineno-0-35)import os [](#__codelineno-0-36)import time [](#__codelineno-0-37)[](#__codelineno-0-38)from vllm import LLM, SamplingParams [](#__codelineno-0-39)from vllm.distributed import cleanup_dist_env_and_memory [](#__codelineno-0-40)[](#__codelineno-0-41)# NOTE: This is just a running example. For benchmarking purpose, [](#__codelineno-0-42)# please see benchmarks/benchmark_prefix_caching.py [](#__codelineno-0-43) [](#__codelineno-0-44)[](#__codelineno-0-45)def parse_args(): [](#__codelineno-0-46) parser = argparse.ArgumentParser( [](#__codelineno-0-47) description="Example of using FlexKV with vLLM for prefix caching." [](#__codelineno-0-48) ) [](#__codelineno-0-49) parser.add_argument( [](#__codelineno-0-50) "--model", [](#__codelineno-0-51) type=str, [](#__codelineno-0-52) required=True, [](#__codelineno-0-53) help="Path or name of the model to use.", [](#__codelineno-0-54) ) [](#__codelineno-0-55) parser.add_argument( [](#__codelineno-0-56) "--tp-size", [](#__codelineno-0-57) type=int, [](#__codelineno-0-58) default=1, [](#__codelineno-0-59) help="Tensor parallel size (default: 1).", [](#__codelineno-0-60) ) [](#__codelineno-0-61) parser.add_argument( [](#__codelineno-0-62) "--gpu-memory-util", [](#__codelineno-0-63) type=float, [](#__codelineno-0-64) default=0.4, [](#__codelineno-0-65) help="GPU memory utilization fraction (default: 0.4).", [](#__codelineno-0-66) ) [](#__codelineno-0-67) return parser.parse_args() [](#__codelineno-0-68) [](#__codelineno-0-69)[](#__codelineno-0-70)def main(): [](#__codelineno-0-71) args = parse_args() [](#__codelineno-0-72) [](#__codelineno-0-73) flexkv_config = { [](#__codelineno-0-74) "server_recv_port": f"ipc:///tmp/flexkv_test_{os.getpid()}", [](#__codelineno-0-75) "cache_config": { [](#__codelineno-0-76) "enable_cpu": True, [](#__codelineno-0-77) "num_cpu_blocks": 10240, [](#__codelineno-0-78) }, [](#__codelineno-0-79) "num_log_interval_requests": 200, [](#__codelineno-0-80) } [](#__codelineno-0-81) flexkv_config_path = f"./flexkv_config_{os.getpid()}.json" [](#__codelineno-0-82) with open(flexkv_config_path, "w") as f: [](#__codelineno-0-83) json.dump(flexkv_config, f) [](#__codelineno-0-84) os.environ["FLEXKV_CONFIG_PATH"] = flexkv_config_path [](#__codelineno-0-85) [](#__codelineno-0-86) try: [](#__codelineno-0-87) _run(args) [](#__codelineno-0-88) finally: [](#__codelineno-0-89) if os.path.exists(flexkv_config_path): [](#__codelineno-0-90) os.remove(flexkv_config_path) [](#__codelineno-0-91) [](#__codelineno-0-92)[](#__codelineno-0-93)def _run(args): [](#__codelineno-0-94) # Common prefix. [](#__codelineno-0-95) prefix = ( [](#__codelineno-0-96) "You are an expert school principal, skilled in effectively managing " [](#__codelineno-0-97) "faculty and staff. Draft 10-15 questions for a potential first grade " [](#__codelineno-0-98) "Head Teacher for my K-12, all-girls', independent school that emphasizes " [](#__codelineno-0-99) "community, joyful discovery, and life-long learning. The candidate is " [](#__codelineno-0-100) "coming in for a first-round panel interview for a 8th grade Math " [](#__codelineno-0-101) "teaching role. They have 5 years of previous teaching experience " [](#__codelineno-0-102) "as an assistant teacher at a co-ed, public school with experience " [](#__codelineno-0-103) "in middle school math teaching. Based on these information, fulfill " [](#__codelineno-0-104) "the following paragraph: " [](#__codelineno-0-105) ) [](#__codelineno-0-106) [](#__codelineno-0-107) # Sample prompts. [](#__codelineno-0-108) prompts = [ [](#__codelineno-0-109) "Hello, my name is", [](#__codelineno-0-110) "The president of the United States is", [](#__codelineno-0-111) "The capital of France is", [](#__codelineno-0-112) "The future of AI is", [](#__codelineno-0-113) ] [](#__codelineno-0-114) [](#__codelineno-0-115) generating_prompts = [prefix + prompt for prompt in prompts] [](#__codelineno-0-116) [](#__codelineno-0-117) # Create a sampling params object. [](#__codelineno-0-118) sampling_params = SamplingParams(temperature=0.0) [](#__codelineno-0-119) [](#__codelineno-0-120) kv_transfer_config = { [](#__codelineno-0-121) "kv_connector": "FlexKVConnectorV1", [](#__codelineno-0-122) "kv_role": "kv_both", [](#__codelineno-0-123) } [](#__codelineno-0-124) [](#__codelineno-0-125) # Create an LLM without prefix caching as a baseline. [](#__codelineno-0-126) regular_llm = LLM( [](#__codelineno-0-127) model=args.model, [](#__codelineno-0-128) enable_prefix_caching=False, [](#__codelineno-0-129) gpu_memory_utilization=args.gpu_memory_util, [](#__codelineno-0-130) tensor_parallel_size=args.tp_size, [](#__codelineno-0-131) ) [](#__codelineno-0-132) [](#__codelineno-0-133) print("Results without `enable_prefix_caching`") [](#__codelineno-0-134) [](#__codelineno-0-135) # ruff: noqa: E501 [](#__codelineno-0-136) # Generate texts from the prompts. The output is a list of RequestOutput [](#__codelineno-0-137) # objects that contain the prompt, generated text, and other information. [](#__codelineno-0-138) outputs = regular_llm.generate(generating_prompts, sampling_params) [](#__codelineno-0-139) [](#__codelineno-0-140) regular_generated_texts = [] [](#__codelineno-0-141) # Print the outputs. [](#__codelineno-0-142) print("-" * 50) [](#__codelineno-0-143) for output in outputs: [](#__codelineno-0-144) prompt = output.prompt [](#__codelineno-0-145) generated_text = output.outputs[0].text [](#__codelineno-0-146) regular_generated_texts.append(generated_text) [](#__codelineno-0-147) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-0-148) print("-" * 50) [](#__codelineno-0-149) [](#__codelineno-0-150) # Destroy the LLM object and free up the GPU memory. [](#__codelineno-0-151) del regular_llm [](#__codelineno-0-152) cleanup_dist_env_and_memory() [](#__codelineno-0-153) [](#__codelineno-0-154) # Create an LLM with prefix caching enabled. [](#__codelineno-0-155) prefix_cached_llm = LLM( [](#__codelineno-0-156) model=args.model, [](#__codelineno-0-157) enable_prefix_caching=True, [](#__codelineno-0-158) gpu_memory_utilization=args.gpu_memory_util, [](#__codelineno-0-159) tensor_parallel_size=args.tp_size, [](#__codelineno-0-160) kv_transfer_config=kv_transfer_config, [](#__codelineno-0-161) ) [](#__codelineno-0-162) [](#__codelineno-0-163) # Warmup so that the shared prompt's KV cache is computed. [](#__codelineno-0-164) prefix_cached_llm.generate(generating_prompts[0], sampling_params) [](#__codelineno-0-165) [](#__codelineno-0-166) # wait for offload kv task finished. [](#__codelineno-0-167) time.sleep(2) [](#__codelineno-0-168) [](#__codelineno-0-169) # Generate with prefix caching. [](#__codelineno-0-170) outputs = prefix_cached_llm.generate(generating_prompts, sampling_params) [](#__codelineno-0-171) [](#__codelineno-0-172) print("Results with `enable_prefix_caching`") [](#__codelineno-0-173) [](#__codelineno-0-174) cached_generated_texts = [] [](#__codelineno-0-175) # Print the outputs. You should see the same outputs as before. [](#__codelineno-0-176) print("-" * 50) [](#__codelineno-0-177) for output in outputs: [](#__codelineno-0-178) prompt = output.prompt [](#__codelineno-0-179) generated_text = output.outputs[0].text [](#__codelineno-0-180) cached_generated_texts.append(generated_text) [](#__codelineno-0-181) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-0-182) print("-" * 50) [](#__codelineno-0-183) [](#__codelineno-0-184) # Compare the results and display the speedup [](#__codelineno-0-185) generated_same = all( [](#__codelineno-0-186) regular_generated_texts[i] == cached_generated_texts[i] [](#__codelineno-0-187) for i in range(len(prompts)) [](#__codelineno-0-188) ) [](#__codelineno-0-189) print(f"Generated answers are the same: {generated_same}") [](#__codelineno-0-190) [](#__codelineno-0-191) # wait for offload kv task finished. [](#__codelineno-0-192) time.sleep(2) [](#__codelineno-0-193) [](#__codelineno-0-194) # reset prefix cache to use flexkv [](#__codelineno-0-195) prefix_cached_llm.reset_prefix_cache() [](#__codelineno-0-196) [](#__codelineno-0-197) # Generate with prefix caching. [](#__codelineno-0-198) outputs = prefix_cached_llm.generate(generating_prompts, sampling_params) [](#__codelineno-0-199) [](#__codelineno-0-200) print("Results with `flexkv`") [](#__codelineno-0-201) [](#__codelineno-0-202) flexkv_generated_texts = [] [](#__codelineno-0-203) # Print the outputs. You should see the same outputs as before. [](#__codelineno-0-204) print("-" * 50) [](#__codelineno-0-205) for output in outputs: [](#__codelineno-0-206) prompt = output.prompt [](#__codelineno-0-207) generated_text = output.outputs[0].text [](#__codelineno-0-208) flexkv_generated_texts.append(generated_text) [](#__codelineno-0-209) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-0-210) print("-" * 50) [](#__codelineno-0-211) [](#__codelineno-0-212) # Compare the results and display the speedup [](#__codelineno-0-213) generated_same = all( [](#__codelineno-0-214) regular_generated_texts[i] == flexkv_generated_texts[i] [](#__codelineno-0-215) for i in range(len(prompts)) [](#__codelineno-0-216) ) [](#__codelineno-0-217) print(f"Generated answers are the same: {generated_same}") [](#__codelineno-0-218) [](#__codelineno-0-219)[](#__codelineno-0-220)if __name__ == "__main__": [](#__codelineno-0-221) main()`` --- # page `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)[](#__codelineno-0-4)import argparse [](#__codelineno-0-5)import asyncio [](#__codelineno-0-6)import ipaddress [](#__codelineno-0-7)import itertools [](#__codelineno-0-8)import os [](#__codelineno-0-9)import urllib [](#__codelineno-0-10)import uuid [](#__codelineno-0-11)from contextlib import asynccontextmanager [](#__codelineno-0-12)from typing import Any [](#__codelineno-0-13)[](#__codelineno-0-14)import httpx [](#__codelineno-0-15)from fastapi import FastAPI, HTTPException, Request [](#__codelineno-0-16)from fastapi.responses import StreamingResponse [](#__codelineno-0-17) [](#__codelineno-0-18)[](#__codelineno-0-19)def maybe_wrap_ipv6_address(address: str) -> str: [](#__codelineno-0-20) try: [](#__codelineno-0-21) ipaddress.IPv6Address(address) [](#__codelineno-0-22) return f"[{address}]" [](#__codelineno-0-23) except ValueError: [](#__codelineno-0-24) return address [](#__codelineno-0-25) [](#__codelineno-0-26)[](#__codelineno-0-27)def make_http_path(host: str, port: int) -> str: [](#__codelineno-0-28) return f"http://{host}:{port}" [](#__codelineno-0-29) [](#__codelineno-0-30)[](#__codelineno-0-31)def prefiller_cycle(prefill_clients: list[Any]): [](#__codelineno-0-32) while True: [](#__codelineno-0-33) for prefill_client in prefill_clients: [](#__codelineno-0-34) for i in range(prefill_client["dp_size"]): [](#__codelineno-0-35) yield prefill_client, i [](#__codelineno-0-36) [](#__codelineno-0-37)[](#__codelineno-0-38)async def get_prefiller_info(prefill_clients: list, ready: asyncio.Event): [](#__codelineno-0-39) for prefill_client in prefill_clients: [](#__codelineno-0-40) while True: [](#__codelineno-0-41) try: [](#__codelineno-0-42) # Wait for prefill service to be ready [](#__codelineno-0-43) response = await prefill_client["client"].get("/health") [](#__codelineno-0-44) response.raise_for_status() [](#__codelineno-0-45) except Exception: [](#__codelineno-0-46) await asyncio.sleep(1) [](#__codelineno-0-47) continue [](#__codelineno-0-48) [](#__codelineno-0-49) response = await prefill_client["client"].get( [](#__codelineno-0-50) prefill_client["bootstrap_addr"] + "/query" [](#__codelineno-0-51) ) [](#__codelineno-0-52) response.raise_for_status() [](#__codelineno-0-53) data = response.json() [](#__codelineno-0-54) break [](#__codelineno-0-55) [](#__codelineno-0-56) for dp_rank, dp_entry in data.items(): [](#__codelineno-0-57) prefill_client["dp_engine_id"][int(dp_rank)] = dp_entry["engine_id"] [](#__codelineno-0-58) dp_size = len(data) [](#__codelineno-0-59) prefill_client["dp_size"] = dp_size [](#__codelineno-0-60) print(f"Inited prefiller {prefill_client['url']} with dp_size={dp_size}") [](#__codelineno-0-61) [](#__codelineno-0-62) ready.set() [](#__codelineno-0-63) print("All prefiller instances are ready.") [](#__codelineno-0-64) [](#__codelineno-0-65)[](#__codelineno-0-66)@asynccontextmanager [](#__codelineno-0-67)async def lifespan(app: FastAPI): [](#__codelineno-0-68) """ [](#__codelineno-0-69) Lifespan context manager to handle startup and shutdown events. [](#__codelineno-0-70) """ [](#__codelineno-0-71) # Startup: Initialize client pools for prefiller and decoder services [](#__codelineno-0-72) app.state.prefill_clients = [] [](#__codelineno-0-73) app.state.decode_clients = [] [](#__codelineno-0-74) app.state.ready = asyncio.Event() [](#__codelineno-0-75) [](#__codelineno-0-76) # Create prefill clients [](#__codelineno-0-77) for i, (url, bootstrap_port) in enumerate(global_args.prefill): [](#__codelineno-0-78) parsed_url = urllib.parse.urlparse(url) [](#__codelineno-0-79) hostname = maybe_wrap_ipv6_address(parsed_url.hostname) [](#__codelineno-0-80) app.state.prefill_clients.append( [](#__codelineno-0-81) { [](#__codelineno-0-82) "client": httpx.AsyncClient( [](#__codelineno-0-83) timeout=None, [](#__codelineno-0-84) base_url=url, [](#__codelineno-0-85) limits=httpx.Limits( [](#__codelineno-0-86) max_connections=None, [](#__codelineno-0-87) max_keepalive_connections=None, [](#__codelineno-0-88) ), [](#__codelineno-0-89) ), [](#__codelineno-0-90) "url": url, [](#__codelineno-0-91) "bootstrap_addr": make_http_path(hostname, bootstrap_port or 8998), [](#__codelineno-0-92) "dp_engine_id": {}, [](#__codelineno-0-93) } [](#__codelineno-0-94) ) [](#__codelineno-0-95) [](#__codelineno-0-96) # Create decode clients [](#__codelineno-0-97) for i, url in enumerate(global_args.decode): [](#__codelineno-0-98) parsed_url = urllib.parse.urlparse(url) [](#__codelineno-0-99) hostname = maybe_wrap_ipv6_address(parsed_url.hostname) [](#__codelineno-0-100) app.state.decode_clients.append( [](#__codelineno-0-101) { [](#__codelineno-0-102) "client": httpx.AsyncClient( [](#__codelineno-0-103) timeout=None, [](#__codelineno-0-104) base_url=url, [](#__codelineno-0-105) limits=httpx.Limits( [](#__codelineno-0-106) max_connections=None, [](#__codelineno-0-107) max_keepalive_connections=None, [](#__codelineno-0-108) ), [](#__codelineno-0-109) ), [](#__codelineno-0-110) } [](#__codelineno-0-111) ) [](#__codelineno-0-112) [](#__codelineno-0-113) asyncio.create_task(get_prefiller_info(app.state.prefill_clients, app.state.ready)) [](#__codelineno-0-114) [](#__codelineno-0-115) # Initialize round-robin iterators [](#__codelineno-0-116) app.state.prefill_iterator = prefiller_cycle(app.state.prefill_clients) [](#__codelineno-0-117) app.state.decode_iterator = itertools.cycle(range(len(app.state.decode_clients))) [](#__codelineno-0-118) [](#__codelineno-0-119) print( [](#__codelineno-0-120) f"Got {len(app.state.prefill_clients)} prefill clients " [](#__codelineno-0-121) f"and {len(app.state.decode_clients)} decode clients." [](#__codelineno-0-122) ) [](#__codelineno-0-123) [](#__codelineno-0-124) yield [](#__codelineno-0-125) [](#__codelineno-0-126) # Shutdown: Close all clients [](#__codelineno-0-127) for client_info in app.state.prefill_clients: [](#__codelineno-0-128) await client_info["client"].aclose() [](#__codelineno-0-129) [](#__codelineno-0-130) for client_info in app.state.decode_clients: [](#__codelineno-0-131) await client_info["client"].aclose() [](#__codelineno-0-132) [](#__codelineno-0-133)[](#__codelineno-0-134)# Update FastAPI app initialization to use lifespan [](#__codelineno-0-135)app = FastAPI(lifespan=lifespan) [](#__codelineno-0-136) [](#__codelineno-0-137)[](#__codelineno-0-138)def parse_args(): [](#__codelineno-0-139) parser = argparse.ArgumentParser() [](#__codelineno-0-140) [](#__codelineno-0-141) parser.add_argument("--port", type=int, default=8000) [](#__codelineno-0-142) # Always use 127.0.0.1 as localhost binds to IPv6 which is blocked on CI [](#__codelineno-0-143) parser.add_argument("--host", type=str, default="127.0.0.1") [](#__codelineno-0-144) [](#__codelineno-0-145) # For prefiller instances [](#__codelineno-0-146) parser.add_argument( [](#__codelineno-0-147) "--prefill", [](#__codelineno-0-148) nargs="+", [](#__codelineno-0-149) action="append", [](#__codelineno-0-150) dest="prefill_raw", [](#__codelineno-0-151) metavar=("URL", "bootstrap_port"), [](#__codelineno-0-152) help=( [](#__codelineno-0-153) "Prefill server URL and optional bootstrap port. " [](#__codelineno-0-154) "Can be specified multiple times. " [](#__codelineno-0-155) "Format: --prefill URL [BOOTSTRAP_PORT]. " [](#__codelineno-0-156) "BOOTSTRAP_PORT can be a port number, " [](#__codelineno-0-157) "'none', or omitted (defaults to none)." [](#__codelineno-0-158) ), [](#__codelineno-0-159) ) [](#__codelineno-0-160) [](#__codelineno-0-161) # For decoder instances [](#__codelineno-0-162) parser.add_argument( [](#__codelineno-0-163) "--decode", [](#__codelineno-0-164) nargs=1, [](#__codelineno-0-165) action="append", [](#__codelineno-0-166) dest="decode_raw", [](#__codelineno-0-167) metavar=("URL",), [](#__codelineno-0-168) help="Decode server URL. Can be specified multiple times.", [](#__codelineno-0-169) ) [](#__codelineno-0-170) [](#__codelineno-0-171) args = parser.parse_args() [](#__codelineno-0-172) args.prefill = _parse_prefill_urls(args.prefill_raw) [](#__codelineno-0-173) args.decode = _parse_decode_urls(args.decode_raw) [](#__codelineno-0-174) [](#__codelineno-0-175) return args [](#__codelineno-0-176) [](#__codelineno-0-177)[](#__codelineno-0-178)# From sglang router_args.py [](#__codelineno-0-179)def _parse_prefill_urls(prefill_list): [](#__codelineno-0-180) """Parse prefill URLs from --prefill arguments. [](#__codelineno-0-181) [](#__codelineno-0-182) Format: --prefill URL [BOOTSTRAP_PORT] [](#__codelineno-0-183) Example: [](#__codelineno-0-184) --prefill http://prefill1:8080 9000 # With bootstrap port [](#__codelineno-0-185) --prefill http://prefill2:8080 none # Explicitly no bootstrap port [](#__codelineno-0-186) --prefill http://prefill3:8080 # Defaults to no bootstrap port [](#__codelineno-0-187) """ [](#__codelineno-0-188) if not prefill_list: [](#__codelineno-0-189) return [] [](#__codelineno-0-190) [](#__codelineno-0-191) prefill_urls = [] [](#__codelineno-0-192) for prefill_args in prefill_list: [](#__codelineno-0-193) url = prefill_args[0] [](#__codelineno-0-194) [](#__codelineno-0-195) # Handle optional bootstrap port [](#__codelineno-0-196) if len(prefill_args) >= 2: [](#__codelineno-0-197) bootstrap_port_str = prefill_args[1] [](#__codelineno-0-198) # Handle 'none' as None [](#__codelineno-0-199) if bootstrap_port_str.lower() == "none": [](#__codelineno-0-200) bootstrap_port = None [](#__codelineno-0-201) else: [](#__codelineno-0-202) try: [](#__codelineno-0-203) bootstrap_port = int(bootstrap_port_str) [](#__codelineno-0-204) except ValueError as e: [](#__codelineno-0-205) raise ValueError( [](#__codelineno-0-206) f"Invalid bootstrap port: {bootstrap_port_str}. Must be a number or 'none'" # noqa: E501 [](#__codelineno-0-207) ) from e [](#__codelineno-0-208) else: [](#__codelineno-0-209) # No bootstrap port specified, default to None [](#__codelineno-0-210) bootstrap_port = None [](#__codelineno-0-211) [](#__codelineno-0-212) prefill_urls.append((url, bootstrap_port)) [](#__codelineno-0-213) [](#__codelineno-0-214) return prefill_urls [](#__codelineno-0-215) [](#__codelineno-0-216)[](#__codelineno-0-217)def _parse_decode_urls(decode_list): [](#__codelineno-0-218) """Parse decode URLs from --decode arguments. [](#__codelineno-0-219) [](#__codelineno-0-220) Format: --decode URL [](#__codelineno-0-221) Example: --decode http://decode1:8081 --decode http://decode2:8081 [](#__codelineno-0-222) """ [](#__codelineno-0-223) if not decode_list: [](#__codelineno-0-224) return [] [](#__codelineno-0-225) [](#__codelineno-0-226) # decode_list is a list of single-element lists due to nargs=1 [](#__codelineno-0-227) return [url[0] for url in decode_list] [](#__codelineno-0-228) [](#__codelineno-0-229)[](#__codelineno-0-230)def get_next_client(app, service_type: str): [](#__codelineno-0-231) """ [](#__codelineno-0-232) Get the next client in round-robin fashion. [](#__codelineno-0-233) [](#__codelineno-0-234) Args: [](#__codelineno-0-235) app: The FastAPI app instance [](#__codelineno-0-236) service_type: Either 'prefill' or 'decode' [](#__codelineno-0-237) [](#__codelineno-0-238) Returns: [](#__codelineno-0-239) The next client to use [](#__codelineno-0-240) """ [](#__codelineno-0-241) if service_type == "prefill": [](#__codelineno-0-242) return next(app.state.prefill_iterator) [](#__codelineno-0-243) elif service_type == "decode": [](#__codelineno-0-244) client_idx = next(app.state.decode_iterator) [](#__codelineno-0-245) return app.state.decode_clients[client_idx] [](#__codelineno-0-246) else: [](#__codelineno-0-247) raise ValueError(f"Unknown service type: {service_type}") [](#__codelineno-0-248) [](#__codelineno-0-249)[](#__codelineno-0-250)async def send_request_to_service( [](#__codelineno-0-251) client_info: dict, dp_rank: int, endpoint: str, req_data: dict, request_id: str [](#__codelineno-0-252)): [](#__codelineno-0-253) """ [](#__codelineno-0-254) Send a request to a service using a client from the pool. [](#__codelineno-0-255) """ [](#__codelineno-0-256) req_data = req_data.copy() [](#__codelineno-0-257) req_data["kv_transfer_params"] = { [](#__codelineno-0-258) "do_remote_decode": True, [](#__codelineno-0-259) "do_remote_prefill": False, [](#__codelineno-0-260) "transfer_id": f"xfer-{request_id}", [](#__codelineno-0-261) } [](#__codelineno-0-262) req_data["stream"] = False [](#__codelineno-0-263) req_data["max_tokens"] = 1 [](#__codelineno-0-264) if "max_completion_tokens" in req_data: [](#__codelineno-0-265) req_data["max_completion_tokens"] = 1 [](#__codelineno-0-266) if "stream_options" in req_data: [](#__codelineno-0-267) del req_data["stream_options"] [](#__codelineno-0-268) headers = { [](#__codelineno-0-269) "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", [](#__codelineno-0-270) "X-Request-Id": request_id, [](#__codelineno-0-271) "X-data-parallel-rank": str(dp_rank), [](#__codelineno-0-272) } [](#__codelineno-0-273) [](#__codelineno-0-274) response = await client_info["client"].post( [](#__codelineno-0-275) endpoint, json=req_data, headers=headers [](#__codelineno-0-276) ) [](#__codelineno-0-277) response.raise_for_status() [](#__codelineno-0-278) [](#__codelineno-0-279) # CRITICAL: Release connection back to pool [](#__codelineno-0-280) await response.aclose() [](#__codelineno-0-281) [](#__codelineno-0-282)[](#__codelineno-0-283)async def stream_service_response( [](#__codelineno-0-284) prefill_client_info: dict, [](#__codelineno-0-285) prefill_dp_rank: int, [](#__codelineno-0-286) decode_client_info: dict, [](#__codelineno-0-287) endpoint: str, [](#__codelineno-0-288) req_data: dict, [](#__codelineno-0-289) request_id: str, [](#__codelineno-0-290)): [](#__codelineno-0-291) """ [](#__codelineno-0-292) Asynchronously stream response from a service using a client from the pool. [](#__codelineno-0-293) """ [](#__codelineno-0-294) headers = { [](#__codelineno-0-295) "Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}", [](#__codelineno-0-296) "X-Request-Id": request_id, [](#__codelineno-0-297) } [](#__codelineno-0-298) [](#__codelineno-0-299) req_data["kv_transfer_params"] = { [](#__codelineno-0-300) "do_remote_decode": False, [](#__codelineno-0-301) "do_remote_prefill": True, [](#__codelineno-0-302) "remote_bootstrap_addr": prefill_client_info["bootstrap_addr"], [](#__codelineno-0-303) "remote_engine_id": prefill_client_info["dp_engine_id"][prefill_dp_rank], [](#__codelineno-0-304) "transfer_id": f"xfer-{request_id}", [](#__codelineno-0-305) } [](#__codelineno-0-306) [](#__codelineno-0-307) async with decode_client_info["client"].stream( [](#__codelineno-0-308) "POST", endpoint, json=req_data, headers=headers [](#__codelineno-0-309) ) as response: [](#__codelineno-0-310) response.raise_for_status() [](#__codelineno-0-311) async for chunk in response.aiter_bytes(): [](#__codelineno-0-312) yield chunk [](#__codelineno-0-313) [](#__codelineno-0-314)[](#__codelineno-0-315)async def _handle_completions(api: str, request: Request): [](#__codelineno-0-316) if not app.state.ready.is_set(): [](#__codelineno-0-317) raise HTTPException(status_code=503, detail="Service Unavailable") [](#__codelineno-0-318) [](#__codelineno-0-319) try: [](#__codelineno-0-320) req_data = await request.json() [](#__codelineno-0-321) request_id = str(uuid.uuid4()) [](#__codelineno-0-322) [](#__codelineno-0-323) # Get the next prefill client in round-robin fashion [](#__codelineno-0-324) prefill_client_info, prefill_dp_rank = get_next_client(request.app, "prefill") [](#__codelineno-0-325) [](#__codelineno-0-326) # Send request to prefill service [](#__codelineno-0-327) asyncio.create_task( [](#__codelineno-0-328) send_request_to_service( [](#__codelineno-0-329) prefill_client_info, prefill_dp_rank, api, req_data, request_id [](#__codelineno-0-330) ) [](#__codelineno-0-331) ) [](#__codelineno-0-332) [](#__codelineno-0-333) decode_client_info = get_next_client(request.app, "decode") [](#__codelineno-0-334) [](#__codelineno-0-335) # Stream response from decode service [](#__codelineno-0-336) async def generate_stream(): [](#__codelineno-0-337) async for chunk in stream_service_response( [](#__codelineno-0-338) prefill_client_info, [](#__codelineno-0-339) prefill_dp_rank, [](#__codelineno-0-340) decode_client_info, [](#__codelineno-0-341) api, [](#__codelineno-0-342) req_data, [](#__codelineno-0-343) request_id=request_id, [](#__codelineno-0-344) ): [](#__codelineno-0-345) yield chunk [](#__codelineno-0-346) [](#__codelineno-0-347) return StreamingResponse(generate_stream(), media_type="application/json") [](#__codelineno-0-348) [](#__codelineno-0-349) except Exception as e: [](#__codelineno-0-350) import sys [](#__codelineno-0-351) import traceback [](#__codelineno-0-352) [](#__codelineno-0-353) exc_info = sys.exc_info() [](#__codelineno-0-354) print(f"Error occurred in disagg prefill proxy server - {api} endpoint") [](#__codelineno-0-355) print(e) [](#__codelineno-0-356) print("".join(traceback.format_exception(*exc_info))) [](#__codelineno-0-357) raise [](#__codelineno-0-358) [](#__codelineno-0-359)[](#__codelineno-0-360)@app.post("/v1/completions") [](#__codelineno-0-361)async def handle_completions(request: Request): [](#__codelineno-0-362) return await _handle_completions("/v1/completions", request) [](#__codelineno-0-363) [](#__codelineno-0-364)[](#__codelineno-0-365)@app.post("/v1/chat/completions") [](#__codelineno-0-366)async def handle_chat_completions(request: Request): [](#__codelineno-0-367) return await _handle_completions("/v1/chat/completions", request) [](#__codelineno-0-368) [](#__codelineno-0-369)[](#__codelineno-0-370)if __name__ == "__main__": [](#__codelineno-0-371) global global_args [](#__codelineno-0-372) global_args = parse_args() [](#__codelineno-0-373) [](#__codelineno-0-374) import uvicorn [](#__codelineno-0-375) [](#__codelineno-0-376) uvicorn.run(app, host=global_args.host, port=global_args.port)` `[](#__codelineno-1-1)#!/bin/bash [](#__codelineno-1-2)[](#__codelineno-1-3)# ============================================================================= [](#__codelineno-1-4)# vLLM Disaggregated Serving Script for Mooncake Connector [](#__codelineno-1-5)# ============================================================================= [](#__codelineno-1-6)# This script demonstrates disaggregated prefill and decode serving using [](#__codelineno-1-7)# Mooncake Connector. [](#__codelineno-1-8)# [](#__codelineno-1-9)# Configuration can be customized via environment variables: [](#__codelineno-1-10)# MODEL: Model to serve [](#__codelineno-1-11)# PREFILL_GPUS: Comma-separated GPU IDs for prefill servers [](#__codelineno-1-12)# DECODE_GPUS: Comma-separated GPU IDs for decode servers [](#__codelineno-1-13)# PREFILL_PORTS: Comma-separated ports for prefill servers [](#__codelineno-1-14)# BOOTSTRAP_PORTS: Bootstrap server port launched by prefill servers [](#__codelineno-1-15)# DECODE_PORTS: Comma-separated ports for decode servers [](#__codelineno-1-16)# PROXY_PORT: Proxy server port used to setup P/D disaggregated connection. [](#__codelineno-1-17)# TIMEOUT_SECONDS: Server startup timeout [](#__codelineno-1-18)# ============================================================================= [](#__codelineno-1-19)[](#__codelineno-1-20)# Configuration - can be overridden via environment variables [](#__codelineno-1-21)MODEL=${MODEL:-Qwen/Qwen2.5-7B-Instruct} [](#__codelineno-1-22)TIMEOUT_SECONDS=${TIMEOUT_SECONDS:-1200} [](#__codelineno-1-23)PROXY_PORT=${PROXY_PORT:-8000} [](#__codelineno-1-24)[](#__codelineno-1-25)PREFILL_GPUS=${PREFILL_GPUS:-0} [](#__codelineno-1-26)DECODE_GPUS=${DECODE_GPUS:-1} [](#__codelineno-1-27)PREFILL_PORTS=${PREFILL_PORTS:-8010} [](#__codelineno-1-28)BOOTSTRAP_PORTS=${BOOTSTRAP_PORTS:-8998} [](#__codelineno-1-29)DECODE_PORTS=${DECODE_PORTS:-8020} [](#__codelineno-1-30)[](#__codelineno-1-31)echo "Warning: Mooncake Connector support for vLLM v1 is experimental and subject to change." [](#__codelineno-1-32)echo "" [](#__codelineno-1-33)echo "Architecture Configuration:" [](#__codelineno-1-34)echo " Model: $MODEL" [](#__codelineno-1-35)echo " Prefill GPUs: $PREFILL_GPUS, Ports: $PREFILL_PORTS, Bootstrap Port:$BOOTSTRAP_PORTS" [](#__codelineno-1-36)echo " Decode GPUs: $DECODE_GPUS, Ports: $DECODE_PORTS" [](#__codelineno-1-37)echo " Proxy Port: $PROXY_PORT" [](#__codelineno-1-38)echo " Timeout: ${TIMEOUT_SECONDS}s" [](#__codelineno-1-39)echo "" [](#__codelineno-1-40)[](#__codelineno-1-41)PIDS=() [](#__codelineno-1-42)[](#__codelineno-1-43)# Switch to the directory of the current script [](#__codelineno-1-44)cd "$(dirname "${BASH_SOURCE[0]}")" [](#__codelineno-1-45)[](#__codelineno-1-46)check_required_files() { [](#__codelineno-1-47) local files=("mooncake_connector_proxy.py") [](#__codelineno-1-48) for file in "${files[@]}"; do [](#__codelineno-1-49) if [[ ! -f "$file" ]]; then [](#__codelineno-1-50) echo "Required file $file not found in $(pwd)" [](#__codelineno-1-51) exit 1 [](#__codelineno-1-52) fi [](#__codelineno-1-53) done [](#__codelineno-1-54)} [](#__codelineno-1-55)[](#__codelineno-1-56)check_hf_token() { [](#__codelineno-1-57) if [ -z "$HF_TOKEN" ]; then [](#__codelineno-1-58) echo "HF_TOKEN is not set. Please set it to your Hugging Face token." [](#__codelineno-1-59) echo "Example: export HF_TOKEN=your_token_here" [](#__codelineno-1-60) exit 1 [](#__codelineno-1-61) fi [](#__codelineno-1-62) if [[ "$HF_TOKEN" != hf_* ]]; then [](#__codelineno-1-63) echo "HF_TOKEN is not a valid Hugging Face token. Please set it to your Hugging Face token." [](#__codelineno-1-64) exit 1 [](#__codelineno-1-65) fi [](#__codelineno-1-66) echo "HF_TOKEN is set and valid." [](#__codelineno-1-67)} [](#__codelineno-1-68)[](#__codelineno-1-69)check_num_gpus() { [](#__codelineno-1-70) # Check if the number of GPUs are >=2 via nvidia-smi [](#__codelineno-1-71) num_gpus=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l) [](#__codelineno-1-72) if [ "$num_gpus" -lt 2 ]; then [](#__codelineno-1-73) echo "You need at least 2 GPUs to run disaggregated prefill." [](#__codelineno-1-74) exit 1 [](#__codelineno-1-75) else [](#__codelineno-1-76) echo "Found $num_gpus GPUs." [](#__codelineno-1-77) fi [](#__codelineno-1-78)} [](#__codelineno-1-79)[](#__codelineno-1-80)ensure_python_library_installed() { [](#__codelineno-1-81) echo "Checking if $1 is installed..." [](#__codelineno-1-82) if ! python3 -c "import $1" > /dev/null 2>&1; then [](#__codelineno-1-83) echo "$1 is not installed. Please install it via pip install $1." [](#__codelineno-1-84) exit 1 [](#__codelineno-1-85) else [](#__codelineno-1-86) echo "$1 is installed." [](#__codelineno-1-87) fi [](#__codelineno-1-88)} [](#__codelineno-1-89)[](#__codelineno-1-90)cleanup() { [](#__codelineno-1-91) echo "Stopping everything…" [](#__codelineno-1-92) trap - INT TERM # prevent re-entrancy [](#__codelineno-1-93) pkill -9 -f "mooncake_connector_proxy.py" [](#__codelineno-1-94) kill -- -$$ # negative PID == "this whole process-group" [](#__codelineno-1-95) wait # reap children so we don't leave zombies [](#__codelineno-1-96) exit 0 [](#__codelineno-1-97)} [](#__codelineno-1-98)[](#__codelineno-1-99)wait_for_server() { [](#__codelineno-1-100) local port=$1 [](#__codelineno-1-101) local timeout_seconds=$TIMEOUT_SECONDS [](#__codelineno-1-102) local start_time=$(date +%s) [](#__codelineno-1-103) [](#__codelineno-1-104) echo "Waiting for server on port $port..." [](#__codelineno-1-105) [](#__codelineno-1-106) while true; do [](#__codelineno-1-107) if curl -s "localhost:${port}/v1/completions" > /dev/null; then [](#__codelineno-1-108) echo "Server on port $port is ready." [](#__codelineno-1-109) return 0 [](#__codelineno-1-110) fi [](#__codelineno-1-111) [](#__codelineno-1-112) local now=$(date +%s) [](#__codelineno-1-113) if (( now - start_time >= timeout_seconds )); then [](#__codelineno-1-114) echo "Timeout waiting for server on port $port" [](#__codelineno-1-115) return 1 [](#__codelineno-1-116) fi [](#__codelineno-1-117) [](#__codelineno-1-118) sleep 1 [](#__codelineno-1-119) done [](#__codelineno-1-120)} [](#__codelineno-1-121)[](#__codelineno-1-122)main() { [](#__codelineno-1-123) check_required_files [](#__codelineno-1-124) check_hf_token [](#__codelineno-1-125) check_num_gpus [](#__codelineno-1-126) ensure_python_library_installed vllm [](#__codelineno-1-127) ensure_python_library_installed mooncake.engine [](#__codelineno-1-128) [](#__codelineno-1-129) trap cleanup INT [](#__codelineno-1-130) trap cleanup USR1 [](#__codelineno-1-131) trap cleanup TERM [](#__codelineno-1-132) [](#__codelineno-1-133) echo "Launching disaggregated serving components..." [](#__codelineno-1-134) echo "Please check the log files for detailed output:" [](#__codelineno-1-135) echo " - prefill*.log: Prefill server logs" [](#__codelineno-1-136) echo " - decode*.log: Decode server logs" [](#__codelineno-1-137) echo " - proxy.log: Proxy server log" [](#__codelineno-1-138) [](#__codelineno-1-139) # Parse GPU and port arrays [](#__codelineno-1-140) IFS=',' read -ra PREFILL_GPU_ARRAY <<< "$PREFILL_GPUS" [](#__codelineno-1-141) IFS=',' read -ra DECODE_GPU_ARRAY <<< "$DECODE_GPUS" [](#__codelineno-1-142) IFS=',' read -ra PREFILL_PORT_ARRAY <<< "$PREFILL_PORTS" [](#__codelineno-1-143) IFS=',' read -ra BOOTSTRAP_PORT_ARRAY <<< "$BOOTSTRAP_PORTS" [](#__codelineno-1-144) IFS=',' read -ra DECODE_PORT_ARRAY <<< "$DECODE_PORTS" [](#__codelineno-1-145) [](#__codelineno-1-146) proxy_args=() [](#__codelineno-1-147) [](#__codelineno-1-148) # ============================================================================= [](#__codelineno-1-149) # Launch Prefill Servers (X Producers) [](#__codelineno-1-150) # ============================================================================= [](#__codelineno-1-151) echo "" [](#__codelineno-1-152) echo "Starting ${#PREFILL_GPU_ARRAY[@]} prefill server(s)..." [](#__codelineno-1-153) for i in "${!PREFILL_GPU_ARRAY[@]}"; do [](#__codelineno-1-154) local gpu_id=${PREFILL_GPU_ARRAY[$i]} [](#__codelineno-1-155) local port=${PREFILL_PORT_ARRAY[$i]} [](#__codelineno-1-156) local bootstrap_port=${BOOTSTRAP_PORT_ARRAY[$i]} [](#__codelineno-1-157) [](#__codelineno-1-158) echo " Prefill server $((i+1)): GPU $gpu_id, Port $port, Bootstrap Port $bootstrap_port" [](#__codelineno-1-159) VLLM_MOONCAKE_BOOTSTRAP_PORT=$bootstrap_port CUDA_VISIBLE_DEVICES=$gpu_id vllm serve "$MODEL" \ [](#__codelineno-1-160) --port "$port" \ [](#__codelineno-1-161) --kv-transfer-config \ [](#__codelineno-1-162) "{\"kv_connector\":\"MooncakeConnector\",\"kv_role\":\"kv_producer\"}" > prefill$((i+1)).log 2>&1 & [](#__codelineno-1-163) PIDS+=($!) [](#__codelineno-1-164) proxy_args+=(--prefill "http://0.0.0.0:${port}" "$bootstrap_port") [](#__codelineno-1-165) done [](#__codelineno-1-166) [](#__codelineno-1-167) # ============================================================================= [](#__codelineno-1-168) # Launch Decode Servers (Y Decoders) [](#__codelineno-1-169) # ============================================================================= [](#__codelineno-1-170) echo "" [](#__codelineno-1-171) echo "Starting ${#DECODE_GPU_ARRAY[@]} decode server(s)..." [](#__codelineno-1-172) for i in "${!DECODE_GPU_ARRAY[@]}"; do [](#__codelineno-1-173) local gpu_id=${DECODE_GPU_ARRAY[$i]} [](#__codelineno-1-174) local port=${DECODE_PORT_ARRAY[$i]} [](#__codelineno-1-175) [](#__codelineno-1-176) echo " Decode server $((i+1)): GPU $gpu_id, Port $port" [](#__codelineno-1-177) CUDA_VISIBLE_DEVICES=$gpu_id vllm serve "$MODEL" \ [](#__codelineno-1-178) --port "$port" \ [](#__codelineno-1-179) --kv-transfer-config \ [](#__codelineno-1-180) "{\"kv_connector\":\"MooncakeConnector\",\"kv_role\":\"kv_consumer\"}" > decode$((i+1)).log 2>&1 & [](#__codelineno-1-181) PIDS+=($!) [](#__codelineno-1-182) proxy_args+=(--decode "http://0.0.0.0:${port}") [](#__codelineno-1-183) done [](#__codelineno-1-184) [](#__codelineno-1-185) # ============================================================================= [](#__codelineno-1-186) # Launch Proxy Server [](#__codelineno-1-187) # ============================================================================= [](#__codelineno-1-188) echo "" [](#__codelineno-1-189) echo "Starting proxy server on port $PROXY_PORT..." [](#__codelineno-1-190) python3 mooncake_connector_proxy.py "${proxy_args[@]}" --port "$PROXY_PORT" > proxy.log 2>&1 & [](#__codelineno-1-191) PIDS+=($!) [](#__codelineno-1-192) [](#__codelineno-1-193) # ============================================================================= [](#__codelineno-1-194) # Wait for All Servers to Start [](#__codelineno-1-195) # ============================================================================= [](#__codelineno-1-196) echo "" [](#__codelineno-1-197) echo "Waiting for all servers to start..." [](#__codelineno-1-198) for port in "${PREFILL_PORT_ARRAY[@]}" "${DECODE_PORT_ARRAY[@]}"; do [](#__codelineno-1-199) if ! wait_for_server "$port"; then [](#__codelineno-1-200) echo "Failed to start server on port $port" [](#__codelineno-1-201) cleanup [](#__codelineno-1-202) # shellcheck disable=SC2317 [](#__codelineno-1-203) exit 1 [](#__codelineno-1-204) fi [](#__codelineno-1-205) done [](#__codelineno-1-206) [](#__codelineno-1-207) echo "" [](#__codelineno-1-208) echo "All servers are up. Starting benchmark..." [](#__codelineno-1-209) [](#__codelineno-1-210) # ============================================================================= [](#__codelineno-1-211) # Run Benchmark [](#__codelineno-1-212) # ============================================================================= [](#__codelineno-1-213) vllm bench serve --port "$PROXY_PORT" --seed "$(date +%s)" \ [](#__codelineno-1-214) --backend vllm --model "$MODEL" \ [](#__codelineno-1-215) --dataset-name random --random-input-len 7500 --random-output-len 200 \ [](#__codelineno-1-216) --num-prompts 200 --burstiness 100 --request-rate 2 | tee benchmark.log [](#__codelineno-1-217) [](#__codelineno-1-218) echo "Benchmarking done. Cleaning up..." [](#__codelineno-1-219) [](#__codelineno-1-220) cleanup [](#__codelineno-1-221)} [](#__codelineno-1-222)[](#__codelineno-1-223)main` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/disaggregated/lmcache.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/lmcache](https://github.com/vllm-project/vllm/tree/main/examples/disaggregated/lmcache). This folder demonstrates how to use LMCache for disaggregated prefilling, CPU offloading and KV cache sharing. ## 1\. Disaggregated Prefill in vLLM v1[¶](#1-disaggregated-prefill-in-vllm-v1 "Permanent link") This example demonstrates how to run LMCache with disaggregated prefill using NIXL on a single node. ### Prerequisites[¶](#prerequisites "Permanent link") - Install [LMCache](https://github.com/LMCache/LMCache). You can simply run `pip install lmcache`. - Install [NIXL](https://github.com/ai-dynamo/nixl). - At least 2 GPUs - Valid Hugging Face token (HF\_TOKEN) for Llama 3.1 8B Instruct. ### Usage[¶](#usage "Permanent link") Run `cd disagg_prefill_lmcache_v1` to get into `disagg_prefill_lmcache_v1` folder, and then run `[](#__codelineno-0-1)bash disagg_example_nixl.sh` to run disaggregated prefill and benchmark the performance. ### Components[¶](#components "Permanent link") #### Server Scripts[¶](#server-scripts "Permanent link") - `disagg_prefill_lmcache_v1/disagg_vllm_launcher.sh` - Launches individual vLLM servers for prefill/decode, and also launches the proxy server. - `disagg_prefill_lmcache_v1/disagg_proxy_server.py` - FastAPI proxy server that coordinates between prefiller and decoder - `disagg_prefill_lmcache_v1/disagg_example_nixl.sh` - Main script to run the example #### Configuration[¶](#configuration "Permanent link") - `disagg_prefill_lmcache_v1/configs/lmcache-prefiller-config.yaml` - Configuration for prefiller server - `disagg_prefill_lmcache_v1/configs/lmcache-decoder-config.yaml` - Configuration for decoder server #### Log Files[¶](#log-files "Permanent link") The main script generates several log files: - `prefiller.log` - Logs from the prefill server - `decoder.log` - Logs from the decode server - `proxy.log` - Logs from the proxy server ## 2\. CPU Offload Examples[¶](#2-cpu-offload-examples "Permanent link") - `python cpu_offload_lmcache.py -v v0` - CPU offloading implementation for vLLM v0 - `python cpu_offload_lmcache.py -v v1` - CPU offloading implementation for vLLM v1 ## 3\. KV Cache Sharing[¶](#3-kv-cache-sharing "Permanent link") The `kv_cache_sharing_lmcache_v1.py` example demonstrates how to share KV caches between vLLM v1 instances. ## 4\. Disaggregated Prefill in vLLM v0[¶](#4-disaggregated-prefill-in-vllm-v0 "Permanent link") The `disaggregated_prefill_lmcache_v0.py` provides an example of how to run disaggregated prefill in vLLM v0. ## Example materials[¶](#example-materials "Permanent link") cpu\_offload\_lmcache.py ``[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)This file demonstrates the example usage of cpu offloading [](#__codelineno-1-5)with LMCache in vLLM v1 or v0. [](#__codelineno-1-6)[](#__codelineno-1-7)Usage: [](#__codelineno-1-8) [](#__codelineno-1-9) Specify vLLM version [](#__codelineno-1-10) [](#__codelineno-1-11) -v v0 : Use LMCacheConnector [](#__codelineno-1-12) model = mistralai/Mistral-7B-Instruct-v0.2 [](#__codelineno-1-13) (Includes enable_chunked_prefill = True) [](#__codelineno-1-14) [](#__codelineno-1-15) -v v1 : Use LMCacheConnectorV1 (default) [](#__codelineno-1-16) model = meta-llama/Meta-Llama-3.1-8B-Instruct [](#__codelineno-1-17) (Without enable_chunked_prefill) [](#__codelineno-1-18)[](#__codelineno-1-19)Note that `lmcache` is needed to run this example. [](#__codelineno-1-20)Requirements: [](#__codelineno-1-21)https://docs.lmcache.ai/getting_started/installation.html#prerequisites [](#__codelineno-1-22)Learn more about LMCache environment setup, please refer to: [](#__codelineno-1-23)https://docs.lmcache.ai/getting_started/installation.html [](#__codelineno-1-24)""" [](#__codelineno-1-25)[](#__codelineno-1-26)import argparse [](#__codelineno-1-27)import contextlib [](#__codelineno-1-28)import os [](#__codelineno-1-29)import time [](#__codelineno-1-30)from dataclasses import asdict [](#__codelineno-1-31)[](#__codelineno-1-32)from lmcache.integration.vllm.utils import ENGINE_NAME [](#__codelineno-1-33)from lmcache.v1.cache_engine import LMCacheEngineBuilder [](#__codelineno-1-34)[](#__codelineno-1-35)from vllm import LLM, SamplingParams [](#__codelineno-1-36)from vllm.config import KVTransferConfig [](#__codelineno-1-37)from vllm.engine.arg_utils import EngineArgs [](#__codelineno-1-38) [](#__codelineno-1-39)[](#__codelineno-1-40)def setup_environment_variables(): [](#__codelineno-1-41) # LMCache-related environment variables [](#__codelineno-1-42) # Use experimental features in LMCache [](#__codelineno-1-43) os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True" [](#__codelineno-1-44) # LMCache is set to use 256 tokens per chunk [](#__codelineno-1-45) os.environ["LMCACHE_CHUNK_SIZE"] = "256" [](#__codelineno-1-46) # Enable local CPU backend in LMCache [](#__codelineno-1-47) os.environ["LMCACHE_LOCAL_CPU"] = "True" [](#__codelineno-1-48) # Set local CPU memory limit to 5.0 GB [](#__codelineno-1-49) os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0" [](#__codelineno-1-50) [](#__codelineno-1-51)[](#__codelineno-1-52)@contextlib.contextmanager [](#__codelineno-1-53)def build_llm_with_lmcache(lmcache_connector: str, model: str): [](#__codelineno-1-54) ktc = KVTransferConfig( [](#__codelineno-1-55) kv_connector=lmcache_connector, [](#__codelineno-1-56) kv_role="kv_both", [](#__codelineno-1-57) ) [](#__codelineno-1-58) # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB [](#__codelineno-1-59) # memory. Reduce the value if your GPU has less memory. [](#__codelineno-1-60) # Note: LMCache supports chunked prefill (see vLLM#14505, LMCache#392). [](#__codelineno-1-61) llm_args = EngineArgs( [](#__codelineno-1-62) model=model, [](#__codelineno-1-63) kv_transfer_config=ktc, [](#__codelineno-1-64) max_model_len=8000, [](#__codelineno-1-65) gpu_memory_utilization=0.8, [](#__codelineno-1-66) ) [](#__codelineno-1-67) [](#__codelineno-1-68) llm = LLM(**asdict(llm_args)) [](#__codelineno-1-69) try: [](#__codelineno-1-70) yield llm [](#__codelineno-1-71) finally: [](#__codelineno-1-72) # Clean up lmcache backend [](#__codelineno-1-73) LMCacheEngineBuilder.destroy(ENGINE_NAME) [](#__codelineno-1-74) [](#__codelineno-1-75)[](#__codelineno-1-76)def print_output( [](#__codelineno-1-77) llm: LLM, [](#__codelineno-1-78) prompt: list[str], [](#__codelineno-1-79) sampling_params: SamplingParams, [](#__codelineno-1-80) req_str: str, [](#__codelineno-1-81)): [](#__codelineno-1-82) # Should be able to see logs like the following: [](#__codelineno-1-83) # `LMCache INFO: Storing KV cache for 6006 out of 6006 tokens for request 0` [](#__codelineno-1-84) # This indicates that the KV cache has been stored in LMCache. [](#__codelineno-1-85) start = time.time() [](#__codelineno-1-86) outputs = llm.generate(prompt, sampling_params) [](#__codelineno-1-87) print("-" * 50) [](#__codelineno-1-88) for output in outputs: [](#__codelineno-1-89) generated_text = output.outputs[0].text [](#__codelineno-1-90) print(f"Generated text: {generated_text!r}") [](#__codelineno-1-91) print(f"Generation took {time.time() - start:.2f} seconds, {req_str} request done.") [](#__codelineno-1-92) print("-" * 50) [](#__codelineno-1-93) [](#__codelineno-1-94)[](#__codelineno-1-95)def parse_args(): [](#__codelineno-1-96) parser = argparse.ArgumentParser() [](#__codelineno-1-97) parser.add_argument( [](#__codelineno-1-98) "-v", [](#__codelineno-1-99) "--version", [](#__codelineno-1-100) choices=["v0", "v1"], [](#__codelineno-1-101) default="v1", [](#__codelineno-1-102) help="Specify vLLM version (default: v1)", [](#__codelineno-1-103) ) [](#__codelineno-1-104) return parser.parse_args() [](#__codelineno-1-105) [](#__codelineno-1-106)[](#__codelineno-1-107)def main(): [](#__codelineno-1-108) lmcache_connector = "LMCacheConnectorV1" [](#__codelineno-1-109) model = "meta-llama/Meta-Llama-3.1-8B-Instruct" [](#__codelineno-1-110) setup_environment_variables() [](#__codelineno-1-111) with build_llm_with_lmcache(lmcache_connector, model) as llm: [](#__codelineno-1-112) # This example script runs two requests with a shared prefix. [](#__codelineno-1-113) # Define the shared prompt and specific prompts [](#__codelineno-1-114) shared_prompt = "Hello, how are you?" * 1000 [](#__codelineno-1-115) first_prompt = [ [](#__codelineno-1-116) shared_prompt + "Hello, my name is", [](#__codelineno-1-117) ] [](#__codelineno-1-118) second_prompt = [ [](#__codelineno-1-119) shared_prompt + "Tell me a very long story", [](#__codelineno-1-120) ] [](#__codelineno-1-121) [](#__codelineno-1-122) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) [](#__codelineno-1-123) [](#__codelineno-1-124) # Print the first output [](#__codelineno-1-125) print_output(llm, first_prompt, sampling_params, "first") [](#__codelineno-1-126) [](#__codelineno-1-127) time.sleep(1) [](#__codelineno-1-128) [](#__codelineno-1-129) # print the second output [](#__codelineno-1-130) print_output(llm, second_prompt, sampling_params, "second") [](#__codelineno-1-131) [](#__codelineno-1-132)[](#__codelineno-1-133)if __name__ == "__main__": [](#__codelineno-1-134) main()`` disagg\_prefill\_lmcache\_v0.py ``[](#__codelineno-2-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-2-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-2-3)""" [](#__codelineno-2-4)This file demonstrates the example usage of disaggregated prefilling [](#__codelineno-2-5)with LMCache. [](#__codelineno-2-6)We will launch 2 vllm instances (GPU 0 for prefill and GPU 1 for decode), [](#__codelineno-2-7)and launch an additional LMCache server. [](#__codelineno-2-8)KV cache is transferred in the following manner: [](#__codelineno-2-9)vLLM prefill node -> LMCache server -> vLLM decode node. [](#__codelineno-2-10)[](#__codelineno-2-11)Note that `pip install lmcache` is needed to run this example. [](#__codelineno-2-12)Learn more about LMCache in https://github.com/LMCache/LMCache. [](#__codelineno-2-13)""" [](#__codelineno-2-14)[](#__codelineno-2-15)import os [](#__codelineno-2-16)import subprocess [](#__codelineno-2-17)import time [](#__codelineno-2-18)from multiprocessing import Event, Process [](#__codelineno-2-19)[](#__codelineno-2-20)from lmcache.experimental.cache_engine import LMCacheEngineBuilder [](#__codelineno-2-21)from lmcache.integration.vllm.utils import ENGINE_NAME [](#__codelineno-2-22)[](#__codelineno-2-23)from vllm import LLM, SamplingParams [](#__codelineno-2-24)from vllm.config import KVTransferConfig [](#__codelineno-2-25)[](#__codelineno-2-26)# LMCache-related environment variables [](#__codelineno-2-27)# The port to start LMCache server [](#__codelineno-2-28)port = 8100 [](#__codelineno-2-29)# Use experimental features in LMCache [](#__codelineno-2-30)os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True" [](#__codelineno-2-31)# LMCache is set to use 256 tokens per chunk [](#__codelineno-2-32)os.environ["LMCACHE_CHUNK_SIZE"] = "256" [](#__codelineno-2-33)# Disable local CPU backend in LMCache [](#__codelineno-2-34)os.environ["LMCACHE_LOCAL_CPU"] = "False" [](#__codelineno-2-35)# Set local CPU memory buffer limit to 5.0 GB [](#__codelineno-2-36)os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0" [](#__codelineno-2-37)# Set the remote URL for LMCache server [](#__codelineno-2-38)os.environ["LMCACHE_REMOTE_URL"] = f"lm://localhost:{port}" [](#__codelineno-2-39)# Set the serializer/deserializer between vllm and LMCache server [](#__codelineno-2-40)# `naive` indicates using raw bytes of the tensor without any compression [](#__codelineno-2-41)os.environ["LMCACHE_REMOTE_SERDE"] = "naive" [](#__codelineno-2-42)[](#__codelineno-2-43)prompts = [ [](#__codelineno-2-44) "Hello, how are you?" * 1000, [](#__codelineno-2-45)] [](#__codelineno-2-46) [](#__codelineno-2-47)[](#__codelineno-2-48)def run_prefill(prefill_done, prompts): [](#__codelineno-2-49) # We use GPU 0 for prefill node. [](#__codelineno-2-50) os.environ["CUDA_VISIBLE_DEVICES"] = "0" [](#__codelineno-2-51) [](#__codelineno-2-52) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=1) [](#__codelineno-2-53) [](#__codelineno-2-54) ktc = KVTransferConfig( [](#__codelineno-2-55) kv_connector="LMCacheConnector", [](#__codelineno-2-56) kv_role="kv_producer", [](#__codelineno-2-57) kv_rank=0, [](#__codelineno-2-58) kv_parallel_size=2, [](#__codelineno-2-59) ) [](#__codelineno-2-60) # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB [](#__codelineno-2-61) # memory. Reduce the value if your GPU has less memory. [](#__codelineno-2-62) llm = LLM( [](#__codelineno-2-63) model="mistralai/Mistral-7B-Instruct-v0.2", [](#__codelineno-2-64) kv_transfer_config=ktc, [](#__codelineno-2-65) max_model_len=8000, [](#__codelineno-2-66) gpu_memory_utilization=0.8, [](#__codelineno-2-67) enforce_eager=True, [](#__codelineno-2-68) ) [](#__codelineno-2-69) [](#__codelineno-2-70) # llm.generate(prompts, sampling_params) [](#__codelineno-2-71) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-2-72) for output in outputs: [](#__codelineno-2-73) generated_text = output.outputs[0].text [](#__codelineno-2-74) print(f"Generated text: {generated_text!r}") [](#__codelineno-2-75) print("Prefill node is finished.") [](#__codelineno-2-76) prefill_done.set() [](#__codelineno-2-77) [](#__codelineno-2-78) # Clean up lmcache backend [](#__codelineno-2-79) LMCacheEngineBuilder.destroy(ENGINE_NAME) [](#__codelineno-2-80) [](#__codelineno-2-81)[](#__codelineno-2-82)def run_decode(prefill_done, prompts, timeout=1): [](#__codelineno-2-83) # We use GPU 1 for decode node. [](#__codelineno-2-84) os.environ["CUDA_VISIBLE_DEVICES"] = "1" [](#__codelineno-2-85) [](#__codelineno-2-86) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) [](#__codelineno-2-87) [](#__codelineno-2-88) ktc = KVTransferConfig( [](#__codelineno-2-89) kv_connector="LMCacheConnector", [](#__codelineno-2-90) kv_role="kv_consumer", [](#__codelineno-2-91) kv_rank=1, [](#__codelineno-2-92) kv_parallel_size=2, [](#__codelineno-2-93) ) [](#__codelineno-2-94) # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB [](#__codelineno-2-95) # of memory. Reduce the value if your GPU has less memory. [](#__codelineno-2-96) llm = LLM( [](#__codelineno-2-97) model="mistralai/Mistral-7B-Instruct-v0.2", [](#__codelineno-2-98) kv_transfer_config=ktc, [](#__codelineno-2-99) max_model_len=8000, [](#__codelineno-2-100) gpu_memory_utilization=0.8, [](#__codelineno-2-101) enforce_eager=True, [](#__codelineno-2-102) ) [](#__codelineno-2-103) [](#__codelineno-2-104) print("Waiting for prefill node to finish...") [](#__codelineno-2-105) prefill_done.wait() [](#__codelineno-2-106) time.sleep(timeout) [](#__codelineno-2-107) [](#__codelineno-2-108) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-2-109) for output in outputs: [](#__codelineno-2-110) generated_text = output.outputs[0].text [](#__codelineno-2-111) print(f"Generated text: {generated_text!r}") [](#__codelineno-2-112) [](#__codelineno-2-113) # Clean up lmcache backend [](#__codelineno-2-114) LMCacheEngineBuilder.destroy(ENGINE_NAME) [](#__codelineno-2-115) [](#__codelineno-2-116)[](#__codelineno-2-117)def run_lmcache_server(port): [](#__codelineno-2-118) server_proc = subprocess.Popen( [](#__codelineno-2-119) ["python", "-m", "lmcache.experimental.server", "localhost", str(port)] [](#__codelineno-2-120) ) [](#__codelineno-2-121) return server_proc [](#__codelineno-2-122) [](#__codelineno-2-123)[](#__codelineno-2-124)def main(): [](#__codelineno-2-125) prefill_done = Event() [](#__codelineno-2-126) prefill_process = Process(target=run_prefill, args=(prefill_done, prompts)) [](#__codelineno-2-127) decode_process = Process(target=run_decode, args=(prefill_done, prompts)) [](#__codelineno-2-128) lmcache_server_process = run_lmcache_server(port) [](#__codelineno-2-129) [](#__codelineno-2-130) # Start prefill node [](#__codelineno-2-131) prefill_process.start() [](#__codelineno-2-132) [](#__codelineno-2-133) # Start decode node [](#__codelineno-2-134) decode_process.start() [](#__codelineno-2-135) [](#__codelineno-2-136) # Clean up the processes [](#__codelineno-2-137) decode_process.join() [](#__codelineno-2-138) prefill_process.terminate() [](#__codelineno-2-139) lmcache_server_process.terminate() [](#__codelineno-2-140) lmcache_server_process.wait() [](#__codelineno-2-141) [](#__codelineno-2-142)[](#__codelineno-2-143)if __name__ == "__main__": [](#__codelineno-2-144) main()`` disagg\_prefill\_lmcache\_v1/configs/lmcache-decoder-config.yaml `[](#__codelineno-3-1)local_cpu: False [](#__codelineno-3-2)max_local_cpu_size: 0 [](#__codelineno-3-3)#local_disk: [](#__codelineno-3-4)max_local_disk_size: 0 [](#__codelineno-3-5)remote_serde: NULL [](#__codelineno-3-6)[](#__codelineno-3-7)enable_nixl: True [](#__codelineno-3-8)nixl_role: "receiver" [](#__codelineno-3-9)nixl_peer_host: "localhost" [](#__codelineno-3-10)nixl_peer_port: 55555 [](#__codelineno-3-11)nixl_buffer_size: 1073741824 # 1GB [](#__codelineno-3-12)nixl_buffer_device: "cuda" [](#__codelineno-3-13)nixl_enable_gc: True` disagg\_prefill\_lmcache\_v1/configs/lmcache-prefiller-config.yaml `[](#__codelineno-4-1)local_cpu: False [](#__codelineno-4-2)max_local_cpu_size: 0 [](#__codelineno-4-3)#local_disk: [](#__codelineno-4-4)max_local_disk_size: 0 [](#__codelineno-4-5)remote_serde: NULL [](#__codelineno-4-6)[](#__codelineno-4-7)enable_nixl: True [](#__codelineno-4-8)nixl_role: "sender" [](#__codelineno-4-9)nixl_peer_host: "localhost" [](#__codelineno-4-10)nixl_peer_port: 55555 [](#__codelineno-4-11)nixl_buffer_size: 1073741824 # 1GB [](#__codelineno-4-12)nixl_buffer_device: "cuda" [](#__codelineno-4-13)nixl_enable_gc: True` disagg\_prefill\_lmcache\_v1/disagg\_example\_nixl.sh `[](#__codelineno-5-1)#!/bin/bash [](#__codelineno-5-2)[](#__codelineno-5-3)echo "Warning: LMCache disaggregated prefill support for vLLM v1 is experimental and subject to change." [](#__codelineno-5-4) [](#__codelineno-5-5)[](#__codelineno-5-6)PIDS=() [](#__codelineno-5-7)[](#__codelineno-5-8)# Switch to the directory of the current script [](#__codelineno-5-9)cd "$(dirname "${BASH_SOURCE[0]}")" [](#__codelineno-5-10)[](#__codelineno-5-11)check_hf_token() { [](#__codelineno-5-12) if [ -z "$HF_TOKEN" ]; then [](#__codelineno-5-13) echo "HF_TOKEN is not set. Please set it to your Hugging Face token." [](#__codelineno-5-14) exit 1 [](#__codelineno-5-15) fi [](#__codelineno-5-16) if [[ "$HF_TOKEN" != hf_* ]]; then [](#__codelineno-5-17) echo "HF_TOKEN is not a valid Hugging Face token. Please set it to your Hugging Face token." [](#__codelineno-5-18) exit 1 [](#__codelineno-5-19) fi [](#__codelineno-5-20) echo "HF_TOKEN is set and valid." [](#__codelineno-5-21)} [](#__codelineno-5-22)[](#__codelineno-5-23)check_num_gpus() { [](#__codelineno-5-24) # can you check if the number of GPUs are >=2 via nvidia-smi/rocm-smi? [](#__codelineno-5-25) if ! which rocm-smi > /dev/null 2>&1; then [](#__codelineno-5-26) num_gpus=$(nvidia-smi --query-gpu=name --format=csv,noheader | wc -l) [](#__codelineno-5-27) else [](#__codelineno-5-28) num_gpus=$(rocm-smi --showid | grep -c Instinct) [](#__codelineno-5-29) fi [](#__codelineno-5-30) [](#__codelineno-5-31) if [ "$num_gpus" -lt 2 ]; then [](#__codelineno-5-32) echo "You need at least 2 GPUs to run disaggregated prefill." [](#__codelineno-5-33) exit 1 [](#__codelineno-5-34) else [](#__codelineno-5-35) echo "Found $num_gpus GPUs." [](#__codelineno-5-36) fi [](#__codelineno-5-37)} [](#__codelineno-5-38)[](#__codelineno-5-39)ensure_python_library_installed() { [](#__codelineno-5-40) echo "Checking if $1 is installed..." [](#__codelineno-5-41) if ! python3 -c "import $1" > /dev/null 2>&1; then [](#__codelineno-5-42) if [ "$1" == "nixl" ]; then [](#__codelineno-5-43) echo "$1 is not installed. Please refer to https://github.com/ai-dynamo/nixl for installation." [](#__codelineno-5-44) else [](#__codelineno-5-45) echo "$1 is not installed. Please install it via pip install $1." [](#__codelineno-5-46) fi [](#__codelineno-5-47) exit 1 [](#__codelineno-5-48) else [](#__codelineno-5-49) echo "$1 is installed." [](#__codelineno-5-50) fi [](#__codelineno-5-51)} [](#__codelineno-5-52)[](#__codelineno-5-53)cleanup() { [](#__codelineno-5-54) echo "Stopping everything…" [](#__codelineno-5-55) trap - INT TERM # prevent re-entrancy [](#__codelineno-5-56) kill -- -$$ # negative PID == “this whole process-group” [](#__codelineno-5-57) wait # reap children so we don't leave zombies [](#__codelineno-5-58) exit 0 [](#__codelineno-5-59)} [](#__codelineno-5-60)[](#__codelineno-5-61)wait_for_server() { [](#__codelineno-5-62) local port=$1 [](#__codelineno-5-63) local timeout_seconds=1200 [](#__codelineno-5-64) local start_time=$(date +%s) [](#__codelineno-5-65) [](#__codelineno-5-66) echo "Waiting for server on port $port..." [](#__codelineno-5-67) [](#__codelineno-5-68) while true; do [](#__codelineno-5-69) if curl -s "localhost:${port}/v1/completions" > /dev/null; then [](#__codelineno-5-70) return 0 [](#__codelineno-5-71) fi [](#__codelineno-5-72) [](#__codelineno-5-73) local now=$(date +%s) [](#__codelineno-5-74) if (( now - start_time >= timeout_seconds )); then [](#__codelineno-5-75) echo "Timeout waiting for server" [](#__codelineno-5-76) return 1 [](#__codelineno-5-77) fi [](#__codelineno-5-78) [](#__codelineno-5-79) sleep 1 [](#__codelineno-5-80) done [](#__codelineno-5-81)} [](#__codelineno-5-82) [](#__codelineno-5-83)[](#__codelineno-5-84)main() { [](#__codelineno-5-85) check_hf_token [](#__codelineno-5-86) check_num_gpus [](#__codelineno-5-87) ensure_python_library_installed lmcache [](#__codelineno-5-88) ensure_python_library_installed nixl [](#__codelineno-5-89) ensure_python_library_installed pandas [](#__codelineno-5-90) ensure_python_library_installed datasets [](#__codelineno-5-91) ensure_python_library_installed vllm [](#__codelineno-5-92) [](#__codelineno-5-93) trap cleanup INT [](#__codelineno-5-94) trap cleanup USR1 [](#__codelineno-5-95) trap cleanup TERM [](#__codelineno-5-96) [](#__codelineno-5-97) echo "Launching prefiller, decoder and proxy..." [](#__codelineno-5-98) echo "Please check prefiller.log, decoder.log and proxy.log for logs." [](#__codelineno-5-99) [](#__codelineno-5-100) bash disagg_vllm_launcher.sh prefiller \ [](#__codelineno-5-101) > >(tee prefiller.log) 2>&1 & [](#__codelineno-5-102) prefiller_pid=$! [](#__codelineno-5-103) PIDS+=("$prefiller_pid") [](#__codelineno-5-104) [](#__codelineno-5-105) bash disagg_vllm_launcher.sh decoder \ [](#__codelineno-5-106) > >(tee decoder.log) 2>&1 & [](#__codelineno-5-107) decoder_pid=$! [](#__codelineno-5-108) PIDS+=("$decoder_pid") [](#__codelineno-5-109) [](#__codelineno-5-110) python3 disagg_proxy_server.py \ [](#__codelineno-5-111) --host localhost \ [](#__codelineno-5-112) --port 9000 \ [](#__codelineno-5-113) --prefiller-host localhost \ [](#__codelineno-5-114) --prefiller-port 8100 \ [](#__codelineno-5-115) --decoder-host localhost \ [](#__codelineno-5-116) --decoder-port 8200 \ [](#__codelineno-5-117) > >(tee proxy.log) 2>&1 & [](#__codelineno-5-118) proxy_pid=$! [](#__codelineno-5-119) PIDS+=("$proxy_pid") [](#__codelineno-5-120) [](#__codelineno-5-121) wait_for_server 8100 [](#__codelineno-5-122) wait_for_server 8200 [](#__codelineno-5-123) wait_for_server 9000 [](#__codelineno-5-124) [](#__codelineno-5-125) echo "All servers are up. Starting benchmark..." [](#__codelineno-5-126) [](#__codelineno-5-127) # begin benchmark [](#__codelineno-5-128) cd ../../../../benchmarks/ [](#__codelineno-5-129) vllm bench serve --port 9000 --seed "$(date +%s)" \ [](#__codelineno-5-130) --model meta-llama/Llama-3.1-8B-Instruct \ [](#__codelineno-5-131) --dataset-name random --random-input-len 7500 --random-output-len 200 \ [](#__codelineno-5-132) --num-prompts 200 --burstiness 100 --request-rate 3.6 | tee benchmark.log [](#__codelineno-5-133) [](#__codelineno-5-134) echo "Benchmarking done. Cleaning up..." [](#__codelineno-5-135) [](#__codelineno-5-136) cleanup [](#__codelineno-5-137)[](#__codelineno-5-138)} [](#__codelineno-5-139)[](#__codelineno-5-140)main` disagg\_prefill\_lmcache\_v1/disagg\_proxy\_server.py `[](#__codelineno-6-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-6-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-6-3)[](#__codelineno-6-4)import argparse [](#__codelineno-6-5)import os [](#__codelineno-6-6)import time [](#__codelineno-6-7)from contextlib import asynccontextmanager [](#__codelineno-6-8)[](#__codelineno-6-9)import httpx [](#__codelineno-6-10)import numpy as np [](#__codelineno-6-11)from fastapi import FastAPI, Request [](#__codelineno-6-12)from fastapi.responses import StreamingResponse [](#__codelineno-6-13) [](#__codelineno-6-14)[](#__codelineno-6-15)@asynccontextmanager [](#__codelineno-6-16)async def lifespan(app: FastAPI): [](#__codelineno-6-17) """ [](#__codelineno-6-18) Lifespan context manager to handle startup and shutdown events. [](#__codelineno-6-19) """ [](#__codelineno-6-20) # Startup: Initialize clients [](#__codelineno-6-21) prefiller_base_url = ( [](#__codelineno-6-22) f"http://{global_args.prefiller_host}:{global_args.prefiller_port}/v1" [](#__codelineno-6-23) ) [](#__codelineno-6-24) decoder_base_url = ( [](#__codelineno-6-25) f"http://{global_args.decoder_host}:{global_args.decoder_port}/v1" [](#__codelineno-6-26) ) [](#__codelineno-6-27) [](#__codelineno-6-28) app.state.prefill_client = httpx.AsyncClient( [](#__codelineno-6-29) timeout=None, [](#__codelineno-6-30) base_url=prefiller_base_url, [](#__codelineno-6-31) limits=httpx.Limits( [](#__codelineno-6-32) max_connections=None, [](#__codelineno-6-33) max_keepalive_connections=None, [](#__codelineno-6-34) ), [](#__codelineno-6-35) ) [](#__codelineno-6-36) app.state.decode_client = httpx.AsyncClient( [](#__codelineno-6-37) timeout=None, [](#__codelineno-6-38) base_url=decoder_base_url, [](#__codelineno-6-39) limits=httpx.Limits( [](#__codelineno-6-40) max_connections=None, [](#__codelineno-6-41) max_keepalive_connections=None, [](#__codelineno-6-42) ), [](#__codelineno-6-43) ) [](#__codelineno-6-44) [](#__codelineno-6-45) yield [](#__codelineno-6-46) [](#__codelineno-6-47) # Shutdown: Close clients [](#__codelineno-6-48) await app.state.prefill_client.aclose() [](#__codelineno-6-49) await app.state.decode_client.aclose() [](#__codelineno-6-50) [](#__codelineno-6-51)[](#__codelineno-6-52)# Update FastAPI app initialization to use lifespan [](#__codelineno-6-53)app = FastAPI(lifespan=lifespan) [](#__codelineno-6-54) [](#__codelineno-6-55)[](#__codelineno-6-56)class StatsCalculator: [](#__codelineno-6-57) def __init__(self): [](#__codelineno-6-58) self._stats = [] [](#__codelineno-6-59) self._last_log_time = time.time() [](#__codelineno-6-60) [](#__codelineno-6-61) def add(self, value): [](#__codelineno-6-62) self._stats.append(value) [](#__codelineno-6-63) if time.time() - self._last_log_time > 5: [](#__codelineno-6-64) self._log_stats() [](#__codelineno-6-65) self._last_log_time = time.time() [](#__codelineno-6-66) [](#__codelineno-6-67) def _log_stats(self): [](#__codelineno-6-68) # Print average, median, and 99th percentile [](#__codelineno-6-69) np_arr = np.array(self._stats) [](#__codelineno-6-70) output_str = ( [](#__codelineno-6-71) f"\nNum requests: {len(self._stats)}" [](#__codelineno-6-72) "\nPrefill node TTFT stats:" [](#__codelineno-6-73) f"\n - Average (ms): {np.mean(np_arr)}" [](#__codelineno-6-74) f"\n - Median (ms): {np.median(np_arr)}" [](#__codelineno-6-75) f"\n - 99th Percentile (ms): {np.percentile(np_arr, 99)}\n" [](#__codelineno-6-76) ) [](#__codelineno-6-77) print( [](#__codelineno-6-78) "===============================", [](#__codelineno-6-79) output_str, [](#__codelineno-6-80) "===============================", [](#__codelineno-6-81) ) [](#__codelineno-6-82) [](#__codelineno-6-83)[](#__codelineno-6-84)stats_calculator = StatsCalculator() [](#__codelineno-6-85)counter = 0 [](#__codelineno-6-86) [](#__codelineno-6-87)[](#__codelineno-6-88)def parse_args(): [](#__codelineno-6-89) parser = argparse.ArgumentParser() [](#__codelineno-6-90) [](#__codelineno-6-91) parser.add_argument("--port", type=int, default=8000) [](#__codelineno-6-92) parser.add_argument("--host", type=str, default="localhost") [](#__codelineno-6-93) parser.add_argument("--prefiller-host", type=str, default="localhost") [](#__codelineno-6-94) parser.add_argument("--prefiller-port", type=int, default=8100) [](#__codelineno-6-95) parser.add_argument("--decoder-host", type=str, default="localhost") [](#__codelineno-6-96) parser.add_argument("--decoder-port", type=int, default=8200) [](#__codelineno-6-97) args = parser.parse_args() [](#__codelineno-6-98) return args [](#__codelineno-6-99) [](#__codelineno-6-100)[](#__codelineno-6-101)# Initialize variables to hold the persistent clients [](#__codelineno-6-102)app.state.prefill_client = None [](#__codelineno-6-103)app.state.decode_client = None [](#__codelineno-6-104) [](#__codelineno-6-105)[](#__codelineno-6-106)async def send_request_to_service( [](#__codelineno-6-107) client: httpx.AsyncClient, endpoint: str, req_data: dict [](#__codelineno-6-108)): [](#__codelineno-6-109) """ [](#__codelineno-6-110) Send a request to a service using a persistent client. [](#__codelineno-6-111) """ [](#__codelineno-6-112) req_data = req_data.copy() [](#__codelineno-6-113) req_data["max_tokens"] = 1 [](#__codelineno-6-114) if "max_completion_tokens" in req_data: [](#__codelineno-6-115) req_data["max_completion_tokens"] = 1 [](#__codelineno-6-116) [](#__codelineno-6-117) headers = {"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"} [](#__codelineno-6-118) response = await client.post(endpoint, json=req_data, headers=headers) [](#__codelineno-6-119) response.raise_for_status() [](#__codelineno-6-120) [](#__codelineno-6-121) # read/consume the response body to release the connection [](#__codelineno-6-122) # otherwise, it would http.ReadError [](#__codelineno-6-123) await response.aread() [](#__codelineno-6-124) [](#__codelineno-6-125) return response [](#__codelineno-6-126) [](#__codelineno-6-127)[](#__codelineno-6-128)async def stream_service_response( [](#__codelineno-6-129) client: httpx.AsyncClient, endpoint: str, req_data: dict [](#__codelineno-6-130)): [](#__codelineno-6-131) """ [](#__codelineno-6-132) Asynchronously stream the response from a service using a persistent client. [](#__codelineno-6-133) """ [](#__codelineno-6-134) headers = {"Authorization": f"Bearer {os.environ.get('OPENAI_API_KEY')}"} [](#__codelineno-6-135) async with client.stream( [](#__codelineno-6-136) "POST", endpoint, json=req_data, headers=headers [](#__codelineno-6-137) ) as response: [](#__codelineno-6-138) response.raise_for_status() [](#__codelineno-6-139) async for chunk in response.aiter_bytes(): [](#__codelineno-6-140) yield chunk [](#__codelineno-6-141) [](#__codelineno-6-142)[](#__codelineno-6-143)@app.post("/v1/completions") [](#__codelineno-6-144)async def handle_completions(request: Request): [](#__codelineno-6-145) global counter, stats_calculator [](#__codelineno-6-146) counter += 1 [](#__codelineno-6-147) [](#__codelineno-6-148) st = time.time() [](#__codelineno-6-149) try: [](#__codelineno-6-150) req_data = await request.json() [](#__codelineno-6-151) [](#__codelineno-6-152) # Send request to prefill service, ignore the response [](#__codelineno-6-153) await send_request_to_service( [](#__codelineno-6-154) app.state.prefill_client, "/completions", req_data [](#__codelineno-6-155) ) [](#__codelineno-6-156) [](#__codelineno-6-157) et = time.time() [](#__codelineno-6-158) stats_calculator.add(et - st) [](#__codelineno-6-159) [](#__codelineno-6-160) # Stream response from decode service [](#__codelineno-6-161) async def generate_stream(): [](#__codelineno-6-162) async for chunk in stream_service_response( [](#__codelineno-6-163) app.state.decode_client, "/completions", req_data [](#__codelineno-6-164) ): [](#__codelineno-6-165) yield chunk [](#__codelineno-6-166) [](#__codelineno-6-167) return StreamingResponse(generate_stream(), media_type="text/event-stream") [](#__codelineno-6-168) [](#__codelineno-6-169) except Exception as e: [](#__codelineno-6-170) import sys [](#__codelineno-6-171) import traceback [](#__codelineno-6-172) [](#__codelineno-6-173) exc_info = sys.exc_info() [](#__codelineno-6-174) print("Error occurred in disagg prefill proxy server - completions endpoint") [](#__codelineno-6-175) print(e) [](#__codelineno-6-176) print("".join(traceback.format_exception(*exc_info))) [](#__codelineno-6-177) raise [](#__codelineno-6-178) [](#__codelineno-6-179)[](#__codelineno-6-180)@app.post("/v1/chat/completions") [](#__codelineno-6-181)async def handle_chat_completions(request: Request): [](#__codelineno-6-182) global counter, stats_calculator [](#__codelineno-6-183) counter += 1 [](#__codelineno-6-184) [](#__codelineno-6-185) st = time.time() [](#__codelineno-6-186) try: [](#__codelineno-6-187) req_data = await request.json() [](#__codelineno-6-188) [](#__codelineno-6-189) # Send request to prefill service, ignore the response [](#__codelineno-6-190) await send_request_to_service( [](#__codelineno-6-191) app.state.prefill_client, "/chat/completions", req_data [](#__codelineno-6-192) ) [](#__codelineno-6-193) [](#__codelineno-6-194) et = time.time() [](#__codelineno-6-195) stats_calculator.add(et - st) [](#__codelineno-6-196) [](#__codelineno-6-197) # Stream response from decode service [](#__codelineno-6-198) async def generate_stream(): [](#__codelineno-6-199) async for chunk in stream_service_response( [](#__codelineno-6-200) app.state.decode_client, "/chat/completions", req_data [](#__codelineno-6-201) ): [](#__codelineno-6-202) yield chunk [](#__codelineno-6-203) [](#__codelineno-6-204) return StreamingResponse(generate_stream(), media_type="text/event-stream") [](#__codelineno-6-205) [](#__codelineno-6-206) except Exception as e: [](#__codelineno-6-207) import sys [](#__codelineno-6-208) import traceback [](#__codelineno-6-209) [](#__codelineno-6-210) exc_info = sys.exc_info() [](#__codelineno-6-211) print( [](#__codelineno-6-212) "Error occurred in disagg prefill proxy server - chat completions endpoint" [](#__codelineno-6-213) ) [](#__codelineno-6-214) print(e) [](#__codelineno-6-215) print("".join(traceback.format_exception(*exc_info))) [](#__codelineno-6-216) raise [](#__codelineno-6-217) [](#__codelineno-6-218)[](#__codelineno-6-219)if __name__ == "__main__": [](#__codelineno-6-220) global global_args [](#__codelineno-6-221) global_args = parse_args() [](#__codelineno-6-222) [](#__codelineno-6-223) import uvicorn [](#__codelineno-6-224) [](#__codelineno-6-225) uvicorn.run(app, host=global_args.host, port=global_args.port)` disagg\_prefill\_lmcache\_v1/disagg\_vllm\_launcher.sh `[](#__codelineno-7-1)#!/bin/bash [](#__codelineno-7-2)[](#__codelineno-7-3)SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" [](#__codelineno-7-4)[](#__codelineno-7-5)if [[ $# -lt 1 ]]; then [](#__codelineno-7-6) echo "Usage: $0 [model]" [](#__codelineno-7-7) exit 1 [](#__codelineno-7-8)fi [](#__codelineno-7-9)[](#__codelineno-7-10)if [[ $# -eq 1 ]]; then [](#__codelineno-7-11) echo "Using default model: meta-llama/Llama-3.1-8B-Instruct" [](#__codelineno-7-12) MODEL="meta-llama/Llama-3.1-8B-Instruct" [](#__codelineno-7-13)else [](#__codelineno-7-14) echo "Using model: $2" [](#__codelineno-7-15) MODEL=$2 [](#__codelineno-7-16)fi [](#__codelineno-7-17)[](#__codelineno-7-18)# The prefillers and decoders in LMCache use the same hash seed for all chunk keys. [](#__codelineno-7-19)# This seed must be aligned so that decoders can identify and retrieve KV cache [](#__codelineno-7-20)# entries stored by prefillers. [](#__codelineno-7-21)# [](#__codelineno-7-22)# WARNING: Using a fixed hash seed is insecure and makes the application vulnerable to [](#__codelineno-7-23)# denial-of-service attacks. In a production environment, this should be set to a [](#__codelineno-7-24)# secure random value. This is set to a fixed value for demonstration purposes only. [](#__codelineno-7-25)export PYTHONHASHSEED=${VLLM_PYTHON_HASH_SEED:-123} [](#__codelineno-7-26)[](#__codelineno-7-27)if [[ $1 == "prefiller" ]]; then [](#__codelineno-7-28) # Prefiller listens on port 8100 [](#__codelineno-7-29) prefill_config_file=$SCRIPT_DIR/configs/lmcache-prefiller-config.yaml [](#__codelineno-7-30) [](#__codelineno-7-31) UCX_TLS=cuda_ipc,cuda_copy,tcp \ [](#__codelineno-7-32) LMCACHE_CONFIG_FILE=$prefill_config_file \ [](#__codelineno-7-33) LMCACHE_USE_EXPERIMENTAL=True \ [](#__codelineno-7-34) VLLM_ENABLE_V1_MULTIPROCESSING=1 \ [](#__codelineno-7-35) VLLM_WORKER_MULTIPROC_METHOD=spawn \ [](#__codelineno-7-36) CUDA_VISIBLE_DEVICES=0 \ [](#__codelineno-7-37) vllm serve "$MODEL" \ [](#__codelineno-7-38) --port 8100 \ [](#__codelineno-7-39) --enforce-eager \ [](#__codelineno-7-40) --kv-transfer-config \ [](#__codelineno-7-41) '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_producer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "producer1"}}' [](#__codelineno-7-42) [](#__codelineno-7-43)[](#__codelineno-7-44)elif [[ $1 == "decoder" ]]; then [](#__codelineno-7-45) # Decoder listens on port 8200 [](#__codelineno-7-46) decode_config_file=$SCRIPT_DIR/configs/lmcache-decoder-config.yaml [](#__codelineno-7-47) [](#__codelineno-7-48) UCX_TLS=cuda_ipc,cuda_copy,tcp \ [](#__codelineno-7-49) LMCACHE_CONFIG_FILE=$decode_config_file \ [](#__codelineno-7-50) LMCACHE_USE_EXPERIMENTAL=True \ [](#__codelineno-7-51) VLLM_ENABLE_V1_MULTIPROCESSING=1 \ [](#__codelineno-7-52) VLLM_WORKER_MULTIPROC_METHOD=spawn \ [](#__codelineno-7-53) CUDA_VISIBLE_DEVICES=1 \ [](#__codelineno-7-54) vllm serve "$MODEL" \ [](#__codelineno-7-55) --port 8200 \ [](#__codelineno-7-56) --enforce-eager \ [](#__codelineno-7-57) --kv-transfer-config \ [](#__codelineno-7-58) '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_consumer","kv_connector_extra_config": {"discard_partial_chunks": false, "lmcache_rpc_port": "consumer1"}}' [](#__codelineno-7-59) [](#__codelineno-7-60)[](#__codelineno-7-61)else [](#__codelineno-7-62) echo "Invalid role: $1" [](#__codelineno-7-63) echo "Should be either prefiller, decoder" [](#__codelineno-7-64) exit 1 [](#__codelineno-7-65)fi` kv\_cache\_sharing\_lmcache\_v1.py ``[](#__codelineno-8-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-8-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-8-3)""" [](#__codelineno-8-4)This file demonstrates the example usage of remote KV cache sharing [](#__codelineno-8-5)with LMCache. [](#__codelineno-8-6)We will launch 2 vllm instances, and launch an additional LMCache server. [](#__codelineno-8-7)KV cache is transferred in the following manner: [](#__codelineno-8-8)(1) vLLM instance 1 -> LMCache server (KV cache store). [](#__codelineno-8-9)(2) LMCache server -> vLLM instance 2 (KV cache reuse/retrieve). [](#__codelineno-8-10)[](#__codelineno-8-11)Note that lmcache needs to be installed to run this example. [](#__codelineno-8-12)Learn more about LMCache in https://github.com/LMCache/LMCache. [](#__codelineno-8-13)""" [](#__codelineno-8-14)[](#__codelineno-8-15)import os [](#__codelineno-8-16)import subprocess [](#__codelineno-8-17)import time [](#__codelineno-8-18)from multiprocessing import Event, Process [](#__codelineno-8-19)[](#__codelineno-8-20)from lmcache.integration.vllm.utils import ENGINE_NAME [](#__codelineno-8-21)from lmcache.v1.cache_engine import LMCacheEngineBuilder [](#__codelineno-8-22)[](#__codelineno-8-23)from vllm import LLM, SamplingParams [](#__codelineno-8-24)from vllm.config import KVTransferConfig [](#__codelineno-8-25)[](#__codelineno-8-26)# LMCache-related environment variables [](#__codelineno-8-27)# The port to start LMCache server [](#__codelineno-8-28)port = 8100 [](#__codelineno-8-29)# Use experimental features in LMCache [](#__codelineno-8-30)os.environ["LMCACHE_USE_EXPERIMENTAL"] = "True" [](#__codelineno-8-31)# LMCache is set to use 256 tokens per chunk [](#__codelineno-8-32)os.environ["LMCACHE_CHUNK_SIZE"] = "256" [](#__codelineno-8-33)# Disable local CPU backend in LMCache [](#__codelineno-8-34)os.environ["LMCACHE_LOCAL_CPU"] = "False" [](#__codelineno-8-35)# Set local CPU memory buffer limit to 5.0 GB [](#__codelineno-8-36)os.environ["LMCACHE_MAX_LOCAL_CPU_SIZE"] = "5.0" [](#__codelineno-8-37)# Set the remote URL for LMCache server [](#__codelineno-8-38)os.environ["LMCACHE_REMOTE_URL"] = f"lm://localhost:{port}" [](#__codelineno-8-39)# Set the serializer/deserializer between vllm and LMCache server [](#__codelineno-8-40)# `naive` indicates using raw bytes of the tensor without any compression [](#__codelineno-8-41)os.environ["LMCACHE_REMOTE_SERDE"] = "naive" [](#__codelineno-8-42)[](#__codelineno-8-43)prompts = [ [](#__codelineno-8-44) "Hello, how are you?" * 1000, [](#__codelineno-8-45)] [](#__codelineno-8-46) [](#__codelineno-8-47)[](#__codelineno-8-48)def run_store(store_done, prompts): [](#__codelineno-8-49) # We use GPU 0 for KV cache store process. [](#__codelineno-8-50) os.environ["CUDA_VISIBLE_DEVICES"] = "0" [](#__codelineno-8-51) [](#__codelineno-8-52) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) [](#__codelineno-8-53) [](#__codelineno-8-54) ktc = KVTransferConfig(kv_connector="LMCacheConnectorV1", kv_role="kv_both") [](#__codelineno-8-55) # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB [](#__codelineno-8-56) # memory. Reduce the value if your GPU has less memory. [](#__codelineno-8-57) llm = LLM( [](#__codelineno-8-58) model="mistralai/Mistral-7B-Instruct-v0.2", [](#__codelineno-8-59) kv_transfer_config=ktc, [](#__codelineno-8-60) max_model_len=8000, [](#__codelineno-8-61) gpu_memory_utilization=0.8, [](#__codelineno-8-62) enforce_eager=True, [](#__codelineno-8-63) ) [](#__codelineno-8-64) [](#__codelineno-8-65) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-8-66) for output in outputs: [](#__codelineno-8-67) generated_text = output.outputs[0].text [](#__codelineno-8-68) print(f"Generated text: {generated_text!r}") [](#__codelineno-8-69) print("KV cache store is finished.") [](#__codelineno-8-70) store_done.set() [](#__codelineno-8-71) [](#__codelineno-8-72) # Clean up lmcache backend [](#__codelineno-8-73) LMCacheEngineBuilder.destroy(ENGINE_NAME) [](#__codelineno-8-74) [](#__codelineno-8-75)[](#__codelineno-8-76)def run_retrieve(store_done, prompts, timeout=1): [](#__codelineno-8-77) # We use GPU 1 for KV cache retrieve process. [](#__codelineno-8-78) os.environ["CUDA_VISIBLE_DEVICES"] = "1" [](#__codelineno-8-79) [](#__codelineno-8-80) sampling_params = SamplingParams(temperature=0, top_p=0.95, max_tokens=10) [](#__codelineno-8-81) [](#__codelineno-8-82) ktc = KVTransferConfig(kv_connector="LMCacheConnectorV1", kv_role="kv_both") [](#__codelineno-8-83) # Set GPU memory utilization to 0.8 for an A40 GPU with 40GB [](#__codelineno-8-84) # of memory. Reduce the value if your GPU has less memory. [](#__codelineno-8-85) llm = LLM( [](#__codelineno-8-86) model="mistralai/Mistral-7B-Instruct-v0.2", [](#__codelineno-8-87) kv_transfer_config=ktc, [](#__codelineno-8-88) max_model_len=8000, [](#__codelineno-8-89) gpu_memory_utilization=0.8, [](#__codelineno-8-90) enforce_eager=True, [](#__codelineno-8-91) ) [](#__codelineno-8-92) [](#__codelineno-8-93) print("Waiting for KV cache store to finish...") [](#__codelineno-8-94) store_done.wait() [](#__codelineno-8-95) time.sleep(timeout) [](#__codelineno-8-96) [](#__codelineno-8-97) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-8-98) for output in outputs: [](#__codelineno-8-99) generated_text = output.outputs[0].text [](#__codelineno-8-100) print(f"Generated text: {generated_text!r}") [](#__codelineno-8-101) [](#__codelineno-8-102) # Clean up lmcache backend [](#__codelineno-8-103) LMCacheEngineBuilder.destroy(ENGINE_NAME) [](#__codelineno-8-104) [](#__codelineno-8-105)[](#__codelineno-8-106)def run_lmcache_server(port): [](#__codelineno-8-107) server_proc = subprocess.Popen( [](#__codelineno-8-108) ["python", "-m", "lmcache.v1.server", "localhost", str(port)] [](#__codelineno-8-109) ) [](#__codelineno-8-110) return server_proc [](#__codelineno-8-111) [](#__codelineno-8-112)[](#__codelineno-8-113)def main(): [](#__codelineno-8-114) store_done = Event() [](#__codelineno-8-115) store_process = Process(target=run_store, args=(store_done, prompts)) [](#__codelineno-8-116) retrieve_process = Process(target=run_retrieve, args=(store_done, prompts)) [](#__codelineno-8-117) lmcache_server_process = run_lmcache_server(port) [](#__codelineno-8-118) [](#__codelineno-8-119) # Start KV cache store process [](#__codelineno-8-120) store_process.start() [](#__codelineno-8-121) [](#__codelineno-8-122) # Start KV cache retrieve process [](#__codelineno-8-123) retrieve_process.start() [](#__codelineno-8-124) [](#__codelineno-8-125) # Clean up the processes [](#__codelineno-8-126) store_process.join() [](#__codelineno-8-127) retrieve_process.terminate() [](#__codelineno-8-128) lmcache_server_process.terminate() [](#__codelineno-8-129) lmcache_server_process.wait() [](#__codelineno-8-130) [](#__codelineno-8-131)[](#__codelineno-8-132)if __name__ == "__main__": [](#__codelineno-8-133) main()`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/automatic_prefix_caching.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/automatic\_prefix\_caching](https://github.com/vllm-project/vllm/tree/main/examples/features/automatic_prefix_caching). ## Automatic Prefix Caching Offline[¶](#automatic-prefix-caching-offline "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Demonstration script for Automatic Prefix Caching (APC) in vLLM. [](#__codelineno-0-5)[](#__codelineno-0-6)Automatic Prefix Caching (APC) allows the vLLM engine to reuse cached [](#__codelineno-0-7)KV (key-value) pairs from previous prompts if a new query shares the same [](#__codelineno-0-8)prefix. This reduces redundant computation and improves inference speed. [](#__codelineno-0-9)[](#__codelineno-0-10)To enable APC, set `enable_prefix_caching=True` when initializing the [](#__codelineno-0-11)vLLM engine. [](#__codelineno-0-12)[](#__codelineno-0-13)This script uses a long Markdown table as the shared prompt prefix and [](#__codelineno-0-14)compares the generation time for two queries that share the same prefix [](#__codelineno-0-15)but ask different questions. [](#__codelineno-0-16)[](#__codelineno-0-17)Run: [](#__codelineno-0-18)python examples/features/automatic_prefix_caching/automatic_prefix_caching_offline.py [](#__codelineno-0-19)""" [](#__codelineno-0-20)[](#__codelineno-0-21)import time [](#__codelineno-0-22)[](#__codelineno-0-23)from vllm import LLM, SamplingParams [](#__codelineno-0-24)[](#__codelineno-0-25)# ruff: noqa: E501 [](#__codelineno-0-26)# A prompt containing a large markdown table. The table is randomly generated by GPT-4. [](#__codelineno-0-27)LONG_PROMPT = ( [](#__codelineno-0-28) "You are a helpful assistant in recognizes the content of tables in markdown format. Here is a table as follows.\n# Table\n" [](#__codelineno-0-29) """ [](#__codelineno-0-30)| ID | Name | Age | Occupation | Country | Email | Phone Number | Address | [](#__codelineno-0-31)|-----|---------------|-----|---------------|---------------|------------------------|----------------|------------------------------| [](#__codelineno-0-32)| 1 | John Doe | 29 | Engineer | USA | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-1234 | 123 Elm St, Springfield, IL | [](#__codelineno-0-33)| 2 | Jane Smith | 34 | Doctor | Canada | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-5678 | 456 Oak St, Toronto, ON | [](#__codelineno-0-34)| 3 | Alice Johnson | 27 | Teacher | UK | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-8765 | 789 Pine St, London, UK | [](#__codelineno-0-35)| 4 | Bob Brown | 45 | Artist | Australia | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-4321 | 321 Maple St, Sydney, NSW | [](#__codelineno-0-36)| 5 | Carol White | 31 | Scientist | New Zealand | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-6789 | 654 Birch St, Wellington, NZ | [](#__codelineno-0-37)| 6 | Dave Green | 28 | Lawyer | Ireland | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-3456 | 987 Cedar St, Dublin, IE | [](#__codelineno-0-38)| 7 | Emma Black | 40 | Musician | USA | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-1111 | 246 Ash St, New York, NY | [](#__codelineno-0-39)| 8 | Frank Blue | 37 | Chef | Canada | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-2222 | 135 Spruce St, Vancouver, BC | [](#__codelineno-0-40)| 9 | Grace Yellow | 50 | Engineer | UK | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-3333 | 864 Fir St, Manchester, UK | [](#__codelineno-0-41)| 10 | Henry Violet | 32 | Artist | Australia | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-4444 | 753 Willow St, Melbourne, VIC| [](#__codelineno-0-42)| 11 | Irene Orange | 26 | Scientist | New Zealand | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-5555 | 912 Poplar St, Auckland, NZ | [](#__codelineno-0-43)| 12 | Jack Indigo | 38 | Teacher | Ireland | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-6666 | 159 Elm St, Cork, IE | [](#__codelineno-0-44)| 13 | Karen Red | 41 | Lawyer | USA | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-7777 | 357 Cedar St, Boston, MA | [](#__codelineno-0-45)| 14 | Leo Brown | 30 | Chef | Canada | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-8888 | 246 Oak St, Calgary, AB | [](#__codelineno-0-46)| 15 | Mia Green | 33 | Musician | UK | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-9999 | 975 Pine St, Edinburgh, UK | [](#__codelineno-0-47)| 16 | Noah Yellow | 29 | Doctor | Australia | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-0000 | 864 Birch St, Brisbane, QLD | [](#__codelineno-0-48)| 17 | Olivia Blue | 35 | Engineer | New Zealand | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-1212 | 753 Maple St, Hamilton, NZ | [](#__codelineno-0-49)| 18 | Peter Black | 42 | Artist | Ireland | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-3434 | 912 Fir St, Limerick, IE | [](#__codelineno-0-50)| 19 | Quinn White | 28 | Scientist | USA | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-5656 | 159 Willow St, Seattle, WA | [](#__codelineno-0-51)| 20 | Rachel Red | 31 | Teacher | Canada | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-7878 | 357 Poplar St, Ottawa, ON | [](#__codelineno-0-52)| 21 | Steve Green | 44 | Lawyer | UK | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-9090 | 753 Elm St, Birmingham, UK | [](#__codelineno-0-53)| 22 | Tina Blue | 36 | Musician | Australia | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-1213 | 864 Cedar St, Perth, WA | [](#__codelineno-0-54)| 23 | Umar Black | 39 | Chef | New Zealand | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-3435 | 975 Spruce St, Christchurch, NZ| [](#__codelineno-0-55)| 24 | Victor Yellow | 43 | Engineer | Ireland | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-5657 | 246 Willow St, Galway, IE | [](#__codelineno-0-56)| 25 | Wendy Orange | 27 | Artist | USA | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-7879 | 135 Elm St, Denver, CO | [](#__codelineno-0-57)| 26 | Xavier Green | 34 | Scientist | Canada | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-9091 | 357 Oak St, Montreal, QC | [](#__codelineno-0-58)| 27 | Yara Red | 41 | Teacher | UK | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-1214 | 975 Pine St, Leeds, UK | [](#__codelineno-0-59)| 28 | Zack Blue | 30 | Lawyer | Australia | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-3436 | 135 Birch St, Adelaide, SA | [](#__codelineno-0-60)| 29 | Amy White | 33 | Musician | New Zealand | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-5658 | 159 Maple St, Wellington, NZ | [](#__codelineno-0-61)| 30 | Ben Black | 38 | Chef | Ireland | [[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection) | 555-7870 | 246 Fir St, Waterford, IE | [](#__codelineno-0-62)""" [](#__codelineno-0-63)) [](#__codelineno-0-64) [](#__codelineno-0-65)[](#__codelineno-0-66)def get_generation_time(llm, sampling_params, prompts): [](#__codelineno-0-67) # time the generation [](#__codelineno-0-68) start_time = time.time() [](#__codelineno-0-69) output = llm.generate(prompts, sampling_params=sampling_params) [](#__codelineno-0-70) end_time = time.time() [](#__codelineno-0-71) # print the output and generation time [](#__codelineno-0-72) print("-" * 30) [](#__codelineno-0-73) print(f"Output: {output[0].outputs[0].text}") [](#__codelineno-0-74) print(f"Generation time: {end_time - start_time} seconds.") [](#__codelineno-0-75) print("-" * 30) [](#__codelineno-0-76) [](#__codelineno-0-77)[](#__codelineno-0-78)def main(): [](#__codelineno-0-79) # set enable_prefix_caching=True to enable APC [](#__codelineno-0-80) llm = LLM(model="lmsys/longchat-13b-16k", enable_prefix_caching=True) [](#__codelineno-0-81) [](#__codelineno-0-82) sampling_params = SamplingParams(temperature=0, max_tokens=100) [](#__codelineno-0-83) [](#__codelineno-0-84) # Querying the age of John Doe [](#__codelineno-0-85) get_generation_time( [](#__codelineno-0-86) llm, [](#__codelineno-0-87) sampling_params, [](#__codelineno-0-88) LONG_PROMPT [](#__codelineno-0-89) + "Question: what is the age of John Doe? Your answer: The age of John Doe is ", [](#__codelineno-0-90) ) [](#__codelineno-0-91) [](#__codelineno-0-92) # Querying the age of Zack Blue [](#__codelineno-0-93) # This query will be faster since vllm avoids computing the KV cache of LONG_PROMPT again. [](#__codelineno-0-94) get_generation_time( [](#__codelineno-0-95) llm, [](#__codelineno-0-96) sampling_params, [](#__codelineno-0-97) LONG_PROMPT [](#__codelineno-0-98) + "Question: what is the age of Zack Blue? Your answer: The age of Zack Blue is ", [](#__codelineno-0-99) ) [](#__codelineno-0-100) [](#__codelineno-0-101)[](#__codelineno-0-102)if __name__ == "__main__": [](#__codelineno-0-103) main()`` ## Prefix Caching Offline[¶](#prefix-caching-offline "Permanent link") ``[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)[](#__codelineno-1-4)from vllm import LLM, SamplingParams [](#__codelineno-1-5)from vllm.distributed import cleanup_dist_env_and_memory [](#__codelineno-1-6)[](#__codelineno-1-7)# NOTE: This is just a running example. For benchmarking purpose, [](#__codelineno-1-8)# please see benchmarks/benchmark_prefix_caching.py [](#__codelineno-1-9)[](#__codelineno-1-10)# Common prefix. [](#__codelineno-1-11)prefix = ( [](#__codelineno-1-12) "You are an expert school principal, skilled in effectively managing " [](#__codelineno-1-13) "faculty and staff. Draft 10-15 questions for a potential first grade " [](#__codelineno-1-14) "Head Teacher for my K-12, all-girls', independent school that emphasizes " [](#__codelineno-1-15) "community, joyful discovery, and life-long learning. The candidate is " [](#__codelineno-1-16) "coming in for a first-round panel interview for a 8th grade Math " [](#__codelineno-1-17) "teaching role. They have 5 years of previous teaching experience " [](#__codelineno-1-18) "as an assistant teacher at a co-ed, public school with experience " [](#__codelineno-1-19) "in middle school math teaching. Based on these information, fulfill " [](#__codelineno-1-20) "the following paragraph: " [](#__codelineno-1-21)) [](#__codelineno-1-22)[](#__codelineno-1-23)# Sample prompts. [](#__codelineno-1-24)prompts = [ [](#__codelineno-1-25) "Hello, my name is", [](#__codelineno-1-26) "The president of the United States is", [](#__codelineno-1-27) "The capital of France is", [](#__codelineno-1-28) "The future of AI is", [](#__codelineno-1-29)] [](#__codelineno-1-30)[](#__codelineno-1-31)generating_prompts = [prefix + prompt for prompt in prompts] [](#__codelineno-1-32)[](#__codelineno-1-33)# Create a sampling params object. [](#__codelineno-1-34)sampling_params = SamplingParams(temperature=0.0) [](#__codelineno-1-35) [](#__codelineno-1-36)[](#__codelineno-1-37)def main(): [](#__codelineno-1-38) # Create an LLM without prefix caching as a baseline. [](#__codelineno-1-39) regular_llm = LLM(model="facebook/opt-125m", gpu_memory_utilization=0.4) [](#__codelineno-1-40) [](#__codelineno-1-41) print("Results without `enable_prefix_caching`") [](#__codelineno-1-42) [](#__codelineno-1-43) # ruff: noqa: E501 [](#__codelineno-1-44) # Generate texts from the prompts. The output is a list of RequestOutput objects [](#__codelineno-1-45) # that contain the prompt, generated text, and other information. [](#__codelineno-1-46) outputs = regular_llm.generate(generating_prompts, sampling_params) [](#__codelineno-1-47) [](#__codelineno-1-48) regular_generated_texts = [] [](#__codelineno-1-49) # Print the outputs. [](#__codelineno-1-50) print("-" * 50) [](#__codelineno-1-51) for output in outputs: [](#__codelineno-1-52) prompt = output.prompt [](#__codelineno-1-53) generated_text = output.outputs[0].text [](#__codelineno-1-54) regular_generated_texts.append(generated_text) [](#__codelineno-1-55) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-1-56) print("-" * 50) [](#__codelineno-1-57) [](#__codelineno-1-58) # Destroy the LLM object and free up the GPU memory. [](#__codelineno-1-59) del regular_llm [](#__codelineno-1-60) cleanup_dist_env_and_memory() [](#__codelineno-1-61) [](#__codelineno-1-62) # Create an LLM with prefix caching enabled. [](#__codelineno-1-63) prefix_cached_llm = LLM( [](#__codelineno-1-64) model="facebook/opt-125m", [](#__codelineno-1-65) enable_prefix_caching=True, [](#__codelineno-1-66) gpu_memory_utilization=0.4, [](#__codelineno-1-67) ) [](#__codelineno-1-68) [](#__codelineno-1-69) # Warmup so that the shared prompt's KV cache is computed. [](#__codelineno-1-70) prefix_cached_llm.generate(generating_prompts[0], sampling_params) [](#__codelineno-1-71) [](#__codelineno-1-72) # Generate with prefix caching. [](#__codelineno-1-73) outputs = prefix_cached_llm.generate(generating_prompts, sampling_params) [](#__codelineno-1-74) [](#__codelineno-1-75) print("Results with `enable_prefix_caching`") [](#__codelineno-1-76) [](#__codelineno-1-77) cached_generated_texts = [] [](#__codelineno-1-78) # Print the outputs. You should see the same outputs as before. [](#__codelineno-1-79) print("-" * 50) [](#__codelineno-1-80) for output in outputs: [](#__codelineno-1-81) prompt = output.prompt [](#__codelineno-1-82) generated_text = output.outputs[0].text [](#__codelineno-1-83) cached_generated_texts.append(generated_text) [](#__codelineno-1-84) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-1-85) print("-" * 50) [](#__codelineno-1-86) [](#__codelineno-1-87) # Compare the results and display the speedup [](#__codelineno-1-88) generated_same = all( [](#__codelineno-1-89) [ [](#__codelineno-1-90) regular_generated_texts[i] == cached_generated_texts[i] [](#__codelineno-1-91) for i in range(len(prompts)) [](#__codelineno-1-92) ] [](#__codelineno-1-93) ) [](#__codelineno-1-94) print(f"Generated answers are the same: {generated_same}") [](#__codelineno-1-95) [](#__codelineno-1-96)[](#__codelineno-1-97)if __name__ == "__main__": [](#__codelineno-1-98) main()`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/context_extension.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/context\_extension](https://github.com/vllm-project/vllm/tree/main/examples/features/context_extension). ## Context Extension Offline[¶](#context-extension-offline "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This script demonstrates how to extend the context length [](#__codelineno-0-5)of a Qwen model using the YARN method (rope_parameters) [](#__codelineno-0-6)and run a simple chat example. [](#__codelineno-0-7)[](#__codelineno-0-8)Usage: [](#__codelineno-0-9) python examples/features/context_extension/context_extension_offline.py [](#__codelineno-0-10)""" [](#__codelineno-0-11)[](#__codelineno-0-12)from vllm import LLM, RequestOutput, SamplingParams [](#__codelineno-0-13) [](#__codelineno-0-14)[](#__codelineno-0-15)def create_llm(): [](#__codelineno-0-16) rope_theta = 1000000 [](#__codelineno-0-17) original_max_position_embeddings = 32768 [](#__codelineno-0-18) factor = 4.0 [](#__codelineno-0-19) [](#__codelineno-0-20) # Use yarn to extend context [](#__codelineno-0-21) hf_overrides = { [](#__codelineno-0-22) "rope_parameters": { [](#__codelineno-0-23) "rope_theta": rope_theta, [](#__codelineno-0-24) "rope_type": "yarn", [](#__codelineno-0-25) "factor": factor, [](#__codelineno-0-26) "original_max_position_embeddings": original_max_position_embeddings, [](#__codelineno-0-27) }, [](#__codelineno-0-28) "max_model_len": int(original_max_position_embeddings * factor), [](#__codelineno-0-29) } [](#__codelineno-0-30) [](#__codelineno-0-31) llm = LLM(model="Qwen/Qwen3-0.6B", hf_overrides=hf_overrides) [](#__codelineno-0-32) return llm [](#__codelineno-0-33) [](#__codelineno-0-34)[](#__codelineno-0-35)def run_llm_chat(llm): [](#__codelineno-0-36) sampling_params = SamplingParams( [](#__codelineno-0-37) temperature=0.8, [](#__codelineno-0-38) top_p=0.95, [](#__codelineno-0-39) max_tokens=128, [](#__codelineno-0-40) ) [](#__codelineno-0-41) [](#__codelineno-0-42) conversation = [ [](#__codelineno-0-43) {"role": "system", "content": "You are a helpful assistant"}, [](#__codelineno-0-44) {"role": "user", "content": "Hello"}, [](#__codelineno-0-45) {"role": "assistant", "content": "Hello! How can I assist you today?"}, [](#__codelineno-0-46) ] [](#__codelineno-0-47) outputs = llm.chat(conversation, sampling_params, use_tqdm=False) [](#__codelineno-0-48) return outputs, [ [](#__codelineno-0-49) conversation, [](#__codelineno-0-50) ] [](#__codelineno-0-51) [](#__codelineno-0-52)[](#__codelineno-0-53)def print_outputs(outputs: list[RequestOutput], conversations: list): [](#__codelineno-0-54) print("\nGenerated Outputs:\n" + "-" * 80) [](#__codelineno-0-55) for i, output in enumerate(outputs): [](#__codelineno-0-56) prompt = conversations[i] [](#__codelineno-0-57) generated_text = output.outputs[0].text [](#__codelineno-0-58) print(f"Prompt: {prompt!r}\n") [](#__codelineno-0-59) print(f"Generated text: {generated_text!r}") [](#__codelineno-0-60) print("-" * 80) [](#__codelineno-0-61) [](#__codelineno-0-62)[](#__codelineno-0-63)def main(): [](#__codelineno-0-64) llm = create_llm() [](#__codelineno-0-65) outputs, conversations = run_llm_chat(llm) [](#__codelineno-0-66) print_outputs(outputs, conversations) [](#__codelineno-0-67) [](#__codelineno-0-68)[](#__codelineno-0-69)if __name__ == "__main__": [](#__codelineno-0-70) main()` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/data_parallel.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/data\_parallel](https://github.com/vllm-project/vllm/tree/main/examples/features/data_parallel). ## Data Parallel Offline[¶](#data-parallel-offline "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Usage: [](#__codelineno-0-5)Single node: [](#__codelineno-0-6) python examples/features/data_parallel/data_parallel_offline.py \ [](#__codelineno-0-7) --model="ibm-research/PowerMoE-3b" \ [](#__codelineno-0-8) -dp=2 \ [](#__codelineno-0-9) -tp=2 [](#__codelineno-0-10)[](#__codelineno-0-11)Multi-node: [](#__codelineno-0-12) Node 0 (assume the node has ip of 10.99.48.128): [](#__codelineno-0-13) python examples/features/data_parallel/data_parallel_offline.py \ [](#__codelineno-0-14) --model="ibm-research/PowerMoE-3b" \ [](#__codelineno-0-15) -dp=2 \ [](#__codelineno-0-16) -tp=2 \ [](#__codelineno-0-17) --dp-num-nodes=2 \ [](#__codelineno-0-18) --dp-node-rank=0 \ [](#__codelineno-0-19) --dp-master-addr=10.99.48.128 \ [](#__codelineno-0-20) --dp-master-port=13345 [](#__codelineno-0-21) Node 1: [](#__codelineno-0-22) python examples/features/data_parallel/data_parallel_offline.py \ [](#__codelineno-0-23) --model="ibm-research/PowerMoE-3b" \ [](#__codelineno-0-24) -dp=2 \ [](#__codelineno-0-25) -tp=2 \ [](#__codelineno-0-26) --dp-num-nodes=2 \ [](#__codelineno-0-27) --dp-node-rank=1 \ [](#__codelineno-0-28) --dp-master-addr=10.99.48.128 \ [](#__codelineno-0-29) --dp-master-port=13345 [](#__codelineno-0-30)""" [](#__codelineno-0-31)[](#__codelineno-0-32)import os [](#__codelineno-0-33)from time import sleep [](#__codelineno-0-34)[](#__codelineno-0-35)from vllm import LLM, EngineArgs, SamplingParams [](#__codelineno-0-36)from vllm.platforms import current_platform [](#__codelineno-0-37)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-0-38)from vllm.utils.network_utils import get_open_port [](#__codelineno-0-39) [](#__codelineno-0-40)[](#__codelineno-0-41)def create_parser(): [](#__codelineno-0-42) parser = FlexibleArgumentParser(description="Data Parallel Inference") [](#__codelineno-0-43) [](#__codelineno-0-44) # Add all engine args [](#__codelineno-0-45) EngineArgs.add_cli_args(parser) [](#__codelineno-0-46) parser.set_defaults( [](#__codelineno-0-47) model="ibm-research/PowerMoE-3b", [](#__codelineno-0-48) enable_expert_parallel=True, [](#__codelineno-0-49) ) [](#__codelineno-0-50) [](#__codelineno-0-51) # Add DP-specific args (separate from engine args to avoid conflicts) [](#__codelineno-0-52) parser.add_argument( [](#__codelineno-0-53) "--dp-num-nodes", [](#__codelineno-0-54) type=int, [](#__codelineno-0-55) default=1, [](#__codelineno-0-56) help="Total number of nodes for data parallel.", [](#__codelineno-0-57) ) [](#__codelineno-0-58) parser.add_argument( [](#__codelineno-0-59) "--dp-node-rank", [](#__codelineno-0-60) type=int, [](#__codelineno-0-61) default=0, [](#__codelineno-0-62) help="Rank of the current node for data parallel.", [](#__codelineno-0-63) ) [](#__codelineno-0-64) parser.add_argument( [](#__codelineno-0-65) "--dp-master-addr", [](#__codelineno-0-66) type=str, [](#__codelineno-0-67) default="", [](#__codelineno-0-68) help="Master node IP address for DP coordination.", [](#__codelineno-0-69) ) [](#__codelineno-0-70) parser.add_argument( [](#__codelineno-0-71) "--dp-master-port", [](#__codelineno-0-72) type=int, [](#__codelineno-0-73) default=0, [](#__codelineno-0-74) help="Master node port for DP coordination.", [](#__codelineno-0-75) ) [](#__codelineno-0-76) parser.add_argument( [](#__codelineno-0-77) "--timeout", [](#__codelineno-0-78) type=int, [](#__codelineno-0-79) default=300, [](#__codelineno-0-80) help="Number of seconds before unresponsive process is killed.", [](#__codelineno-0-81) ) [](#__codelineno-0-82) [](#__codelineno-0-83) return parser [](#__codelineno-0-84) [](#__codelineno-0-85)[](#__codelineno-0-86)def main( [](#__codelineno-0-87) dp_size, [](#__codelineno-0-88) local_dp_rank, [](#__codelineno-0-89) global_dp_rank, [](#__codelineno-0-90) dp_master_ip, [](#__codelineno-0-91) dp_master_port, [](#__codelineno-0-92) engine_args, [](#__codelineno-0-93)): [](#__codelineno-0-94) os.environ["VLLM_DP_RANK"] = str(global_dp_rank) [](#__codelineno-0-95) os.environ["VLLM_DP_RANK_LOCAL"] = str(local_dp_rank) [](#__codelineno-0-96) os.environ["VLLM_DP_SIZE"] = str(dp_size) [](#__codelineno-0-97) os.environ["VLLM_DP_MASTER_IP"] = dp_master_ip [](#__codelineno-0-98) os.environ["VLLM_DP_MASTER_PORT"] = str(dp_master_port) [](#__codelineno-0-99) [](#__codelineno-0-100) # CUDA_VISIBLE_DEVICES for each DP rank is set automatically inside the [](#__codelineno-0-101) # engine processes. [](#__codelineno-0-102) [](#__codelineno-0-103) # Sample prompts. [](#__codelineno-0-104) prompts = [ [](#__codelineno-0-105) "Hello, my name is", [](#__codelineno-0-106) "The president of the United States is", [](#__codelineno-0-107) "The capital of France is", [](#__codelineno-0-108) "The future of AI is", [](#__codelineno-0-109) ] * 100 [](#__codelineno-0-110) [](#__codelineno-0-111) # with DP, each rank should process different prompts. [](#__codelineno-0-112) # usually all the DP ranks process a full dataset, [](#__codelineno-0-113) # and each rank processes a different part of the dataset. [](#__codelineno-0-114) floor = len(prompts) // dp_size [](#__codelineno-0-115) remainder = len(prompts) % dp_size [](#__codelineno-0-116) [](#__codelineno-0-117) # Distribute prompts into even groups. [](#__codelineno-0-118) def start(rank): [](#__codelineno-0-119) return rank * floor + min(rank, remainder) [](#__codelineno-0-120) [](#__codelineno-0-121) prompts = prompts[start(global_dp_rank) : start(global_dp_rank + 1)] [](#__codelineno-0-122) if len(prompts) == 0: [](#__codelineno-0-123) # if any rank has no prompts to process, [](#__codelineno-0-124) # we need to set a placeholder prompt [](#__codelineno-0-125) prompts = ["Placeholder"] [](#__codelineno-0-126) print(f"DP rank {global_dp_rank} needs to process {len(prompts)} prompts") [](#__codelineno-0-127) [](#__codelineno-0-128) # Create a sampling params object. [](#__codelineno-0-129) # since we are doing data parallel, every rank can have different [](#__codelineno-0-130) # sampling params. here we set different max_tokens for different [](#__codelineno-0-131) # ranks for demonstration. [](#__codelineno-0-132) sampling_params = SamplingParams( [](#__codelineno-0-133) temperature=0.8, top_p=0.95, max_tokens=[16, 20][global_dp_rank % 2] [](#__codelineno-0-134) ) [](#__codelineno-0-135) [](#__codelineno-0-136) # Create an LLM. [](#__codelineno-0-137) llm = LLM(**engine_args) [](#__codelineno-0-138) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-139) # Print the outputs. [](#__codelineno-0-140) for i, output in enumerate(outputs): [](#__codelineno-0-141) if i >= 5: [](#__codelineno-0-142) # print only 5 outputs [](#__codelineno-0-143) break [](#__codelineno-0-144) prompt = output.prompt [](#__codelineno-0-145) generated_text = output.outputs[0].text [](#__codelineno-0-146) print( [](#__codelineno-0-147) f"DP rank {global_dp_rank}, Prompt: {prompt!r}, " [](#__codelineno-0-148) f"Generated text: {generated_text!r}" [](#__codelineno-0-149) ) [](#__codelineno-0-150) [](#__codelineno-0-151) # Give engines time to pause their processing loops before exiting. [](#__codelineno-0-152) sleep(1) [](#__codelineno-0-153) [](#__codelineno-0-154)[](#__codelineno-0-155)if __name__ == "__main__": [](#__codelineno-0-156) parser = create_parser() [](#__codelineno-0-157) args = vars(parser.parse_args()) [](#__codelineno-0-158) [](#__codelineno-0-159) # Extract DP-specific args (pop to remove from engine_args) [](#__codelineno-0-160) dp_size = args.pop("data_parallel_size") [](#__codelineno-0-161) dp_num_nodes = args.pop("dp_num_nodes") [](#__codelineno-0-162) dp_node_rank = args.pop("dp_node_rank") [](#__codelineno-0-163) dp_master_addr = args.pop("dp_master_addr") [](#__codelineno-0-164) dp_master_port = args.pop("dp_master_port") [](#__codelineno-0-165) timeout = args.pop("timeout") [](#__codelineno-0-166) [](#__codelineno-0-167) # Remaining args are engine args [](#__codelineno-0-168) engine_args = args [](#__codelineno-0-169) [](#__codelineno-0-170) if dp_num_nodes == 1: [](#__codelineno-0-171) dp_master_ip = "127.0.0.1" [](#__codelineno-0-172) dp_master_port_val = get_open_port() [](#__codelineno-0-173) else: [](#__codelineno-0-174) dp_master_ip = dp_master_addr [](#__codelineno-0-175) dp_master_port_val = dp_master_port [](#__codelineno-0-176) [](#__codelineno-0-177) assert dp_size % dp_num_nodes == 0, "dp_size should be divisible by dp_num_nodes" [](#__codelineno-0-178) dp_per_node = dp_size // dp_num_nodes [](#__codelineno-0-179) [](#__codelineno-0-180) from multiprocessing import Process [](#__codelineno-0-181) [](#__codelineno-0-182) if current_platform.is_rocm(): [](#__codelineno-0-183) from multiprocessing import set_start_method [](#__codelineno-0-184) [](#__codelineno-0-185) set_start_method("spawn", force=True) [](#__codelineno-0-186) [](#__codelineno-0-187) procs = [] [](#__codelineno-0-188) for local_dp_rank, global_dp_rank in enumerate( [](#__codelineno-0-189) range(dp_node_rank * dp_per_node, (dp_node_rank + 1) * dp_per_node) [](#__codelineno-0-190) ): [](#__codelineno-0-191) proc = Process( [](#__codelineno-0-192) target=main, [](#__codelineno-0-193) args=( [](#__codelineno-0-194) dp_size, [](#__codelineno-0-195) local_dp_rank, [](#__codelineno-0-196) global_dp_rank, [](#__codelineno-0-197) dp_master_ip, [](#__codelineno-0-198) dp_master_port_val, [](#__codelineno-0-199) engine_args, [](#__codelineno-0-200) ), [](#__codelineno-0-201) ) [](#__codelineno-0-202) proc.start() [](#__codelineno-0-203) procs.append(proc) [](#__codelineno-0-204) exit_code = 0 [](#__codelineno-0-205) for proc in procs: [](#__codelineno-0-206) proc.join(timeout=timeout) [](#__codelineno-0-207) if proc.exitcode is None: [](#__codelineno-0-208) print(f"Killing process {proc.pid} that didn't stop within 5 minutes.") [](#__codelineno-0-209) proc.kill() [](#__codelineno-0-210) exit_code = 1 [](#__codelineno-0-211) elif proc.exitcode: [](#__codelineno-0-212) exit_code = proc.exitcode [](#__codelineno-0-213) [](#__codelineno-0-214) exit(exit_code)` ## Multi Instance Data Parallel[¶](#multi-instance-data-parallel "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)import asyncio [](#__codelineno-1-4)import threading [](#__codelineno-1-5)[](#__codelineno-1-6)from vllm.engine.arg_utils import AsyncEngineArgs [](#__codelineno-1-7)from vllm.engine.async_llm_engine import AsyncLLMEngine [](#__codelineno-1-8)from vllm.outputs import RequestOutput [](#__codelineno-1-9)from vllm.sampling_params import SamplingParams [](#__codelineno-1-10)from vllm.v1.metrics.loggers import AggregatedLoggingStatLogger [](#__codelineno-1-11)[](#__codelineno-1-12)""" [](#__codelineno-1-13)To run this example, run the following commands simultaneously with [](#__codelineno-1-14)different CUDA_VISIBLE_DEVICES: [](#__codelineno-1-15) python examples/features/data_parallel/multi_instance_data_parallel.py [](#__codelineno-1-16) [](#__codelineno-1-17) vllm serve ibm-research/PowerMoE-3b -dp 2 -dpr 1 \ [](#__codelineno-1-18) --data-parallel-address 127.0.0.1 --data-parallel-rpc-port 62300 \ [](#__codelineno-1-19) --data-parallel-size-local 1 --enforce-eager --headless [](#__codelineno-1-20)[](#__codelineno-1-21)Once both instances have completed the handshake, this example will [](#__codelineno-1-22)send a request to the instance with DP rank 1. [](#__codelineno-1-23)""" [](#__codelineno-1-24) [](#__codelineno-1-25)[](#__codelineno-1-26)def _do_background_logging(engine, interval, stop_event): [](#__codelineno-1-27) try: [](#__codelineno-1-28) while not stop_event.is_set(): [](#__codelineno-1-29) asyncio.run(engine.do_log_stats()) [](#__codelineno-1-30) stop_event.wait(interval) [](#__codelineno-1-31) except Exception as e: [](#__codelineno-1-32) print(f"vLLM background logging shutdown: {e}") [](#__codelineno-1-33) pass [](#__codelineno-1-34) [](#__codelineno-1-35)[](#__codelineno-1-36)async def main(): [](#__codelineno-1-37) engine_args = AsyncEngineArgs( [](#__codelineno-1-38) model="ibm-research/PowerMoE-3b", [](#__codelineno-1-39) data_parallel_size=2, [](#__codelineno-1-40) tensor_parallel_size=1, [](#__codelineno-1-41) dtype="auto", [](#__codelineno-1-42) max_model_len=2048, [](#__codelineno-1-43) data_parallel_address="127.0.0.1", [](#__codelineno-1-44) data_parallel_rpc_port=62300, [](#__codelineno-1-45) data_parallel_size_local=1, [](#__codelineno-1-46) enforce_eager=True, [](#__codelineno-1-47) enable_log_requests=True, [](#__codelineno-1-48) disable_custom_all_reduce=True, [](#__codelineno-1-49) ) [](#__codelineno-1-50) [](#__codelineno-1-51) engine_client = AsyncLLMEngine.from_engine_args( [](#__codelineno-1-52) engine_args, [](#__codelineno-1-53) # Example: Using aggregated logger [](#__codelineno-1-54) stat_loggers=[AggregatedLoggingStatLogger], [](#__codelineno-1-55) ) [](#__codelineno-1-56) stop_logging_event = threading.Event() [](#__codelineno-1-57) logging_thread = threading.Thread( [](#__codelineno-1-58) target=_do_background_logging, [](#__codelineno-1-59) args=(engine_client, 5, stop_logging_event), [](#__codelineno-1-60) daemon=True, [](#__codelineno-1-61) ) [](#__codelineno-1-62) logging_thread.start() [](#__codelineno-1-63) sampling_params = SamplingParams( [](#__codelineno-1-64) temperature=0.7, [](#__codelineno-1-65) top_p=0.9, [](#__codelineno-1-66) max_tokens=100, [](#__codelineno-1-67) ) [](#__codelineno-1-68) num_prompts = 10 [](#__codelineno-1-69) for i in range(num_prompts): [](#__codelineno-1-70) prompt = "Who won the 2004 World Series?" [](#__codelineno-1-71) final_output: RequestOutput | None = None [](#__codelineno-1-72) async for output in engine_client.generate( [](#__codelineno-1-73) prompt=prompt, [](#__codelineno-1-74) sampling_params=sampling_params, [](#__codelineno-1-75) request_id=f"abcdef-{i}", [](#__codelineno-1-76) data_parallel_rank=1, [](#__codelineno-1-77) ): [](#__codelineno-1-78) final_output = output [](#__codelineno-1-79) if final_output: [](#__codelineno-1-80) print(final_output.outputs[0].text) [](#__codelineno-1-81) [](#__codelineno-1-82) stop_logging_event.set() [](#__codelineno-1-83) logging_thread.join() [](#__codelineno-1-84) [](#__codelineno-1-85)[](#__codelineno-1-86)if __name__ == "__main__": [](#__codelineno-1-87) asyncio.run(main())` --- # page 1. [Home](https://docs.vllm.ai/en/) 2. [User Guide](https://docs.vllm.ai/en/usage/) 3. [Getting Started](https://docs.vllm.ai/en/getting_started/quickstart/) 4. [Examples](https://docs.vllm.ai/en/latest/) 5. [Features](https://docs.vllm.ai/en/latest/examples/automatic_prefix_caching/) [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/kv_events.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/kv\_events](https://github.com/vllm-project/vllm/tree/main/examples/features/kv_events). ## Kv Events Subscriber[¶](#kv-events-subscriber "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)from typing import Any [](#__codelineno-0-4)[](#__codelineno-0-5)import msgspec [](#__codelineno-0-6)import zmq [](#__codelineno-0-7)from msgspec.msgpack import Decoder [](#__codelineno-0-8)[](#__codelineno-0-9)from vllm.v1.core.kv_cache_utils import ExternalBlockHash [](#__codelineno-0-10) [](#__codelineno-0-11)[](#__codelineno-0-12)# [](#__codelineno-0-13)# Types copied from vllm.distributed.kv_events [](#__codelineno-0-14)# [](#__codelineno-0-15)class EventBatch(msgspec.Struct, array_like=True, omit_defaults=True, gc=False): [](#__codelineno-0-16) ts: float [](#__codelineno-0-17) events: list[Any] [](#__codelineno-0-18) [](#__codelineno-0-19)[](#__codelineno-0-20)class KVCacheEvent( [](#__codelineno-0-21) msgspec.Struct, array_like=True, omit_defaults=True, gc=False, tag=True [](#__codelineno-0-22)): [](#__codelineno-0-23) """Base class for all KV cache-related events""" [](#__codelineno-0-24) [](#__codelineno-0-25)[](#__codelineno-0-26)class BlockStored(KVCacheEvent): [](#__codelineno-0-27) block_hashes: list[ExternalBlockHash] [](#__codelineno-0-28) parent_block_hash: ExternalBlockHash | None [](#__codelineno-0-29) token_ids: list[int] [](#__codelineno-0-30) block_size: int [](#__codelineno-0-31) [](#__codelineno-0-32) lora_id: int | None [](#__codelineno-0-33) """Deprecated: use `lora_name` for KV block key hash. [](#__codelineno-0-34) Retained for backward compatibility. [](#__codelineno-0-35) """ [](#__codelineno-0-36) [](#__codelineno-0-37) medium: str | None [](#__codelineno-0-38) lora_name: str | None [](#__codelineno-0-39) [](#__codelineno-0-40) extra_keys: list[tuple[Any, ...] | None] | None = None [](#__codelineno-0-41) """Extra keys used in block hash computation, one entry per block in [](#__codelineno-0-42) block_hashes. Each entry contains MM identifiers, LoRA name, cache_salt, [](#__codelineno-0-43) prompt embeddings data, etc. for that specific block. [](#__codelineno-0-44) """ [](#__codelineno-0-45) [](#__codelineno-0-46) group_idx: int | None = None [](#__codelineno-0-47) [](#__codelineno-0-48)[](#__codelineno-0-49)class BlockRemoved(KVCacheEvent): [](#__codelineno-0-50) block_hashes: list[ExternalBlockHash] [](#__codelineno-0-51) medium: str | None [](#__codelineno-0-52) group_idx: int | None = None [](#__codelineno-0-53) [](#__codelineno-0-54)[](#__codelineno-0-55)class AllBlocksCleared(KVCacheEvent): [](#__codelineno-0-56) pass [](#__codelineno-0-57) [](#__codelineno-0-58)[](#__codelineno-0-59)class KVEventBatch(EventBatch): [](#__codelineno-0-60) events: list[BlockStored | BlockRemoved | AllBlocksCleared] [](#__codelineno-0-61) [](#__codelineno-0-62)[](#__codelineno-0-63)def process_event(event_batch): [](#__codelineno-0-64) print(f"Received event batch at {event_batch.ts}:") [](#__codelineno-0-65) for event in event_batch.events: [](#__codelineno-0-66) print(f" - {event}") [](#__codelineno-0-67) [](#__codelineno-0-68)[](#__codelineno-0-69)def main(): [](#__codelineno-0-70) decoder = Decoder(type=KVEventBatch) [](#__codelineno-0-71) last_seq = -1 [](#__codelineno-0-72) [](#__codelineno-0-73) context = zmq.Context() [](#__codelineno-0-74) [](#__codelineno-0-75) # Set up the main subscription socket [](#__codelineno-0-76) sub = context.socket(zmq.SUB) [](#__codelineno-0-77) sub.connect("tcp://localhost:5557") [](#__codelineno-0-78) topic = "kv-events" [](#__codelineno-0-79) sub.setsockopt_string(zmq.SUBSCRIBE, topic) [](#__codelineno-0-80) [](#__codelineno-0-81) # Initialize replay socket [](#__codelineno-0-82) replay = context.socket(zmq.REQ) [](#__codelineno-0-83) replay.connect("tcp://localhost:5558") [](#__codelineno-0-84) poller = zmq.Poller() [](#__codelineno-0-85) poller.register(replay, zmq.POLLIN) [](#__codelineno-0-86) [](#__codelineno-0-87) print("Listening for KV cache events on topic:", topic) [](#__codelineno-0-88) [](#__codelineno-0-89) while True: [](#__codelineno-0-90) try: [](#__codelineno-0-91) if sub.poll(50): [](#__codelineno-0-92) _, seq_bytes, payload = sub.recv_multipart() [](#__codelineno-0-93) seq = int.from_bytes(seq_bytes, "big") [](#__codelineno-0-94) [](#__codelineno-0-95) if last_seq >= 0 and seq > last_seq + 1: [](#__codelineno-0-96) missed = seq - last_seq - 1 [](#__codelineno-0-97) print( [](#__codelineno-0-98) f"Missed {missed} messages (last: {last_seq}, current: {seq})" [](#__codelineno-0-99) ) [](#__codelineno-0-100) [](#__codelineno-0-101) replay.send((last_seq + 1).to_bytes(8, "big")) [](#__codelineno-0-102) [](#__codelineno-0-103) while poller.poll(timeout=200): [](#__codelineno-0-104) seq_bytes, replay_payload = replay.recv_multipart() [](#__codelineno-0-105) if not replay_payload: [](#__codelineno-0-106) # End of replay marker is sent as an empty frame [](#__codelineno-0-107) # for the payload [](#__codelineno-0-108) break [](#__codelineno-0-109) [](#__codelineno-0-110) replay_seq = int.from_bytes(seq_bytes, "big") [](#__codelineno-0-111) [](#__codelineno-0-112) if replay_seq > last_seq: [](#__codelineno-0-113) event_batch = decoder.decode(replay_payload) [](#__codelineno-0-114) process_event(event_batch) [](#__codelineno-0-115) last_seq = replay_seq [](#__codelineno-0-116) if replay_seq >= seq - 1: [](#__codelineno-0-117) break [](#__codelineno-0-118) [](#__codelineno-0-119) event_batch = decoder.decode(payload) [](#__codelineno-0-120) process_event(event_batch) [](#__codelineno-0-121) [](#__codelineno-0-122) # ... do other periodic work or check for shutdown ... [](#__codelineno-0-123) [](#__codelineno-0-124) except KeyboardInterrupt: [](#__codelineno-0-125) print("Interrupted") [](#__codelineno-0-126) break [](#__codelineno-0-127) except Exception as e: [](#__codelineno-0-128) print("Error decoding message:", e) [](#__codelineno-0-129) [](#__codelineno-0-130)[](#__codelineno-0-131)if __name__ == "__main__": [](#__codelineno-0-132) main()`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/logging_configuration.md "Edit this page") Source [https://github.com/vllm-project/vllm/blob/main/examples/features/logging\_configuration.md](https://github.com/vllm-project/vllm/blob/main/examples/features/logging_configuration.md). vLLM leverages Python's `logging.config.dictConfig` functionality to enable robust and flexible configuration of the various loggers used by vLLM. vLLM offers two environment variables that can be used to accommodate a range of logging configurations that range from simple-and-inflexible to more-complex-and-more-flexible. - No vLLM logging (simple and inflexible) - Set `VLLM_CONFIGURE_LOGGING=0` (leaving `VLLM_LOGGING_CONFIG_PATH` unset) - vLLM's default logging configuration (simple and inflexible) - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` - Fine-grained custom logging configuration (more complex, more flexible) - Leave `VLLM_CONFIGURE_LOGGING` unset or set `VLLM_CONFIGURE_LOGGING=1` and set `VLLM_LOGGING_CONFIG_PATH=` ## Logging Configuration Environment Variables[¶](#logging-configuration-environment-variables "Permanent link") ### `VLLM_CONFIGURE_LOGGING`[¶](#vllm_configure_logging "Permanent link") `VLLM_CONFIGURE_LOGGING` controls whether or not vLLM takes any action to configure the loggers used by vLLM. This functionality is enabled by default, but can be disabled by setting `VLLM_CONFIGURE_LOGGING=0` when running vLLM. If `VLLM_CONFIGURE_LOGGING` is enabled and no value is given for `VLLM_LOGGING_CONFIG_PATH`, vLLM will use built-in default configuration to configure the root vLLM logger. By default, no other vLLM loggers are configured and, as such, all vLLM loggers defer to the root vLLM logger to make all logging decisions. If `VLLM_CONFIGURE_LOGGING` is disabled and a value is given for `VLLM_LOGGING_CONFIG_PATH`, an error will occur while starting vLLM. ### `VLLM_LOGGING_CONFIG_PATH`[¶](#vllm_logging_config_path "Permanent link") `VLLM_LOGGING_CONFIG_PATH` allows users to specify a path to a JSON file of alternative, custom logging configuration that will be used instead of vLLM's built-in default logging configuration. The logging configuration should be provided in JSON format following the schema specified by Python's [logging configuration dictionary schema](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details). If `VLLM_LOGGING_CONFIG_PATH` is specified, but `VLLM_CONFIGURE_LOGGING` is disabled, an error will occur while starting vLLM. ## Examples[¶](#examples "Permanent link") ### Example 1: Customize vLLM root logger[¶](#example-1-customize-vllm-root-logger "Permanent link") For this example, we will customize the vLLM root logger to use [`python-json-logger`](https://github.com/nhairs/python-json-logger) (which is part of the container image) to log to STDOUT of the console in JSON format with a log level of `INFO`. To begin, first, create an appropriate JSON logging configuration file: /path/to/logging\_config.json `[](#__codelineno-0-1){ [](#__codelineno-0-2) "formatters": { [](#__codelineno-0-3) "json": { [](#__codelineno-0-4) "class": "pythonjsonlogger.jsonlogger.JsonFormatter" [](#__codelineno-0-5) } [](#__codelineno-0-6) }, [](#__codelineno-0-7) "handlers": { [](#__codelineno-0-8) "console": { [](#__codelineno-0-9) "class" : "logging.StreamHandler", [](#__codelineno-0-10) "formatter": "json", [](#__codelineno-0-11) "level": "INFO", [](#__codelineno-0-12) "stream": "ext://sys.stdout" [](#__codelineno-0-13) } [](#__codelineno-0-14) }, [](#__codelineno-0-15) "loggers": { [](#__codelineno-0-16) "vllm": { [](#__codelineno-0-17) "handlers": ["console"], [](#__codelineno-0-18) "level": "INFO", [](#__codelineno-0-19) "propagate": false [](#__codelineno-0-20) } [](#__codelineno-0-21) }, [](#__codelineno-0-22) "version": 1 [](#__codelineno-0-23)}` Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set to the path of the custom logging configuration JSON file: `[](#__codelineno-1-1)VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \ [](#__codelineno-1-2) vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048` ### Example 2: Silence a particular vLLM logger[¶](#example-2-silence-a-particular-vllm-logger "Permanent link") To silence a particular vLLM logger, it is necessary to provide custom logging configuration for the target logger that configures the logger so that it won't propagate its log messages to the root vLLM logger. When custom configuration is provided for any logger, it is also necessary to provide configuration for the root vLLM logger since any custom logger configuration overrides the built-in default logging configuration used by vLLM. First, create an appropriate JSON logging configuration file that includes configuration for the root vLLM logger and for the logger you wish to silence: /path/to/logging\_config.json `[](#__codelineno-2-1){ [](#__codelineno-2-2) "formatters": { [](#__codelineno-2-3) "vllm": { [](#__codelineno-2-4) "class": "vllm.logging_utils.NewLineFormatter", [](#__codelineno-2-5) "datefmt": "%m-%d %H:%M:%S", [](#__codelineno-2-6) "format": "%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s" [](#__codelineno-2-7) } [](#__codelineno-2-8) }, [](#__codelineno-2-9) "handlers": { [](#__codelineno-2-10) "vllm": { [](#__codelineno-2-11) "class" : "logging.StreamHandler", [](#__codelineno-2-12) "formatter": "vllm", [](#__codelineno-2-13) "level": "INFO", [](#__codelineno-2-14) "stream": "ext://sys.stdout" [](#__codelineno-2-15) } [](#__codelineno-2-16) }, [](#__codelineno-2-17) "loggers": { [](#__codelineno-2-18) "vllm": { [](#__codelineno-2-19) "handlers": ["vllm"], [](#__codelineno-2-20) "level": "DEBUG", [](#__codelineno-2-21) "propagate": false [](#__codelineno-2-22) }, [](#__codelineno-2-23) "vllm.example_noisy_logger": { [](#__codelineno-2-24) "propagate": false [](#__codelineno-2-25) } [](#__codelineno-2-26) }, [](#__codelineno-2-27) "version": 1 [](#__codelineno-2-28)}` Finally, run vLLM with the `VLLM_LOGGING_CONFIG_PATH` environment variable set to the path of the custom logging configuration JSON file: `[](#__codelineno-3-1)VLLM_LOGGING_CONFIG_PATH=/path/to/logging_config.json \ [](#__codelineno-3-2) vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048` ### Example 3: Disable vLLM default logging configuration[¶](#example-3-disable-vllm-default-logging-configuration "Permanent link") To disable vLLM's default logging configuration and silence all vLLM loggers, simple set `VLLM_CONFIGURE_LOGGING=0` when running vLLM. This will prevent vLLM for configuring the root vLLM logger, which in turn, silences all other vLLM loggers. `[](#__codelineno-4-1)VLLM_CONFIGURE_LOGGING=0 \ [](#__codelineno-4-2) vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048` ### Example 4: Disable access logs for health check endpoints[¶](#example-4-disable-access-logs-for-health-check-endpoints "Permanent link") In production environments, health check endpoints like `/health`, `/metrics`, and `/ping` are frequently called by load balancers and monitoring systems, generating a large volume of repetitive access logs. To reduce log noise while keeping logs for other endpoints, use the `--disable-access-log-for-endpoints` option. **Disable access logs for health and metrics endpoints:** `[](#__codelineno-5-1)vllm serve mistralai/Mistral-7B-v0.1 --max-model-len 2048 \ [](#__codelineno-5-2) --disable-access-log-for-endpoints /health,/metrics,/ping` **Common endpoints to consider filtering:** Endpoint Description Typical Caller `/health` Health check Kubernetes liveness/readiness probes, load balancers `/metrics` Prometheus metrics Prometheus scraper (every 15-60s) `/ping` SageMaker health check SageMaker infrastructure `/load` Server load metrics Custom monitoring **Notes:** - This option only affects uvicorn access logs, not vLLM application logs - Specify multiple endpoints by separating them with commas (no spaces) - The filter uses exact path matching, query parameters are ignored (e.g., `/health?verbose=true` matches `/health`) - If you need to completely disable all access logs, use `--disable-uvicorn-access-log` instead ## Additional resources[¶](#additional-resources "Permanent link") - [`logging.config` Dictionary Schema Details](https://docs.python.org/3/library/logging.config.html#dictionary-schema-details) --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/logits_processor.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/logits\_processor](https://github.com/vllm-project/vllm/tree/main/examples/features/logits_processor). This directory contains examples demonstrating how to use custom logits processors with vLLM's offline inference API. Logits processors allow you to modify the model's output distribution before sampling, enabling controlled generation behaviors like token masking, constrained decoding, and custom sampling strategies. ## Scripts[¶](#scripts "Permanent link") ### `custom.py` — Engine-level logits processor[¶](#custompy-engine-level-logits-processor "Permanent link") Demonstrates how to instantiate vLLM with a custom logits processor class that operates at the batch level. The example uses a `DummyLogitsProcessor` that masks out all tokens except a specified `target_token` when passed via `SamplingParams.extra_args`. `[](#__codelineno-0-1)python examples/features/logits_processor/custom.py` ### `custom_req.py` — Request-level logits processor wrapper[¶](#custom_reqpy-request-level-logits-processor-wrapper "Permanent link") Shows how to wrap a request-level logits processor (which operates on individual requests) to be compatible with vLLM's batch-level logits processing interface. `[](#__codelineno-1-1)python examples/features/logits_processor/custom_req.py` ### `custom_req_init.py` — Request-level processor with engine config[¶](#custom_req_initpy-request-level-processor-with-engine-config "Permanent link") A special case of wrapping a request-level logits processor where the processor needs access to engine configuration or model metadata during initialization (e.g., vocabulary size, tokenizer info). `[](#__codelineno-2-1)python examples/features/logits_processor/custom_req_init.py` ## Key Concepts[¶](#key-concepts "Permanent link") - **Batch-level vs. request-level**: vLLM processes logits at the batch level for efficiency. If you have a per-request processor, you need to wrap it using the patterns shown in `custom_req.py` and `custom_req_init.py`. - **`SamplingParams.extra_args`**: Use this to pass custom keyword arguments to your logits processor on a per-request basis (e.g., `target_token`). - **`DummyLogitsProcessor`**: A reference implementation available in `vllm/test_utils.py` that can be used as a starting point for custom processors. ## Further Reading[¶](#further-reading "Permanent link") - [vLLM Sampling Parameters](https://docs.vllm.ai/en/latest/api/inference_params.html#sampling-parameters) - [vLLM LLM API](https://docs.vllm.ai/en/latest/api/offline_inference/llm.html) ## Example materials[¶](#example-materials "Permanent link") custom.py ``[](#__codelineno-3-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-3-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-3-3)[](#__codelineno-3-4)"""This example demonstrates instantiating vLLM with a custom logits processor [](#__codelineno-3-5)class object. [](#__codelineno-3-6)[](#__codelineno-3-7)For a basic example of implementing a custom logits processor, see [](#__codelineno-3-8)the `DummyLogitsProcessor` implementation in `vllm/test_utils.py`. [](#__codelineno-3-9)[](#__codelineno-3-10)For testing purposes, a dummy logits processor is employed which, if [](#__codelineno-3-11)`target_token` is passed as a keyword argument to `SamplingParams.extra_args`, [](#__codelineno-3-12)will mask out all tokens except `target_token`. [](#__codelineno-3-13)[](#__codelineno-3-14)A batch is constructed with `temperature=0.0` and 50% of requests specifying [](#__codelineno-3-15)`target_token`, and for these requests - and *only* these requests - we [](#__codelineno-3-16)expect the `target_token` to be decoded in each step, yielding an output [](#__codelineno-3-17)similar to that shown below: [](#__codelineno-3-18)[](#__codelineno-3-19)Generated Outputs: [](#__codelineno-3-20)------------------------------------------------------------ [](#__codelineno-3-21)Prompt: 'Hello, my name is' [](#__codelineno-3-22)Output: " ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '" [](#__codelineno-3-23)------------------------------------------------------------ [](#__codelineno-3-24)Prompt: 'The president of the United States is' [](#__codelineno-3-25)Output: " not a racist. He is a racist.\nHe's a racist because he" [](#__codelineno-3-26)------------------------------------------------------------ [](#__codelineno-3-27)Prompt: 'The capital of France is' [](#__codelineno-3-28)Output: ' also also also also also also also also also also also also also [](#__codelineno-3-29) also also also' [](#__codelineno-3-30)------------------------------------------------------------ [](#__codelineno-3-31)Prompt: 'The future of AI is' [](#__codelineno-3-32)Output: ' in the hands of the people.\n\nThe future of AI is in the' [](#__codelineno-3-33)------------------------------------------------------------ [](#__codelineno-3-34)""" [](#__codelineno-3-35)[](#__codelineno-3-36)from typing import Any [](#__codelineno-3-37)[](#__codelineno-3-38)import torch [](#__codelineno-3-39)[](#__codelineno-3-40)from vllm import LLM, SamplingParams [](#__codelineno-3-41)from vllm.config import VllmConfig [](#__codelineno-3-42)from vllm.v1.sample.logits_processor import ( [](#__codelineno-3-43) BatchUpdate, [](#__codelineno-3-44) LogitsProcessor, [](#__codelineno-3-45)) [](#__codelineno-3-46)from vllm.v1.sample.logits_processor.builtin import process_dict_updates [](#__codelineno-3-47) [](#__codelineno-3-48)[](#__codelineno-3-49)# Hypothetical custom logits processor [](#__codelineno-3-50)class DummyLogitsProcessor(LogitsProcessor): [](#__codelineno-3-51) """Fake logit processor to support unit testing and examples""" [](#__codelineno-3-52) [](#__codelineno-3-53) @classmethod [](#__codelineno-3-54) def validate_params(cls, params: SamplingParams): [](#__codelineno-3-55) target_token: Any | None = params.extra_args and params.extra_args.get( [](#__codelineno-3-56) "target_token" [](#__codelineno-3-57) ) [](#__codelineno-3-58) if target_token is not None and not isinstance(target_token, int): [](#__codelineno-3-59) raise ValueError( [](#__codelineno-3-60) f"target_token value {target_token} {type(target_token)} is not int" [](#__codelineno-3-61) ) [](#__codelineno-3-62) [](#__codelineno-3-63) def __init__( [](#__codelineno-3-64) self, vllm_config: VllmConfig, device: torch.device, is_pin_memory: bool [](#__codelineno-3-65) ): [](#__codelineno-3-66) self.req_info: dict[int, int] = {} [](#__codelineno-3-67) [](#__codelineno-3-68) def is_argmax_invariant(self) -> bool: [](#__codelineno-3-69) return False [](#__codelineno-3-70) [](#__codelineno-3-71) def update_state(self, batch_update: BatchUpdate | None): [](#__codelineno-3-72) def extract_extra_arg(params: SamplingParams) -> int | None: [](#__codelineno-3-73) self.validate_params(params) [](#__codelineno-3-74) return params.extra_args and params.extra_args.get("target_token") [](#__codelineno-3-75) [](#__codelineno-3-76) process_dict_updates( [](#__codelineno-3-77) self.req_info, [](#__codelineno-3-78) batch_update, [](#__codelineno-3-79) # This function returns the LP's per-request state based on the [](#__codelineno-3-80) # request details, or None if this LP does not apply to the [](#__codelineno-3-81) # request. [](#__codelineno-3-82) lambda params, _, __: extract_extra_arg(params), [](#__codelineno-3-83) ) [](#__codelineno-3-84) [](#__codelineno-3-85) def apply(self, logits: torch.Tensor) -> torch.Tensor: [](#__codelineno-3-86) if not self.req_info: [](#__codelineno-3-87) return logits [](#__codelineno-3-88) [](#__codelineno-3-89) # Save target values before modification [](#__codelineno-3-90) cols = torch.tensor( [](#__codelineno-3-91) list(self.req_info.values()), dtype=torch.long, device=logits.device [](#__codelineno-3-92) ) [](#__codelineno-3-93) rows = torch.tensor( [](#__codelineno-3-94) list(self.req_info.keys()), dtype=torch.long, device=logits.device [](#__codelineno-3-95) ) [](#__codelineno-3-96) values_to_keep = logits[rows, cols].clone() [](#__codelineno-3-97) [](#__codelineno-3-98) # Mask all but target tokens [](#__codelineno-3-99) logits[rows] = float("-inf") [](#__codelineno-3-100) logits[rows, cols] = values_to_keep [](#__codelineno-3-101) [](#__codelineno-3-102) return logits [](#__codelineno-3-103) [](#__codelineno-3-104)[](#__codelineno-3-105)# Sample prompts. [](#__codelineno-3-106)prompts = [ [](#__codelineno-3-107) "Hello, my name is", [](#__codelineno-3-108) "The president of the United States is", [](#__codelineno-3-109) "The capital of France is", [](#__codelineno-3-110) "The future of AI is", [](#__codelineno-3-111)] [](#__codelineno-3-112)# Create a mixture of requests which do and don't utilize the dummy logitproc [](#__codelineno-3-113)sampling_params_list = [ [](#__codelineno-3-114) SamplingParams(temperature=0.0, extra_args={"target_token": 128}), [](#__codelineno-3-115) SamplingParams(temperature=0.0), [](#__codelineno-3-116) SamplingParams(temperature=0.0, extra_args={"target_token": 67}), [](#__codelineno-3-117) SamplingParams(temperature=0.0), [](#__codelineno-3-118)] [](#__codelineno-3-119) [](#__codelineno-3-120)[](#__codelineno-3-121)def main(): [](#__codelineno-3-122) # Create an LLM. [](#__codelineno-3-123) llm = LLM( [](#__codelineno-3-124) model="facebook/opt-125m", [](#__codelineno-3-125) logits_processors=[DummyLogitsProcessor], [](#__codelineno-3-126) ) [](#__codelineno-3-127) # Generate texts from the prompts. [](#__codelineno-3-128) # The output is a list of RequestOutput objects [](#__codelineno-3-129) # that contain the prompt, generated text, and other information. [](#__codelineno-3-130) outputs = llm.generate(prompts, sampling_params_list) [](#__codelineno-3-131) # Print the outputs. [](#__codelineno-3-132) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-3-133) for output in outputs: [](#__codelineno-3-134) prompt = output.prompt [](#__codelineno-3-135) generated_text = output.outputs[0].text [](#__codelineno-3-136) print(f"Prompt: {prompt!r}") [](#__codelineno-3-137) print(f"Output: {generated_text!r}") [](#__codelineno-3-138) print("-" * 60) [](#__codelineno-3-139) [](#__codelineno-3-140)[](#__codelineno-3-141)if __name__ == "__main__": [](#__codelineno-3-142) main()`` custom\_req.py ``[](#__codelineno-4-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-4-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-4-3)[](#__codelineno-4-4)"""This example demonstrates wrapping a request-level logits processor to be [](#__codelineno-4-5)compatible with vLLM's batch-level logits processing [](#__codelineno-4-6)[](#__codelineno-4-7)For demo purposes, a dummy logits processor is employed which, if [](#__codelineno-4-8)`target_token` is passed as a keyword argument to `SamplingParams.extra_args`, [](#__codelineno-4-9)will mask out all tokens except `target_token`. This logits processor can be [](#__codelineno-4-10)applied to a vector of logits associated with a single decode step for a single [](#__codelineno-4-11)request. The logits processor cannot be applied to a request which does not [](#__codelineno-4-12)pass in a `target_token` custom argument. [](#__codelineno-4-13)[](#__codelineno-4-14)The request-level dummy logits processor is wrapped to create a batch-level [](#__codelineno-4-15)logits processor, which can apply the logits processor to output logits from [](#__codelineno-4-16)all requests in the persistent batch in a given decode step. For requests which [](#__codelineno-4-17)do not provide a `target_token` argument, the corresponding row of `logits` [](#__codelineno-4-18)will not be modified. [](#__codelineno-4-19)[](#__codelineno-4-20)A batch is constructed with `temperature=0.0` and 50% of requests specifying [](#__codelineno-4-21)`target_token`, and for these requests - and *only* these requests - we [](#__codelineno-4-22)expect the `target_token` to be decoded in each step, yielding an output [](#__codelineno-4-23)similar to that shown below: [](#__codelineno-4-24)[](#__codelineno-4-25)Generated Outputs: [](#__codelineno-4-26)------------------------------------------------------------ [](#__codelineno-4-27)Prompt: 'Hello, my name is' [](#__codelineno-4-28)Output: " ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '" [](#__codelineno-4-29)------------------------------------------------------------ [](#__codelineno-4-30)Prompt: 'The president of the United States is' [](#__codelineno-4-31)Output: " not a racist. He is a racist.\nHe's a racist because he" [](#__codelineno-4-32)------------------------------------------------------------ [](#__codelineno-4-33)Prompt: 'The capital of France is' [](#__codelineno-4-34)Output: ' also also also also also also also also also also also also also [](#__codelineno-4-35) also also also' [](#__codelineno-4-36)------------------------------------------------------------ [](#__codelineno-4-37)Prompt: 'The future of AI is' [](#__codelineno-4-38)Output: ' in the hands of the people.\n\nThe future of AI is in the' [](#__codelineno-4-39)------------------------------------------------------------ [](#__codelineno-4-40)""" [](#__codelineno-4-41)[](#__codelineno-4-42)from typing import Any [](#__codelineno-4-43)[](#__codelineno-4-44)import torch [](#__codelineno-4-45)[](#__codelineno-4-46)from vllm import LLM, SamplingParams [](#__codelineno-4-47)from vllm.logger import init_logger [](#__codelineno-4-48)from vllm.v1.sample.logits_processor import ( [](#__codelineno-4-49) AdapterLogitsProcessor, [](#__codelineno-4-50) RequestLogitsProcessor, [](#__codelineno-4-51)) [](#__codelineno-4-52)[](#__codelineno-4-53)logger = init_logger(__name__) [](#__codelineno-4-54) [](#__codelineno-4-55)[](#__codelineno-4-56)class DummyPerReqLogitsProcessor: [](#__codelineno-4-57) """The request-level logits processor masks out all logits except the [](#__codelineno-4-58) token id identified by `target_token`""" [](#__codelineno-4-59) [](#__codelineno-4-60) def __init__(self, target_token: int) -> None: [](#__codelineno-4-61) """Specify `target_token`""" [](#__codelineno-4-62) self.target_token = target_token [](#__codelineno-4-63) [](#__codelineno-4-64) def __call__( [](#__codelineno-4-65) self, [](#__codelineno-4-66) output_ids: list[int], [](#__codelineno-4-67) logits: torch.Tensor, [](#__codelineno-4-68) ) -> torch.Tensor: [](#__codelineno-4-69) val_to_keep = logits[self.target_token].item() [](#__codelineno-4-70) logits[:] = float("-inf") [](#__codelineno-4-71) logits[self.target_token] = val_to_keep [](#__codelineno-4-72) return logits [](#__codelineno-4-73) [](#__codelineno-4-74)[](#__codelineno-4-75)class WrappedPerReqLogitsProcessor(AdapterLogitsProcessor): [](#__codelineno-4-76) """Example of wrapping a fake request-level logit processor to create a [](#__codelineno-4-77) batch-level logits processor""" [](#__codelineno-4-78) [](#__codelineno-4-79) @classmethod [](#__codelineno-4-80) def validate_params(cls, params: SamplingParams): [](#__codelineno-4-81) target_token: Any | None = params.extra_args and params.extra_args.get( [](#__codelineno-4-82) "target_token" [](#__codelineno-4-83) ) [](#__codelineno-4-84) if target_token is not None and not isinstance(target_token, int): [](#__codelineno-4-85) raise ValueError(f"target_token value {target_token} is not int") [](#__codelineno-4-86) [](#__codelineno-4-87) def is_argmax_invariant(self) -> bool: [](#__codelineno-4-88) return False [](#__codelineno-4-89) [](#__codelineno-4-90) def new_req_logits_processor( [](#__codelineno-4-91) self, [](#__codelineno-4-92) params: SamplingParams, [](#__codelineno-4-93) ) -> RequestLogitsProcessor | None: [](#__codelineno-4-94) """This method returns a new request-level logits processor, customized [](#__codelineno-4-95) to the `target_token` value associated with a particular request. [](#__codelineno-4-96) [](#__codelineno-4-97) Returns None if the logits processor should not be applied to the [](#__codelineno-4-98) particular request. To use the logits processor the request must have [](#__codelineno-4-99) a "target_token" custom argument with an integer value. [](#__codelineno-4-100) [](#__codelineno-4-101) Args: [](#__codelineno-4-102) params: per-request sampling params [](#__codelineno-4-103) [](#__codelineno-4-104) Returns: [](#__codelineno-4-105) `Callable` request logits processor, or None [](#__codelineno-4-106) """ [](#__codelineno-4-107) target_token: Any | None = params.extra_args and params.extra_args.get( [](#__codelineno-4-108) "target_token" [](#__codelineno-4-109) ) [](#__codelineno-4-110) if target_token is None: [](#__codelineno-4-111) return None [](#__codelineno-4-112) return DummyPerReqLogitsProcessor(target_token) [](#__codelineno-4-113) [](#__codelineno-4-114)[](#__codelineno-4-115)# Sample prompts. [](#__codelineno-4-116)prompts = [ [](#__codelineno-4-117) "Hello, my name is", [](#__codelineno-4-118) "The president of the United States is", [](#__codelineno-4-119) "The capital of France is", [](#__codelineno-4-120) "The future of AI is", [](#__codelineno-4-121)] [](#__codelineno-4-122)# Create a mixture of requests which do and don't utilize the dummy logitproc [](#__codelineno-4-123)sampling_params_list = [ [](#__codelineno-4-124) SamplingParams(temperature=0.0, extra_args={"target_token": 128}), [](#__codelineno-4-125) SamplingParams(temperature=0.0), [](#__codelineno-4-126) SamplingParams(temperature=0.0, extra_args={"target_token": 67}), [](#__codelineno-4-127) SamplingParams(temperature=0.0), [](#__codelineno-4-128)] [](#__codelineno-4-129) [](#__codelineno-4-130)[](#__codelineno-4-131)def main(): [](#__codelineno-4-132) # Create an LLM. [](#__codelineno-4-133) llm = LLM( [](#__codelineno-4-134) model="facebook/opt-125m", [](#__codelineno-4-135) logits_processors=[WrappedPerReqLogitsProcessor], [](#__codelineno-4-136) ) [](#__codelineno-4-137) # Generate texts from the prompts. [](#__codelineno-4-138) # The output is a list of RequestOutput objects [](#__codelineno-4-139) # that contain the prompt, generated text, and other information. [](#__codelineno-4-140) outputs = llm.generate(prompts, sampling_params_list) [](#__codelineno-4-141) # Print the outputs. [](#__codelineno-4-142) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-4-143) for output in outputs: [](#__codelineno-4-144) prompt = output.prompt [](#__codelineno-4-145) generated_text = output.outputs[0].text [](#__codelineno-4-146) print(f"Prompt: {prompt!r}") [](#__codelineno-4-147) print(f"Output: {generated_text!r}") [](#__codelineno-4-148) print("-" * 60) [](#__codelineno-4-149) [](#__codelineno-4-150)[](#__codelineno-4-151)if __name__ == "__main__": [](#__codelineno-4-152) main()`` custom\_req\_init.py ``[](#__codelineno-5-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-5-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-5-3)[](#__codelineno-5-4)"""This example demonstrates a special case of wrapping a request-level logits [](#__codelineno-5-5)processor, namely the case where it is necessary to utilize engine config or [](#__codelineno-5-6)environment info passed to the constructor. The subclass must override the [](#__codelineno-5-7)wrapper base class `__init__()` method to access the engine config, the device [](#__codelineno-5-8)identifier, or the flag which indicates whether pinned memory is available. [](#__codelineno-5-9)[](#__codelineno-5-10)For demo purposes, a request-level dummy logits processor is employed which [](#__codelineno-5-11)causes the same token (`target_token`) to be decoded in each step. The [](#__codelineno-5-12)request-level dummy logits processor is wrapped to create a batch-level logits [](#__codelineno-5-13)processor, which can apply the logits processor to output logits from all [](#__codelineno-5-14)requests in the persistent batch in a given decode step. [](#__codelineno-5-15)[](#__codelineno-5-16)The wrapped dummy logits processor below models a scenario where we must [](#__codelineno-5-17)disable the logits processor on non-"cuda" platforms. The wrapper base class [](#__codelineno-5-18)`__init__()` is overridden in order to check this condition and set a flag. [](#__codelineno-5-19)[](#__codelineno-5-20)A batch is constructed with `temperature=0.0` and 50% of requests specifying [](#__codelineno-5-21)`target_token`, and for these requests - and *only* these requests - we [](#__codelineno-5-22)expect that on a "cuda" device the output will look something like: [](#__codelineno-5-23)[](#__codelineno-5-24)Generated Outputs: [](#__codelineno-5-25)------------------------------------------------------------ [](#__codelineno-5-26)Prompt: 'Hello, my name is' [](#__codelineno-5-27)Output: " ' ' ' ' ' ' ' ' ' ' ' ' ' ' ' '" [](#__codelineno-5-28)------------------------------------------------------------ [](#__codelineno-5-29)Prompt: 'The president of the United States is' [](#__codelineno-5-30)Output: " not a racist. He is a racist.\nHe's a racist because he" [](#__codelineno-5-31)------------------------------------------------------------ [](#__codelineno-5-32)Prompt: 'The capital of France is' [](#__codelineno-5-33)Output: ' also also also also also also also also also also also also also [](#__codelineno-5-34) also also also' [](#__codelineno-5-35)------------------------------------------------------------ [](#__codelineno-5-36)Prompt: 'The future of AI is' [](#__codelineno-5-37)Output: ' in the hands of the people.\n\nThe future of AI is in the' [](#__codelineno-5-38)------------------------------------------------------------ [](#__codelineno-5-39)[](#__codelineno-5-40)which indicates that the logits processor is running. However, on a non-"cuda" [](#__codelineno-5-41)device, the first and third requests would not repeat the same token. [](#__codelineno-5-42)""" [](#__codelineno-5-43)[](#__codelineno-5-44)import torch [](#__codelineno-5-45)[](#__codelineno-5-46)from vllm import LLM, SamplingParams [](#__codelineno-5-47)from vllm.config import VllmConfig [](#__codelineno-5-48)from vllm.logger import init_logger [](#__codelineno-5-49)from vllm.v1.sample.logits_processor import ( [](#__codelineno-5-50) AdapterLogitsProcessor, [](#__codelineno-5-51) RequestLogitsProcessor, [](#__codelineno-5-52)) [](#__codelineno-5-53)[](#__codelineno-5-54)logger = init_logger(__name__) [](#__codelineno-5-55) [](#__codelineno-5-56)[](#__codelineno-5-57)class DummyPerReqLogitsProcessor: [](#__codelineno-5-58) """The request-level logits processor masks out all logits except the [](#__codelineno-5-59) token id identified by `target_token`""" [](#__codelineno-5-60) [](#__codelineno-5-61) def __init__(self, target_token: int) -> None: [](#__codelineno-5-62) """Specify `target_token`""" [](#__codelineno-5-63) self.target_token = target_token [](#__codelineno-5-64) [](#__codelineno-5-65) def __call__( [](#__codelineno-5-66) self, [](#__codelineno-5-67) output_ids: list[int], [](#__codelineno-5-68) logits: torch.Tensor, [](#__codelineno-5-69) ) -> torch.Tensor: [](#__codelineno-5-70) val_to_keep = logits[self.target_token].item() [](#__codelineno-5-71) logits[:] = float("-inf") [](#__codelineno-5-72) logits[self.target_token] = val_to_keep [](#__codelineno-5-73) return logits [](#__codelineno-5-74) [](#__codelineno-5-75)[](#__codelineno-5-76)class WrappedPerReqLogitsProcessor(AdapterLogitsProcessor): [](#__codelineno-5-77) """Example of overriding the wrapper class `__init__()` in order to utilize [](#__codelineno-5-78) info about the device type""" [](#__codelineno-5-79) [](#__codelineno-5-80) @classmethod [](#__codelineno-5-81) def validate_params(cls, params: SamplingParams): [](#__codelineno-5-82) target_token = params.extra_args and params.extra_args.get("target_token") [](#__codelineno-5-83) if target_token is not None and not isinstance(target_token, int): [](#__codelineno-5-84) raise ValueError( [](#__codelineno-5-85) f"`target_token` has to be an integer, got {target_token}." [](#__codelineno-5-86) ) [](#__codelineno-5-87) [](#__codelineno-5-88) def __init__( [](#__codelineno-5-89) self, vllm_config: VllmConfig, device: torch.device, is_pin_memory: bool [](#__codelineno-5-90) ): [](#__codelineno-5-91) super().__init__(vllm_config, device, is_pin_memory) [](#__codelineno-5-92) self.is_cuda = device.type == "cuda" [](#__codelineno-5-93) [](#__codelineno-5-94) def is_argmax_invariant(self) -> bool: [](#__codelineno-5-95) return False [](#__codelineno-5-96) [](#__codelineno-5-97) def new_req_logits_processor( [](#__codelineno-5-98) self, [](#__codelineno-5-99) params: SamplingParams, [](#__codelineno-5-100) ) -> RequestLogitsProcessor | None: [](#__codelineno-5-101) """This method returns a new request-level logits processor, customized [](#__codelineno-5-102) to the `target_token` value associated with a particular request. [](#__codelineno-5-103) [](#__codelineno-5-104) Returns None if the logits processor should not be applied to the [](#__codelineno-5-105) particular request. To use the logits processor the request must have [](#__codelineno-5-106) a "target_token" custom argument with an integer value, and the device [](#__codelineno-5-107) must be "cuda"-type [](#__codelineno-5-108) [](#__codelineno-5-109) Args: [](#__codelineno-5-110) params: per-request sampling params [](#__codelineno-5-111) [](#__codelineno-5-112) Returns: [](#__codelineno-5-113) `Callable` request logits processor, or None [](#__codelineno-5-114) """ [](#__codelineno-5-115) if ( [](#__codelineno-5-116) not self.is_cuda [](#__codelineno-5-117) or ( [](#__codelineno-5-118) target_token := params.extra_args [](#__codelineno-5-119) and params.extra_args.get("target_token") [](#__codelineno-5-120) ) [](#__codelineno-5-121) is None [](#__codelineno-5-122) ): [](#__codelineno-5-123) return None [](#__codelineno-5-124) return DummyPerReqLogitsProcessor(target_token) [](#__codelineno-5-125) [](#__codelineno-5-126)[](#__codelineno-5-127)# Sample prompts. [](#__codelineno-5-128)prompts = [ [](#__codelineno-5-129) "Hello, my name is", [](#__codelineno-5-130) "The president of the United States is", [](#__codelineno-5-131) "The capital of France is", [](#__codelineno-5-132) "The future of AI is", [](#__codelineno-5-133)] [](#__codelineno-5-134)# Create a mixture of requests which do and don't utilize the dummy logitproc [](#__codelineno-5-135)sampling_params_list = [ [](#__codelineno-5-136) SamplingParams(temperature=0.0, extra_args={"target_token": 128}), [](#__codelineno-5-137) SamplingParams(temperature=0.0), [](#__codelineno-5-138) SamplingParams(temperature=0.0, extra_args={"target_token": 67}), [](#__codelineno-5-139) SamplingParams(temperature=0.0), [](#__codelineno-5-140)] [](#__codelineno-5-141) [](#__codelineno-5-142)[](#__codelineno-5-143)def main(): [](#__codelineno-5-144) # Create an LLM. [](#__codelineno-5-145) llm = LLM( [](#__codelineno-5-146) model="facebook/opt-125m", [](#__codelineno-5-147) logits_processors=[WrappedPerReqLogitsProcessor], [](#__codelineno-5-148) ) [](#__codelineno-5-149) # Generate texts from the prompts. [](#__codelineno-5-150) # The output is a list of RequestOutput objects [](#__codelineno-5-151) # that contain the prompt, generated text, and other information. [](#__codelineno-5-152) outputs = llm.generate(prompts, sampling_params_list) [](#__codelineno-5-153) # Print the outputs. [](#__codelineno-5-154) print("\nGenerated Outputs:\n" + "-" * 60) [](#__codelineno-5-155) for output in outputs: [](#__codelineno-5-156) prompt = output.prompt [](#__codelineno-5-157) generated_text = output.outputs[0].text [](#__codelineno-5-158) print(f"Prompt: {prompt!r}") [](#__codelineno-5-159) print(f"Output: {generated_text!r}") [](#__codelineno-5-160) print("-" * 60) [](#__codelineno-5-161) [](#__codelineno-5-162)[](#__codelineno-5-163)if __name__ == "__main__": [](#__codelineno-5-164) main()`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/lora.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/lora](https://github.com/vllm-project/vllm/tree/main/examples/features/lora). ## LoRA With Quantization Offline[¶](#lora-with-quantization-offline "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This example shows how to use LoRA with different quantization techniques [](#__codelineno-0-5)for offline inference. [](#__codelineno-0-6)[](#__codelineno-0-7)Requires HuggingFace credentials for access. [](#__codelineno-0-8)""" [](#__codelineno-0-9)[](#__codelineno-0-10)import gc [](#__codelineno-0-11)[](#__codelineno-0-12)import torch [](#__codelineno-0-13)from huggingface_hub import snapshot_download [](#__codelineno-0-14)[](#__codelineno-0-15)from vllm import EngineArgs, LLMEngine, RequestOutput, SamplingParams [](#__codelineno-0-16)from vllm.lora.request import LoRARequest [](#__codelineno-0-17) [](#__codelineno-0-18)[](#__codelineno-0-19)def create_test_prompts( [](#__codelineno-0-20) lora_path: str, [](#__codelineno-0-21)) -> list[tuple[str, SamplingParams, LoRARequest | None]]: [](#__codelineno-0-22) return [ [](#__codelineno-0-23) # this is an example of using quantization without LoRA [](#__codelineno-0-24) ( [](#__codelineno-0-25) "My name is", [](#__codelineno-0-26) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-0-27) None, [](#__codelineno-0-28) ), [](#__codelineno-0-29) # the next three examples use quantization with LoRA [](#__codelineno-0-30) ( [](#__codelineno-0-31) "my name is", [](#__codelineno-0-32) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-0-33) LoRARequest("lora-test-1", 1, lora_path), [](#__codelineno-0-34) ), [](#__codelineno-0-35) ( [](#__codelineno-0-36) "The capital of USA is", [](#__codelineno-0-37) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-0-38) LoRARequest("lora-test-2", 1, lora_path), [](#__codelineno-0-39) ), [](#__codelineno-0-40) ( [](#__codelineno-0-41) "The capital of France is", [](#__codelineno-0-42) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-0-43) LoRARequest("lora-test-3", 1, lora_path), [](#__codelineno-0-44) ), [](#__codelineno-0-45) ] [](#__codelineno-0-46) [](#__codelineno-0-47)[](#__codelineno-0-48)def process_requests( [](#__codelineno-0-49) engine: LLMEngine, [](#__codelineno-0-50) test_prompts: list[tuple[str, SamplingParams, LoRARequest | None]], [](#__codelineno-0-51)): [](#__codelineno-0-52) """Continuously process a list of prompts and handle the outputs.""" [](#__codelineno-0-53) request_id = 0 [](#__codelineno-0-54) [](#__codelineno-0-55) while test_prompts or engine.has_unfinished_requests(): [](#__codelineno-0-56) if test_prompts: [](#__codelineno-0-57) prompt, sampling_params, lora_request = test_prompts.pop(0) [](#__codelineno-0-58) engine.add_request( [](#__codelineno-0-59) str(request_id), prompt, sampling_params, lora_request=lora_request [](#__codelineno-0-60) ) [](#__codelineno-0-61) request_id += 1 [](#__codelineno-0-62) [](#__codelineno-0-63) request_outputs: list[RequestOutput] = engine.step() [](#__codelineno-0-64) for request_output in request_outputs: [](#__codelineno-0-65) if request_output.finished: [](#__codelineno-0-66) print("----------------------------------------------------") [](#__codelineno-0-67) print(f"Prompt: {request_output.prompt}") [](#__codelineno-0-68) print(f"Output: {request_output.outputs[0].text}") [](#__codelineno-0-69) [](#__codelineno-0-70)[](#__codelineno-0-71)def initialize_engine( [](#__codelineno-0-72) model: str, quantization: str, lora_repo: str | None [](#__codelineno-0-73)) -> LLMEngine: [](#__codelineno-0-74) """Initialize the LLMEngine.""" [](#__codelineno-0-75) [](#__codelineno-0-76) engine_args = EngineArgs( [](#__codelineno-0-77) model=model, [](#__codelineno-0-78) quantization=quantization, [](#__codelineno-0-79) enable_lora=True, [](#__codelineno-0-80) max_lora_rank=64, [](#__codelineno-0-81) max_loras=4, [](#__codelineno-0-82) ) [](#__codelineno-0-83) return LLMEngine.from_engine_args(engine_args) [](#__codelineno-0-84) [](#__codelineno-0-85)[](#__codelineno-0-86)def main(): [](#__codelineno-0-87) """Main function that sets up and runs the prompt processing.""" [](#__codelineno-0-88) [](#__codelineno-0-89) test_configs = [ [](#__codelineno-0-90) # QLoRA (https://arxiv.org/abs/2305.14314) [](#__codelineno-0-91) { [](#__codelineno-0-92) "name": "qlora_inference_example", [](#__codelineno-0-93) "model": "huggyllama/llama-7b", [](#__codelineno-0-94) "quantization": "bitsandbytes", [](#__codelineno-0-95) "lora_repo": "timdettmers/qlora-flan-7b", [](#__codelineno-0-96) }, [](#__codelineno-0-97) { [](#__codelineno-0-98) "name": "AWQ_inference_with_lora_example", [](#__codelineno-0-99) "model": "TheBloke/TinyLlama-1.1B-Chat-v0.3-AWQ", [](#__codelineno-0-100) "quantization": "awq", [](#__codelineno-0-101) "lora_repo": "jashing/tinyllama-colorist-lora", [](#__codelineno-0-102) }, [](#__codelineno-0-103) { [](#__codelineno-0-104) "name": "GPTQ_inference_with_lora_example", [](#__codelineno-0-105) "model": "TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ", [](#__codelineno-0-106) "quantization": "gptq", [](#__codelineno-0-107) "lora_repo": "jashing/tinyllama-colorist-lora", [](#__codelineno-0-108) }, [](#__codelineno-0-109) ] [](#__codelineno-0-110) [](#__codelineno-0-111) for test_config in test_configs: [](#__codelineno-0-112) print(f"~~~~~~~~~~~~~~~~ Running: {test_config['name']} ~~~~~~~~~~~~~~~~") [](#__codelineno-0-113) engine = initialize_engine( [](#__codelineno-0-114) test_config["model"], test_config["quantization"], test_config["lora_repo"] [](#__codelineno-0-115) ) [](#__codelineno-0-116) lora_path = snapshot_download(repo_id=test_config["lora_repo"]) [](#__codelineno-0-117) test_prompts = create_test_prompts(lora_path) [](#__codelineno-0-118) process_requests(engine, test_prompts) [](#__codelineno-0-119) [](#__codelineno-0-120) # Clean up the GPU memory for the next test [](#__codelineno-0-121) del engine [](#__codelineno-0-122) gc.collect() [](#__codelineno-0-123) torch.accelerator.empty_cache() [](#__codelineno-0-124) [](#__codelineno-0-125)[](#__codelineno-0-126)if __name__ == "__main__": [](#__codelineno-0-127) main()` ## MultiLoRA Offline[¶](#multilora-offline "Permanent link") ``[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)This example shows how to use the multi-LoRA functionality [](#__codelineno-1-5)for offline inference. [](#__codelineno-1-6)[](#__codelineno-1-7)Requires HuggingFace credentials for access to Llama2. [](#__codelineno-1-8)""" [](#__codelineno-1-9)[](#__codelineno-1-10)from huggingface_hub import snapshot_download [](#__codelineno-1-11)[](#__codelineno-1-12)from vllm import EngineArgs, LLMEngine, RequestOutput, SamplingParams [](#__codelineno-1-13)from vllm.lora.request import LoRARequest [](#__codelineno-1-14) [](#__codelineno-1-15)[](#__codelineno-1-16)def create_test_prompts( [](#__codelineno-1-17) lora_path: str, [](#__codelineno-1-18)) -> list[tuple[str, SamplingParams, LoRARequest | None]]: [](#__codelineno-1-19) """Create a list of test prompts with their sampling parameters. [](#__codelineno-1-20) [](#__codelineno-1-21) 2 requests for base model, 4 requests for the LoRA. We define 2 [](#__codelineno-1-22) different LoRA adapters (using the same model for demo purposes). [](#__codelineno-1-23) Since we also set `max_loras=1`, the expectation is that the requests [](#__codelineno-1-24) with the second LoRA adapter will be run after all requests with the [](#__codelineno-1-25) first adapter have finished. [](#__codelineno-1-26) """ [](#__codelineno-1-27) return [ [](#__codelineno-1-28) ( [](#__codelineno-1-29) "A robot may not injure a human being", [](#__codelineno-1-30) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-1-31) None, [](#__codelineno-1-32) ), [](#__codelineno-1-33) ( [](#__codelineno-1-34) "To be or not to be,", [](#__codelineno-1-35) SamplingParams( [](#__codelineno-1-36) temperature=0.8, top_k=5, presence_penalty=0.2, max_tokens=128 [](#__codelineno-1-37) ), [](#__codelineno-1-38) None, [](#__codelineno-1-39) ), [](#__codelineno-1-40) ( [](#__codelineno-1-41) "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]", # noqa: E501 [](#__codelineno-1-42) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-1-43) LoRARequest("sql-lora", 1, lora_path), [](#__codelineno-1-44) ), [](#__codelineno-1-45) ( [](#__codelineno-1-46) "[user] Write a SQL query to answer the question based on the table schema.\n\n context: CREATE TABLE table_name_74 (icao VARCHAR, airport VARCHAR)\n\n question: Name the ICAO for lilongwe international airport [/user] [assistant]", # noqa: E501 [](#__codelineno-1-47) SamplingParams(temperature=0.0, logprobs=1, max_tokens=128), [](#__codelineno-1-48) LoRARequest("sql-lora2", 2, lora_path), [](#__codelineno-1-49) ), [](#__codelineno-1-50) ] [](#__codelineno-1-51) [](#__codelineno-1-52)[](#__codelineno-1-53)def process_requests( [](#__codelineno-1-54) engine: LLMEngine, [](#__codelineno-1-55) test_prompts: list[tuple[str, SamplingParams, LoRARequest | None]], [](#__codelineno-1-56)): [](#__codelineno-1-57) """Continuously process a list of prompts and handle the outputs.""" [](#__codelineno-1-58) request_id = 0 [](#__codelineno-1-59) [](#__codelineno-1-60) print("-" * 50) [](#__codelineno-1-61) while test_prompts or engine.has_unfinished_requests(): [](#__codelineno-1-62) if test_prompts: [](#__codelineno-1-63) prompt, sampling_params, lora_request = test_prompts.pop(0) [](#__codelineno-1-64) engine.add_request( [](#__codelineno-1-65) str(request_id), prompt, sampling_params, lora_request=lora_request [](#__codelineno-1-66) ) [](#__codelineno-1-67) request_id += 1 [](#__codelineno-1-68) [](#__codelineno-1-69) request_outputs: list[RequestOutput] = engine.step() [](#__codelineno-1-70) [](#__codelineno-1-71) for request_output in request_outputs: [](#__codelineno-1-72) if request_output.finished: [](#__codelineno-1-73) print(request_output) [](#__codelineno-1-74) print("-" * 50) [](#__codelineno-1-75) [](#__codelineno-1-76)[](#__codelineno-1-77)def initialize_engine() -> LLMEngine: [](#__codelineno-1-78) """Initialize the LLMEngine.""" [](#__codelineno-1-79) # max_loras: controls the number of LoRAs that can be used in the same [](#__codelineno-1-80) # batch. Larger numbers will cause higher memory usage, as each LoRA [](#__codelineno-1-81) # slot requires its own preallocated tensor. [](#__codelineno-1-82) # max_lora_rank: controls the maximum supported rank of all LoRAs. Larger [](#__codelineno-1-83) # numbers will cause higher memory usage. If you know that all LoRAs will [](#__codelineno-1-84) # use the same rank, it is recommended to set this as low as possible. [](#__codelineno-1-85) # max_cpu_loras: controls the size of the CPU LoRA cache. [](#__codelineno-1-86) engine_args = EngineArgs( [](#__codelineno-1-87) model="meta-llama/Llama-3.2-3B-Instruct", [](#__codelineno-1-88) enable_lora=True, [](#__codelineno-1-89) max_loras=1, [](#__codelineno-1-90) max_lora_rank=8, [](#__codelineno-1-91) max_cpu_loras=2, [](#__codelineno-1-92) max_num_seqs=256, [](#__codelineno-1-93) ) [](#__codelineno-1-94) return LLMEngine.from_engine_args(engine_args) [](#__codelineno-1-95) [](#__codelineno-1-96)[](#__codelineno-1-97)def main(): [](#__codelineno-1-98) """Main function that sets up and runs the prompt processing.""" [](#__codelineno-1-99) engine = initialize_engine() [](#__codelineno-1-100) lora_path = snapshot_download(repo_id="jeeejeee/llama32-3b-text2sql-spider") [](#__codelineno-1-101) test_prompts = create_test_prompts(lora_path) [](#__codelineno-1-102) process_requests(engine, test_prompts) [](#__codelineno-1-103) [](#__codelineno-1-104)[](#__codelineno-1-105)if __name__ == "__main__": [](#__codelineno-1-106) main()`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/openai_batch.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/openai\_batch](https://github.com/vllm-project/vllm/tree/main/examples/features/openai_batch). `[](#__codelineno-0-1)This is a guide to performing batch inference using the OpenAI batch file format, **not** the complete Batch (REST) API.` ## File Format[¶](#file-format "Permanent link") The OpenAI batch file format consists of a series of json objects on new lines. [See here for an example file.](https://github.com/vllm-project/vllm/blob/main/examples/features/openai_batch/openai_example_batch.jsonl) Each line represents a separate request. See the [OpenAI package reference](https://platform.openai.com/docs/api-reference/batch/requestInput) for more details. ``[](#__codelineno-1-1)We currently support `/v1/chat/completions`, `/v1/embeddings`, and `/v1/score` endpoints (completions coming soon).`` ## Pre-requisites[¶](#pre-requisites "Permanent link") - The examples in this document use `meta-llama/Meta-Llama-3-8B-Instruct`. - Create a [user access token](https://huggingface.co/docs/hub/en/security-tokens) - Install the token on your machine (Run `hf auth login`). - Get access to the gated model by [visiting the model card](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) and agreeing to the terms and conditions. ## Example 1: Running with a local file[¶](#example-1-running-with-a-local-file "Permanent link") ### Step 1: Create your batch file[¶](#step-1-create-your-batch-file "Permanent link") To follow along with this example, you can download the example batch, or create your own batch file in your working directory. `[](#__codelineno-2-1)wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl` Once you've created your batch file it should look like this `[](#__codelineno-3-1)cat features/openai_batch/openai_example_batch.jsonl [](#__codelineno-3-2){"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}} [](#__codelineno-3-3){"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}` ### Step 2: Run the batch[¶](#step-2-run-the-batch "Permanent link") The batch running tool is designed to be used from the command line. You can run the batch with the following command, which will write its results to a file called `results.jsonl` `[](#__codelineno-4-1)python -m vllm.entrypoints.openai.run_batch \ [](#__codelineno-4-2) -i features/openai_batch/openai_example_batch.jsonl \ [](#__codelineno-4-3) -o results.jsonl \ [](#__codelineno-4-4) --model meta-llama/Meta-Llama-3-8B-Instruct` or use command-line: `[](#__codelineno-5-1)vllm run-batch \ [](#__codelineno-5-2) -i features/openai_batch/openai_example_batch.jsonl \ [](#__codelineno-5-3) -o results.jsonl \ [](#__codelineno-5-4) --model meta-llama/Meta-Llama-3-8B-Instruct` ### Step 3: Check your results[¶](#step-3-check-your-results "Permanent link") You should now have your results at `results.jsonl`. You can check your results by running `cat results.jsonl` `[](#__codelineno-6-1)cat results.jsonl [](#__codelineno-6-2){"id":"vllm-383d1c59835645aeb2e07d004d62a826","custom_id":"request-1","response":{"id":"cmpl-61c020e54b964d5a98fa7527bfcdd378","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! It's great to meet you! I'm here to help with any questions or tasks you may have. What's on your mind today?"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":25,"total_tokens":56,"completion_tokens":31}},"error":null} [](#__codelineno-6-3){"id":"vllm-42e3d09b14b04568afa3f1797751a267","custom_id":"request-2","response":{"id":"cmpl-f44d049f6b3a42d4b2d7850bb1e31bcc","object":"chat.completion","created":1715633336,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"message":{"role":"assistant","content":"*silence*"},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":27,"total_tokens":32,"completion_tokens":5}},"error":null}` ## Example 2: Using remote files[¶](#example-2-using-remote-files "Permanent link") The batch runner supports remote input and output urls that are accessible via http/https. For example, to run against our example input file located at `https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl`, you can run `[](#__codelineno-7-1)python -m vllm.entrypoints.openai.run_batch \ [](#__codelineno-7-2) -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl \ [](#__codelineno-7-3) -o results.jsonl \ [](#__codelineno-7-4) --model meta-llama/Meta-Llama-3-8B-Instruct` or use command-line: `[](#__codelineno-8-1)vllm run-batch \ [](#__codelineno-8-2) -i https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl \ [](#__codelineno-8-3) -o results.jsonl \ [](#__codelineno-8-4) --model meta-llama/Meta-Llama-3-8B-Instruct` ## Example 3: Integrating with AWS S3[¶](#example-3-integrating-with-aws-s3 "Permanent link") To integrate with cloud blob storage, we recommend using presigned urls. \[Learn more about S3 presigned urls here\] ### Additional prerequisites[¶](#additional-prerequisites "Permanent link") - [Create an S3 bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html). - The `awscli` package (Run `pip install awscli`) to configure your credentials and interactively use s3. - [Configure your credentials](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-quickstart.html). - The `boto3` python package (Run `pip install boto3`) to generate presigned urls. ### Step 1: Upload your input script[¶](#step-1-upload-your-input-script "Permanent link") To follow along with this example, you can download the example batch, or create your own batch file in your working directory. `[](#__codelineno-9-1)wget https://raw.githubusercontent.com/vllm-project/vllm/main/examples/features/openai_batch/openai_example_batch.jsonl` Once you've created your batch file it should look like this `[](#__codelineno-10-1)cat features/openai_batch/openai_example_batch.jsonl [](#__codelineno-10-2){"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}} [](#__codelineno-10-3){"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}` Now upload your batch file to your S3 bucket. `[](#__codelineno-11-1)aws s3 cp features/openai_batch/openai_example_batch.jsonl s3://MY_BUCKET/MY_INPUT_FILE.jsonl` ### Step 2: Generate your presigned urls[¶](#step-2-generate-your-presigned-urls "Permanent link") Presigned urls can only be generated via the SDK. You can run the following python script to generate your presigned urls. Be sure to replace the `MY_BUCKET`, `MY_INPUT_FILE.jsonl`, and `MY_OUTPUT_FILE.jsonl` placeholders with your bucket and file names. (The script is adapted from [https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example\_code/s3/s3\_basics/presigned\_url.py](https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/python/example_code/s3/s3_basics/presigned_url.py)) `[](#__codelineno-12-1)import boto3 [](#__codelineno-12-2)from botocore.exceptions import ClientError [](#__codelineno-12-3)[](#__codelineno-12-4)def generate_presigned_url(s3_client, client_method, method_parameters, expires_in): [](#__codelineno-12-5) """ [](#__codelineno-12-6) Generate a presigned Amazon S3 URL that can be used to perform an action. [](#__codelineno-12-7) [](#__codelineno-12-8) :param s3_client: A Boto3 Amazon S3 client. [](#__codelineno-12-9) :param client_method: The name of the client method that the URL performs. [](#__codelineno-12-10) :param method_parameters: The parameters of the specified client method. [](#__codelineno-12-11) :param expires_in: The number of seconds the presigned URL is valid for. [](#__codelineno-12-12) :return: The presigned URL. [](#__codelineno-12-13) """ [](#__codelineno-12-14) try: [](#__codelineno-12-15) url = s3_client.generate_presigned_url( [](#__codelineno-12-16) ClientMethod=client_method, [](#__codelineno-12-17) Params=method_parameters, [](#__codelineno-12-18) ExpiresIn=expires_in, [](#__codelineno-12-19) ) [](#__codelineno-12-20) except ClientError: [](#__codelineno-12-21) raise [](#__codelineno-12-22) return url [](#__codelineno-12-23) [](#__codelineno-12-24)[](#__codelineno-12-25)s3_client = boto3.client("s3") [](#__codelineno-12-26)input_url = generate_presigned_url( [](#__codelineno-12-27) s3_client, [](#__codelineno-12-28) "get_object", [](#__codelineno-12-29) {"Bucket": "MY_BUCKET", "Key": "MY_INPUT_FILE.jsonl"}, [](#__codelineno-12-30) expires_in=3600, [](#__codelineno-12-31)) [](#__codelineno-12-32)output_url = generate_presigned_url( [](#__codelineno-12-33) s3_client, [](#__codelineno-12-34) "put_object", [](#__codelineno-12-35) {"Bucket": "MY_BUCKET", "Key": "MY_OUTPUT_FILE.jsonl"}, [](#__codelineno-12-36) expires_in=3600, [](#__codelineno-12-37)) [](#__codelineno-12-38)print(f"{input_url=}") [](#__codelineno-12-39)print(f"{output_url=}")` This script should output `[](#__codelineno-13-1)input_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091' [](#__codelineno-13-2)output_url='https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091'` ### Step 3: Run the batch runner using your presigned urls[¶](#step-3-run-the-batch-runner-using-your-presigned-urls "Permanent link") You can now run the batch runner, using the urls generated in the previous section. `[](#__codelineno-14-1)python -m vllm.entrypoints.openai.run_batch \ [](#__codelineno-14-2) -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \ [](#__codelineno-14-3) -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \ [](#__codelineno-14-4) --model meta-llama/Meta-Llama-3-8B-Instruct` or use command-line: `[](#__codelineno-15-1)vllm run-batch \ [](#__codelineno-15-2) -i "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_INPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \ [](#__codelineno-15-3) -o "https://s3.us-west-2.amazonaws.com/MY_BUCKET/MY_OUTPUT_FILE.jsonl?AWSAccessKeyId=ABCDEFGHIJKLMNOPQRST&Signature=abcdefghijklmnopqrstuvwxyz12345&Expires=1715800091" \ [](#__codelineno-15-4) --model meta-llama/Meta-Llama-3-8B-Instruct` ### Step 4: View your results[¶](#step-4-view-your-results "Permanent link") Your results are now on S3. You can view them in your terminal by running `[](#__codelineno-16-1)aws s3 cp s3://MY_BUCKET/MY_OUTPUT_FILE.jsonl -` ## Example 4: Using embeddings endpoint[¶](#example-4-using-embeddings-endpoint "Permanent link") ### Additional prerequisites[¶](#additional-prerequisites_1 "Permanent link") - Ensure you are using `vllm >= 0.5.5`. ### Step 1: Create your batch file[¶](#step-1-create-your-batch-file_1 "Permanent link") Add embedding requests to your batch file. The following is an example: `[](#__codelineno-17-1){"custom_id": "request-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are a helpful assistant."}} [](#__codelineno-17-2){"custom_id": "request-2", "method": "POST", "url": "/v1/embeddings", "body": {"model": "intfloat/e5-mistral-7b-instruct", "input": "You are an unhelpful assistant."}}` You can even mix chat completion and embedding requests in the batch file, as long as the model you are using supports both chat completion and embeddings (note that all requests must use the same model). ### Step 2: Run the batch[¶](#step-2-run-the-batch_1 "Permanent link") You can run the batch using the same command as in earlier examples. ### Step 3: Check your results[¶](#step-3-check-your-results_1 "Permanent link") You can check your results by running `cat results.jsonl` `[](#__codelineno-18-1)cat results.jsonl [](#__codelineno-18-2){"id":"vllm-db0f71f7dec244e6bce530e0b4ef908b","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-3580bf4d4ae54d52b67eee266a6eab20","body":{"id":"embd-33ac2efa7996430184461f2e38529746","object":"list","created":444647,"model":"intfloat/e5-mistral-7b-instruct","data":[{"index":0,"object":"embedding","embedding":[0.016204833984375,0.0092010498046875,0.0018358230590820312,-0.0028228759765625,0.001422882080078125,-0.0031147003173828125,...]}],"usage":{"prompt_tokens":8,"total_tokens":8,"completion_tokens":0}}},"error":null} [](#__codelineno-18-3)...` ## Example 5: Using score endpoint[¶](#example-5-using-score-endpoint "Permanent link") ### Additional prerequisites[¶](#additional-prerequisites_2 "Permanent link") - Ensure you are using `vllm >= 0.7.0`. ### Step 1: Create your batch file[¶](#step-1-create-your-batch-file_2 "Permanent link") Add score requests to your batch file. The following is an example: `[](#__codelineno-19-1){"custom_id": "request-1", "method": "POST", "url": "/v1/score", "body": {"model": "BAAI/bge-reranker-v2-m3", "queries": "What is the capital of France?", "documents": ["The capital of Brazil is Brasilia.", "The capital of France is Paris."]}} [](#__codelineno-19-2){"custom_id": "request-2", "method": "POST", "url": "/v1/score", "body": {"model": "BAAI/bge-reranker-v2-m3", "queries": "What is the capital of France?", "documents": ["The capital of Brazil is Brasilia.", "The capital of France is Paris."]}}` You can mix chat completion, embedding, and score requests in the batch file, as long as the model you are using supports them all (note that all requests must use the same model). ### Step 2: Run the batch[¶](#step-2-run-the-batch_2 "Permanent link") You can run the batch using the same command as in earlier examples. ### Step 3: Check your results[¶](#step-3-check-your-results_2 "Permanent link") You can check your results by running `cat results.jsonl` `[](#__codelineno-20-1)cat results.jsonl [](#__codelineno-20-2){"id":"vllm-f87c5c4539184f618e555744a2965987","custom_id":"request-1","response":{"status_code":200,"request_id":"vllm-batch-806ab64512e44071b37d3f7ccd291413","body":{"id":"score-4ee45236897b4d29907d49b01298cdb1","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.0010900497436523438},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null} [](#__codelineno-20-3){"id":"vllm-41990c51a26d4fac8419077f12871099","custom_id":"request-2","response":{"status_code":200,"request_id":"vllm-batch-73ce66379026482699f81974e14e1e99","body":{"id":"score-13f2ffe6ba40460fbf9f7f00ad667d75","object":"list","created":1737847944,"model":"BAAI/bge-reranker-v2-m3","data":[{"index":0,"object":"score","score":0.001094818115234375},{"index":1,"object":"score","score":1.0}],"usage":{"prompt_tokens":37,"total_tokens":37,"completion_tokens":0,"prompt_tokens_details":null}}},"error":null}` ## Example materials[¶](#example-materials "Permanent link") openai\_example\_batch.jsonl `[](#__codelineno-21-1){"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}} [](#__codelineno-21-2){"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_completion_tokens": 1000}}` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/pause_resume.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/pause\_resume](https://github.com/vllm-project/vllm/tree/main/examples/features/pause_resume). ## Data Parallel Pause Resume[¶](#data-parallel-pause-resume "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Test pause/resume with Data Parallel (DP) via HTTP API. [](#__codelineno-0-5)[](#__codelineno-0-6)This example demonstrates coordinated pause/resume across multiple DP ranks. [](#__codelineno-0-7)The pause synchronizes across all DP engines via all-reduce. [](#__codelineno-0-8)[](#__codelineno-0-9)Prerequisites: [](#__codelineno-0-10) Start a vLLM server with data parallelism: [](#__codelineno-0-11) [](#__codelineno-0-12) $ VLLM_SERVER_DEV_MODE=1 vllm serve facebook/opt-125m \ [](#__codelineno-0-13) --enforce-eager \ [](#__codelineno-0-14) --data-parallel-size 4 \ [](#__codelineno-0-15) --tensor-parallel-size 1 [](#__codelineno-0-16) [](#__codelineno-0-17) Then run this script: [](#__codelineno-0-18) [](#__codelineno-0-19) $ python data_parallel_pause_resume.py [](#__codelineno-0-20)[](#__codelineno-0-21)The test verifies pause works by: [](#__codelineno-0-22)1. Starting a streaming generation request [](#__codelineno-0-23)2. Pausing the server mid-generation [](#__codelineno-0-24)3. Sleeping for PAUSE_DURATION seconds [](#__codelineno-0-25)4. Resuming the server [](#__codelineno-0-26)5. Verifying there was a gap in token generation matching the pause duration [](#__codelineno-0-27)""" [](#__codelineno-0-28)[](#__codelineno-0-29)import argparse [](#__codelineno-0-30)import threading [](#__codelineno-0-31)import time [](#__codelineno-0-32)[](#__codelineno-0-33)import requests [](#__codelineno-0-34)from openai import OpenAI [](#__codelineno-0-35)[](#__codelineno-0-36)BASE_URL = "http://localhost:8000" [](#__codelineno-0-37)MODEL_NAME = "facebook/opt-125m" [](#__codelineno-0-38)PAUSE_DURATION = 3.0 [](#__codelineno-0-39) [](#__codelineno-0-40)[](#__codelineno-0-41)def pause_generation(base_url: str, mode: str = "keep") -> None: [](#__codelineno-0-42) """Pause generation via HTTP endpoint.""" [](#__codelineno-0-43) url = f"{base_url}/pause" [](#__codelineno-0-44) response = requests.post(url, params={"mode": mode}, timeout=60) [](#__codelineno-0-45) response.raise_for_status() [](#__codelineno-0-46) print("Server paused") [](#__codelineno-0-47) [](#__codelineno-0-48)[](#__codelineno-0-49)def resume_generation(base_url: str) -> None: [](#__codelineno-0-50) """Resume generation via HTTP endpoint.""" [](#__codelineno-0-51) url = f"{base_url}/resume" [](#__codelineno-0-52) response = requests.post(url, timeout=60) [](#__codelineno-0-53) response.raise_for_status() [](#__codelineno-0-54) print("Server resumed") [](#__codelineno-0-55) [](#__codelineno-0-56)[](#__codelineno-0-57)def main(): [](#__codelineno-0-58) parser = argparse.ArgumentParser() [](#__codelineno-0-59) parser.add_argument("--base-url", default=BASE_URL) [](#__codelineno-0-60) parser.add_argument("--model", default=MODEL_NAME) [](#__codelineno-0-61) args = parser.parse_args() [](#__codelineno-0-62) [](#__codelineno-0-63) client = OpenAI( [](#__codelineno-0-64) base_url=f"{args.base_url}/v1", [](#__codelineno-0-65) api_key="EMPTY", [](#__codelineno-0-66) ) [](#__codelineno-0-67) [](#__codelineno-0-68) prompt = "Write a long story about a dragon. Once upon a time" [](#__codelineno-0-69) token_times: list[float] = [] [](#__codelineno-0-70) pause_token_idx = 0 [](#__codelineno-0-71) pause_triggered = threading.Event() [](#__codelineno-0-72) [](#__codelineno-0-73) def generator_thread(): [](#__codelineno-0-74) """Stream tokens and record timestamps.""" [](#__codelineno-0-75) stream = client.completions.create( [](#__codelineno-0-76) model=args.model, [](#__codelineno-0-77) prompt=prompt, [](#__codelineno-0-78) max_tokens=50, [](#__codelineno-0-79) stream=True, [](#__codelineno-0-80) ) [](#__codelineno-0-81) for chunk in stream: [](#__codelineno-0-82) if chunk.choices[0].text: [](#__codelineno-0-83) token_times.append(time.monotonic()) [](#__codelineno-0-84) token_count = len(token_times) [](#__codelineno-0-85) print(f"Token {token_count}: {chunk.choices[0].text!r}") [](#__codelineno-0-86) [](#__codelineno-0-87) # Signal controller after some tokens [](#__codelineno-0-88) if token_count >= 5 and not pause_triggered.is_set(): [](#__codelineno-0-89) pause_triggered.set() [](#__codelineno-0-90) [](#__codelineno-0-91) def controller_thread(): [](#__codelineno-0-92) """Pause and resume the server.""" [](#__codelineno-0-93) nonlocal pause_token_idx [](#__codelineno-0-94) [](#__codelineno-0-95) # Wait for some tokens [](#__codelineno-0-96) pause_triggered.wait() [](#__codelineno-0-97) [](#__codelineno-0-98) print(f"\nPausing server (keep mode) at token {len(token_times)}...") [](#__codelineno-0-99) pause_generation(args.base_url, mode="keep") [](#__codelineno-0-100) pause_token_idx = len(token_times) [](#__codelineno-0-101) print(f"Sleeping for {PAUSE_DURATION}s...") [](#__codelineno-0-102) [](#__codelineno-0-103) time.sleep(PAUSE_DURATION) [](#__codelineno-0-104) [](#__codelineno-0-105) print("Resuming server...") [](#__codelineno-0-106) resume_generation(args.base_url) [](#__codelineno-0-107) print("Resumed!\n") [](#__codelineno-0-108) [](#__codelineno-0-109) # Run both threads [](#__codelineno-0-110) gen_thread = threading.Thread(target=generator_thread) [](#__codelineno-0-111) ctrl_thread = threading.Thread(target=controller_thread) [](#__codelineno-0-112) [](#__codelineno-0-113) gen_thread.start() [](#__codelineno-0-114) ctrl_thread.start() [](#__codelineno-0-115) [](#__codelineno-0-116) gen_thread.join() [](#__codelineno-0-117) ctrl_thread.join() [](#__codelineno-0-118) [](#__codelineno-0-119) # Check gap at the pause point [](#__codelineno-0-120) if pause_token_idx < len(token_times): [](#__codelineno-0-121) pause_gap = token_times[pause_token_idx] - token_times[pause_token_idx - 1] [](#__codelineno-0-122) print( [](#__codelineno-0-123) f"\nGap after pause (token {pause_token_idx} -> " [](#__codelineno-0-124) f"{pause_token_idx + 1}): {pause_gap:.3f}s" [](#__codelineno-0-125) ) [](#__codelineno-0-126) if pause_gap >= PAUSE_DURATION * 0.9: [](#__codelineno-0-127) print("Test passed! Pause synchronized across DP ranks.") [](#__codelineno-0-128) else: [](#__codelineno-0-129) print(f"Test failed! Expected ~{PAUSE_DURATION}s gap, got {pause_gap:.3f}s") [](#__codelineno-0-130) else: [](#__codelineno-0-131) print("Test failed! No tokens were generated after resuming.") [](#__codelineno-0-132) [](#__codelineno-0-133)[](#__codelineno-0-134)if __name__ == "__main__": [](#__codelineno-0-135) main()` ## Pause Resume Offline[¶](#pause-resume-offline "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)"""Test for pause/resume with keep mode. [](#__codelineno-1-4)[](#__codelineno-1-5)This test uses concurrent tasks to verify the engine truly stops generating [](#__codelineno-1-6)during pause: [](#__codelineno-1-7)1. Generator task: continuously generates and logs time between tokens [](#__codelineno-1-8)2. Controller task: sends pause/resume commands [](#__codelineno-1-9)[](#__codelineno-1-10)If the engine properly pauses, we should see a gap in token timestamps [](#__codelineno-1-11)matching the pause duration. [](#__codelineno-1-12)""" [](#__codelineno-1-13)[](#__codelineno-1-14)import asyncio [](#__codelineno-1-15)import time [](#__codelineno-1-16)[](#__codelineno-1-17)from vllm import SamplingParams [](#__codelineno-1-18)from vllm.engine.arg_utils import AsyncEngineArgs [](#__codelineno-1-19)from vllm.v1.engine.async_llm import AsyncLLM [](#__codelineno-1-20)[](#__codelineno-1-21)PAUSE_DURATION = 3.0 # seconds [](#__codelineno-1-22) [](#__codelineno-1-23)[](#__codelineno-1-24)async def main(): [](#__codelineno-1-25) # Create engine with a small model [](#__codelineno-1-26) engine_args = AsyncEngineArgs( [](#__codelineno-1-27) model="facebook/opt-125m", [](#__codelineno-1-28) enforce_eager=True, [](#__codelineno-1-29) ) [](#__codelineno-1-30) engine = AsyncLLM.from_engine_args(engine_args) [](#__codelineno-1-31) [](#__codelineno-1-32) prompt = "Write a story about a dragon. Once upon a time" [](#__codelineno-1-33) sampling_params = SamplingParams(max_tokens=30, ignore_eos=True) [](#__codelineno-1-34) [](#__codelineno-1-35) # Track token arrival times [](#__codelineno-1-36) token_times: list[tuple[int, float]] = [] # (token_count, timestamp) [](#__codelineno-1-37) pause_time: float = 0 [](#__codelineno-1-38) resume_time: float = 0 [](#__codelineno-1-39) pause_token_idx: int = 0 # Index in token_times when pause occurred [](#__codelineno-1-40) [](#__codelineno-1-41) async def generator_task(): [](#__codelineno-1-42) """Generate tokens and record timestamps.""" [](#__codelineno-1-43) async for output in engine.generate( [](#__codelineno-1-44) request_id="test-req", [](#__codelineno-1-45) prompt=prompt, [](#__codelineno-1-46) sampling_params=sampling_params, [](#__codelineno-1-47) ): [](#__codelineno-1-48) token_count = len(output.outputs[0].token_ids) [](#__codelineno-1-49) token_times.append((token_count, time.monotonic())) [](#__codelineno-1-50) print( [](#__codelineno-1-51) f"Token {token_count} arrived:" [](#__codelineno-1-52) f"T={token_times[-1][1] - token_times[0][1]:.3f}s" [](#__codelineno-1-53) ) [](#__codelineno-1-54) return output [](#__codelineno-1-55) [](#__codelineno-1-56) async def controller_task(): [](#__codelineno-1-57) """Pause and resume the engine after some tokens generated.""" [](#__codelineno-1-58) nonlocal pause_time, resume_time, pause_token_idx [](#__codelineno-1-59) [](#__codelineno-1-60) # Wait for some tokens to be generated [](#__codelineno-1-61) while len(token_times) < 5: [](#__codelineno-1-62) await asyncio.sleep(0.01) [](#__codelineno-1-63) [](#__codelineno-1-64) print(f"\nPausing engine (keep mode) at token {len(token_times)}") [](#__codelineno-1-65) pause_time = time.monotonic() [](#__codelineno-1-66) await engine.pause_generation(mode="keep") [](#__codelineno-1-67) pause_token_idx = len(token_times) [](#__codelineno-1-68) print(f"Paused! Sleeping for {PAUSE_DURATION}s...") [](#__codelineno-1-69) [](#__codelineno-1-70) # Sleep while paused - no tokens should be generated during this time [](#__codelineno-1-71) await asyncio.sleep(PAUSE_DURATION) [](#__codelineno-1-72) [](#__codelineno-1-73) print("Resuming engine...") [](#__codelineno-1-74) await engine.resume_generation() [](#__codelineno-1-75) resume_time = time.monotonic() [](#__codelineno-1-76) print("Resumed!\n") [](#__codelineno-1-77) [](#__codelineno-1-78) # Run both tasks concurrently [](#__codelineno-1-79) gen_task = asyncio.create_task(generator_task()) [](#__codelineno-1-80) ctrl_task = asyncio.create_task(controller_task()) [](#__codelineno-1-81) [](#__codelineno-1-82) final_output, _ = await asyncio.gather(gen_task, ctrl_task) [](#__codelineno-1-83) [](#__codelineno-1-84) # Verify the pause actually stopped generation. [](#__codelineno-1-85) # The gap after the pause token should be approximately the sleep duration. [](#__codelineno-1-86) pause_gap = token_times[pause_token_idx][1] - token_times[pause_token_idx - 1][1] [](#__codelineno-1-87) print( [](#__codelineno-1-88) f"\nGap after pause (token {pause_token_idx - 1} -> {pause_token_idx}): " [](#__codelineno-1-89) f"{pause_gap:.3f}s" [](#__codelineno-1-90) ) [](#__codelineno-1-91) if pause_gap >= PAUSE_DURATION * 0.9: [](#__codelineno-1-92) print(f"✓ Test passed! Engine paused for ~{pause_gap:.1f}s") [](#__codelineno-1-93) else: [](#__codelineno-1-94) print( [](#__codelineno-1-95) f"✗ Test failed! Expected ~{PAUSE_DURATION}s gap after pause, " [](#__codelineno-1-96) f"got {pause_gap:.3f}s" [](#__codelineno-1-97) ) [](#__codelineno-1-98) raise AssertionError("Engine did not properly pause") [](#__codelineno-1-99) [](#__codelineno-1-100) # Verify request completed [](#__codelineno-1-101) assert final_output.finished, "Request should have finished" [](#__codelineno-1-102) assert len(final_output.outputs[0].token_ids) == 30, "Should have all tokens" [](#__codelineno-1-103) [](#__codelineno-1-104) engine.shutdown() [](#__codelineno-1-105) [](#__codelineno-1-106)[](#__codelineno-1-107)if __name__ == "__main__": [](#__codelineno-1-108) asyncio.run(main())` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/profiling.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/profiling](https://github.com/vllm-project/vllm/tree/main/examples/features/profiling). ## Run One Batch Offline[¶](#run-one-batch-offline "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)[](#__codelineno-0-4)from __future__ import annotations [](#__codelineno-0-5)[](#__codelineno-0-6)from vllm import LLM, EngineArgs [](#__codelineno-0-7)from vllm.config import ProfilerConfig [](#__codelineno-0-8)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-0-9)[](#__codelineno-0-10)DEFAULT_MAX_TOKENS = 16 [](#__codelineno-0-11) [](#__codelineno-0-12)[](#__codelineno-0-13)def create_parser() -> FlexibleArgumentParser: [](#__codelineno-0-14) parser = FlexibleArgumentParser() [](#__codelineno-0-15) EngineArgs.add_cli_args(parser) [](#__codelineno-0-16) parser.set_defaults(model="meta-llama/Llama-3.2-1B-Instruct") [](#__codelineno-0-17) [](#__codelineno-0-18) batch_group = parser.add_argument_group("Batch parameters") [](#__codelineno-0-19) batch_group.add_argument("--batch-size", type=int, default=1) [](#__codelineno-0-20) batch_group.add_argument("--prompt-size", type=int, default=128) [](#__codelineno-0-21) batch_group.add_argument("--prompt-prefix", type=str, default="Hello, my name is") [](#__codelineno-0-22) [](#__codelineno-0-23) profile_group = parser.add_argument_group("Profiling parameters") [](#__codelineno-0-24) profile_group.add_argument( [](#__codelineno-0-25) "--profile", [](#__codelineno-0-26) choices=["none", "prefill", "decode", "both"], [](#__codelineno-0-27) default="none", [](#__codelineno-0-28) ) [](#__codelineno-0-29) profile_group.add_argument( [](#__codelineno-0-30) "--profile-dir", [](#__codelineno-0-31) type=str, [](#__codelineno-0-32) default="", [](#__codelineno-0-33) help="Required when --profile is not 'none'.", [](#__codelineno-0-34) ) [](#__codelineno-0-35) [](#__codelineno-0-36) return parser [](#__codelineno-0-37) [](#__codelineno-0-38)[](#__codelineno-0-39)def _build_prompt(prefix: str, prompt_size: int) -> str: [](#__codelineno-0-40) if prompt_size <= 0: [](#__codelineno-0-41) return "" [](#__codelineno-0-42) if not prefix: [](#__codelineno-0-43) prefix = " " [](#__codelineno-0-44) if len(prefix) >= prompt_size: [](#__codelineno-0-45) return prefix[:prompt_size] [](#__codelineno-0-46) repeat_count = (prompt_size + len(prefix) - 1) // len(prefix) [](#__codelineno-0-47) return (prefix * repeat_count)[:prompt_size] [](#__codelineno-0-48) [](#__codelineno-0-49)[](#__codelineno-0-50)def _build_profiler_config( [](#__codelineno-0-51) profile: str, profile_dir: str, max_tokens: int [](#__codelineno-0-52)) -> ProfilerConfig | None: [](#__codelineno-0-53) if profile == "none": [](#__codelineno-0-54) return None [](#__codelineno-0-55) if not profile_dir: [](#__codelineno-0-56) raise ValueError("--profile-dir must be set when profiling is enabled.") [](#__codelineno-0-57) if profile == "prefill": [](#__codelineno-0-58) delay_iterations = 0 [](#__codelineno-0-59) max_iterations = 1 [](#__codelineno-0-60) elif profile == "decode": [](#__codelineno-0-61) delay_iterations = 1 [](#__codelineno-0-62) max_iterations = max(1, max_tokens) [](#__codelineno-0-63) else: [](#__codelineno-0-64) delay_iterations = 0 [](#__codelineno-0-65) max_iterations = 0 [](#__codelineno-0-66) [](#__codelineno-0-67) return ProfilerConfig( [](#__codelineno-0-68) profiler="torch", [](#__codelineno-0-69) torch_profiler_dir=profile_dir, [](#__codelineno-0-70) delay_iterations=delay_iterations, [](#__codelineno-0-71) max_iterations=max_iterations, [](#__codelineno-0-72) ) [](#__codelineno-0-73) [](#__codelineno-0-74)[](#__codelineno-0-75)def main(args: dict) -> None: [](#__codelineno-0-76) max_tokens = DEFAULT_MAX_TOKENS [](#__codelineno-0-77) batch_size = args.pop("batch_size") [](#__codelineno-0-78) prompt_size = args.pop("prompt_size") [](#__codelineno-0-79) prompt_prefix = args.pop("prompt_prefix") [](#__codelineno-0-80) profile = args.pop("profile") [](#__codelineno-0-81) profile_dir = args.pop("profile_dir") [](#__codelineno-0-82) [](#__codelineno-0-83) profiler_config = _build_profiler_config(profile, profile_dir, max_tokens) [](#__codelineno-0-84) if profiler_config is not None: [](#__codelineno-0-85) args["profiler_config"] = profiler_config [](#__codelineno-0-86) [](#__codelineno-0-87) llm = LLM(**args) [](#__codelineno-0-88) [](#__codelineno-0-89) sampling_params = llm.get_default_sampling_params() [](#__codelineno-0-90) sampling_params.max_tokens = max_tokens [](#__codelineno-0-91) sampling_params.min_tokens = max_tokens [](#__codelineno-0-92) sampling_params.ignore_eos = True [](#__codelineno-0-93) [](#__codelineno-0-94) prompt = _build_prompt(prompt_prefix, prompt_size) [](#__codelineno-0-95) prompts = [prompt] * batch_size [](#__codelineno-0-96) [](#__codelineno-0-97) if profile != "none": [](#__codelineno-0-98) llm.start_profile() [](#__codelineno-0-99) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-100) if profile != "none": [](#__codelineno-0-101) llm.stop_profile() [](#__codelineno-0-102) [](#__codelineno-0-103) print("-" * 50) [](#__codelineno-0-104) for output in outputs: [](#__codelineno-0-105) generated_text = output.outputs[0].text [](#__codelineno-0-106) print(f"Prompt: {output.prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-0-107) print("-" * 50) [](#__codelineno-0-108) [](#__codelineno-0-109)[](#__codelineno-0-110)if __name__ == "__main__": [](#__codelineno-0-111) parser = create_parser() [](#__codelineno-0-112) main(vars(parser.parse_args()))` ## Simple Profiling Offline[¶](#simple-profiling-offline "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)[](#__codelineno-1-4)import time [](#__codelineno-1-5)[](#__codelineno-1-6)from vllm import LLM, SamplingParams [](#__codelineno-1-7)[](#__codelineno-1-8)# Sample prompts. [](#__codelineno-1-9)prompts = [ [](#__codelineno-1-10) "Hello, my name is", [](#__codelineno-1-11) "The president of the United States is", [](#__codelineno-1-12) "The capital of France is", [](#__codelineno-1-13) "The future of AI is", [](#__codelineno-1-14)] [](#__codelineno-1-15)# Create a sampling params object. [](#__codelineno-1-16)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-1-17) [](#__codelineno-1-18)[](#__codelineno-1-19)def main(): [](#__codelineno-1-20) # Create an LLM. [](#__codelineno-1-21) llm = LLM( [](#__codelineno-1-22) model="facebook/opt-125m", [](#__codelineno-1-23) tensor_parallel_size=1, [](#__codelineno-1-24) profiler_config={ [](#__codelineno-1-25) "profiler": "torch", [](#__codelineno-1-26) "torch_profiler_dir": "./vllm_profile", [](#__codelineno-1-27) }, [](#__codelineno-1-28) ) [](#__codelineno-1-29) [](#__codelineno-1-30) llm.start_profile() [](#__codelineno-1-31) [](#__codelineno-1-32) # Generate texts from the prompts. The output is a list of RequestOutput [](#__codelineno-1-33) # objects that contain the prompt, generated text, and other information. [](#__codelineno-1-34) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-1-35) [](#__codelineno-1-36) llm.stop_profile() [](#__codelineno-1-37) [](#__codelineno-1-38) # Print the outputs. [](#__codelineno-1-39) print("-" * 50) [](#__codelineno-1-40) for output in outputs: [](#__codelineno-1-41) prompt = output.prompt [](#__codelineno-1-42) generated_text = output.outputs[0].text [](#__codelineno-1-43) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}") [](#__codelineno-1-44) print("-" * 50) [](#__codelineno-1-45) [](#__codelineno-1-46) # Add a buffer to wait for profiler in the background process [](#__codelineno-1-47) # (in case MP is on) to finish writing profiling output. [](#__codelineno-1-48) time.sleep(10) [](#__codelineno-1-49) [](#__codelineno-1-50)[](#__codelineno-1-51)if __name__ == "__main__": [](#__codelineno-1-52) main()` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/prompt_embed.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/prompt\_embed](https://github.com/vllm-project/vllm/tree/main/examples/features/prompt_embed). ## Prompt Embed Inference With OpenAI Client[¶](#prompt-embed-inference-with-openai-client "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)"""vLLM OpenAI-Compatible Client with Prompt Embeddings. [](#__codelineno-0-4)[](#__codelineno-0-5)This script demonstrates how to: [](#__codelineno-0-6)1. Generate prompt embeddings using Hugging Face Transformers. [](#__codelineno-0-7)2. Encode them in base64 format. [](#__codelineno-0-8)3. Send them to a vLLM server for inference via both: [](#__codelineno-0-9) - OpenAI-compatible Chat Completions API [](#__codelineno-0-10) - OpenAI-compatible Completions API [](#__codelineno-0-11)[](#__codelineno-0-12)Important distinction between the two APIs: [](#__codelineno-0-13)[](#__codelineno-0-14)- Chat Completions API: `prompt_embeds` content parts should encode ONLY [](#__codelineno-0-15) the user-provided content, not a templated conversation. The server [](#__codelineno-0-16) renders the surrounding chat template around the embedded content at [](#__codelineno-0-17) request time, the same way it would for a plain text `content` string. [](#__codelineno-0-18) Embedding a full templated conversation here would double-apply the [](#__codelineno-0-19) template and likely produce undesirable results. [](#__codelineno-0-20)[](#__codelineno-0-21)- Completions API: the server does NOT apply a chat template to [](#__codelineno-0-22) `prompt_embeds`. The caller is responsible for producing embeddings for [](#__codelineno-0-23) the full, already-templated prompt (i.e. apply the chat template first, [](#__codelineno-0-24) then embed the resulting token IDs). Anything the model would normally [](#__codelineno-0-25) need (system prompt, role markers, generation prompt, etc.) must already [](#__codelineno-0-26) be baked into the embedded tokens. [](#__codelineno-0-27)[](#__codelineno-0-28)Run the vLLM server first: [](#__codelineno-0-29)vllm serve meta-llama/Llama-3.2-1B-Instruct \ [](#__codelineno-0-30) --runner generate \ [](#__codelineno-0-31) --max-model-len 4096 \ [](#__codelineno-0-32) --enable-prompt-embeds [](#__codelineno-0-33)[](#__codelineno-0-34)Run the client: [](#__codelineno-0-35)python examples/features/prompt_embed/prompt_embed_inference_with_openai_client.py [](#__codelineno-0-36)[](#__codelineno-0-37)Model: meta-llama/Llama-3.2-1B-Instruct [](#__codelineno-0-38)Note: This model is gated on Hugging Face Hub. [](#__codelineno-0-39) You must request access to use it: [](#__codelineno-0-40) https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct [](#__codelineno-0-41)[](#__codelineno-0-42)Dependencies: [](#__codelineno-0-43)- transformers [](#__codelineno-0-44)- torch [](#__codelineno-0-45)- openai [](#__codelineno-0-46)""" [](#__codelineno-0-47)[](#__codelineno-0-48)import transformers [](#__codelineno-0-49)from openai import OpenAI [](#__codelineno-0-50)[](#__codelineno-0-51)from vllm.utils.serial_utils import tensor2base64 [](#__codelineno-0-52) [](#__codelineno-0-53)[](#__codelineno-0-54)def run_chat_completion_prompt_embeds( [](#__codelineno-0-55) client: OpenAI, [](#__codelineno-0-56) model_name: str, [](#__codelineno-0-57) tokenizer: transformers.PreTrainedTokenizerBase, [](#__codelineno-0-58) embedding_layer, [](#__codelineno-0-59) messages: list[dict], [](#__codelineno-0-60)) -> None: [](#__codelineno-0-61) """Run a Chat Completions API request using prompt_embeds content parts. [](#__codelineno-0-62) [](#__codelineno-0-63) This example embeds ONLY the user-provided content of the final user turn, the [](#__codelineno-0-64) vLLM server applies the chat template around it at request time. [](#__codelineno-0-65) """ [](#__codelineno-0-66) user_content = messages[-1]["content"] [](#__codelineno-0-67) content_token_ids = tokenizer( [](#__codelineno-0-68) user_content, return_tensors="pt", add_special_tokens=False [](#__codelineno-0-69) ).input_ids [](#__codelineno-0-70) content_prompt_embeds = embedding_layer(content_token_ids).squeeze(0) [](#__codelineno-0-71) encoded_embeds = tensor2base64(content_prompt_embeds) [](#__codelineno-0-72) [](#__codelineno-0-73) api_messages = [ [](#__codelineno-0-74) *messages[:-1], [](#__codelineno-0-75) { [](#__codelineno-0-76) "role": messages[-1]["role"], [](#__codelineno-0-77) "content": [{"type": "prompt_embeds", "data": encoded_embeds}], [](#__codelineno-0-78) }, [](#__codelineno-0-79) ] [](#__codelineno-0-80) [](#__codelineno-0-81) chat_completion = client.chat.completions.create( [](#__codelineno-0-82) model=model_name, [](#__codelineno-0-83) max_tokens=6, [](#__codelineno-0-84) temperature=0.0, [](#__codelineno-0-85) messages=api_messages, [](#__codelineno-0-86) ) [](#__codelineno-0-87) [](#__codelineno-0-88) print("-" * 30) [](#__codelineno-0-89) print("Chat Completions API") [](#__codelineno-0-90) print(chat_completion.choices[0].message.content) [](#__codelineno-0-91) print("-" * 30) [](#__codelineno-0-92) [](#__codelineno-0-93)[](#__codelineno-0-94)def run_completion_prompt_embeds( [](#__codelineno-0-95) client: OpenAI, [](#__codelineno-0-96) model_name: str, [](#__codelineno-0-97) tokenizer: transformers.PreTrainedTokenizerBase, [](#__codelineno-0-98) embedding_layer, [](#__codelineno-0-99) messages: list[dict], [](#__codelineno-0-100)) -> None: [](#__codelineno-0-101) """Run a Completions API request using prompt embeddings. [](#__codelineno-0-102) [](#__codelineno-0-103) The Completions endpoint does not apply a chat template, [](#__codelineno-0-104) so the caller must apply it and embed the full templated prompt. [](#__codelineno-0-105) """ [](#__codelineno-0-106) templated_token_ids = tokenizer.apply_chat_template( [](#__codelineno-0-107) messages, add_generation_prompt=True, return_tensors="pt", return_dict=True [](#__codelineno-0-108) ).input_ids [](#__codelineno-0-109) templated_prompt_embeds = embedding_layer(templated_token_ids).squeeze(0) [](#__codelineno-0-110) encoded_embeds = tensor2base64(templated_prompt_embeds) [](#__codelineno-0-111) [](#__codelineno-0-112) completion = client.completions.create( [](#__codelineno-0-113) model=model_name, [](#__codelineno-0-114) prompt=None, [](#__codelineno-0-115) max_tokens=6, [](#__codelineno-0-116) temperature=0.0, [](#__codelineno-0-117) # NOTE: The OpenAI client allows passing in extra JSON body via the [](#__codelineno-0-118) # `extra_body` argument. [](#__codelineno-0-119) extra_body={"prompt_embeds": encoded_embeds}, [](#__codelineno-0-120) ) [](#__codelineno-0-121) [](#__codelineno-0-122) print("-" * 30) [](#__codelineno-0-123) print("Completions API") [](#__codelineno-0-124) print(completion.choices[0].text) [](#__codelineno-0-125) print("-" * 30) [](#__codelineno-0-126) [](#__codelineno-0-127)[](#__codelineno-0-128)def main() -> None: [](#__codelineno-0-129) client = OpenAI( [](#__codelineno-0-130) api_key="EMPTY", [](#__codelineno-0-131) base_url="http://localhost:8000/v1", [](#__codelineno-0-132) ) [](#__codelineno-0-133) [](#__codelineno-0-134) model_name = "meta-llama/Llama-3.2-1B-Instruct" [](#__codelineno-0-135) [](#__codelineno-0-136) tokenizer = transformers.AutoTokenizer.from_pretrained(model_name) [](#__codelineno-0-137) transformers_model = transformers.AutoModelForCausalLM.from_pretrained(model_name) [](#__codelineno-0-138) embedding_layer = transformers_model.get_input_embeddings() [](#__codelineno-0-139) [](#__codelineno-0-140) messages = [ [](#__codelineno-0-141) {"role": "user", "content": "Please tell me about the capital of France."} [](#__codelineno-0-142) ] [](#__codelineno-0-143) [](#__codelineno-0-144) # Chat Completions API: embed ONLY the user content. The server wraps [](#__codelineno-0-145) # the embedding in the chat template when it renders the messages. [](#__codelineno-0-146) run_chat_completion_prompt_embeds( [](#__codelineno-0-147) client, model_name, tokenizer, embedding_layer, messages [](#__codelineno-0-148) ) [](#__codelineno-0-149) [](#__codelineno-0-150) # Completions API: embed the FULL templated prompt. The caller must [](#__codelineno-0-151) # apply the chat template up-front. [](#__codelineno-0-152) run_completion_prompt_embeds( [](#__codelineno-0-153) client, model_name, tokenizer, embedding_layer, messages [](#__codelineno-0-154) ) [](#__codelineno-0-155) [](#__codelineno-0-156)[](#__codelineno-0-157)if __name__ == "__main__": [](#__codelineno-0-158) main()`` ## Prompt Embed Offline[¶](#prompt-embed-offline "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)Demonstrates how to generate prompt embeddings using [](#__codelineno-1-5)Hugging Face Transformers and use them as input to vLLM [](#__codelineno-1-6)for both single and batch inference. [](#__codelineno-1-7)[](#__codelineno-1-8)Model: meta-llama/Llama-3.2-1B-Instruct [](#__codelineno-1-9)Note: This model is gated on Hugging Face Hub. [](#__codelineno-1-10) You must request access to use it: [](#__codelineno-1-11) https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct [](#__codelineno-1-12)[](#__codelineno-1-13)Requirements: [](#__codelineno-1-14)- vLLM [](#__codelineno-1-15)- transformers [](#__codelineno-1-16)[](#__codelineno-1-17)Run: [](#__codelineno-1-18) python examples/features/prompt_embed/prompt_embed_offline.py [](#__codelineno-1-19)""" [](#__codelineno-1-20)[](#__codelineno-1-21)import torch [](#__codelineno-1-22)from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedTokenizer [](#__codelineno-1-23)[](#__codelineno-1-24)from vllm import LLM [](#__codelineno-1-25) [](#__codelineno-1-26)[](#__codelineno-1-27)def init_tokenizer_and_llm(model_name: str): [](#__codelineno-1-28) tokenizer = AutoTokenizer.from_pretrained(model_name) [](#__codelineno-1-29) transformers_model = AutoModelForCausalLM.from_pretrained(model_name) [](#__codelineno-1-30) embedding_layer = transformers_model.get_input_embeddings() [](#__codelineno-1-31) llm = LLM(model=model_name, enable_prompt_embeds=True) [](#__codelineno-1-32) return tokenizer, embedding_layer, llm [](#__codelineno-1-33) [](#__codelineno-1-34)[](#__codelineno-1-35)def get_prompt_embeds( [](#__codelineno-1-36) chat: list[dict[str, str]], [](#__codelineno-1-37) tokenizer: PreTrainedTokenizer, [](#__codelineno-1-38) embedding_layer: torch.nn.Module, [](#__codelineno-1-39)): [](#__codelineno-1-40) token_ids = tokenizer.apply_chat_template( [](#__codelineno-1-41) chat, add_generation_prompt=True, return_tensors="pt", return_dict=True [](#__codelineno-1-42) ).input_ids [](#__codelineno-1-43) prompt_embeds = embedding_layer(token_ids).squeeze(0) [](#__codelineno-1-44) return prompt_embeds [](#__codelineno-1-45) [](#__codelineno-1-46)[](#__codelineno-1-47)def single_prompt_inference( [](#__codelineno-1-48) llm: LLM, tokenizer: PreTrainedTokenizer, embedding_layer: torch.nn.Module [](#__codelineno-1-49)): [](#__codelineno-1-50) chat = [{"role": "user", "content": "Please tell me about the capital of France."}] [](#__codelineno-1-51) prompt_embeds = get_prompt_embeds(chat, tokenizer, embedding_layer) [](#__codelineno-1-52) [](#__codelineno-1-53) outputs = llm.generate( [](#__codelineno-1-54) { [](#__codelineno-1-55) "prompt_embeds": prompt_embeds, [](#__codelineno-1-56) } [](#__codelineno-1-57) ) [](#__codelineno-1-58) [](#__codelineno-1-59) print("\n[Single Inference Output]") [](#__codelineno-1-60) print("-" * 30) [](#__codelineno-1-61) for o in outputs: [](#__codelineno-1-62) print(o.outputs[0].text) [](#__codelineno-1-63) print("-" * 30) [](#__codelineno-1-64) [](#__codelineno-1-65)[](#__codelineno-1-66)def batch_prompt_inference( [](#__codelineno-1-67) llm: LLM, tokenizer: PreTrainedTokenizer, embedding_layer: torch.nn.Module [](#__codelineno-1-68)): [](#__codelineno-1-69) chats = [ [](#__codelineno-1-70) [{"role": "user", "content": "Please tell me about the capital of France."}], [](#__codelineno-1-71) [{"role": "user", "content": "When is the day longest during the year?"}], [](#__codelineno-1-72) [{"role": "user", "content": "Where is bigger, the moon or the sun?"}], [](#__codelineno-1-73) ] [](#__codelineno-1-74) [](#__codelineno-1-75) prompt_embeds_list = [ [](#__codelineno-1-76) get_prompt_embeds(chat, tokenizer, embedding_layer) for chat in chats [](#__codelineno-1-77) ] [](#__codelineno-1-78) [](#__codelineno-1-79) outputs = llm.generate([{"prompt_embeds": embeds} for embeds in prompt_embeds_list]) [](#__codelineno-1-80) [](#__codelineno-1-81) print("\n[Batch Inference Outputs]") [](#__codelineno-1-82) print("-" * 30) [](#__codelineno-1-83) for i, o in enumerate(outputs): [](#__codelineno-1-84) print(f"Q{i + 1}: {chats[i][0]['content']}") [](#__codelineno-1-85) print(f"A{i + 1}: {o.outputs[0].text}\n") [](#__codelineno-1-86) print("-" * 30) [](#__codelineno-1-87) [](#__codelineno-1-88)[](#__codelineno-1-89)def main(): [](#__codelineno-1-90) model_name = "meta-llama/Llama-3.2-1B-Instruct" [](#__codelineno-1-91) tokenizer, embedding_layer, llm = init_tokenizer_and_llm(model_name) [](#__codelineno-1-92) single_prompt_inference(llm, tokenizer, embedding_layer) [](#__codelineno-1-93) batch_prompt_inference(llm, tokenizer, embedding_layer) [](#__codelineno-1-94) [](#__codelineno-1-95)[](#__codelineno-1-96)if __name__ == "__main__": [](#__codelineno-1-97) main()` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/reset_kv.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/reset\_kv](https://github.com/vllm-project/vllm/tree/main/examples/features/reset_kv). ## Reset Kv Offline[¶](#reset-kv-offline "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This file demonstrates preempt requests when using the `LLMEngine` [](#__codelineno-0-5)for processing prompts with various sampling parameters. [](#__codelineno-0-6)""" [](#__codelineno-0-7)[](#__codelineno-0-8)import argparse [](#__codelineno-0-9)[](#__codelineno-0-10)from vllm import EngineArgs, LLMEngine, RequestOutput, SamplingParams [](#__codelineno-0-11)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-0-12) [](#__codelineno-0-13)[](#__codelineno-0-14)def create_test_prompts() -> list[tuple[str, SamplingParams]]: [](#__codelineno-0-15) """Create a list of test prompts with their sampling parameters.""" [](#__codelineno-0-16) return [ [](#__codelineno-0-17) ( [](#__codelineno-0-18) "A robot may not injure a human being " * 50, [](#__codelineno-0-19) SamplingParams( [](#__codelineno-0-20) temperature=0.0, logprobs=1, prompt_logprobs=1, max_tokens=16 [](#__codelineno-0-21) ), [](#__codelineno-0-22) ), [](#__codelineno-0-23) ( [](#__codelineno-0-24) "A robot may not injure a human being " * 50, [](#__codelineno-0-25) SamplingParams( [](#__codelineno-0-26) temperature=0.0, logprobs=1, prompt_logprobs=1, max_tokens=16 [](#__codelineno-0-27) ), [](#__codelineno-0-28) ), [](#__codelineno-0-29) ( [](#__codelineno-0-30) "To be or not to be,", [](#__codelineno-0-31) SamplingParams( [](#__codelineno-0-32) temperature=0.8, top_k=5, presence_penalty=0.2, max_tokens=128 [](#__codelineno-0-33) ), [](#__codelineno-0-34) ), [](#__codelineno-0-35) ( [](#__codelineno-0-36) "What is the meaning of life?", [](#__codelineno-0-37) SamplingParams( [](#__codelineno-0-38) n=2, temperature=0.8, top_p=0.95, frequency_penalty=0.1, max_tokens=128 [](#__codelineno-0-39) ), [](#__codelineno-0-40) ), [](#__codelineno-0-41) ] [](#__codelineno-0-42) [](#__codelineno-0-43)[](#__codelineno-0-44)def process_requests(engine: LLMEngine, test_prompts: list[tuple[str, SamplingParams]]): [](#__codelineno-0-45) """Continuously process a list of prompts and handle the outputs.""" [](#__codelineno-0-46) request_id = 0 [](#__codelineno-0-47) [](#__codelineno-0-48) print("-" * 50) [](#__codelineno-0-49) step_id = 0 [](#__codelineno-0-50) while test_prompts or engine.has_unfinished_requests(): [](#__codelineno-0-51) print("-" * 50) [](#__codelineno-0-52) import os [](#__codelineno-0-53) [](#__codelineno-0-54) print(f"Step {step_id} (pid={os.getpid()})") [](#__codelineno-0-55) [](#__codelineno-0-56) if test_prompts: [](#__codelineno-0-57) prompt, sampling_params = test_prompts.pop(0) [](#__codelineno-0-58) engine.add_request(str(request_id), prompt, sampling_params) [](#__codelineno-0-59) request_id += 1 [](#__codelineno-0-60) [](#__codelineno-0-61) if step_id == 10: [](#__codelineno-0-62) print(f"Resetting prefix cache at {step_id}") [](#__codelineno-0-63) engine.reset_prefix_cache(reset_running_requests=True) [](#__codelineno-0-64) [](#__codelineno-0-65) request_outputs: list[RequestOutput] = engine.step() [](#__codelineno-0-66) [](#__codelineno-0-67) for request_output in request_outputs: [](#__codelineno-0-68) if request_output.finished: [](#__codelineno-0-69) print("-" * 50) [](#__codelineno-0-70) print(request_output) [](#__codelineno-0-71) print("-" * 50) [](#__codelineno-0-72) step_id += 1 [](#__codelineno-0-73) [](#__codelineno-0-74)[](#__codelineno-0-75)def initialize_engine(args: argparse.Namespace) -> LLMEngine: [](#__codelineno-0-76) """Initialize the LLMEngine from the command line arguments.""" [](#__codelineno-0-77) engine_args = EngineArgs.from_cli_args(args) [](#__codelineno-0-78) return LLMEngine.from_engine_args(engine_args) [](#__codelineno-0-79) [](#__codelineno-0-80)[](#__codelineno-0-81)def parse_args(): [](#__codelineno-0-82) parser = FlexibleArgumentParser( [](#__codelineno-0-83) description="Demo on using the LLMEngine class directly" [](#__codelineno-0-84) ) [](#__codelineno-0-85) parser = EngineArgs.add_cli_args(parser) [](#__codelineno-0-86) return parser.parse_args() [](#__codelineno-0-87) [](#__codelineno-0-88)[](#__codelineno-0-89)def main(args: argparse.Namespace): [](#__codelineno-0-90) """Main function that sets up and runs the prompt processing.""" [](#__codelineno-0-91) engine = initialize_engine(args) [](#__codelineno-0-92) test_prompts = create_test_prompts() [](#__codelineno-0-93) process_requests(engine, test_prompts) [](#__codelineno-0-94) [](#__codelineno-0-95)[](#__codelineno-0-96)if __name__ == "__main__": [](#__codelineno-0-97) args = parse_args() [](#__codelineno-0-98) main(args)`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/sharded_state.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/sharded\_state](https://github.com/vllm-project/vllm/tree/main/examples/features/sharded_state). ## Load Sharded State Offline[¶](#load-sharded-state-offline "Permanent link") `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)Validates the loading of a model saved with the sharded_state format. [](#__codelineno-0-5)This script demonstrates how to load a model that was previously saved [](#__codelineno-0-6)using save_sharded_state_offline.py and validates it by running inference. [](#__codelineno-0-7)Example usage: [](#__codelineno-0-8)(First need to save a sharded_state mode) [](#__codelineno-0-9)[](#__codelineno-0-10)python save_sharded_state_offline.py \ [](#__codelineno-0-11) --model /path/to/load \ [](#__codelineno-0-12) --tensor-parallel-size 8 \ [](#__codelineno-0-13) --output /path/to/save/sharded/model [](#__codelineno-0-14)[](#__codelineno-0-15)python load_sharded_state_offline.py \ [](#__codelineno-0-16) --model /path/to/saved/sharded/model \ [](#__codelineno-0-17) --load-format sharded_state \ [](#__codelineno-0-18) --tensor-parallel-size 8 \ [](#__codelineno-0-19) --prompt "Hello, my name is" \ [](#__codelineno-0-20) --max-tokens 50 [](#__codelineno-0-21)""" [](#__codelineno-0-22)[](#__codelineno-0-23)from vllm import LLM, EngineArgs, SamplingParams [](#__codelineno-0-24)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-0-25) [](#__codelineno-0-26)[](#__codelineno-0-27)def parse_args(): [](#__codelineno-0-28) parser = FlexibleArgumentParser() [](#__codelineno-0-29) # Add engine arguments [](#__codelineno-0-30) EngineArgs.add_cli_args(parser) [](#__codelineno-0-31) [](#__codelineno-0-32) # Override default load_format for clarity [](#__codelineno-0-33) parser.set_defaults(load_format="sharded_state") [](#__codelineno-0-34) [](#__codelineno-0-35) # Add validation arguments [](#__codelineno-0-36) parser.add_argument( [](#__codelineno-0-37) "--prompt", type=str, default="Hello, world!", help="Prompt for validation" [](#__codelineno-0-38) ) [](#__codelineno-0-39) parser.add_argument( [](#__codelineno-0-40) "--max-tokens", [](#__codelineno-0-41) type=int, [](#__codelineno-0-42) default=100, [](#__codelineno-0-43) help="Maximum number of tokens to generate", [](#__codelineno-0-44) ) [](#__codelineno-0-45) parser.add_argument( [](#__codelineno-0-46) "--temperature", type=float, default=0.7, help="Sampling temperature" [](#__codelineno-0-47) ) [](#__codelineno-0-48) parser.add_argument( [](#__codelineno-0-49) "--top-p", type=float, default=1.0, help="Top-p sampling parameter" [](#__codelineno-0-50) ) [](#__codelineno-0-51) [](#__codelineno-0-52) return parser.parse_args() [](#__codelineno-0-53) [](#__codelineno-0-54)[](#__codelineno-0-55)def main(): [](#__codelineno-0-56) args = parse_args() [](#__codelineno-0-57) engine_args = EngineArgs.from_cli_args(args) [](#__codelineno-0-58) [](#__codelineno-0-59) print( [](#__codelineno-0-60) f"Loading model from {engine_args.model} using format {engine_args.load_format}" [](#__codelineno-0-61) ) [](#__codelineno-0-62) print(f"Tensor parallel size: {engine_args.tensor_parallel_size}") [](#__codelineno-0-63) [](#__codelineno-0-64) # Load the model using engine args [](#__codelineno-0-65) llm = LLM.from_engine_args(engine_args) [](#__codelineno-0-66) [](#__codelineno-0-67) # Prepare sampling parameters [](#__codelineno-0-68) sampling_params = SamplingParams( [](#__codelineno-0-69) temperature=args.temperature, [](#__codelineno-0-70) top_p=args.top_p, [](#__codelineno-0-71) max_tokens=args.max_tokens, [](#__codelineno-0-72) ) [](#__codelineno-0-73) [](#__codelineno-0-74) print("\nRunning inference:") [](#__codelineno-0-75) print(f"Prompt: {args.prompt}") [](#__codelineno-0-76) [](#__codelineno-0-77) # Generate completion [](#__codelineno-0-78) outputs = llm.generate(args.prompt, sampling_params) [](#__codelineno-0-79) [](#__codelineno-0-80) # Display generated text [](#__codelineno-0-81) print("\nGenerated outputs:") [](#__codelineno-0-82) for output in outputs: [](#__codelineno-0-83) generated_text = output.outputs[0].text [](#__codelineno-0-84) print("-" * 50) [](#__codelineno-0-85) print(f"Full output: {args.prompt}{generated_text}") [](#__codelineno-0-86) print("-" * 50) [](#__codelineno-0-87) [](#__codelineno-0-88)[](#__codelineno-0-89)if __name__ == "__main__": [](#__codelineno-0-90) main()` ## Save Sharded State Offline[¶](#save-sharded-state-offline "Permanent link") `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)Saves each worker's model state dict directly to a checkpoint, which enables a [](#__codelineno-1-5)fast load path for large tensor-parallel models where each worker only needs to [](#__codelineno-1-6)read its own shard rather than the entire checkpoint. [](#__codelineno-1-7)[](#__codelineno-1-8)Example usage: [](#__codelineno-1-9)[](#__codelineno-1-10)python save_sharded_state_offline.py \ [](#__codelineno-1-11) --model /path/to/load \ [](#__codelineno-1-12) --tensor-parallel-size 8 \ [](#__codelineno-1-13) --output /path/to/save [](#__codelineno-1-14)[](#__codelineno-1-15)Then, the model can be loaded with [](#__codelineno-1-16)[](#__codelineno-1-17)llm = LLM( [](#__codelineno-1-18) model="/path/to/save", [](#__codelineno-1-19) load_format="sharded_state", [](#__codelineno-1-20) tensor_parallel_size=8, [](#__codelineno-1-21)) [](#__codelineno-1-22)""" [](#__codelineno-1-23)[](#__codelineno-1-24)import os [](#__codelineno-1-25)import shutil [](#__codelineno-1-26)from pathlib import Path [](#__codelineno-1-27)[](#__codelineno-1-28)from vllm import LLM, EngineArgs [](#__codelineno-1-29)from vllm.model_executor.model_loader import ShardedStateLoader [](#__codelineno-1-30)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-1-31) [](#__codelineno-1-32)[](#__codelineno-1-33)def parse_args(): [](#__codelineno-1-34) parser = FlexibleArgumentParser() [](#__codelineno-1-35) EngineArgs.add_cli_args(parser) [](#__codelineno-1-36) parser.add_argument( [](#__codelineno-1-37) "--output", "-o", required=True, type=str, help="path to output checkpoint" [](#__codelineno-1-38) ) [](#__codelineno-1-39) parser.add_argument( [](#__codelineno-1-40) "--file-pattern", [](#__codelineno-1-41) type=str, [](#__codelineno-1-42) default=ShardedStateLoader.DEFAULT_PATTERN, [](#__codelineno-1-43) help="string pattern of saved filenames", [](#__codelineno-1-44) ) [](#__codelineno-1-45) parser.add_argument( [](#__codelineno-1-46) "--max-file-size", [](#__codelineno-1-47) type=int, [](#__codelineno-1-48) default=5 * 1024**3, [](#__codelineno-1-49) help="max size (in bytes) of each safetensors file", [](#__codelineno-1-50) ) [](#__codelineno-1-51) return parser.parse_args() [](#__codelineno-1-52) [](#__codelineno-1-53)[](#__codelineno-1-54)def main(args): [](#__codelineno-1-55) engine_args = EngineArgs.from_cli_args(args) [](#__codelineno-1-56) if engine_args.enable_lora: [](#__codelineno-1-57) raise ValueError("Saving with enable_lora=True is not supported!") [](#__codelineno-1-58) model_path = engine_args.model [](#__codelineno-1-59) if not Path(model_path).is_dir(): [](#__codelineno-1-60) raise ValueError("model path must be a local directory") [](#__codelineno-1-61) # Create LLM instance from arguments [](#__codelineno-1-62) llm = LLM.from_engine_args(engine_args) [](#__codelineno-1-63) # Prepare output directory [](#__codelineno-1-64) Path(args.output).mkdir(exist_ok=True) [](#__codelineno-1-65) # Dump worker states to output directory [](#__codelineno-1-66) [](#__codelineno-1-67) llm.llm_engine.engine_core.save_sharded_state( [](#__codelineno-1-68) path=args.output, pattern=args.file_pattern, max_size=args.max_file_size [](#__codelineno-1-69) ) [](#__codelineno-1-70) [](#__codelineno-1-71) # Copy metadata files to output directory [](#__codelineno-1-72) for file in os.listdir(model_path): [](#__codelineno-1-73) if os.path.splitext(file)[1] not in (".bin", ".pt", ".safetensors"): [](#__codelineno-1-74) if os.path.isdir(os.path.join(model_path, file)): [](#__codelineno-1-75) shutil.copytree( [](#__codelineno-1-76) os.path.join(model_path, file), os.path.join(args.output, file) [](#__codelineno-1-77) ) [](#__codelineno-1-78) else: [](#__codelineno-1-79) shutil.copy(os.path.join(model_path, file), args.output) [](#__codelineno-1-80) [](#__codelineno-1-81)[](#__codelineno-1-82)if __name__ == "__main__": [](#__codelineno-1-83) args = parse_args() [](#__codelineno-1-84) main(args)` --- # page `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)import tempfile [](#__codelineno-0-4)[](#__codelineno-0-5)from vllm import LLM, SamplingParams [](#__codelineno-0-6)from vllm.config.kv_transfer import KVTransferConfig [](#__codelineno-0-7)from vllm.distributed.kv_transfer.kv_connector.v1 import ( [](#__codelineno-0-8) example_hidden_states_connector, [](#__codelineno-0-9)) [](#__codelineno-0-10)[](#__codelineno-0-11)# NOTE: If changing the interface of the ExampleHiddenStatesConnector, please also [](#__codelineno-0-12)# update the benchmark in benchmarks/benchmark_hidden_state_extraction.py [](#__codelineno-0-13)# and the docs in docs/features/speculative_decoding/extract_hidden_states.md [](#__codelineno-0-14)[](#__codelineno-0-15)# Example: Using the custom "extract_hidden_states" speculator method and [](#__codelineno-0-16)# ExampleHiddenStatesConnector to extract and save hidden states from vllm [](#__codelineno-0-17)[](#__codelineno-0-18)with tempfile.TemporaryDirectory() as tmpdirname: [](#__codelineno-0-19) llm = LLM( [](#__codelineno-0-20) model="Qwen/Qwen3-8B", # Your target model [](#__codelineno-0-21) enable_chunked_prefill=False, # required [](#__codelineno-0-22) speculative_config={ [](#__codelineno-0-23) "method": "extract_hidden_states", [](#__codelineno-0-24) "num_speculative_tokens": 1, [](#__codelineno-0-25) "draft_model_config": { [](#__codelineno-0-26) "hf_config": { [](#__codelineno-0-27) "eagle_aux_hidden_state_layer_ids": [ # Target model layer indices [](#__codelineno-0-28) 1, [](#__codelineno-0-29) 2, [](#__codelineno-0-30) 3, [](#__codelineno-0-31) 4, [](#__codelineno-0-32) ], [](#__codelineno-0-33) }, [](#__codelineno-0-34) }, [](#__codelineno-0-35) }, [](#__codelineno-0-36) kv_transfer_config=KVTransferConfig( [](#__codelineno-0-37) kv_connector="ExampleHiddenStatesConnector", [](#__codelineno-0-38) kv_role="kv_producer", [](#__codelineno-0-39) kv_connector_extra_config={ [](#__codelineno-0-40) "shared_storage_path": tmpdirname, [](#__codelineno-0-41) }, [](#__codelineno-0-42) ), [](#__codelineno-0-43) ) [](#__codelineno-0-44) [](#__codelineno-0-45) prompts = ["Generate a sentence with hidden states", "Write a python function"] [](#__codelineno-0-46) sampling_params = SamplingParams(max_tokens=1) [](#__codelineno-0-47) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-48) [](#__codelineno-0-49) for output in outputs: [](#__codelineno-0-50) print("\nPrompt:", output.prompt) [](#__codelineno-0-51) print("Prompt token ids:", output.prompt_token_ids) [](#__codelineno-0-52) [](#__codelineno-0-53) hidden_states_path = output.kv_transfer_params.get("hidden_states_path") [](#__codelineno-0-54) assert hidden_states_path is not None [](#__codelineno-0-55) print("Prompt hidden states path:", hidden_states_path) [](#__codelineno-0-56) [](#__codelineno-0-57) obj = example_hidden_states_connector.load_hidden_states(hidden_states_path) [](#__codelineno-0-58) token_ids = obj["token_ids"] [](#__codelineno-0-59) hidden_states = obj["hidden_states"] [](#__codelineno-0-60) [](#__codelineno-0-61) print("Extracted token ids:", token_ids) # Matches prompt token ids [](#__codelineno-0-62) print( [](#__codelineno-0-63) "Extracted hidden states shape:", hidden_states.shape [](#__codelineno-0-64) ) # [prompt_len, num_extracted_layers, hidden_size] [](#__codelineno-0-65) print("Extracted hidden states:", hidden_states) [](#__codelineno-0-66) [](#__codelineno-0-67) example_hidden_states_connector.cleanup_hidden_states(hidden_states_path)` `[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)This file demonstrates the usage of text generation with an LLM model, [](#__codelineno-1-5)comparing the performance with and without speculative decoding. [](#__codelineno-1-6)[](#__codelineno-1-7)Note that this example is out of date and not supported in vLLM v1. [](#__codelineno-1-8)""" [](#__codelineno-1-9)[](#__codelineno-1-10)import gc [](#__codelineno-1-11)import time [](#__codelineno-1-12)[](#__codelineno-1-13)from vllm import LLM, SamplingParams [](#__codelineno-1-14) [](#__codelineno-1-15)[](#__codelineno-1-16)def time_generation( [](#__codelineno-1-17) llm: LLM, prompts: list[str], sampling_params: SamplingParams, title: str [](#__codelineno-1-18)): [](#__codelineno-1-19) # Generate texts from the prompts. The output is a list of RequestOutput [](#__codelineno-1-20) # objects that contain the prompt, generated text, and other information. [](#__codelineno-1-21) # Warmup first [](#__codelineno-1-22) llm.generate(prompts, sampling_params) [](#__codelineno-1-23) llm.generate(prompts, sampling_params) [](#__codelineno-1-24) start = time.time() [](#__codelineno-1-25) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-1-26) end = time.time() [](#__codelineno-1-27) print("-" * 50) [](#__codelineno-1-28) print(title) [](#__codelineno-1-29) print("time: ", (end - start) / sum(len(o.outputs[0].token_ids) for o in outputs)) [](#__codelineno-1-30) # Print the outputs. [](#__codelineno-1-31) for output in outputs: [](#__codelineno-1-32) generated_text = output.outputs[0].text [](#__codelineno-1-33) print(f"text: {generated_text!r}") [](#__codelineno-1-34) print("-" * 50) [](#__codelineno-1-35) [](#__codelineno-1-36)[](#__codelineno-1-37)def main(): [](#__codelineno-1-38) template = ( [](#__codelineno-1-39) "Below is an instruction that describes a task. Write a response " [](#__codelineno-1-40) "that appropriately completes the request.\n\n### Instruction:\n{}" [](#__codelineno-1-41) "\n\n### Response:\n" [](#__codelineno-1-42) ) [](#__codelineno-1-43) [](#__codelineno-1-44) # Sample prompts. [](#__codelineno-1-45) prompts = [ [](#__codelineno-1-46) "Write about the president of the United States.", [](#__codelineno-1-47) ] [](#__codelineno-1-48) prompts = [template.format(prompt) for prompt in prompts] [](#__codelineno-1-49) # Create a sampling params object. [](#__codelineno-1-50) sampling_params = SamplingParams(temperature=0.0, max_tokens=200) [](#__codelineno-1-51) [](#__codelineno-1-52) # Create an LLM without spec decoding [](#__codelineno-1-53) llm = LLM(model="meta-llama/Llama-2-13b-chat-hf") [](#__codelineno-1-54) [](#__codelineno-1-55) time_generation(llm, prompts, sampling_params, "Without speculation") [](#__codelineno-1-56) [](#__codelineno-1-57) del llm [](#__codelineno-1-58) gc.collect() [](#__codelineno-1-59) [](#__codelineno-1-60) # Create an LLM with spec decoding [](#__codelineno-1-61) llm = LLM( [](#__codelineno-1-62) model="meta-llama/Llama-2-13b-chat-hf", [](#__codelineno-1-63) speculative_config={ [](#__codelineno-1-64) "model": "ibm-ai-platform/llama-13b-accelerator", [](#__codelineno-1-65) }, [](#__codelineno-1-66) ) [](#__codelineno-1-67) [](#__codelineno-1-68) time_generation(llm, prompts, sampling_params, "With speculation") [](#__codelineno-1-69) [](#__codelineno-1-70)[](#__codelineno-1-71)if __name__ == "__main__": [](#__codelineno-1-72) main()` `[](#__codelineno-2-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-2-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-2-3)[](#__codelineno-2-4)from transformers import AutoTokenizer [](#__codelineno-2-5)[](#__codelineno-2-6)from vllm import LLM, SamplingParams [](#__codelineno-2-7)from vllm.benchmarks.datasets import add_dataset_parser, get_samples [](#__codelineno-2-8)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-2-9)from vllm.v1.metrics.reader import Counter, Vector [](#__codelineno-2-10)[](#__codelineno-2-11)QUESTION = "What is the content of each image?" [](#__codelineno-2-12)IMAGE_URLS = [ [](#__codelineno-2-13) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg", [](#__codelineno-2-14) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg", [](#__codelineno-2-15) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/flycatcher.jpeg", [](#__codelineno-2-16) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/somefish.jpg", [](#__codelineno-2-17) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/starfish.jpg", [](#__codelineno-2-18) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/snail.jpg", [](#__codelineno-2-19) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/thistle.jpg", [](#__codelineno-2-20) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/husky.jpg", [](#__codelineno-2-21) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/orangetabbycat.jpg", [](#__codelineno-2-22) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/guineapig.jpg", [](#__codelineno-2-23) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/rabbit.jpg", [](#__codelineno-2-24) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/horsepony.jpg", [](#__codelineno-2-25)] [](#__codelineno-2-26) [](#__codelineno-2-27)[](#__codelineno-2-28)def get_custom_mm_prompts(num_prompts): [](#__codelineno-2-29) prompts = [] [](#__codelineno-2-30) for url in IMAGE_URLS: [](#__codelineno-2-31) prompts.append( [](#__codelineno-2-32) [ [](#__codelineno-2-33) {"type": "image_url", "image_url": {"url": url}}, [](#__codelineno-2-34) {"type": "text", "text": QUESTION}, [](#__codelineno-2-35) ] [](#__codelineno-2-36) ) [](#__codelineno-2-37) if num_prompts > len(IMAGE_URLS): [](#__codelineno-2-38) prompts = prompts * (num_prompts // len(IMAGE_URLS) + 1) [](#__codelineno-2-39) [](#__codelineno-2-40) return [[{"role": "user", "content": prompt}] for prompt in prompts[:num_prompts]] [](#__codelineno-2-41) [](#__codelineno-2-42)[](#__codelineno-2-43)def parse_args(): [](#__codelineno-2-44) parser = FlexibleArgumentParser() [](#__codelineno-2-45) add_dataset_parser(parser) [](#__codelineno-2-46) parser.add_argument("--test", action="store_true") [](#__codelineno-2-47) parser.add_argument( [](#__codelineno-2-48) "--method", [](#__codelineno-2-49) type=str, [](#__codelineno-2-50) default="eagle", [](#__codelineno-2-51) choices=["ngram", "eagle", "eagle3", "mtp", "draft_model"], [](#__codelineno-2-52) ) [](#__codelineno-2-53) parser.add_argument("--backend", type=str, default="openai") [](#__codelineno-2-54) parser.add_argument("--num-spec-tokens", type=int, default=2) [](#__codelineno-2-55) parser.add_argument("--prompt-lookup-max", type=int, default=5) [](#__codelineno-2-56) parser.add_argument("--prompt-lookup-min", type=int, default=2) [](#__codelineno-2-57) parser.add_argument("--tp", type=int, default=1) [](#__codelineno-2-58) parser.add_argument("--enforce-eager", action="store_true") [](#__codelineno-2-59) parser.add_argument("--enable-chunked-prefill", action="store_true") [](#__codelineno-2-60) parser.add_argument("--max-model-len", type=int, default=16384) [](#__codelineno-2-61) parser.add_argument("--temp", type=float, default=0) [](#__codelineno-2-62) parser.add_argument("--top-p", type=float, default=1.0) [](#__codelineno-2-63) parser.add_argument("--top-k", type=int, default=-1) [](#__codelineno-2-64) parser.add_argument("--print-output", action="store_true") [](#__codelineno-2-65) parser.add_argument("--output-len", type=int, default=256) [](#__codelineno-2-66) parser.add_argument("--model-dir", type=str, default=None) [](#__codelineno-2-67) parser.add_argument("--eagle-dir", type=str, default=None) [](#__codelineno-2-68) parser.add_argument("--draft-model", type=str, default=None) [](#__codelineno-2-69) parser.add_argument("--custom-mm-prompts", action="store_true") [](#__codelineno-2-70) parser.add_argument("--gpu-memory-utilization", type=float, default=0.9) [](#__codelineno-2-71) parser.add_argument("--disable-padded-drafter-batch", action="store_true") [](#__codelineno-2-72) parser.add_argument("--max-num-seqs", type=int, default=None) [](#__codelineno-2-73) parser.add_argument("--parallel-drafting", action="store_true") [](#__codelineno-2-74) parser.add_argument("--allowed-local-media-path", type=str, default="") [](#__codelineno-2-75) return parser.parse_args() [](#__codelineno-2-76) [](#__codelineno-2-77)[](#__codelineno-2-78)def main(args): [](#__codelineno-2-79) model_dir = args.model_dir [](#__codelineno-2-80) if args.model_dir is None: [](#__codelineno-2-81) if args.custom_mm_prompts: [](#__codelineno-2-82) raise ValueError( [](#__codelineno-2-83) "custom_mm_prompts requires mm based models" [](#__codelineno-2-84) "default llama3.1-8b-instruct is not mm based" [](#__codelineno-2-85) "please specify model_dir to give a mm based model" [](#__codelineno-2-86) ) [](#__codelineno-2-87) model_dir = "meta-llama/Llama-3.1-8B-Instruct" [](#__codelineno-2-88) tokenizer = AutoTokenizer.from_pretrained(model_dir) [](#__codelineno-2-89) [](#__codelineno-2-90) if args.custom_mm_prompts: [](#__codelineno-2-91) prompts = llm_prompts = get_custom_mm_prompts(args.num_prompts) [](#__codelineno-2-92) else: [](#__codelineno-2-93) prompts = get_samples(args, tokenizer) [](#__codelineno-2-94) if args.enable_multimodal_chat: [](#__codelineno-2-95) llm_prompts = [p.prompt for p in prompts] [](#__codelineno-2-96) else: [](#__codelineno-2-97) # add_special_tokens is False to avoid adding bos twice [](#__codelineno-2-98) # when using chat templates [](#__codelineno-2-99) llm_prompts = [ [](#__codelineno-2-100) { [](#__codelineno-2-101) "prompt_token_ids": tokenizer.encode( [](#__codelineno-2-102) prompt.prompt, add_special_tokens=False [](#__codelineno-2-103) ), [](#__codelineno-2-104) "multi_modal_data": prompt.multi_modal_data, [](#__codelineno-2-105) } [](#__codelineno-2-106) for prompt in prompts [](#__codelineno-2-107) ] [](#__codelineno-2-108) if args.method == "eagle" or args.method == "eagle3": [](#__codelineno-2-109) eagle_dir = args.eagle_dir [](#__codelineno-2-110) if args.method == "eagle" and eagle_dir is None: [](#__codelineno-2-111) eagle_dir = "yuhuili/EAGLE-LLaMA3.1-Instruct-8B" [](#__codelineno-2-112) [](#__codelineno-2-113) elif args.method == "eagle3" and eagle_dir is None: [](#__codelineno-2-114) eagle_dir = "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B" [](#__codelineno-2-115) speculative_config = { [](#__codelineno-2-116) "method": args.method, [](#__codelineno-2-117) "model": eagle_dir, [](#__codelineno-2-118) "num_speculative_tokens": args.num_spec_tokens, [](#__codelineno-2-119) "disable_padded_drafter_batch": args.disable_padded_drafter_batch, [](#__codelineno-2-120) "parallel_drafting": args.parallel_drafting, [](#__codelineno-2-121) } [](#__codelineno-2-122) elif args.method == "ngram": [](#__codelineno-2-123) speculative_config = { [](#__codelineno-2-124) "method": "ngram", [](#__codelineno-2-125) "num_speculative_tokens": args.num_spec_tokens, [](#__codelineno-2-126) "prompt_lookup_max": args.prompt_lookup_max, [](#__codelineno-2-127) "prompt_lookup_min": args.prompt_lookup_min, [](#__codelineno-2-128) } [](#__codelineno-2-129) elif args.method == "draft_model": [](#__codelineno-2-130) assert args.draft_model is not None and args.draft_model != "" [](#__codelineno-2-131) speculative_config = { [](#__codelineno-2-132) "method": args.method, [](#__codelineno-2-133) "model": args.draft_model, [](#__codelineno-2-134) "num_speculative_tokens": args.num_spec_tokens, [](#__codelineno-2-135) "enforce_eager": args.enforce_eager, [](#__codelineno-2-136) "max_model_len": args.max_model_len, [](#__codelineno-2-137) "parallel_drafting": args.parallel_drafting, [](#__codelineno-2-138) } [](#__codelineno-2-139) elif args.method == "mtp": [](#__codelineno-2-140) speculative_config = { [](#__codelineno-2-141) "method": "mtp", [](#__codelineno-2-142) "num_speculative_tokens": args.num_spec_tokens, [](#__codelineno-2-143) } [](#__codelineno-2-144) else: [](#__codelineno-2-145) raise ValueError(f"unknown method: {args.method}") [](#__codelineno-2-146) [](#__codelineno-2-147) llm = LLM( [](#__codelineno-2-148) model=model_dir, [](#__codelineno-2-149) trust_remote_code=True, [](#__codelineno-2-150) tensor_parallel_size=args.tp, [](#__codelineno-2-151) enable_chunked_prefill=args.enable_chunked_prefill, [](#__codelineno-2-152) enforce_eager=args.enforce_eager, [](#__codelineno-2-153) gpu_memory_utilization=args.gpu_memory_utilization, [](#__codelineno-2-154) speculative_config=speculative_config, [](#__codelineno-2-155) disable_log_stats=False, [](#__codelineno-2-156) max_model_len=args.max_model_len, [](#__codelineno-2-157) limit_mm_per_prompt={"image": 5}, [](#__codelineno-2-158) disable_chunked_mm_input=True, [](#__codelineno-2-159) max_num_seqs=args.max_num_seqs, [](#__codelineno-2-160) allowed_local_media_path=args.allowed_local_media_path, [](#__codelineno-2-161) ) [](#__codelineno-2-162) [](#__codelineno-2-163) sampling_params = SamplingParams(temperature=args.temp, max_tokens=args.output_len) [](#__codelineno-2-164) if args.backend == "openai-chat": [](#__codelineno-2-165) outputs = llm.chat(llm_prompts, sampling_params=sampling_params) [](#__codelineno-2-166) else: [](#__codelineno-2-167) outputs = llm.generate( [](#__codelineno-2-168) llm_prompts, [](#__codelineno-2-169) sampling_params=sampling_params, [](#__codelineno-2-170) ) [](#__codelineno-2-171) [](#__codelineno-2-172) # print the generated text [](#__codelineno-2-173) if args.print_output: [](#__codelineno-2-174) for i, output in enumerate(outputs): [](#__codelineno-2-175) print("-" * 50) [](#__codelineno-2-176) if not args.custom_mm_prompts: [](#__codelineno-2-177) print(f"prompt: {prompts[i].prompt}") [](#__codelineno-2-178) else: [](#__codelineno-2-179) print(f"prompt: {prompts[i]}") [](#__codelineno-2-180) print(f"generated text: {output.outputs[0].text}") [](#__codelineno-2-181) print("-" * 50) [](#__codelineno-2-182) [](#__codelineno-2-183) metrics = llm.get_metrics() [](#__codelineno-2-184) [](#__codelineno-2-185) total_num_output_tokens = sum( [](#__codelineno-2-186) len(output.outputs[0].token_ids) for output in outputs [](#__codelineno-2-187) ) [](#__codelineno-2-188) num_drafts = 0 [](#__codelineno-2-189) num_draft_tokens = 0 [](#__codelineno-2-190) num_accepted_tokens = 0 [](#__codelineno-2-191) acceptance_counts = [0] * args.num_spec_tokens [](#__codelineno-2-192) for metric in metrics: [](#__codelineno-2-193) if metric.name == "vllm:spec_decode_num_drafts": [](#__codelineno-2-194) assert isinstance(metric, Counter) [](#__codelineno-2-195) num_drafts += metric.value [](#__codelineno-2-196) elif metric.name == "vllm:spec_decode_num_draft_tokens": [](#__codelineno-2-197) assert isinstance(metric, Counter) [](#__codelineno-2-198) num_draft_tokens += metric.value [](#__codelineno-2-199) elif metric.name == "vllm:spec_decode_num_accepted_tokens": [](#__codelineno-2-200) assert isinstance(metric, Counter) [](#__codelineno-2-201) num_accepted_tokens += metric.value [](#__codelineno-2-202) elif metric.name == "vllm:spec_decode_num_accepted_tokens_per_pos": [](#__codelineno-2-203) assert isinstance(metric, Vector) [](#__codelineno-2-204) for pos in range(len(metric.values)): [](#__codelineno-2-205) acceptance_counts[pos] += metric.values[pos] [](#__codelineno-2-206) [](#__codelineno-2-207) print("-" * 50) [](#__codelineno-2-208) print(f"total_num_output_tokens: {total_num_output_tokens}") [](#__codelineno-2-209) print(f"num_drafts: {num_drafts}") [](#__codelineno-2-210) print(f"num_draft_tokens: {num_draft_tokens}") [](#__codelineno-2-211) print(f"num_accepted_tokens: {num_accepted_tokens}") [](#__codelineno-2-212) acceptance_length = 1 + (num_accepted_tokens / num_drafts) if num_drafts > 0 else 1 [](#__codelineno-2-213) print(f"mean acceptance length: {acceptance_length:.2f}") [](#__codelineno-2-214) print("-" * 50) [](#__codelineno-2-215) [](#__codelineno-2-216) # print acceptance at each token position [](#__codelineno-2-217) for i in range(len(acceptance_counts)): [](#__codelineno-2-218) acceptance_rate = acceptance_counts[i] / num_drafts if num_drafts > 0 else 0 [](#__codelineno-2-219) print(f"acceptance at token {i}: {acceptance_rate:.2f}") [](#__codelineno-2-220) [](#__codelineno-2-221) return acceptance_length [](#__codelineno-2-222) [](#__codelineno-2-223)[](#__codelineno-2-224)if __name__ == "__main__": [](#__codelineno-2-225) args = parse_args() [](#__codelineno-2-226) args.enable_multimodal_chat = args.backend == "openai-chat" [](#__codelineno-2-227) [](#__codelineno-2-228) acceptance_length = main(args) [](#__codelineno-2-229) [](#__codelineno-2-230) if args.test: [](#__codelineno-2-231) # takes ~30s to run on 1xH100 [](#__codelineno-2-232) assert args.method in ["eagle", "eagle3"] [](#__codelineno-2-233) assert args.tp == 1 [](#__codelineno-2-234) assert args.num_spec_tokens == 3 [](#__codelineno-2-235) assert args.dataset_name == "hf" [](#__codelineno-2-236) assert args.dataset_path == "philschmid/mt-bench" [](#__codelineno-2-237) assert args.num_prompts == 80 [](#__codelineno-2-238) assert args.temp == 0 [](#__codelineno-2-239) assert args.top_p == 1.0 [](#__codelineno-2-240) assert args.top_k == -1 [](#__codelineno-2-241) assert args.enable_chunked_prefill [](#__codelineno-2-242) [](#__codelineno-2-243) # check acceptance length is within 2% of expected value [](#__codelineno-2-244) rtol = 0.02 [](#__codelineno-2-245) expected_acceptance_length = 2.296 if args.method == "eagle" else 2.811 [](#__codelineno-2-246) [](#__codelineno-2-247) assert ( [](#__codelineno-2-248) acceptance_length <= (1 + rtol) * expected_acceptance_length [](#__codelineno-2-249) and acceptance_length >= (1 - rtol) * expected_acceptance_length [](#__codelineno-2-250) ), ( [](#__codelineno-2-251) f"acceptance_length {acceptance_length} is not " [](#__codelineno-2-252) f"within {rtol * 100}% of {expected_acceptance_length}" [](#__codelineno-2-253) ) [](#__codelineno-2-254) [](#__codelineno-2-255) print( [](#__codelineno-2-256) f"Test passed! Expected AL: " [](#__codelineno-2-257) f"{expected_acceptance_length}, got {acceptance_length}" [](#__codelineno-2-258) )` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/structured_outputs.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/structured\_outputs](https://github.com/vllm-project/vllm/tree/main/examples/features/structured_outputs). This script demonstrates various structured output capabilities of vLLM's OpenAI-compatible server. It can run individual constraint type or all of them. It supports both streaming responses and concurrent non-streaming requests. To use this example, you must start an vLLM server with any model of your choice. `[](#__codelineno-0-1)vllm serve Qwen/Qwen2.5-3B-Instruct` To serve a reasoning model, you can use the following command: `[](#__codelineno-1-1)vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B \ [](#__codelineno-1-2) --reasoning-parser deepseek_r1` If you want to run this script standalone with `uv`, you can use the following: `[](#__codelineno-2-1)uvx --from git+https://github.com/vllm-project/vllm#subdirectory=examples/features/structured_outputs \ [](#__codelineno-2-2) structured-outputs` See [feature docs](https://docs.vllm.ai/en/latest/features/structured_outputs.html) for more information. Tip If vLLM is running remotely, then set `OPENAI_BASE_URL=` before running the script. ## Usage[¶](#usage "Permanent link") Run all constraints, non-streaming: `[](#__codelineno-3-1)uv run structured_outputs_offline.py` Run all constraints, streaming: `[](#__codelineno-4-1)uv run structured_outputs_offline.py --stream` Run certain constraints, for example `structural_tag` and `regex`, streaming: `[](#__codelineno-5-1)uv run structured_outputs_offline.py \ [](#__codelineno-5-2) --constraint structural_tag regex \ [](#__codelineno-5-3) --stream` Run all constraints, with reasoning models and streaming: `[](#__codelineno-6-1)uv run structured_outputs_offline.py --reasoning --stream` ## Example materials[¶](#example-materials "Permanent link") pyproject.toml `[](#__codelineno-7-1)[project] [](#__codelineno-7-2)name = "examples-online-structured-outputs" [](#__codelineno-7-3)requires-python = ">=3.10, <3.14" [](#__codelineno-7-4)dependencies = ["openai==1.78.1", "pydantic==2.11.4"] [](#__codelineno-7-5)version = "0.0.0" [](#__codelineno-7-6)[](#__codelineno-7-7)[project.scripts] [](#__codelineno-7-8)structured-outputs = "structured_outputs:main"` structured\_outputs\_client.py ``[](#__codelineno-8-1)# ruff: noqa: E501 [](#__codelineno-8-2)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-8-3)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-8-4)import argparse [](#__codelineno-8-5)import asyncio [](#__codelineno-8-6)import enum [](#__codelineno-8-7)import os [](#__codelineno-8-8)from typing import Any, Literal [](#__codelineno-8-9)[](#__codelineno-8-10)import openai [](#__codelineno-8-11)import pydantic [](#__codelineno-8-12)from openai.types.chat import ChatCompletionChunk [](#__codelineno-8-13)[](#__codelineno-8-14)ConstraintsFormat = Literal[ [](#__codelineno-8-15) "choice", [](#__codelineno-8-16) "regex", [](#__codelineno-8-17) "json", [](#__codelineno-8-18) "grammar", [](#__codelineno-8-19) "structural_tag", [](#__codelineno-8-20)] [](#__codelineno-8-21) [](#__codelineno-8-22)[](#__codelineno-8-23)async def print_stream_response( [](#__codelineno-8-24) stream_response: openai.AsyncStream[ChatCompletionChunk], [](#__codelineno-8-25) title: str, [](#__codelineno-8-26) args: argparse.Namespace, [](#__codelineno-8-27)): [](#__codelineno-8-28) print(f"\n\n{title} (Streaming):") [](#__codelineno-8-29) [](#__codelineno-8-30) local_reasoning_header_printed = False [](#__codelineno-8-31) local_content_header_printed = False [](#__codelineno-8-32) [](#__codelineno-8-33) async for chunk in stream_response: [](#__codelineno-8-34) delta = chunk.choices[0].delta [](#__codelineno-8-35) [](#__codelineno-8-36) reasoning_chunk_text: str | None = getattr(delta, "reasoning", None) [](#__codelineno-8-37) content_chunk_text = delta.content [](#__codelineno-8-38) [](#__codelineno-8-39) if args.reasoning: [](#__codelineno-8-40) if reasoning_chunk_text: [](#__codelineno-8-41) if not local_reasoning_header_printed: [](#__codelineno-8-42) print(" Reasoning: ", end="") [](#__codelineno-8-43) local_reasoning_header_printed = True [](#__codelineno-8-44) print(reasoning_chunk_text, end="", flush=True) [](#__codelineno-8-45) [](#__codelineno-8-46) if content_chunk_text: [](#__codelineno-8-47) if not local_content_header_printed: [](#__codelineno-8-48) if local_reasoning_header_printed: [](#__codelineno-8-49) print() [](#__codelineno-8-50) print(" Content: ", end="") [](#__codelineno-8-51) local_content_header_printed = True [](#__codelineno-8-52) print(content_chunk_text, end="", flush=True) [](#__codelineno-8-53) else: [](#__codelineno-8-54) if content_chunk_text: [](#__codelineno-8-55) if not local_content_header_printed: [](#__codelineno-8-56) print(" Content: ", end="") [](#__codelineno-8-57) local_content_header_printed = True [](#__codelineno-8-58) print(content_chunk_text, end="", flush=True) [](#__codelineno-8-59) print() [](#__codelineno-8-60) [](#__codelineno-8-61)[](#__codelineno-8-62)class CarType(str, enum.Enum): [](#__codelineno-8-63) SEDAN = "SEDAN" [](#__codelineno-8-64) SUV = "SUV" [](#__codelineno-8-65) TRUCK = "TRUCK" [](#__codelineno-8-66) COUPE = "COUPE" [](#__codelineno-8-67) [](#__codelineno-8-68)[](#__codelineno-8-69)class CarDescription(pydantic.BaseModel): [](#__codelineno-8-70) brand: str [](#__codelineno-8-71) model: str [](#__codelineno-8-72) car_type: CarType [](#__codelineno-8-73) [](#__codelineno-8-74)[](#__codelineno-8-75)PARAMS: dict[ConstraintsFormat, dict[str, Any]] = { [](#__codelineno-8-76) "choice": { [](#__codelineno-8-77) "messages": [ [](#__codelineno-8-78) { [](#__codelineno-8-79) "role": "user", [](#__codelineno-8-80) "content": "Classify this sentiment: vLLM is wonderful!", [](#__codelineno-8-81) } [](#__codelineno-8-82) ], [](#__codelineno-8-83) "extra_body": {"structured_outputs": {"choice": ["positive", "negative"]}}, [](#__codelineno-8-84) }, [](#__codelineno-8-85) "regex": { [](#__codelineno-8-86) "messages": [ [](#__codelineno-8-87) { [](#__codelineno-8-88) "role": "user", [](#__codelineno-8-89) "content": "Generate an email address for Alan Turing, who works in Enigma. End in .com and new line. Example result: '[[email protected]](https://docs.vllm.ai/cdn-cgi/l/email-protection)\n'", [](#__codelineno-8-90) } [](#__codelineno-8-91) ], [](#__codelineno-8-92) "extra_body": { [](#__codelineno-8-93) "structured_outputs": {"regex": r"[a-z0-9.]{1,20}@\w{6,10}\.com\n"}, [](#__codelineno-8-94) }, [](#__codelineno-8-95) }, [](#__codelineno-8-96) "json": { [](#__codelineno-8-97) "messages": [ [](#__codelineno-8-98) { [](#__codelineno-8-99) "role": "user", [](#__codelineno-8-100) "content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's", [](#__codelineno-8-101) } [](#__codelineno-8-102) ], [](#__codelineno-8-103) "response_format": { [](#__codelineno-8-104) "type": "json_schema", [](#__codelineno-8-105) "json_schema": { [](#__codelineno-8-106) "name": "car-description", [](#__codelineno-8-107) "schema": CarDescription.model_json_schema(), [](#__codelineno-8-108) }, [](#__codelineno-8-109) }, [](#__codelineno-8-110) }, [](#__codelineno-8-111) "grammar": { [](#__codelineno-8-112) "messages": [ [](#__codelineno-8-113) { [](#__codelineno-8-114) "role": "user", [](#__codelineno-8-115) "content": "Generate an SQL query to show the 'username' and 'email' from the 'users' table.", [](#__codelineno-8-116) } [](#__codelineno-8-117) ], [](#__codelineno-8-118) "extra_body": { [](#__codelineno-8-119) "structured_outputs": { [](#__codelineno-8-120) "grammar": """ [](#__codelineno-8-121)root ::= select_statement [](#__codelineno-8-122)[](#__codelineno-8-123)select_statement ::= "SELECT " column " from " table " where " condition [](#__codelineno-8-124)[](#__codelineno-8-125)column ::= "col_1 " | "col_2 " [](#__codelineno-8-126)[](#__codelineno-8-127)table ::= "table_1 " | "table_2 " [](#__codelineno-8-128)[](#__codelineno-8-129)condition ::= column "= " number [](#__codelineno-8-130)[](#__codelineno-8-131)number ::= "1 " | "2 " [](#__codelineno-8-132)""", [](#__codelineno-8-133) } [](#__codelineno-8-134) }, [](#__codelineno-8-135) }, [](#__codelineno-8-136) "structural_tag": { [](#__codelineno-8-137) "messages": [ [](#__codelineno-8-138) { [](#__codelineno-8-139) "role": "user", [](#__codelineno-8-140) "content": """ [](#__codelineno-8-141)You have access to the following function to retrieve the weather in a city: [](#__codelineno-8-142)[](#__codelineno-8-143){ [](#__codelineno-8-144) "name": "get_weather", [](#__codelineno-8-145) "parameters": { [](#__codelineno-8-146) "city": { [](#__codelineno-8-147) "param_type": "string", [](#__codelineno-8-148) "description": "The city to get the weather for", [](#__codelineno-8-149) "required": True [](#__codelineno-8-150) } [](#__codelineno-8-151) } [](#__codelineno-8-152)} [](#__codelineno-8-153)[](#__codelineno-8-154)If a you choose to call a function ONLY reply in the following format: [](#__codelineno-8-155)<{start_tag}={function_name}>{parameters}{end_tag} [](#__codelineno-8-156)where [](#__codelineno-8-157)[](#__codelineno-8-158)start_tag => ` a JSON dict with the function argument name as key and function [](#__codelineno-8-160) argument value as value. [](#__codelineno-8-161)end_tag => `` [](#__codelineno-8-162)[](#__codelineno-8-163)Here is an example, [](#__codelineno-8-164){"example_name": "example_value"} [](#__codelineno-8-165)[](#__codelineno-8-166)Reminder: [](#__codelineno-8-167)- Function calls MUST follow the specified format [](#__codelineno-8-168)- Required parameters MUST be specified [](#__codelineno-8-169)- Only call one function at a time [](#__codelineno-8-170)- Put the entire function call reply on one line [](#__codelineno-8-171)- Always add your sources when using search results to answer the user query [](#__codelineno-8-172)[](#__codelineno-8-173)You are a helpful assistant. [](#__codelineno-8-174)[](#__codelineno-8-175)Given the previous instructions, what is the weather in New York City, Boston, [](#__codelineno-8-176)and San Francisco?""", [](#__codelineno-8-177) }, [](#__codelineno-8-178) ], [](#__codelineno-8-179) "response_format": { [](#__codelineno-8-180) "type": "structural_tag", [](#__codelineno-8-181) "structures": [ [](#__codelineno-8-182) { [](#__codelineno-8-183) "begin": "", [](#__codelineno-8-184) "schema": { [](#__codelineno-8-185) "type": "object", [](#__codelineno-8-186) "properties": {"city": {"type": "string"}}, [](#__codelineno-8-187) "required": ["city"], [](#__codelineno-8-188) }, [](#__codelineno-8-189) "end": "", [](#__codelineno-8-190) } [](#__codelineno-8-191) ], [](#__codelineno-8-192) "triggers": [" 1: [](#__codelineno-0-358) model_path = f"{base_path}/model-rank-%03d.tensors" [](#__codelineno-0-359) else: [](#__codelineno-0-360) model_path = f"{base_path}/model.tensors" [](#__codelineno-0-361) [](#__codelineno-0-362) tensorizer_config = TensorizerConfig( [](#__codelineno-0-363) tensorizer_uri=model_path, [](#__codelineno-0-364) encryption_keyfile=keyfile, [](#__codelineno-0-365) serialization_kwargs=args.serialization_kwargs or {}, [](#__codelineno-0-366) **credentials, [](#__codelineno-0-367) ) [](#__codelineno-0-368) [](#__codelineno-0-369) if args.lora_path: [](#__codelineno-0-370) tensorizer_config.lora_dir = tensorizer_config.tensorizer_dir [](#__codelineno-0-371) tensorize_lora_adapter(args.lora_path, tensorizer_config) [](#__codelineno-0-372) [](#__codelineno-0-373) merge_extra_config_with_tensorizer_config(extra_config, tensorizer_config) [](#__codelineno-0-374) tensorize_vllm_model(engine_args, tensorizer_config) [](#__codelineno-0-375) [](#__codelineno-0-376) elif args.command == "deserialize": [](#__codelineno-0-377) tensorizer_config = TensorizerConfig( [](#__codelineno-0-378) tensorizer_uri=args.path_to_tensors, [](#__codelineno-0-379) tensorizer_dir=args.serialized_directory, [](#__codelineno-0-380) encryption_keyfile=keyfile, [](#__codelineno-0-381) deserialization_kwargs=args.deserialization_kwargs or {}, [](#__codelineno-0-382) **credentials, [](#__codelineno-0-383) ) [](#__codelineno-0-384) [](#__codelineno-0-385) merge_extra_config_with_tensorizer_config(extra_config, tensorizer_config) [](#__codelineno-0-386) deserialize(args, tensorizer_config) [](#__codelineno-0-387) else: [](#__codelineno-0-388) raise ValueError("Either serialize or deserialize must be specified.") [](#__codelineno-0-389) [](#__codelineno-0-390)[](#__codelineno-0-391)if __name__ == "__main__": [](#__codelineno-0-392) main()`` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/features/torchrun.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/features/torchrun](https://github.com/vllm-project/vllm/tree/main/examples/features/torchrun). ## Torchrun Dp Example Offline[¶](#torchrun-dp-example-offline "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)experimental support for data-parallel inference with torchrun [](#__codelineno-0-5)Note the data load balancing and distribution is done out of the vllm engine, [](#__codelineno-0-6)no internal lb supported in external_launcher mode. [](#__codelineno-0-7)[](#__codelineno-0-8)To run this example: [](#__codelineno-0-9)```bash [](#__codelineno-0-10)$ torchrun --nproc-per-node=2 examples/features/torchrun/torchrun_dp_example_offline.py [](#__codelineno-0-11)``` [](#__codelineno-0-12)[](#__codelineno-0-13)With custom parallelism settings: [](#__codelineno-0-14)```bash [](#__codelineno-0-15)$ torchrun --nproc-per-node=8 examples/features/torchrun/torchrun_dp_example_offline.py \ [](#__codelineno-0-16) --tp-size=2 --pp-size=1 --dp-size=4 --enable-ep [](#__codelineno-0-17)``` [](#__codelineno-0-18)""" # noqa: E501 [](#__codelineno-0-19)[](#__codelineno-0-20)import argparse [](#__codelineno-0-21)[](#__codelineno-0-22)from vllm import LLM, SamplingParams [](#__codelineno-0-23) [](#__codelineno-0-24)[](#__codelineno-0-25)def parse_args(): [](#__codelineno-0-26) parser = argparse.ArgumentParser( [](#__codelineno-0-27) description="Data-parallel inference with torchrun" [](#__codelineno-0-28) ) [](#__codelineno-0-29) parser.add_argument( [](#__codelineno-0-30) "--tp-size", [](#__codelineno-0-31) type=int, [](#__codelineno-0-32) default=1, [](#__codelineno-0-33) help="Tensor parallel size (default: 1)", [](#__codelineno-0-34) ) [](#__codelineno-0-35) parser.add_argument( [](#__codelineno-0-36) "--pp-size", [](#__codelineno-0-37) type=int, [](#__codelineno-0-38) default=1, [](#__codelineno-0-39) help="Pipeline parallel size (default: 1)", [](#__codelineno-0-40) ) [](#__codelineno-0-41) parser.add_argument( [](#__codelineno-0-42) "--dp-size", [](#__codelineno-0-43) type=int, [](#__codelineno-0-44) default=2, [](#__codelineno-0-45) help="Data parallel size (default: 2)", [](#__codelineno-0-46) ) [](#__codelineno-0-47) parser.add_argument( [](#__codelineno-0-48) "--enable-ep", [](#__codelineno-0-49) action="store_true", [](#__codelineno-0-50) help="Enable expert parallel (default: False)", [](#__codelineno-0-51) ) [](#__codelineno-0-52) parser.add_argument( [](#__codelineno-0-53) "--model", [](#__codelineno-0-54) type=str, [](#__codelineno-0-55) default="microsoft/Phi-mini-MoE-instruct", [](#__codelineno-0-56) help="Model name or path (default: microsoft/Phi-mini-MoE-instruct)", [](#__codelineno-0-57) ) [](#__codelineno-0-58) parser.add_argument( [](#__codelineno-0-59) "--max-model-len", [](#__codelineno-0-60) type=int, [](#__codelineno-0-61) default=4096, [](#__codelineno-0-62) help="Maximum model length (default: 4096)", [](#__codelineno-0-63) ) [](#__codelineno-0-64) parser.add_argument( [](#__codelineno-0-65) "--gpu-memory-utilization", [](#__codelineno-0-66) type=float, [](#__codelineno-0-67) default=0.6, [](#__codelineno-0-68) help="GPU memory utilization (default: 0.6)", [](#__codelineno-0-69) ) [](#__codelineno-0-70) parser.add_argument( [](#__codelineno-0-71) "--seed", [](#__codelineno-0-72) type=int, [](#__codelineno-0-73) default=1, [](#__codelineno-0-74) help="Random seed (default: 1)", [](#__codelineno-0-75) ) [](#__codelineno-0-76) return parser.parse_args() [](#__codelineno-0-77) [](#__codelineno-0-78)[](#__codelineno-0-79)args = parse_args() [](#__codelineno-0-80) [](#__codelineno-0-81)[](#__codelineno-0-82)# Create prompts, the same across all ranks [](#__codelineno-0-83)prompts = [ [](#__codelineno-0-84) "Hello, my name is", [](#__codelineno-0-85) "The president of the United States is", [](#__codelineno-0-86) "The capital of France is", [](#__codelineno-0-87) "The future of AI is", [](#__codelineno-0-88)] [](#__codelineno-0-89)[](#__codelineno-0-90)# Create sampling parameters, the same across all ranks [](#__codelineno-0-91)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-0-92)[](#__codelineno-0-93)# Use `distributed_executor_backend="external_launcher"` so that [](#__codelineno-0-94)# this llm engine/instance only creates one worker. [](#__codelineno-0-95)# it is important to set an explicit seed to make sure that [](#__codelineno-0-96)# all ranks have the same random seed, so that sampling can be [](#__codelineno-0-97)# deterministic across ranks. [](#__codelineno-0-98)llm = LLM( [](#__codelineno-0-99) model=args.model, [](#__codelineno-0-100) tensor_parallel_size=args.tp_size, [](#__codelineno-0-101) data_parallel_size=args.dp_size, [](#__codelineno-0-102) pipeline_parallel_size=args.pp_size, [](#__codelineno-0-103) enable_expert_parallel=args.enable_ep, [](#__codelineno-0-104) distributed_executor_backend="external_launcher", [](#__codelineno-0-105) max_model_len=args.max_model_len, [](#__codelineno-0-106) gpu_memory_utilization=args.gpu_memory_utilization, [](#__codelineno-0-107) seed=args.seed, [](#__codelineno-0-108)) [](#__codelineno-0-109)[](#__codelineno-0-110)dp_rank = llm.llm_engine.vllm_config.parallel_config.data_parallel_rank [](#__codelineno-0-111)dp_size = llm.llm_engine.vllm_config.parallel_config.data_parallel_size [](#__codelineno-0-112)[](#__codelineno-0-113)prompts = [ [](#__codelineno-0-114) f"{idx}.{prompt}" for idx, prompt in enumerate(prompts) if idx % dp_size == dp_rank [](#__codelineno-0-115)] [](#__codelineno-0-116)[](#__codelineno-0-117)outputs = llm.generate(prompts, sampling_params) [](#__codelineno-0-118)[](#__codelineno-0-119)for output in outputs: [](#__codelineno-0-120) prompt = output.prompt [](#__codelineno-0-121) generated_text = output.outputs[0].text [](#__codelineno-0-122) print( [](#__codelineno-0-123) f"DP Rank: {dp_rank} Prompt: {prompt!r}\nGenerated text: {generated_text!r}\n" [](#__codelineno-0-124) ) [](#__codelineno-0-125)[](#__codelineno-0-126)""" [](#__codelineno-0-127)Further tips: [](#__codelineno-0-128)[](#__codelineno-0-129)1. to communicate control messages across all ranks, use the cpu group, [](#__codelineno-0-130)a PyTorch ProcessGroup with GLOO backend. [](#__codelineno-0-131)[](#__codelineno-0-132)```python [](#__codelineno-0-133)from vllm.distributed.parallel_state import get_world_group [](#__codelineno-0-134)cpu_group = get_world_group().cpu_group [](#__codelineno-0-135)torch_rank = dist.get_rank(group=cpu_group) [](#__codelineno-0-136)if torch_rank == 0: [](#__codelineno-0-137) # do something for rank 0, e.g. saving the results to disk. [](#__codelineno-0-138)``` [](#__codelineno-0-139)[](#__codelineno-0-140)2. to communicate data across all ranks, use the model's device group, [](#__codelineno-0-141)a PyTorch ProcessGroup with NCCL backend. [](#__codelineno-0-142)```python [](#__codelineno-0-143)from vllm.distributed.parallel_state import get_world_group [](#__codelineno-0-144)device_group = get_world_group().device_group [](#__codelineno-0-145)``` [](#__codelineno-0-146)[](#__codelineno-0-147)3. to access the model directly in every rank, use the following code: [](#__codelineno-0-148)```python [](#__codelineno-0-149)llm.llm_engine.model_executor.driver_worker.worker.model_runner.model [](#__codelineno-0-150)``` [](#__codelineno-0-151)"""`` ## Torchrun Example Offline[¶](#torchrun-example-offline "Permanent link") ``[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)experimental support for tensor-parallel inference with torchrun, [](#__codelineno-1-5)see https://github.com/vllm-project/vllm/issues/11400 for [](#__codelineno-1-6)the motivation and use case for this example. [](#__codelineno-1-7)run the script with `torchrun --nproc-per-node=4 torchrun_example_offline.py`, [](#__codelineno-1-8)the argument `4` should match the product of `tensor_parallel_size` and [](#__codelineno-1-9)`pipeline_parallel_size` below. see `tests/distributed/test_torchrun_example.py` [](#__codelineno-1-10)for the unit test. [](#__codelineno-1-11)""" [](#__codelineno-1-12)[](#__codelineno-1-13)import torch.distributed as dist [](#__codelineno-1-14)[](#__codelineno-1-15)from vllm import LLM, SamplingParams [](#__codelineno-1-16)[](#__codelineno-1-17)# Create prompts, the same across all ranks [](#__codelineno-1-18)prompts = [ [](#__codelineno-1-19) "Hello, my name is", [](#__codelineno-1-20) "The president of the United States is", [](#__codelineno-1-21) "The capital of France is", [](#__codelineno-1-22) "The future of AI is", [](#__codelineno-1-23)] [](#__codelineno-1-24)[](#__codelineno-1-25)# Create sampling parameters, the same across all ranks [](#__codelineno-1-26)sampling_params = SamplingParams(temperature=0.8, top_p=0.95) [](#__codelineno-1-27)[](#__codelineno-1-28)# Use `distributed_executor_backend="external_launcher"` so that [](#__codelineno-1-29)# this llm engine/instance only creates one worker. [](#__codelineno-1-30)# it is important to set an explicit seed to make sure that [](#__codelineno-1-31)# all ranks have the same random seed, so that sampling can be [](#__codelineno-1-32)# deterministic across ranks. [](#__codelineno-1-33)llm = LLM( [](#__codelineno-1-34) model="meta-llama/Llama-3.1-8B", [](#__codelineno-1-35) tensor_parallel_size=2, [](#__codelineno-1-36) pipeline_parallel_size=2, [](#__codelineno-1-37) distributed_executor_backend="external_launcher", [](#__codelineno-1-38) max_model_len=32768, [](#__codelineno-1-39) seed=1, [](#__codelineno-1-40)) [](#__codelineno-1-41)[](#__codelineno-1-42)outputs = llm.generate(prompts, sampling_params) [](#__codelineno-1-43)[](#__codelineno-1-44)# all ranks will have the same outputs [](#__codelineno-1-45)if dist.get_rank() == 0: [](#__codelineno-1-46) print("-" * 50) [](#__codelineno-1-47) for output in outputs: [](#__codelineno-1-48) prompt = output.prompt [](#__codelineno-1-49) generated_text = output.outputs[0].text [](#__codelineno-1-50) print(f"Prompt: {prompt!r}\nGenerated text: {generated_text!r}\n") [](#__codelineno-1-51) print("-" * 50) [](#__codelineno-1-52) """ [](#__codelineno-1-53)Further tips: [](#__codelineno-1-54)[](#__codelineno-1-55)1. to communicate control messages across all ranks, use the cpu group, [](#__codelineno-1-56)a PyTorch ProcessGroup with GLOO backend. [](#__codelineno-1-57)[](#__codelineno-1-58)```python [](#__codelineno-1-59)from vllm.distributed.parallel_state import get_world_group [](#__codelineno-1-60)cpu_group = get_world_group().cpu_group [](#__codelineno-1-61)torch_rank = dist.get_rank(group=cpu_group) [](#__codelineno-1-62)if torch_rank == 0: [](#__codelineno-1-63) # do something for rank 0, e.g. saving the results to disk. [](#__codelineno-1-64)``` [](#__codelineno-1-65)[](#__codelineno-1-66)2. to communicate data across all ranks, use the model's device group, [](#__codelineno-1-67)a PyTorch ProcessGroup with NCCL backend. [](#__codelineno-1-68)```python [](#__codelineno-1-69)from vllm.distributed.parallel_state import get_world_group [](#__codelineno-1-70)device_group = get_world_group().device_group [](#__codelineno-1-71)``` [](#__codelineno-1-72)[](#__codelineno-1-73)3. to access the model directly in every rank, use the following code: [](#__codelineno-1-74)```python [](#__codelineno-1-75)llm.llm_engine.model_executor.driver_worker.worker.model_runner.model [](#__codelineno-1-76)``` [](#__codelineno-1-77)"""`` --- # page `[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)"""Examples of batched chat completions via the vLLM OpenAI-compatible API. [](#__codelineno-0-4)[](#__codelineno-0-5)The /v1/chat/completions/batch endpoint accepts ``messages`` as a list of [](#__codelineno-0-6)conversations. Each conversation is processed independently and the response [](#__codelineno-0-7)contains one choice per conversation, indexed 0, 1, ..., N-1. [](#__codelineno-0-8)[](#__codelineno-0-9)Start a server first, e.g.: [](#__codelineno-0-10) vllm serve Qwen/Qwen2.5-1.5B-Instruct --port 8000 [](#__codelineno-0-11)[](#__codelineno-0-12)Current limitations compared to /v1/chat/completions: [](#__codelineno-0-13) - Streaming is not supported. [](#__codelineno-0-14) - Tool use is not supported. [](#__codelineno-0-15) - Beam search is not supported. [](#__codelineno-0-16)""" [](#__codelineno-0-17)[](#__codelineno-0-18)import json [](#__codelineno-0-19)import os [](#__codelineno-0-20)[](#__codelineno-0-21)import httpx [](#__codelineno-0-22)[](#__codelineno-0-23)BASE_URL = os.environ.get("VLLM_BASE_URL", "http://localhost:8000") [](#__codelineno-0-24)MODEL = os.environ.get("VLLM_MODEL", "Qwen/Qwen2.5-1.5B-Instruct") [](#__codelineno-0-25)BATCH_URL = f"{BASE_URL}/v1/chat/completions/batch" [](#__codelineno-0-26) [](#__codelineno-0-27)[](#__codelineno-0-28)def post_batch(payload: dict) -> dict: [](#__codelineno-0-29) response = httpx.post(BATCH_URL, json=payload, timeout=60) [](#__codelineno-0-30) response.raise_for_status() [](#__codelineno-0-31) return response.json() [](#__codelineno-0-32) [](#__codelineno-0-33)[](#__codelineno-0-34)def main() -> None: [](#__codelineno-0-35) print("=== Example 1a: single conversation (standard endpoint) ===") [](#__codelineno-0-36) response = httpx.post( [](#__codelineno-0-37) f"{BASE_URL}/v1/chat/completions", [](#__codelineno-0-38) json={ [](#__codelineno-0-39) "model": MODEL, [](#__codelineno-0-40) "messages": [{"role": "user", "content": "What is the capital of Japan?"}], [](#__codelineno-0-41) }, [](#__codelineno-0-42) timeout=60, [](#__codelineno-0-43) ) [](#__codelineno-0-44) response.raise_for_status() [](#__codelineno-0-45) data = response.json() [](#__codelineno-0-46) for choice in data["choices"]: [](#__codelineno-0-47) print(f" [{choice['index']}] {choice['message']['content']}") [](#__codelineno-0-48) [](#__codelineno-0-49) print("\n=== Example 1b: batched plain text (2 conversations) ===") [](#__codelineno-0-50) data = post_batch( [](#__codelineno-0-51) { [](#__codelineno-0-52) "model": MODEL, [](#__codelineno-0-53) "messages": [ [](#__codelineno-0-54) [{"role": "user", "content": "What is the capital of France?"}], [](#__codelineno-0-55) [{"role": "user", "content": "What is the capital of Japan?"}], [](#__codelineno-0-56) ], [](#__codelineno-0-57) } [](#__codelineno-0-58) ) [](#__codelineno-0-59) for choice in data["choices"]: [](#__codelineno-0-60) print(f" [{choice['index']}] {choice['message']['content']}") [](#__codelineno-0-61) [](#__codelineno-0-62) print("\n=== Example 2: batch with regex constraint (yes|no) ===") [](#__codelineno-0-63) data = post_batch( [](#__codelineno-0-64) { [](#__codelineno-0-65) "model": MODEL, [](#__codelineno-0-66) "messages": [ [](#__codelineno-0-67) [{"role": "user", "content": "Is the sky blue? Answer yes or no."}], [](#__codelineno-0-68) [{"role": "user", "content": "Is fire cold? Answer yes or no."}], [](#__codelineno-0-69) ], [](#__codelineno-0-70) "structured_outputs": {"regex": "(yes|no)"}, [](#__codelineno-0-71) } [](#__codelineno-0-72) ) [](#__codelineno-0-73) for choice in data["choices"]: [](#__codelineno-0-74) print(f" [{choice['index']}] {choice['message']['content']}") [](#__codelineno-0-75) [](#__codelineno-0-76) print("\n=== Example 3: batch with json_schema ===") [](#__codelineno-0-77) person_schema = { [](#__codelineno-0-78) "type": "object", [](#__codelineno-0-79) "properties": { [](#__codelineno-0-80) "name": {"type": "string", "description": "Full name of the person"}, [](#__codelineno-0-81) "age": {"type": "integer", "description": "Age in years"}, [](#__codelineno-0-82) }, [](#__codelineno-0-83) "required": ["name", "age"], [](#__codelineno-0-84) } [](#__codelineno-0-85) data = post_batch( [](#__codelineno-0-86) { [](#__codelineno-0-87) "model": MODEL, [](#__codelineno-0-88) "messages": [ [](#__codelineno-0-89) [ [](#__codelineno-0-90) { [](#__codelineno-0-91) "role": "user", [](#__codelineno-0-92) "content": "Describe the person: name Alice, age 30.", [](#__codelineno-0-93) } [](#__codelineno-0-94) ], [](#__codelineno-0-95) [{"role": "user", "content": "Describe the person: name Bob, age 25."}], [](#__codelineno-0-96) ], [](#__codelineno-0-97) "response_format": { [](#__codelineno-0-98) "type": "json_schema", [](#__codelineno-0-99) "json_schema": { [](#__codelineno-0-100) "name": "person", [](#__codelineno-0-101) "strict": True, [](#__codelineno-0-102) "schema": person_schema, [](#__codelineno-0-103) }, [](#__codelineno-0-104) }, [](#__codelineno-0-105) } [](#__codelineno-0-106) ) [](#__codelineno-0-107) for choice in data["choices"]: [](#__codelineno-0-108) person = json.loads(choice["message"]["content"]) [](#__codelineno-0-109) print(f" [{choice['index']}] {person}") [](#__codelineno-0-110) [](#__codelineno-0-111) print("\n=== Example 4: batch book summaries ===") [](#__codelineno-0-112) book_schema = { [](#__codelineno-0-113) "type": "object", [](#__codelineno-0-114) "properties": { [](#__codelineno-0-115) "author": { [](#__codelineno-0-116) "type": "string", [](#__codelineno-0-117) "description": "Full name of the author", [](#__codelineno-0-118) }, [](#__codelineno-0-119) "num_pages": { [](#__codelineno-0-120) "type": "integer", [](#__codelineno-0-121) "description": "Number of pages in the book", [](#__codelineno-0-122) }, [](#__codelineno-0-123) "short_summary": { [](#__codelineno-0-124) "type": "string", [](#__codelineno-0-125) "description": "A one-sentence summary of the book", [](#__codelineno-0-126) }, [](#__codelineno-0-127) "long_summary": { [](#__codelineno-0-128) "type": "string", [](#__codelineno-0-129) "description": ( [](#__codelineno-0-130) "A detailed two to three sentence summary covering " [](#__codelineno-0-131) "the main themes and plot" [](#__codelineno-0-132) ), [](#__codelineno-0-133) }, [](#__codelineno-0-134) }, [](#__codelineno-0-135) "required": ["author", "num_pages", "short_summary", "long_summary"], [](#__codelineno-0-136) } [](#__codelineno-0-137) system_msg = { [](#__codelineno-0-138) "role": "system", [](#__codelineno-0-139) "content": ( [](#__codelineno-0-140) "You are a literary analyst. Extract structured information " [](#__codelineno-0-141) "from book descriptions." [](#__codelineno-0-142) ), [](#__codelineno-0-143) } [](#__codelineno-0-144) data = post_batch( [](#__codelineno-0-145) { [](#__codelineno-0-146) "model": MODEL, [](#__codelineno-0-147) "messages": [ [](#__codelineno-0-148) [ [](#__codelineno-0-149) system_msg, [](#__codelineno-0-150) { [](#__codelineno-0-151) "role": "user", [](#__codelineno-0-152) "content": ( [](#__codelineno-0-153) "Extract information from this book: '1984' by George" [](#__codelineno-0-154) " Orwell, published in 1949, 328 pages. A dystopian" [](#__codelineno-0-155) " novel set in a totalitarian society ruled by Big" [](#__codelineno-0-156) " Brother, following Winston Smith as he secretly" [](#__codelineno-0-157) " rebels against the oppressive Party that surveils" [](#__codelineno-0-158) " and controls every aspect of life." [](#__codelineno-0-159) ), [](#__codelineno-0-160) }, [](#__codelineno-0-161) ], [](#__codelineno-0-162) [ [](#__codelineno-0-163) system_msg, [](#__codelineno-0-164) { [](#__codelineno-0-165) "role": "user", [](#__codelineno-0-166) "content": ( [](#__codelineno-0-167) "Extract information from this book: 'The Hitchhiker's" [](#__codelineno-0-168) " Guide to the Galaxy' by Douglas Adams, published in" [](#__codelineno-0-169) " 1979, 193 pages. A comedic science fiction novel" [](#__codelineno-0-170) " following Arthur Dent, an ordinary Englishman who is" [](#__codelineno-0-171) " whisked off Earth moments before it is demolished to" [](#__codelineno-0-172) " make way for a hyperspace bypass, and his subsequent" [](#__codelineno-0-173) " absurd adventures across the universe." [](#__codelineno-0-174) ), [](#__codelineno-0-175) }, [](#__codelineno-0-176) ], [](#__codelineno-0-177) ], [](#__codelineno-0-178) "response_format": { [](#__codelineno-0-179) "type": "json_schema", [](#__codelineno-0-180) "json_schema": { [](#__codelineno-0-181) "name": "book_summary", [](#__codelineno-0-182) "strict": True, [](#__codelineno-0-183) "schema": book_schema, [](#__codelineno-0-184) }, [](#__codelineno-0-185) }, [](#__codelineno-0-186) } [](#__codelineno-0-187) ) [](#__codelineno-0-188) for choice in data["choices"]: [](#__codelineno-0-189) book = json.loads(choice["message"]["content"]) [](#__codelineno-0-190) print(f" [{choice['index']}] {book}") [](#__codelineno-0-191) [](#__codelineno-0-192)[](#__codelineno-0-193)if __name__ == "__main__": [](#__codelineno-0-194) main()` --- # page [](https://github.com/vllm-project/vllm/edit/main/docs/examples/generate/multimodal.md "Edit this page") Source [https://github.com/vllm-project/vllm/tree/main/examples/generate/multimodal](https://github.com/vllm-project/vllm/tree/main/examples/generate/multimodal). ## Audio Language Offline[¶](#audio-language-offline "Permanent link") ``[](#__codelineno-0-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-0-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-0-3)""" [](#__codelineno-0-4)This example shows how to use vLLM for running offline inference [](#__codelineno-0-5)with the correct prompt format on audio language models. [](#__codelineno-0-6)[](#__codelineno-0-7)For most models, the prompt format should follow corresponding examples [](#__codelineno-0-8)on HuggingFace model repository. [](#__codelineno-0-9)""" [](#__codelineno-0-10)[](#__codelineno-0-11)import os [](#__codelineno-0-12)from typing import Any, NamedTuple [](#__codelineno-0-13)[](#__codelineno-0-14)from huggingface_hub import snapshot_download [](#__codelineno-0-15)from transformers import AutoTokenizer [](#__codelineno-0-16)[](#__codelineno-0-17)from vllm import LLM, EngineArgs, SamplingParams [](#__codelineno-0-18)from vllm.assets.audio import AudioAsset [](#__codelineno-0-19)from vllm.lora.request import LoRARequest [](#__codelineno-0-20)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-0-21)[](#__codelineno-0-22)audio_assets = [AudioAsset("mary_had_lamb"), AudioAsset("winning_call")] [](#__codelineno-0-23)question_per_audio_count = { [](#__codelineno-0-24) 0: "What is 1+1?", [](#__codelineno-0-25) 1: "What is recited in the audio?", [](#__codelineno-0-26) 2: "What sport and what nursery rhyme are referenced?", [](#__codelineno-0-27)} [](#__codelineno-0-28) [](#__codelineno-0-29)[](#__codelineno-0-30)class ModelRequestData(NamedTuple): [](#__codelineno-0-31) engine_args: EngineArgs [](#__codelineno-0-32) prompt: str | None = None [](#__codelineno-0-33) prompt_token_ids: dict[str, list[int]] | None = None [](#__codelineno-0-34) multi_modal_data: dict[str, Any] | None = None [](#__codelineno-0-35) stop_token_ids: list[int] | None = None [](#__codelineno-0-36) lora_requests: list[LoRARequest] | None = None [](#__codelineno-0-37) [](#__codelineno-0-38)[](#__codelineno-0-39)# NOTE: The default `max_num_seqs` and `max_model_len` may result in OOM on [](#__codelineno-0-40)# lower-end GPUs. [](#__codelineno-0-41)# Unless specified, these settings have been tested to work on a single L4. [](#__codelineno-0-42) [](#__codelineno-0-43)[](#__codelineno-0-44)# AudioFlamingo3 [](#__codelineno-0-45)def run_audioflamingo3(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-46) model_name = "nvidia/audio-flamingo-3-hf" [](#__codelineno-0-47) engine_args = EngineArgs( [](#__codelineno-0-48) model=model_name, [](#__codelineno-0-49) max_model_len=4096, [](#__codelineno-0-50) max_num_seqs=2, [](#__codelineno-0-51) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-52) enforce_eager=True, [](#__codelineno-0-53) ) [](#__codelineno-0-54) [](#__codelineno-0-55) # AudioFlamingo3 uses token for audio [](#__codelineno-0-56) audio_placeholder = "" * audio_count [](#__codelineno-0-57) [](#__codelineno-0-58) prompt = ( [](#__codelineno-0-59) "<|im_start|>system\n" [](#__codelineno-0-60) "You are a helpful assistant.<|im_end|>\n" [](#__codelineno-0-61) "<|im_start|>user\n" [](#__codelineno-0-62) f"{audio_placeholder}{question}<|im_end|>\n" [](#__codelineno-0-63) "<|im_start|>assistant\n" [](#__codelineno-0-64) ) [](#__codelineno-0-65) [](#__codelineno-0-66) return ModelRequestData( [](#__codelineno-0-67) engine_args=engine_args, [](#__codelineno-0-68) prompt=prompt, [](#__codelineno-0-69) ) [](#__codelineno-0-70) [](#__codelineno-0-71)[](#__codelineno-0-72)# CohereASR [](#__codelineno-0-73)def run_cohere_asr(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-74) assert audio_count == 1, "CohereASR only support single audio input per prompt" [](#__codelineno-0-75) model_name = "CohereLabs/cohere-transcribe-03-2026" [](#__codelineno-0-76) [](#__codelineno-0-77) prompt = ( [](#__codelineno-0-78) "<|startofcontext|><|startoftranscript|>" [](#__codelineno-0-79) "<|emo:undefined|><|en|><|en|><|pnc|><|noitn|>" [](#__codelineno-0-80) "<|notimestamp|><|nodiarize|>" [](#__codelineno-0-81) ) [](#__codelineno-0-82) engine_args = EngineArgs( [](#__codelineno-0-83) model=model_name, [](#__codelineno-0-84) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-85) trust_remote_code=True, [](#__codelineno-0-86) ) [](#__codelineno-0-87) [](#__codelineno-0-88) return ModelRequestData( [](#__codelineno-0-89) engine_args=engine_args, [](#__codelineno-0-90) prompt=prompt, [](#__codelineno-0-91) ) [](#__codelineno-0-92) [](#__codelineno-0-93)[](#__codelineno-0-94)# MusicFlamingo [](#__codelineno-0-95)def run_musicflamingo(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-96) model_name = "nvidia/music-flamingo-2601-hf" [](#__codelineno-0-97) engine_args = EngineArgs( [](#__codelineno-0-98) model=model_name, [](#__codelineno-0-99) max_model_len=4096, [](#__codelineno-0-100) max_num_seqs=2, [](#__codelineno-0-101) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-102) enforce_eager=True, [](#__codelineno-0-103) ) [](#__codelineno-0-104) [](#__codelineno-0-105) # MusicFlamingo prompt placeholders use ; vLLM's MusicFlamingo [](#__codelineno-0-106) # multimodal processor expands each one into <|sound_bos|> + audio tokens + [](#__codelineno-0-107) # <|sound_eos|> based on extracted audio feature lengths. [](#__codelineno-0-108) audio_placeholder = "" * audio_count [](#__codelineno-0-109) system_prompt = ( [](#__codelineno-0-110) "You are Music Flamingo, a multimodal assistant for language and music. " [](#__codelineno-0-111) "On each turn you receive an audio clip which contains music and optional " [](#__codelineno-0-112) "text, you will receive at least one or both; use your world knowledge and " [](#__codelineno-0-113) "reasoning to help the user with any task. Interpret the entirety of the " [](#__codelineno-0-114) "content any input music--regardlenss of whether the user calls it audio, " [](#__codelineno-0-115) "music, or sound." [](#__codelineno-0-116) ) [](#__codelineno-0-117) [](#__codelineno-0-118) prompt = ( [](#__codelineno-0-119) "<|im_start|>system\n" [](#__codelineno-0-120) f"{system_prompt}<|im_end|>\n" [](#__codelineno-0-121) "<|im_start|>user\n" [](#__codelineno-0-122) f"{audio_placeholder}{question}<|im_end|>\n" [](#__codelineno-0-123) "<|im_start|>assistant\n" [](#__codelineno-0-124) ) [](#__codelineno-0-125) [](#__codelineno-0-126) return ModelRequestData( [](#__codelineno-0-127) engine_args=engine_args, [](#__codelineno-0-128) prompt=prompt, [](#__codelineno-0-129) ) [](#__codelineno-0-130) [](#__codelineno-0-131)[](#__codelineno-0-132)# Gemma3N [](#__codelineno-0-133)def run_gemma3n(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-134) model_name = "google/gemma-3n-E2B-it" [](#__codelineno-0-135) engine_args = EngineArgs( [](#__codelineno-0-136) model=model_name, [](#__codelineno-0-137) max_model_len=2048, [](#__codelineno-0-138) max_num_batched_tokens=2048, [](#__codelineno-0-139) max_num_seqs=2, [](#__codelineno-0-140) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-141) enforce_eager=True, [](#__codelineno-0-142) ) [](#__codelineno-0-143) prompt = f"user\n{question}" [](#__codelineno-0-144) "\nmodel\n" [](#__codelineno-0-145) return ModelRequestData( [](#__codelineno-0-146) engine_args=engine_args, [](#__codelineno-0-147) prompt=prompt, [](#__codelineno-0-148) ) [](#__codelineno-0-149) [](#__codelineno-0-150)[](#__codelineno-0-151)# GLM-ASR [](#__codelineno-0-152)def run_glmasr(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-153) model_name = "zai-org/GLM-ASR-Nano-2512" [](#__codelineno-0-154) [](#__codelineno-0-155) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-0-156) [](#__codelineno-0-157) # GLM-ASR uses <|pad|> token for audio [](#__codelineno-0-158) audio_placeholder = "<|pad|>" * audio_count [](#__codelineno-0-159) [](#__codelineno-0-160) messages = [{"role": "user", "content": f"{audio_placeholder}{question}"}] [](#__codelineno-0-161) prompt = tokenizer.apply_chat_template( [](#__codelineno-0-162) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-0-163) ) [](#__codelineno-0-164) [](#__codelineno-0-165) engine_args = EngineArgs( [](#__codelineno-0-166) model=model_name, [](#__codelineno-0-167) trust_remote_code=True, [](#__codelineno-0-168) max_model_len=4096, [](#__codelineno-0-169) max_num_seqs=2, [](#__codelineno-0-170) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-171) ) [](#__codelineno-0-172) [](#__codelineno-0-173) return ModelRequestData( [](#__codelineno-0-174) engine_args=engine_args, [](#__codelineno-0-175) prompt=prompt, [](#__codelineno-0-176) ) [](#__codelineno-0-177) [](#__codelineno-0-178)[](#__codelineno-0-179)# FunAudioChat [](#__codelineno-0-180)def run_funaudiochat(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-181) # NOTE: FunAudioChat is not available on the HuggingFace Hub at the time of [](#__codelineno-0-182) # writing. Pass a local model path via `--model`. [](#__codelineno-0-183) model_name = "funaudiochat" [](#__codelineno-0-184) [](#__codelineno-0-185) engine_args = EngineArgs( [](#__codelineno-0-186) model=model_name, [](#__codelineno-0-187) max_model_len=4096, [](#__codelineno-0-188) max_num_seqs=2, [](#__codelineno-0-189) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-190) enforce_eager=True, [](#__codelineno-0-191) ) [](#__codelineno-0-192) [](#__codelineno-0-193) audio_in_prompt = "".join( [](#__codelineno-0-194) ["<|audio_bos|><|AUDIO|><|audio_eos|>\n" for _ in range(audio_count)] [](#__codelineno-0-195) ) [](#__codelineno-0-196) prompt = f"{audio_in_prompt}{question}" [](#__codelineno-0-197) [](#__codelineno-0-198) return ModelRequestData( [](#__codelineno-0-199) engine_args=engine_args, [](#__codelineno-0-200) prompt=prompt, [](#__codelineno-0-201) ) [](#__codelineno-0-202) [](#__codelineno-0-203)[](#__codelineno-0-204)# Granite Speech [](#__codelineno-0-205)def run_granite_speech(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-206) # NOTE - the setting in this example are somewhat different from what is [](#__codelineno-0-207) # optimal for granite speech, and it is generally recommended to use beam [](#__codelineno-0-208) # search. Check the model README for suggested settings. [](#__codelineno-0-209) # https://huggingface.co/ibm-granite/granite-speech-3.3-8b [](#__codelineno-0-210) model_name = "ibm-granite/granite-speech-3.3-8b" [](#__codelineno-0-211) [](#__codelineno-0-212) engine_args = EngineArgs( [](#__codelineno-0-213) model=model_name, [](#__codelineno-0-214) trust_remote_code=True, [](#__codelineno-0-215) max_model_len=2048, [](#__codelineno-0-216) max_num_seqs=2, [](#__codelineno-0-217) enable_lora=True, [](#__codelineno-0-218) max_lora_rank=64, [](#__codelineno-0-219) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-220) ) [](#__codelineno-0-221) [](#__codelineno-0-222) # The model has an audio-specific lora directly in its model dir; [](#__codelineno-0-223) # it should be enabled whenever you pass audio inputs to the model. [](#__codelineno-0-224) speech_lora_path = model_name [](#__codelineno-0-225) audio_placeholder = "<|audio|>" * audio_count [](#__codelineno-0-226) prompts = f"<|start_of_role|>system<|end_of_role|>Knowledge Cutoff Date: April 2024.\nToday's Date: December 19, 2024.\nYou are Granite, developed by IBM. You are a helpful AI assistant<|end_of_text|>\n<|start_of_role|>user<|end_of_role|>{audio_placeholder}{question}<|end_of_text|>\n<|start_of_role|>assistant<|end_of_role|>" # noqa: E501 [](#__codelineno-0-227) [](#__codelineno-0-228) return ModelRequestData( [](#__codelineno-0-229) engine_args=engine_args, [](#__codelineno-0-230) prompt=prompts, [](#__codelineno-0-231) lora_requests=[LoRARequest("speech", 1, speech_lora_path)], [](#__codelineno-0-232) ) [](#__codelineno-0-233) [](#__codelineno-0-234)[](#__codelineno-0-235)# Kimi-Audio-7B-Instruct [](#__codelineno-0-236)def run_kimi_audio(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-237) """Kimi-Audio-7B-Instruct for audio transcription and understanding.""" [](#__codelineno-0-238) model_name = "moonshotai/Kimi-Audio-7B-Instruct" [](#__codelineno-0-239) [](#__codelineno-0-240) engine_args = EngineArgs( [](#__codelineno-0-241) model=model_name, [](#__codelineno-0-242) trust_remote_code=True, [](#__codelineno-0-243) max_model_len=4096, [](#__codelineno-0-244) max_num_seqs=2, [](#__codelineno-0-245) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-246) ) [](#__codelineno-0-247) [](#__codelineno-0-248) # Kimi-Audio uses <|im_kimia_text_blank|> as placeholder for audio features [](#__codelineno-0-249) audio_placeholder = "<|im_kimia_text_blank|>" * audio_count [](#__codelineno-0-250) # Default prompt for transcription [](#__codelineno-0-251) if not question: [](#__codelineno-0-252) question = "Please transcribe the audio" [](#__codelineno-0-253) prompt = f"{audio_placeholder}{question}" [](#__codelineno-0-254) [](#__codelineno-0-255) # Stop at EOS token (151644) to prevent repetition [](#__codelineno-0-256) return ModelRequestData( [](#__codelineno-0-257) engine_args=engine_args, [](#__codelineno-0-258) prompt=prompt, [](#__codelineno-0-259) stop_token_ids=[151644], [](#__codelineno-0-260) ) [](#__codelineno-0-261) [](#__codelineno-0-262)[](#__codelineno-0-263)# MiDashengLM [](#__codelineno-0-264)def run_midashenglm(question: str, audio_count: int): [](#__codelineno-0-265) model_name = "mispeech/midashenglm-7b" [](#__codelineno-0-266) [](#__codelineno-0-267) engine_args = EngineArgs( [](#__codelineno-0-268) model=model_name, [](#__codelineno-0-269) trust_remote_code=True, [](#__codelineno-0-270) max_model_len=4096, [](#__codelineno-0-271) max_num_seqs=5, [](#__codelineno-0-272) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-273) ) [](#__codelineno-0-274) [](#__codelineno-0-275) audio_in_prompt = "".join( [](#__codelineno-0-276) ["<|audio_bos|><|AUDIO|><|audio_eos|>" for idx in range(audio_count)] [](#__codelineno-0-277) ) [](#__codelineno-0-278) [](#__codelineno-0-279) default_system = "You are a helpful language and speech assistant." [](#__codelineno-0-280) [](#__codelineno-0-281) prompt = ( [](#__codelineno-0-282) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-0-283) "<|im_start|>user\n" [](#__codelineno-0-284) f"{audio_in_prompt}{question}<|im_end|>\n" [](#__codelineno-0-285) "<|im_start|>assistant\n" [](#__codelineno-0-286) ) [](#__codelineno-0-287) return ModelRequestData( [](#__codelineno-0-288) engine_args=engine_args, [](#__codelineno-0-289) prompt=prompt, [](#__codelineno-0-290) ) [](#__codelineno-0-291) [](#__codelineno-0-292)[](#__codelineno-0-293)# MiniCPM-O [](#__codelineno-0-294)def run_minicpmo(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-295) model_name = "openbmb/MiniCPM-o-2_6" [](#__codelineno-0-296) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-0-297) engine_args = EngineArgs( [](#__codelineno-0-298) model=model_name, [](#__codelineno-0-299) trust_remote_code=True, [](#__codelineno-0-300) max_model_len=4096, [](#__codelineno-0-301) max_num_seqs=2, [](#__codelineno-0-302) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-303) ) [](#__codelineno-0-304) [](#__codelineno-0-305) stop_tokens = ["<|im_end|>", "<|endoftext|>"] [](#__codelineno-0-306) stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] [](#__codelineno-0-307) [](#__codelineno-0-308) audio_placeholder = "()" * audio_count [](#__codelineno-0-309) audio_chat_template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n<|spk_bos|><|spk|><|spk_eos|><|tts_bos|>' }}{% endif %}" # noqa: E501 [](#__codelineno-0-310) messages = [{"role": "user", "content": f"{audio_placeholder}\n{question}"}] [](#__codelineno-0-311) prompt = tokenizer.apply_chat_template( [](#__codelineno-0-312) messages, [](#__codelineno-0-313) tokenize=False, [](#__codelineno-0-314) add_generation_prompt=True, [](#__codelineno-0-315) chat_template=audio_chat_template, [](#__codelineno-0-316) ) [](#__codelineno-0-317) [](#__codelineno-0-318) return ModelRequestData( [](#__codelineno-0-319) engine_args=engine_args, [](#__codelineno-0-320) prompt=prompt, [](#__codelineno-0-321) stop_token_ids=stop_token_ids, [](#__codelineno-0-322) ) [](#__codelineno-0-323) [](#__codelineno-0-324)[](#__codelineno-0-325)# Phi-4-multimodal-instruct [](#__codelineno-0-326)def run_phi4mm(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-327) """ [](#__codelineno-0-328) Phi-4-multimodal-instruct supports both image and audio inputs. Here, we [](#__codelineno-0-329) show how to process audio inputs. [](#__codelineno-0-330) """ [](#__codelineno-0-331) model_path = snapshot_download("microsoft/Phi-4-multimodal-instruct") [](#__codelineno-0-332) # Since the vision-lora and speech-lora co-exist with the base model, [](#__codelineno-0-333) # we have to manually specify the path of the lora weights. [](#__codelineno-0-334) speech_lora_path = os.path.join(model_path, "speech-lora") [](#__codelineno-0-335) placeholders = "".join([f"<|audio_{i + 1}|>" for i in range(audio_count)]) [](#__codelineno-0-336) [](#__codelineno-0-337) prompts = f"<|user|>{placeholders}{question}<|end|><|assistant|>" [](#__codelineno-0-338) [](#__codelineno-0-339) engine_args = EngineArgs( [](#__codelineno-0-340) model=model_path, [](#__codelineno-0-341) trust_remote_code=True, [](#__codelineno-0-342) max_model_len=12800, [](#__codelineno-0-343) max_num_seqs=2, [](#__codelineno-0-344) enable_lora=True, [](#__codelineno-0-345) max_lora_rank=320, [](#__codelineno-0-346) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-347) ) [](#__codelineno-0-348) [](#__codelineno-0-349) return ModelRequestData( [](#__codelineno-0-350) engine_args=engine_args, [](#__codelineno-0-351) prompt=prompts, [](#__codelineno-0-352) lora_requests=[LoRARequest("speech", 1, speech_lora_path)], [](#__codelineno-0-353) ) [](#__codelineno-0-354) [](#__codelineno-0-355)[](#__codelineno-0-356)# Qwen2-Audio [](#__codelineno-0-357)def run_qwen2_audio(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-358) model_name = "Qwen/Qwen2-Audio-7B-Instruct" [](#__codelineno-0-359) [](#__codelineno-0-360) engine_args = EngineArgs( [](#__codelineno-0-361) model=model_name, [](#__codelineno-0-362) max_model_len=4096, [](#__codelineno-0-363) max_num_seqs=5, [](#__codelineno-0-364) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-365) ) [](#__codelineno-0-366) [](#__codelineno-0-367) audio_in_prompt = "".join( [](#__codelineno-0-368) [ [](#__codelineno-0-369) f"Audio {idx + 1}: <|audio_bos|><|AUDIO|><|audio_eos|>\n" [](#__codelineno-0-370) for idx in range(audio_count) [](#__codelineno-0-371) ] [](#__codelineno-0-372) ) [](#__codelineno-0-373) [](#__codelineno-0-374) prompt = ( [](#__codelineno-0-375) "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" [](#__codelineno-0-376) "<|im_start|>user\n" [](#__codelineno-0-377) f"{audio_in_prompt}{question}<|im_end|>\n" [](#__codelineno-0-378) "<|im_start|>assistant\n" [](#__codelineno-0-379) ) [](#__codelineno-0-380) [](#__codelineno-0-381) return ModelRequestData( [](#__codelineno-0-382) engine_args=engine_args, [](#__codelineno-0-383) prompt=prompt, [](#__codelineno-0-384) ) [](#__codelineno-0-385) [](#__codelineno-0-386)[](#__codelineno-0-387)# Qwen2.5-Omni [](#__codelineno-0-388)def run_qwen2_5_omni(question: str, audio_count: int): [](#__codelineno-0-389) model_name = "Qwen/Qwen2.5-Omni-7B" [](#__codelineno-0-390) [](#__codelineno-0-391) engine_args = EngineArgs( [](#__codelineno-0-392) model=model_name, [](#__codelineno-0-393) max_model_len=4096, [](#__codelineno-0-394) max_num_seqs=5, [](#__codelineno-0-395) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-396) ) [](#__codelineno-0-397) [](#__codelineno-0-398) audio_in_prompt = "".join( [](#__codelineno-0-399) ["<|audio_bos|><|AUDIO|><|audio_eos|>\n" for idx in range(audio_count)] [](#__codelineno-0-400) ) [](#__codelineno-0-401) [](#__codelineno-0-402) default_system = ( [](#__codelineno-0-403) "You are Qwen, a virtual human developed by the Qwen Team, Alibaba " [](#__codelineno-0-404) "Group, capable of perceiving auditory and visual inputs, as well as " [](#__codelineno-0-405) "generating text and speech." [](#__codelineno-0-406) ) [](#__codelineno-0-407) [](#__codelineno-0-408) prompt = ( [](#__codelineno-0-409) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-0-410) "<|im_start|>user\n" [](#__codelineno-0-411) f"{audio_in_prompt}{question}<|im_end|>\n" [](#__codelineno-0-412) "<|im_start|>assistant\n" [](#__codelineno-0-413) ) [](#__codelineno-0-414) return ModelRequestData( [](#__codelineno-0-415) engine_args=engine_args, [](#__codelineno-0-416) prompt=prompt, [](#__codelineno-0-417) ) [](#__codelineno-0-418) [](#__codelineno-0-419)[](#__codelineno-0-420)def run_qwen3_asr(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-421) model_name = "Qwen/Qwen3-Asr-1.7B" [](#__codelineno-0-422) [](#__codelineno-0-423) audio_in_prompt = "<|audio_start|><|audio_pad|><|audio_end|>\n" * audio_count [](#__codelineno-0-424) prompt = f"<|im_start|>user\n{audio_in_prompt}<|im_end|>\n<|im_start|>assistant\n" [](#__codelineno-0-425) [](#__codelineno-0-426) engine_args = EngineArgs( [](#__codelineno-0-427) model=model_name, [](#__codelineno-0-428) max_model_len=4096, [](#__codelineno-0-429) max_num_seqs=5, [](#__codelineno-0-430) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-431) ) [](#__codelineno-0-432) [](#__codelineno-0-433) return ModelRequestData( [](#__codelineno-0-434) engine_args=engine_args, [](#__codelineno-0-435) prompt=prompt, [](#__codelineno-0-436) ) [](#__codelineno-0-437) [](#__codelineno-0-438)[](#__codelineno-0-439)# Ultravox 0.5-1B [](#__codelineno-0-440)def run_ultravox(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-441) model_name = "fixie-ai/ultravox-v0_5-llama-3_2-1b" [](#__codelineno-0-442) [](#__codelineno-0-443) tokenizer = AutoTokenizer.from_pretrained(model_name) [](#__codelineno-0-444) messages = [{"role": "user", "content": "<|audio|>\n" * audio_count + question}] [](#__codelineno-0-445) prompt = tokenizer.apply_chat_template( [](#__codelineno-0-446) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-0-447) ) [](#__codelineno-0-448) [](#__codelineno-0-449) engine_args = EngineArgs( [](#__codelineno-0-450) model=model_name, [](#__codelineno-0-451) max_model_len=4096, [](#__codelineno-0-452) max_num_seqs=5, [](#__codelineno-0-453) trust_remote_code=True, [](#__codelineno-0-454) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-455) ) [](#__codelineno-0-456) [](#__codelineno-0-457) return ModelRequestData( [](#__codelineno-0-458) engine_args=engine_args, [](#__codelineno-0-459) prompt=prompt, [](#__codelineno-0-460) ) [](#__codelineno-0-461) [](#__codelineno-0-462)[](#__codelineno-0-463)# Voxtral [](#__codelineno-0-464)# Make sure to install mistral-common[audio]. [](#__codelineno-0-465)def run_voxtral(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-466) from mistral_common.audio import Audio [](#__codelineno-0-467) from mistral_common.protocol.instruct.chunk import ( [](#__codelineno-0-468) AudioChunk, [](#__codelineno-0-469) RawAudio, [](#__codelineno-0-470) TextChunk, [](#__codelineno-0-471) ) [](#__codelineno-0-472) from mistral_common.protocol.instruct.messages import ( [](#__codelineno-0-473) UserMessage, [](#__codelineno-0-474) ) [](#__codelineno-0-475) from mistral_common.protocol.instruct.request import ChatCompletionRequest [](#__codelineno-0-476) from mistral_common.tokens.tokenizers.mistral import MistralTokenizer [](#__codelineno-0-477) [](#__codelineno-0-478) model_name = "mistralai/Voxtral-Mini-3B-2507" [](#__codelineno-0-479) tokenizer = MistralTokenizer.from_hf_hub(model_name) [](#__codelineno-0-480) [](#__codelineno-0-481) engine_args = EngineArgs( [](#__codelineno-0-482) model=model_name, [](#__codelineno-0-483) max_model_len=8192, [](#__codelineno-0-484) max_num_seqs=2, [](#__codelineno-0-485) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-486) config_format="mistral", [](#__codelineno-0-487) load_format="mistral", [](#__codelineno-0-488) tokenizer_mode="mistral", [](#__codelineno-0-489) enforce_eager=True, [](#__codelineno-0-490) enable_chunked_prefill=False, [](#__codelineno-0-491) ) [](#__codelineno-0-492) [](#__codelineno-0-493) text_chunk = TextChunk(text=question) [](#__codelineno-0-494) audios = [ [](#__codelineno-0-495) Audio.from_file(str(audio_assets[i].get_local_path()), strict=False) [](#__codelineno-0-496) for i in range(audio_count) [](#__codelineno-0-497) ] [](#__codelineno-0-498) audio_chunks = [ [](#__codelineno-0-499) AudioChunk(input_audio=RawAudio.from_audio(audio)) for audio in audios [](#__codelineno-0-500) ] [](#__codelineno-0-501) [](#__codelineno-0-502) messages = [UserMessage(content=[*audio_chunks, text_chunk])] [](#__codelineno-0-503) [](#__codelineno-0-504) req = ChatCompletionRequest(messages=messages, model=model_name) [](#__codelineno-0-505) [](#__codelineno-0-506) tokens = tokenizer.encode_chat_completion(req) [](#__codelineno-0-507) prompt_ids, audios = tokens.tokens, tokens.audios [](#__codelineno-0-508) [](#__codelineno-0-509) audios_and_sr = [(au.audio_array, au.sampling_rate) for au in audios] [](#__codelineno-0-510) [](#__codelineno-0-511) multi_modal_data = {"audio": audios_and_sr} [](#__codelineno-0-512) [](#__codelineno-0-513) return ModelRequestData( [](#__codelineno-0-514) engine_args=engine_args, [](#__codelineno-0-515) prompt_token_ids=prompt_ids, [](#__codelineno-0-516) multi_modal_data=multi_modal_data, [](#__codelineno-0-517) ) [](#__codelineno-0-518) [](#__codelineno-0-519)[](#__codelineno-0-520)# Whisper [](#__codelineno-0-521)def run_whisper(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-522) assert audio_count == 1, "Whisper only support single audio input per prompt" [](#__codelineno-0-523) model_name = "openai/whisper-large-v3-turbo" [](#__codelineno-0-524) [](#__codelineno-0-525) prompt = "<|startoftranscript|>" [](#__codelineno-0-526) [](#__codelineno-0-527) engine_args = EngineArgs( [](#__codelineno-0-528) model=model_name, [](#__codelineno-0-529) max_model_len=448, [](#__codelineno-0-530) max_num_seqs=5, [](#__codelineno-0-531) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-532) ) [](#__codelineno-0-533) [](#__codelineno-0-534) return ModelRequestData( [](#__codelineno-0-535) engine_args=engine_args, [](#__codelineno-0-536) prompt=prompt, [](#__codelineno-0-537) ) [](#__codelineno-0-538) [](#__codelineno-0-539)[](#__codelineno-0-540)# FireRedLID [](#__codelineno-0-541)def run_fireredlid(question: str, audio_count: int) -> ModelRequestData: [](#__codelineno-0-542) assert audio_count == 1, "FireRedLID only supports single audio input per prompt" [](#__codelineno-0-543) model_name = "PatchyTisa/FireRedLID-vllm" [](#__codelineno-0-544) [](#__codelineno-0-545) prompt = "" [](#__codelineno-0-546) [](#__codelineno-0-547) engine_args = EngineArgs( [](#__codelineno-0-548) model=model_name, [](#__codelineno-0-549) max_model_len=8, [](#__codelineno-0-550) max_num_seqs=5, [](#__codelineno-0-551) limit_mm_per_prompt={"audio": audio_count}, [](#__codelineno-0-552) ) [](#__codelineno-0-553) [](#__codelineno-0-554) return ModelRequestData( [](#__codelineno-0-555) engine_args=engine_args, [](#__codelineno-0-556) prompt=prompt, [](#__codelineno-0-557) ) [](#__codelineno-0-558) [](#__codelineno-0-559)[](#__codelineno-0-560)model_example_map = { [](#__codelineno-0-561) "audioflamingo3": run_audioflamingo3, [](#__codelineno-0-562) "cohere_asr": run_cohere_asr, [](#__codelineno-0-563) "fireredlid": run_fireredlid, [](#__codelineno-0-564) "funaudiochat": run_funaudiochat, [](#__codelineno-0-565) "gemma3n": run_gemma3n, [](#__codelineno-0-566) "glmasr": run_glmasr, [](#__codelineno-0-567) "granite_speech": run_granite_speech, [](#__codelineno-0-568) "kimi_audio": run_kimi_audio, [](#__codelineno-0-569) "midashenglm": run_midashenglm, [](#__codelineno-0-570) "minicpmo": run_minicpmo, [](#__codelineno-0-571) "musicflamingo": run_musicflamingo, [](#__codelineno-0-572) "phi4_mm": run_phi4mm, [](#__codelineno-0-573) "qwen2_audio": run_qwen2_audio, [](#__codelineno-0-574) "qwen2_5_omni": run_qwen2_5_omni, [](#__codelineno-0-575) "qwen3_asr": run_qwen3_asr, [](#__codelineno-0-576) "ultravox": run_ultravox, [](#__codelineno-0-577) "voxtral": run_voxtral, [](#__codelineno-0-578) "whisper": run_whisper, [](#__codelineno-0-579)} [](#__codelineno-0-580) [](#__codelineno-0-581)[](#__codelineno-0-582)def parse_args(): [](#__codelineno-0-583) parser = FlexibleArgumentParser( [](#__codelineno-0-584) description="Demo on using vLLM for offline inference with " [](#__codelineno-0-585) "audio language models" [](#__codelineno-0-586) ) [](#__codelineno-0-587) parser.add_argument( [](#__codelineno-0-588) "--model-type", [](#__codelineno-0-589) "-m", [](#__codelineno-0-590) type=str, [](#__codelineno-0-591) default="ultravox", [](#__codelineno-0-592) choices=model_example_map.keys(), [](#__codelineno-0-593) help='Huggingface "model_type".', [](#__codelineno-0-594) ) [](#__codelineno-0-595) parser.add_argument( [](#__codelineno-0-596) "--model", [](#__codelineno-0-597) type=str, [](#__codelineno-0-598) default=None, [](#__codelineno-0-599) help="Model ID or local path override. Required for funaudiochat.", [](#__codelineno-0-600) ) [](#__codelineno-0-601) parser.add_argument( [](#__codelineno-0-602) "--num-prompts", type=int, default=1, help="Number of prompts to run." [](#__codelineno-0-603) ) [](#__codelineno-0-604) parser.add_argument( [](#__codelineno-0-605) "--num-audios", [](#__codelineno-0-606) type=int, [](#__codelineno-0-607) default=1, [](#__codelineno-0-608) choices=[0, 1, 2], [](#__codelineno-0-609) help="Number of audio items per prompt.", [](#__codelineno-0-610) ) [](#__codelineno-0-611) parser.add_argument( [](#__codelineno-0-612) "--seed", [](#__codelineno-0-613) type=int, [](#__codelineno-0-614) default=0, [](#__codelineno-0-615) help="Set the seed when initializing `vllm.LLM`.", [](#__codelineno-0-616) ) [](#__codelineno-0-617) parser.add_argument( [](#__codelineno-0-618) "--tensor-parallel-size", [](#__codelineno-0-619) "-tp", [](#__codelineno-0-620) type=int, [](#__codelineno-0-621) default=None, [](#__codelineno-0-622) help="Tensor parallel size to override the model's default setting. ", [](#__codelineno-0-623) ) [](#__codelineno-0-624) [](#__codelineno-0-625) return parser.parse_args() [](#__codelineno-0-626) [](#__codelineno-0-627)[](#__codelineno-0-628)def main(args): [](#__codelineno-0-629) model = args.model_type [](#__codelineno-0-630) if model not in model_example_map: [](#__codelineno-0-631) raise ValueError(f"Model type {model} is not supported.") [](#__codelineno-0-632) [](#__codelineno-0-633) if model == "funaudiochat" and not args.model: [](#__codelineno-0-634) raise ValueError("--model is required when --model-type=funaudiochat") [](#__codelineno-0-635) [](#__codelineno-0-636) if args.tensor_parallel_size is not None and args.tensor_parallel_size < 1: [](#__codelineno-0-637) raise ValueError( [](#__codelineno-0-638) f"tensor_parallel_size must be a positive integer, " [](#__codelineno-0-639) f"got {args.tensor_parallel_size}" [](#__codelineno-0-640) ) [](#__codelineno-0-641) [](#__codelineno-0-642) audio_count = args.num_audios [](#__codelineno-0-643) req_data = model_example_map[model]( [](#__codelineno-0-644) question_per_audio_count[audio_count], audio_count [](#__codelineno-0-645) ) [](#__codelineno-0-646) if model == "funaudiochat": [](#__codelineno-0-647) req_data.engine_args.model = args.model [](#__codelineno-0-648) [](#__codelineno-0-649) # Disable other modalities to save memory [](#__codelineno-0-650) default_limits = {"image": 0, "video": 0, "audio": 0} [](#__codelineno-0-651) req_data.engine_args.limit_mm_per_prompt = default_limits | dict( [](#__codelineno-0-652) req_data.engine_args.limit_mm_per_prompt or {} [](#__codelineno-0-653) ) [](#__codelineno-0-654) [](#__codelineno-0-655) engine_args = vars(req_data.engine_args) | {"seed": args.seed} [](#__codelineno-0-656) if args.tensor_parallel_size is not None: [](#__codelineno-0-657) engine_args["tensor_parallel_size"] = args.tensor_parallel_size [](#__codelineno-0-658) llm = LLM(**engine_args) [](#__codelineno-0-659) [](#__codelineno-0-660) # We set temperature to 0.2 so that outputs can be different [](#__codelineno-0-661) # even when all prompts are identical when running batch inference. [](#__codelineno-0-662) sampling_params = SamplingParams( [](#__codelineno-0-663) temperature=0.2, max_tokens=64, stop_token_ids=req_data.stop_token_ids [](#__codelineno-0-664) ) [](#__codelineno-0-665) [](#__codelineno-0-666) def get_input(start, end): [](#__codelineno-0-667) mm_data = req_data.multi_modal_data [](#__codelineno-0-668) if not mm_data: [](#__codelineno-0-669) mm_data = {} [](#__codelineno-0-670) if end - start > 0: [](#__codelineno-0-671) mm_data = { [](#__codelineno-0-672) "audio": [ [](#__codelineno-0-673) asset.audio_and_sample_rate for asset in audio_assets[start:end] [](#__codelineno-0-674) ] [](#__codelineno-0-675) } [](#__codelineno-0-676) [](#__codelineno-0-677) inputs = {"multi_modal_data": mm_data} [](#__codelineno-0-678) [](#__codelineno-0-679) if req_data.prompt: [](#__codelineno-0-680) inputs["prompt"] = req_data.prompt [](#__codelineno-0-681) else: [](#__codelineno-0-682) inputs["prompt_token_ids"] = req_data.prompt_token_ids [](#__codelineno-0-683) [](#__codelineno-0-684) return inputs [](#__codelineno-0-685) [](#__codelineno-0-686) # Batch inference [](#__codelineno-0-687) assert args.num_prompts > 0 [](#__codelineno-0-688) if audio_count != 1: [](#__codelineno-0-689) inputs = get_input(0, audio_count) [](#__codelineno-0-690) inputs = [inputs] * args.num_prompts [](#__codelineno-0-691) else: [](#__codelineno-0-692) # For single audio input, we need to vary the audio input [](#__codelineno-0-693) # to avoid deduplication in vLLM engine. [](#__codelineno-0-694) inputs = [] [](#__codelineno-0-695) for i in range(args.num_prompts): [](#__codelineno-0-696) start = i % len(audio_assets) [](#__codelineno-0-697) inp = get_input(start, start + 1) [](#__codelineno-0-698) inputs.append(inp) [](#__codelineno-0-699) [](#__codelineno-0-700) # Add LoRA request if applicable [](#__codelineno-0-701) lora_request = ( [](#__codelineno-0-702) req_data.lora_requests * args.num_prompts if req_data.lora_requests else None [](#__codelineno-0-703) ) [](#__codelineno-0-704) [](#__codelineno-0-705) outputs = llm.generate( [](#__codelineno-0-706) inputs, [](#__codelineno-0-707) sampling_params=sampling_params, [](#__codelineno-0-708) lora_request=lora_request, [](#__codelineno-0-709) ) [](#__codelineno-0-710) [](#__codelineno-0-711) for o in outputs: [](#__codelineno-0-712) generated_text = o.outputs[0].text [](#__codelineno-0-713) print(generated_text) [](#__codelineno-0-714) [](#__codelineno-0-715)[](#__codelineno-0-716)if __name__ == "__main__": [](#__codelineno-0-717) args = parse_args() [](#__codelineno-0-718) main(args)`` ## Encoder Decoder Multimodal Offline[¶](#encoder-decoder-multimodal-offline "Permanent link") ```[](#__codelineno-1-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-1-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-1-3)""" [](#__codelineno-1-4)This example shows how to use vLLM for running offline inference with [](#__codelineno-1-5)the explicit/implicit prompt format on enc-dec LMMs for text generation. [](#__codelineno-1-6)""" [](#__codelineno-1-7)[](#__codelineno-1-8)import os [](#__codelineno-1-9)import time [](#__codelineno-1-10)from collections.abc import Sequence [](#__codelineno-1-11)from typing import NamedTuple [](#__codelineno-1-12)[](#__codelineno-1-13)from vllm import LLM, EngineArgs, PromptType, SamplingParams [](#__codelineno-1-14)from vllm.assets.audio import AudioAsset [](#__codelineno-1-15)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-1-16) [](#__codelineno-1-17)[](#__codelineno-1-18)class ModelRequestData(NamedTuple): [](#__codelineno-1-19) engine_args: EngineArgs [](#__codelineno-1-20) prompts: Sequence[PromptType] [](#__codelineno-1-21) [](#__codelineno-1-22)[](#__codelineno-1-23)def run_whisper(): [](#__codelineno-1-24) os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn" [](#__codelineno-1-25) [](#__codelineno-1-26) engine_args = EngineArgs( [](#__codelineno-1-27) model="openai/whisper-large-v3-turbo", [](#__codelineno-1-28) max_model_len=448, [](#__codelineno-1-29) max_num_seqs=16, [](#__codelineno-1-30) limit_mm_per_prompt={"audio": 1}, [](#__codelineno-1-31) dtype="half", [](#__codelineno-1-32) ) [](#__codelineno-1-33) [](#__codelineno-1-34) prompts = [ [](#__codelineno-1-35) { # Test implicit prompt [](#__codelineno-1-36) "prompt": "<|startoftranscript|>", [](#__codelineno-1-37) "multi_modal_data": { [](#__codelineno-1-38) "audio": AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-1-39) }, [](#__codelineno-1-40) }, [](#__codelineno-1-41) { # Test explicit encoder/decoder prompt [](#__codelineno-1-42) "encoder_prompt": { [](#__codelineno-1-43) "prompt": "", [](#__codelineno-1-44) "multi_modal_data": { [](#__codelineno-1-45) "audio": AudioAsset("winning_call").audio_and_sample_rate, [](#__codelineno-1-46) }, [](#__codelineno-1-47) }, [](#__codelineno-1-48) "decoder_prompt": "<|startoftranscript|>", [](#__codelineno-1-49) }, [](#__codelineno-1-50) ] [](#__codelineno-1-51) [](#__codelineno-1-52) return ModelRequestData( [](#__codelineno-1-53) engine_args=engine_args, [](#__codelineno-1-54) prompts=prompts, [](#__codelineno-1-55) ) [](#__codelineno-1-56) [](#__codelineno-1-57)[](#__codelineno-1-58)def run_fireredasr2(): [](#__codelineno-1-59) """ [](#__codelineno-1-60) FireRedASR2 – Automatic Speech Recognition model. [](#__codelineno-1-61) [](#__codelineno-1-62) This model uses a Conformer encoder + Qwen2 LLM decoder architecture [](#__codelineno-1-63) for speech-to-text transcription. Audio is passed via the implicit [](#__codelineno-1-64) prompt format with the ``<|AUDIO|>`` placeholder token. [](#__codelineno-1-65) """ [](#__codelineno-1-66) engine_args = EngineArgs( [](#__codelineno-1-67) model="allendou/FireRedASR2-LLM-vllm", [](#__codelineno-1-68) max_model_len=448, [](#__codelineno-1-69) max_num_seqs=16, [](#__codelineno-1-70) limit_mm_per_prompt={"audio": 1}, [](#__codelineno-1-71) ) [](#__codelineno-1-72) [](#__codelineno-1-73) prompt_str = ( [](#__codelineno-1-74) "<|im_start|>user\n<|AUDIO|>请转写音频为文字<|im_end|>\n<|im_start|>assistant\n" [](#__codelineno-1-75) ) [](#__codelineno-1-76) [](#__codelineno-1-77) prompts = [ [](#__codelineno-1-78) { # Implicit prompt with audio [](#__codelineno-1-79) "prompt": prompt_str, [](#__codelineno-1-80) "multi_modal_data": { [](#__codelineno-1-81) "audio": AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-1-82) }, [](#__codelineno-1-83) }, [](#__codelineno-1-84) { # Another audio sample [](#__codelineno-1-85) "prompt": prompt_str, [](#__codelineno-1-86) "multi_modal_data": { [](#__codelineno-1-87) "audio": AudioAsset("winning_call").audio_and_sample_rate, [](#__codelineno-1-88) }, [](#__codelineno-1-89) }, [](#__codelineno-1-90) ] [](#__codelineno-1-91) [](#__codelineno-1-92) return ModelRequestData( [](#__codelineno-1-93) engine_args=engine_args, [](#__codelineno-1-94) prompts=prompts, [](#__codelineno-1-95) ) [](#__codelineno-1-96) [](#__codelineno-1-97)[](#__codelineno-1-98)def run_fireredlid(): [](#__codelineno-1-99) """ [](#__codelineno-1-100) FireRedLID – Language Identification model. [](#__codelineno-1-101) [](#__codelineno-1-102) This encoder-decoder model identifies the spoken language of an audio [](#__codelineno-1-103) clip. It outputs at most 2 tokens representing the detected language [](#__codelineno-1-104) (e.g. "en", "zh mandarin"). [](#__codelineno-1-105) """ [](#__codelineno-1-106) engine_args = EngineArgs( [](#__codelineno-1-107) model="PatchyTisa/FireRedLID-vllm", [](#__codelineno-1-108) max_model_len=8, [](#__codelineno-1-109) max_num_seqs=16, [](#__codelineno-1-110) limit_mm_per_prompt={"audio": 1}, [](#__codelineno-1-111) ) [](#__codelineno-1-112) [](#__codelineno-1-113) prompts = [ [](#__codelineno-1-114) { # Test explicit encoder/decoder prompt [](#__codelineno-1-115) "encoder_prompt": { [](#__codelineno-1-116) "prompt": "", [](#__codelineno-1-117) "multi_modal_data": { [](#__codelineno-1-118) "audio": AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-1-119) }, [](#__codelineno-1-120) }, [](#__codelineno-1-121) "decoder_prompt": "", [](#__codelineno-1-122) }, [](#__codelineno-1-123) { # Another audio sample [](#__codelineno-1-124) "encoder_prompt": { [](#__codelineno-1-125) "prompt": "", [](#__codelineno-1-126) "multi_modal_data": { [](#__codelineno-1-127) "audio": AudioAsset("winning_call").audio_and_sample_rate, [](#__codelineno-1-128) }, [](#__codelineno-1-129) }, [](#__codelineno-1-130) "decoder_prompt": "", [](#__codelineno-1-131) }, [](#__codelineno-1-132) ] [](#__codelineno-1-133) [](#__codelineno-1-134) return ModelRequestData( [](#__codelineno-1-135) engine_args=engine_args, [](#__codelineno-1-136) prompts=prompts, [](#__codelineno-1-137) ) [](#__codelineno-1-138) [](#__codelineno-1-139)[](#__codelineno-1-140)model_example_map = { [](#__codelineno-1-141) "fireredasr2": run_fireredasr2, [](#__codelineno-1-142) "fireredlid": run_fireredlid, [](#__codelineno-1-143) "whisper": run_whisper, [](#__codelineno-1-144)} [](#__codelineno-1-145) [](#__codelineno-1-146)[](#__codelineno-1-147)def parse_args(): [](#__codelineno-1-148) parser = FlexibleArgumentParser( [](#__codelineno-1-149) description="Demo on using vLLM for offline inference with " [](#__codelineno-1-150) "vision language models for text generation" [](#__codelineno-1-151) ) [](#__codelineno-1-152) parser.add_argument( [](#__codelineno-1-153) "--model-type", [](#__codelineno-1-154) "-m", [](#__codelineno-1-155) type=str, [](#__codelineno-1-156) default="whisper", [](#__codelineno-1-157) choices=model_example_map.keys(), [](#__codelineno-1-158) help='Huggingface "model_type".', [](#__codelineno-1-159) ) [](#__codelineno-1-160) parser.add_argument( [](#__codelineno-1-161) "--seed", [](#__codelineno-1-162) type=int, [](#__codelineno-1-163) default=0, [](#__codelineno-1-164) help="Set the seed when initializing `vllm.LLM`.", [](#__codelineno-1-165) ) [](#__codelineno-1-166) return parser.parse_args() [](#__codelineno-1-167) [](#__codelineno-1-168)[](#__codelineno-1-169)def main(args): [](#__codelineno-1-170) model = args.model_type [](#__codelineno-1-171) if model not in model_example_map: [](#__codelineno-1-172) raise ValueError(f"Model type {model} is not supported.") [](#__codelineno-1-173) [](#__codelineno-1-174) req_data = model_example_map[model]() [](#__codelineno-1-175) [](#__codelineno-1-176) # Disable other modalities to save memory [](#__codelineno-1-177) engine_args = req_data.engine_args [](#__codelineno-1-178) default_limits = {"image": 0, "video": 0, "audio": 0} [](#__codelineno-1-179) limit_mm_per_prompt = default_limits | (engine_args.limit_mm_per_prompt or {}) [](#__codelineno-1-180) engine_args.limit_mm_per_prompt = limit_mm_per_prompt [](#__codelineno-1-181) engine_args.seed = args.seed [](#__codelineno-1-182) llm = LLM.from_engine_args(engine_args) [](#__codelineno-1-183) [](#__codelineno-1-184) prompts = req_data.prompts [](#__codelineno-1-185) [](#__codelineno-1-186) # Create a sampling params object. [](#__codelineno-1-187) sampling_params = SamplingParams( [](#__codelineno-1-188) temperature=0, [](#__codelineno-1-189) top_p=1.0, [](#__codelineno-1-190) max_tokens=64, [](#__codelineno-1-191) skip_special_tokens=False, [](#__codelineno-1-192) ) [](#__codelineno-1-193) [](#__codelineno-1-194) start = time.time() [](#__codelineno-1-195) [](#__codelineno-1-196) # Generate output tokens from the prompts. The output is a list of [](#__codelineno-1-197) # RequestOutput objects that contain the prompt, generated [](#__codelineno-1-198) # text, and other information. [](#__codelineno-1-199) outputs = llm.generate(prompts, sampling_params) [](#__codelineno-1-200) [](#__codelineno-1-201) # Print the outputs. [](#__codelineno-1-202) for output in outputs: [](#__codelineno-1-203) prompt = output.prompt [](#__codelineno-1-204) generated_text = output.outputs[0].text [](#__codelineno-1-205) print(f"Decoder prompt: {prompt!r}, Generated text: {generated_text!r}") [](#__codelineno-1-206) [](#__codelineno-1-207) duration = time.time() - start [](#__codelineno-1-208) [](#__codelineno-1-209) print("Duration:", duration) [](#__codelineno-1-210) print("RPS:", len(prompts) / duration) [](#__codelineno-1-211) [](#__codelineno-1-212)[](#__codelineno-1-213)if __name__ == "__main__": [](#__codelineno-1-214) args = parse_args() [](#__codelineno-1-215) main(args)``` ## Mistral-Small Offline[¶](#mistral-small-offline "Permanent link") `[](#__codelineno-2-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-2-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-2-3)[](#__codelineno-2-4)# ruff: noqa [](#__codelineno-2-5)import argparse [](#__codelineno-2-6)[](#__codelineno-2-7)from vllm import LLM [](#__codelineno-2-8)from vllm.sampling_params import SamplingParams [](#__codelineno-2-9)from vllm.assets.image import ImageAsset [](#__codelineno-2-10)from vllm.multimodal.utils import encode_image_url [](#__codelineno-2-11)[](#__codelineno-2-12)# This script is an offline demo for running Mistral-Small-3.1 [](#__codelineno-2-13)# [](#__codelineno-2-14)# If you want to run a server/client setup, please follow this code: [](#__codelineno-2-15)# [](#__codelineno-2-16)# - Server: [](#__codelineno-2-17)# [](#__codelineno-2-18)# ```bash [](#__codelineno-2-19)# # Mistral format [](#__codelineno-2-20)# vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 \ [](#__codelineno-2-21)# --tokenizer-mode mistral --config-format mistral --load-format mistral \ [](#__codelineno-2-22)# --limit-mm-per-prompt.image 4 --max-model-len 16384 [](#__codelineno-2-23)# [](#__codelineno-2-24)# # HF format [](#__codelineno-2-25)# vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 \ [](#__codelineno-2-26)# --limit-mm-per-prompt.image 4 --max-model-len 16384 [](#__codelineno-2-27)# ``` [](#__codelineno-2-28)# [](#__codelineno-2-29)# - Client: [](#__codelineno-2-30)# [](#__codelineno-2-31)# ```bash [](#__codelineno-2-32)# curl --location 'http://:8000/v1/chat/completions' \ [](#__codelineno-2-33)# --header 'Content-Type: application/json' \ [](#__codelineno-2-34)# --header 'Authorization: Bearer token' \ [](#__codelineno-2-35)# --data '{ [](#__codelineno-2-36)# "model": "mistralai/Mistral-Small-3.1-24B-Instruct-2503", [](#__codelineno-2-37)# "messages": [ [](#__codelineno-2-38)# { [](#__codelineno-2-39)# "role": "user", [](#__codelineno-2-40)# "content": [ [](#__codelineno-2-41)# {"type" : "text", "text": "Describe this image in detail please."}, [](#__codelineno-2-42)# {"type": "image_url", "image_url": {"url": "https://s3.amazonaws.com/cms.ipressroom.com/338/files/201808/5b894ee1a138352221103195_A680%7Ejogging-edit/A680%7Ejogging-edit_hero.jpg"}}, [](#__codelineno-2-43)# {"type" : "text", "text": "and this one as well. Answer in French."}, [](#__codelineno-2-44)# {"type": "image_url", "image_url": {"url": "https://www.wolframcloud.com/obj/resourcesystem/images/a0e/a0ee3983-46c6-4c92-b85d-059044639928/6af8cfb971db031b.png"}} [](#__codelineno-2-45)# ] [](#__codelineno-2-46)# } [](#__codelineno-2-47)# ] [](#__codelineno-2-48)# }' [](#__codelineno-2-49)# ``` [](#__codelineno-2-50)# [](#__codelineno-2-51)# Usage: [](#__codelineno-2-52)# python demo.py simple [](#__codelineno-2-53)# python demo.py advanced [](#__codelineno-2-54)[](#__codelineno-2-55)# Lower max_model_len and/or max_num_seqs on low-VRAM GPUs. [](#__codelineno-2-56)# These scripts have been tested on 2x L40 GPUs [](#__codelineno-2-57) [](#__codelineno-2-58)[](#__codelineno-2-59)def run_simple_demo(args: argparse.Namespace): [](#__codelineno-2-60) model_name = "mistralai/Mistral-Small-3.1-24B-Instruct-2503" [](#__codelineno-2-61) sampling_params = SamplingParams(max_tokens=8192) [](#__codelineno-2-62) [](#__codelineno-2-63) llm = LLM( [](#__codelineno-2-64) model=model_name, [](#__codelineno-2-65) tokenizer_mode="mistral" if args.format == "mistral" else "hf", [](#__codelineno-2-66) config_format="mistral" if args.format == "mistral" else "hf", [](#__codelineno-2-67) load_format="mistral" if args.format == "mistral" else "hf", [](#__codelineno-2-68) limit_mm_per_prompt={"image": 1}, [](#__codelineno-2-69) max_model_len=4096, [](#__codelineno-2-70) max_num_seqs=2, [](#__codelineno-2-71) tensor_parallel_size=2, [](#__codelineno-2-72) mm_processor_cache_gb=0 if args.disable_mm_processor_cache else 4, [](#__codelineno-2-73) ) [](#__codelineno-2-74) [](#__codelineno-2-75) prompt = "Describe this image in one sentence." [](#__codelineno-2-76) [](#__codelineno-2-77) messages = [ [](#__codelineno-2-78) { [](#__codelineno-2-79) "role": "user", [](#__codelineno-2-80) "content": [ [](#__codelineno-2-81) {"type": "text", "text": prompt}, [](#__codelineno-2-82) { [](#__codelineno-2-83) "type": "image_url", [](#__codelineno-2-84) "image_url": { [](#__codelineno-2-85) "url": encode_image_url(ImageAsset("cherry_blossom").pil_image) [](#__codelineno-2-86) }, [](#__codelineno-2-87) }, [](#__codelineno-2-88) ], [](#__codelineno-2-89) }, [](#__codelineno-2-90) ] [](#__codelineno-2-91) outputs = llm.chat(messages, sampling_params=sampling_params) [](#__codelineno-2-92) print("-" * 50) [](#__codelineno-2-93) print(outputs[0].outputs[0].text) [](#__codelineno-2-94) print("-" * 50) [](#__codelineno-2-95) [](#__codelineno-2-96)[](#__codelineno-2-97)def run_advanced_demo(args: argparse.Namespace): [](#__codelineno-2-98) model_name = "mistralai/Mistral-Small-3.1-24B-Instruct-2503" [](#__codelineno-2-99) max_img_per_msg = 3 [](#__codelineno-2-100) max_tokens_per_img = 4096 [](#__codelineno-2-101) [](#__codelineno-2-102) sampling_params = SamplingParams(max_tokens=8192, temperature=0.7) [](#__codelineno-2-103) llm = LLM( [](#__codelineno-2-104) model=model_name, [](#__codelineno-2-105) tokenizer_mode="mistral" if args.format == "mistral" else "hf", [](#__codelineno-2-106) config_format="mistral" if args.format == "mistral" else "hf", [](#__codelineno-2-107) load_format="mistral" if args.format == "mistral" else "hf", [](#__codelineno-2-108) limit_mm_per_prompt={"image": max_img_per_msg}, [](#__codelineno-2-109) max_model_len=max_img_per_msg * max_tokens_per_img, [](#__codelineno-2-110) tensor_parallel_size=2, [](#__codelineno-2-111) mm_processor_cache_gb=0 if args.disable_mm_processor_cache else 4, [](#__codelineno-2-112) ) [](#__codelineno-2-113) [](#__codelineno-2-114) prompt = "Describe the following image." [](#__codelineno-2-115) [](#__codelineno-2-116) url_1 = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png" [](#__codelineno-2-117) url_2 = "https://picsum.photos/seed/picsum/200/300" [](#__codelineno-2-118) url_3 = "https://picsum.photos/id/32/512/512" [](#__codelineno-2-119) [](#__codelineno-2-120) messages = [ [](#__codelineno-2-121) { [](#__codelineno-2-122) "role": "user", [](#__codelineno-2-123) "content": [ [](#__codelineno-2-124) {"type": "text", "text": prompt}, [](#__codelineno-2-125) {"type": "image_url", "image_url": {"url": url_1}}, [](#__codelineno-2-126) {"type": "image_url", "image_url": {"url": url_2}}, [](#__codelineno-2-127) ], [](#__codelineno-2-128) }, [](#__codelineno-2-129) { [](#__codelineno-2-130) "role": "assistant", [](#__codelineno-2-131) "content": "The images show nature.", [](#__codelineno-2-132) }, [](#__codelineno-2-133) { [](#__codelineno-2-134) "role": "user", [](#__codelineno-2-135) "content": "More details please and answer only in French!.", [](#__codelineno-2-136) }, [](#__codelineno-2-137) { [](#__codelineno-2-138) "role": "user", [](#__codelineno-2-139) "content": [ [](#__codelineno-2-140) {"type": "image_url", "image_url": {"url": url_3}}, [](#__codelineno-2-141) ], [](#__codelineno-2-142) }, [](#__codelineno-2-143) ] [](#__codelineno-2-144) [](#__codelineno-2-145) outputs = llm.chat(messages=messages, sampling_params=sampling_params) [](#__codelineno-2-146) print("-" * 50) [](#__codelineno-2-147) print(outputs[0].outputs[0].text) [](#__codelineno-2-148) print("-" * 50) [](#__codelineno-2-149) [](#__codelineno-2-150)[](#__codelineno-2-151)def parse_args(): [](#__codelineno-2-152) parser = argparse.ArgumentParser( [](#__codelineno-2-153) description="Run a demo in simple or advanced mode." [](#__codelineno-2-154) ) [](#__codelineno-2-155) [](#__codelineno-2-156) parser.add_argument( [](#__codelineno-2-157) "mode", [](#__codelineno-2-158) choices=["simple", "advanced"], [](#__codelineno-2-159) help="Specify the demo mode: 'simple' or 'advanced'", [](#__codelineno-2-160) ) [](#__codelineno-2-161) [](#__codelineno-2-162) parser.add_argument( [](#__codelineno-2-163) "--format", [](#__codelineno-2-164) choices=["mistral", "hf"], [](#__codelineno-2-165) default="mistral", [](#__codelineno-2-166) help="Specify the format of the model to load.", [](#__codelineno-2-167) ) [](#__codelineno-2-168) [](#__codelineno-2-169) parser.add_argument( [](#__codelineno-2-170) "--disable-mm-processor-cache", [](#__codelineno-2-171) action="store_true", [](#__codelineno-2-172) help="If True, disables caching of multi-modal processor.", [](#__codelineno-2-173) ) [](#__codelineno-2-174) return parser.parse_args() [](#__codelineno-2-175) [](#__codelineno-2-176)[](#__codelineno-2-177)def main(): [](#__codelineno-2-178) args = parse_args() [](#__codelineno-2-179) [](#__codelineno-2-180) if args.mode == "simple": [](#__codelineno-2-181) print("Running simple demo...") [](#__codelineno-2-182) run_simple_demo(args) [](#__codelineno-2-183) elif args.mode == "advanced": [](#__codelineno-2-184) print("Running advanced demo...") [](#__codelineno-2-185) run_advanced_demo(args) [](#__codelineno-2-186) [](#__codelineno-2-187)[](#__codelineno-2-188)if __name__ == "__main__": [](#__codelineno-2-189) main()` ## OpenAI Chat Completion Client For Multimodal[¶](#openai-chat-completion-client-for-multimodal "Permanent link") ``[](#__codelineno-3-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-3-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-3-3)"""An example showing how to use vLLM to serve multimodal models [](#__codelineno-3-4)and run online serving with OpenAI client. [](#__codelineno-3-5)[](#__codelineno-3-6)Launch the vLLM server with the following command: [](#__codelineno-3-7)[](#__codelineno-3-8)(single image inference with Llava) [](#__codelineno-3-9)vllm serve llava-hf/llava-1.5-7b-hf [](#__codelineno-3-10)[](#__codelineno-3-11)(multi-image inference with Phi-3.5-vision-instruct) [](#__codelineno-3-12)vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \ [](#__codelineno-3-13) --trust-remote-code --max-model-len 4096 --limit-mm-per-prompt.image 2 [](#__codelineno-3-14)[](#__codelineno-3-15)(audio inference with Ultravox) [](#__codelineno-3-16)vllm serve fixie-ai/ultravox-v0_5-llama-3_2-1b \ [](#__codelineno-3-17) --max-model-len 4096 --trust-remote-code [](#__codelineno-3-18)[](#__codelineno-3-19)run the script with [](#__codelineno-3-20)python openai_chat_completion_client_for_multimodal.py --chat-type audio [](#__codelineno-3-21)""" [](#__codelineno-3-22)[](#__codelineno-3-23)import os [](#__codelineno-3-24)[](#__codelineno-3-25)import pybase64 as base64 [](#__codelineno-3-26)import requests [](#__codelineno-3-27)from openai import OpenAI [](#__codelineno-3-28)[](#__codelineno-3-29)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-3-30)[](#__codelineno-3-31)# Modify OpenAI's API key and API base to use vLLM's API server. [](#__codelineno-3-32)openai_api_key = "EMPTY" [](#__codelineno-3-33)openai_api_base = "http://localhost:8000/v1" [](#__codelineno-3-34)[](#__codelineno-3-35)client = OpenAI( [](#__codelineno-3-36) # defaults to os.environ.get("OPENAI_API_KEY") [](#__codelineno-3-37) api_key=openai_api_key, [](#__codelineno-3-38) base_url=openai_api_base, [](#__codelineno-3-39)) [](#__codelineno-3-40)[](#__codelineno-3-41)headers = {"User-Agent": "vLLM Example Client"} [](#__codelineno-3-42) [](#__codelineno-3-43)[](#__codelineno-3-44)def encode_base64_content_from_url(content_url: str) -> str: [](#__codelineno-3-45) """Encode a content retrieved from a remote url to base64 format.""" [](#__codelineno-3-46) [](#__codelineno-3-47) with requests.get(content_url, headers=headers) as response: [](#__codelineno-3-48) response.raise_for_status() [](#__codelineno-3-49) result = base64.b64encode(response.content).decode("utf-8") [](#__codelineno-3-50) [](#__codelineno-3-51) return result [](#__codelineno-3-52) [](#__codelineno-3-53)[](#__codelineno-3-54)def encode_base64_content_from_file(file_path: str) -> str: [](#__codelineno-3-55) """Encode a local file content to base64 format.""" [](#__codelineno-3-56) [](#__codelineno-3-57) with open(file_path, "rb") as file: [](#__codelineno-3-58) file_content = file.read() [](#__codelineno-3-59) result = base64.b64encode(file_content).decode("utf-8") [](#__codelineno-3-60) [](#__codelineno-3-61) return result [](#__codelineno-3-62) [](#__codelineno-3-63)[](#__codelineno-3-64)# Text-only inference [](#__codelineno-3-65)def run_text_only(model: str, max_completion_tokens: int) -> None: [](#__codelineno-3-66) chat_completion = client.chat.completions.create( [](#__codelineno-3-67) messages=[{"role": "user", "content": "What's the capital of France?"}], [](#__codelineno-3-68) model=model, [](#__codelineno-3-69) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-70) ) [](#__codelineno-3-71) [](#__codelineno-3-72) result = chat_completion.choices[0].message.content [](#__codelineno-3-73) print("Chat completion output:\n", result) [](#__codelineno-3-74) [](#__codelineno-3-75)[](#__codelineno-3-76)# Single-image input inference [](#__codelineno-3-77)def run_single_image(model: str, max_completion_tokens: int) -> None: [](#__codelineno-3-78) ## Use image url in the payload [](#__codelineno-3-79) image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" [](#__codelineno-3-80) image_file = "/path/to/image.jpg" # local file [](#__codelineno-3-81) chat_completion_from_url = client.chat.completions.create( [](#__codelineno-3-82) messages=[ [](#__codelineno-3-83) { [](#__codelineno-3-84) "role": "user", [](#__codelineno-3-85) "content": [ [](#__codelineno-3-86) {"type": "text", "text": "What's in this image?"}, [](#__codelineno-3-87) { [](#__codelineno-3-88) "type": "image_url", [](#__codelineno-3-89) "image_url": {"url": image_url}, [](#__codelineno-3-90) }, [](#__codelineno-3-91) ], [](#__codelineno-3-92) } [](#__codelineno-3-93) ], [](#__codelineno-3-94) model=model, [](#__codelineno-3-95) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-96) ) [](#__codelineno-3-97) [](#__codelineno-3-98) result = chat_completion_from_url.choices[0].message.content [](#__codelineno-3-99) print("Chat completion output from image url:\n", result) [](#__codelineno-3-100) [](#__codelineno-3-101) ## Use local image url in the payload [](#__codelineno-3-102) # Launch the API server/engine with the --allowed-local-media-path argument. [](#__codelineno-3-103) if os.path.exists(image_file): [](#__codelineno-3-104) chat_completion_from_local_image_url = client.chat.completions.create( [](#__codelineno-3-105) messages=[ [](#__codelineno-3-106) { [](#__codelineno-3-107) "role": "user", [](#__codelineno-3-108) "content": [ [](#__codelineno-3-109) {"type": "text", "text": "What's in this image?"}, [](#__codelineno-3-110) { [](#__codelineno-3-111) "type": "image_url", [](#__codelineno-3-112) "image_url": {"url": f"file://{image_file}"}, [](#__codelineno-3-113) }, [](#__codelineno-3-114) ], [](#__codelineno-3-115) } [](#__codelineno-3-116) ], [](#__codelineno-3-117) model=model, [](#__codelineno-3-118) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-119) ) [](#__codelineno-3-120) result = chat_completion_from_local_image_url.choices[0].message.content [](#__codelineno-3-121) print("Chat completion output from local image file:\n", result) [](#__codelineno-3-122) else: [](#__codelineno-3-123) print(f"Local image file not found at {image_file}, skipping local file test.") [](#__codelineno-3-124) [](#__codelineno-3-125) ## Use base64 encoded image in the payload [](#__codelineno-3-126) image_base64 = encode_base64_content_from_url(image_url) [](#__codelineno-3-127) chat_completion_from_base64 = client.chat.completions.create( [](#__codelineno-3-128) messages=[ [](#__codelineno-3-129) { [](#__codelineno-3-130) "role": "user", [](#__codelineno-3-131) "content": [ [](#__codelineno-3-132) {"type": "text", "text": "What's in this image?"}, [](#__codelineno-3-133) { [](#__codelineno-3-134) "type": "image_url", [](#__codelineno-3-135) "image_url": {"url": f"data:image/jpeg;base64,{image_base64}"}, [](#__codelineno-3-136) }, [](#__codelineno-3-137) ], [](#__codelineno-3-138) } [](#__codelineno-3-139) ], [](#__codelineno-3-140) model=model, [](#__codelineno-3-141) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-142) ) [](#__codelineno-3-143) [](#__codelineno-3-144) result = chat_completion_from_base64.choices[0].message.content [](#__codelineno-3-145) print("Chat completion output from base64 encoded image:", result) [](#__codelineno-3-146) [](#__codelineno-3-147) ## Use base64 encoded local image in the payload [](#__codelineno-3-148) if os.path.exists(image_file): [](#__codelineno-3-149) local_image_base64 = encode_base64_content_from_file(image_file) [](#__codelineno-3-150) chat_completion_from_local_image_base64 = client.chat.completions.create( [](#__codelineno-3-151) messages=[ [](#__codelineno-3-152) { [](#__codelineno-3-153) "role": "user", [](#__codelineno-3-154) "content": [ [](#__codelineno-3-155) {"type": "text", "text": "What's in this image?"}, [](#__codelineno-3-156) { [](#__codelineno-3-157) "type": "image_url", [](#__codelineno-3-158) "image_url": { [](#__codelineno-3-159) "url": f"data:image/jpeg;base64,{local_image_base64}" [](#__codelineno-3-160) }, [](#__codelineno-3-161) }, [](#__codelineno-3-162) ], [](#__codelineno-3-163) } [](#__codelineno-3-164) ], [](#__codelineno-3-165) model=model, [](#__codelineno-3-166) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-167) ) [](#__codelineno-3-168) [](#__codelineno-3-169) result = chat_completion_from_local_image_base64.choices[0].message.content [](#__codelineno-3-170) print("Chat completion output from base64 encoded local image:", result) [](#__codelineno-3-171) else: [](#__codelineno-3-172) print(f"Local image file not found at {image_file}, skipping local file test.") [](#__codelineno-3-173) [](#__codelineno-3-174)[](#__codelineno-3-175)# Multi-image input inference [](#__codelineno-3-176)def run_multi_image(model: str, max_completion_tokens: int) -> None: [](#__codelineno-3-177) image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg" [](#__codelineno-3-178) image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg" [](#__codelineno-3-179) chat_completion_from_url = client.chat.completions.create( [](#__codelineno-3-180) messages=[ [](#__codelineno-3-181) { [](#__codelineno-3-182) "role": "user", [](#__codelineno-3-183) "content": [ [](#__codelineno-3-184) {"type": "text", "text": "What are the animals in these images?"}, [](#__codelineno-3-185) { [](#__codelineno-3-186) "type": "image_url", [](#__codelineno-3-187) "image_url": {"url": image_url_duck}, [](#__codelineno-3-188) }, [](#__codelineno-3-189) { [](#__codelineno-3-190) "type": "image_url", [](#__codelineno-3-191) "image_url": {"url": image_url_lion}, [](#__codelineno-3-192) }, [](#__codelineno-3-193) ], [](#__codelineno-3-194) } [](#__codelineno-3-195) ], [](#__codelineno-3-196) model=model, [](#__codelineno-3-197) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-198) ) [](#__codelineno-3-199) [](#__codelineno-3-200) result = chat_completion_from_url.choices[0].message.content [](#__codelineno-3-201) print("Chat completion output:\n", result) [](#__codelineno-3-202) [](#__codelineno-3-203)[](#__codelineno-3-204)# Video input inference [](#__codelineno-3-205)def run_video(model: str, max_completion_tokens: int) -> None: [](#__codelineno-3-206) video_url = "https://huggingface.co/datasets/raushan-testing-hf/videos-test/resolve/main/sample_demo_1.mp4" [](#__codelineno-3-207) video_base64 = encode_base64_content_from_url(video_url) [](#__codelineno-3-208) [](#__codelineno-3-209) ## Use video url in the payload [](#__codelineno-3-210) chat_completion_from_url = client.chat.completions.create( [](#__codelineno-3-211) messages=[ [](#__codelineno-3-212) { [](#__codelineno-3-213) "role": "user", [](#__codelineno-3-214) "content": [ [](#__codelineno-3-215) {"type": "text", "text": "What's in this video?"}, [](#__codelineno-3-216) { [](#__codelineno-3-217) "type": "video_url", [](#__codelineno-3-218) "video_url": {"url": video_url}, [](#__codelineno-3-219) }, [](#__codelineno-3-220) ], [](#__codelineno-3-221) } [](#__codelineno-3-222) ], [](#__codelineno-3-223) model=model, [](#__codelineno-3-224) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-225) ) [](#__codelineno-3-226) [](#__codelineno-3-227) result = chat_completion_from_url.choices[0].message.content [](#__codelineno-3-228) print("Chat completion output from video url:\n", result) [](#__codelineno-3-229) [](#__codelineno-3-230) ## Use base64 encoded video in the payload [](#__codelineno-3-231) chat_completion_from_base64 = client.chat.completions.create( [](#__codelineno-3-232) messages=[ [](#__codelineno-3-233) { [](#__codelineno-3-234) "role": "user", [](#__codelineno-3-235) "content": [ [](#__codelineno-3-236) {"type": "text", "text": "What's in this video?"}, [](#__codelineno-3-237) { [](#__codelineno-3-238) "type": "video_url", [](#__codelineno-3-239) "video_url": {"url": f"data:video/mp4;base64,{video_base64}"}, [](#__codelineno-3-240) }, [](#__codelineno-3-241) ], [](#__codelineno-3-242) } [](#__codelineno-3-243) ], [](#__codelineno-3-244) model=model, [](#__codelineno-3-245) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-246) ) [](#__codelineno-3-247) [](#__codelineno-3-248) result = chat_completion_from_base64.choices[0].message.content [](#__codelineno-3-249) print("Chat completion output from base64 encoded video:\n", result) [](#__codelineno-3-250) [](#__codelineno-3-251)[](#__codelineno-3-252)# Audio input inference [](#__codelineno-3-253)def run_audio(model: str, max_completion_tokens: int) -> None: [](#__codelineno-3-254) from vllm.assets.audio import AudioAsset [](#__codelineno-3-255) [](#__codelineno-3-256) audio_url = AudioAsset("winning_call").url [](#__codelineno-3-257) audio_base64 = encode_base64_content_from_url(audio_url) [](#__codelineno-3-258) [](#__codelineno-3-259) # OpenAI-compatible schema (`input_audio`) [](#__codelineno-3-260) chat_completion_from_base64 = client.chat.completions.create( [](#__codelineno-3-261) messages=[ [](#__codelineno-3-262) { [](#__codelineno-3-263) "role": "user", [](#__codelineno-3-264) "content": [ [](#__codelineno-3-265) {"type": "text", "text": "What's in this audio?"}, [](#__codelineno-3-266) { [](#__codelineno-3-267) "type": "input_audio", [](#__codelineno-3-268) "input_audio": { [](#__codelineno-3-269) # Any format supported by soundfile/PyAV is supported [](#__codelineno-3-270) "data": audio_base64, [](#__codelineno-3-271) "format": "wav", [](#__codelineno-3-272) }, [](#__codelineno-3-273) }, [](#__codelineno-3-274) ], [](#__codelineno-3-275) } [](#__codelineno-3-276) ], [](#__codelineno-3-277) model=model, [](#__codelineno-3-278) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-279) ) [](#__codelineno-3-280) [](#__codelineno-3-281) result = chat_completion_from_base64.choices[0].message.content [](#__codelineno-3-282) print("Chat completion output from input audio:\n", result) [](#__codelineno-3-283) [](#__codelineno-3-284) # HTTP URL [](#__codelineno-3-285) chat_completion_from_url = client.chat.completions.create( [](#__codelineno-3-286) messages=[ [](#__codelineno-3-287) { [](#__codelineno-3-288) "role": "user", [](#__codelineno-3-289) "content": [ [](#__codelineno-3-290) {"type": "text", "text": "What's in this audio?"}, [](#__codelineno-3-291) { [](#__codelineno-3-292) "type": "audio_url", [](#__codelineno-3-293) "audio_url": { [](#__codelineno-3-294) # Any format supported by soundfile/PyAV is supported [](#__codelineno-3-295) "url": audio_url [](#__codelineno-3-296) }, [](#__codelineno-3-297) }, [](#__codelineno-3-298) ], [](#__codelineno-3-299) } [](#__codelineno-3-300) ], [](#__codelineno-3-301) model=model, [](#__codelineno-3-302) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-303) ) [](#__codelineno-3-304) [](#__codelineno-3-305) result = chat_completion_from_url.choices[0].message.content [](#__codelineno-3-306) print("Chat completion output from audio url:\n", result) [](#__codelineno-3-307) [](#__codelineno-3-308) # base64 URL [](#__codelineno-3-309) chat_completion_from_base64 = client.chat.completions.create( [](#__codelineno-3-310) messages=[ [](#__codelineno-3-311) { [](#__codelineno-3-312) "role": "user", [](#__codelineno-3-313) "content": [ [](#__codelineno-3-314) {"type": "text", "text": "What's in this audio?"}, [](#__codelineno-3-315) { [](#__codelineno-3-316) "type": "audio_url", [](#__codelineno-3-317) "audio_url": { [](#__codelineno-3-318) # Any format supported by soundfile/PyAV is supported [](#__codelineno-3-319) "url": f"data:audio/ogg;base64,{audio_base64}" [](#__codelineno-3-320) }, [](#__codelineno-3-321) }, [](#__codelineno-3-322) ], [](#__codelineno-3-323) } [](#__codelineno-3-324) ], [](#__codelineno-3-325) model=model, [](#__codelineno-3-326) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-327) ) [](#__codelineno-3-328) [](#__codelineno-3-329) result = chat_completion_from_base64.choices[0].message.content [](#__codelineno-3-330) print("Chat completion output from base64 encoded audio:\n", result) [](#__codelineno-3-331) [](#__codelineno-3-332)[](#__codelineno-3-333)def run_multi_audio(model: str, max_completion_tokens: int) -> None: [](#__codelineno-3-334) from vllm.assets.audio import AudioAsset [](#__codelineno-3-335) [](#__codelineno-3-336) # Two different audios to showcase batched inference. [](#__codelineno-3-337) audio_url = AudioAsset("winning_call").url [](#__codelineno-3-338) audio_base64 = encode_base64_content_from_url(audio_url) [](#__codelineno-3-339) audio_url2 = AudioAsset("azacinto_foscolo").url [](#__codelineno-3-340) audio_base64_2 = encode_base64_content_from_url(audio_url2) [](#__codelineno-3-341) [](#__codelineno-3-342) # OpenAI-compatible schema (`input_audio`) [](#__codelineno-3-343) chat_completion_from_base64 = client.chat.completions.create( [](#__codelineno-3-344) messages=[ [](#__codelineno-3-345) { [](#__codelineno-3-346) "role": "user", [](#__codelineno-3-347) "content": [ [](#__codelineno-3-348) {"type": "text", "text": "Are these two audios the same?"}, [](#__codelineno-3-349) { [](#__codelineno-3-350) "type": "input_audio", [](#__codelineno-3-351) "input_audio": { [](#__codelineno-3-352) "data": audio_base64, [](#__codelineno-3-353) "format": "wav", [](#__codelineno-3-354) }, [](#__codelineno-3-355) }, [](#__codelineno-3-356) { [](#__codelineno-3-357) "type": "input_audio", [](#__codelineno-3-358) "input_audio": { [](#__codelineno-3-359) "data": audio_base64_2, [](#__codelineno-3-360) "format": "wav", [](#__codelineno-3-361) }, [](#__codelineno-3-362) }, [](#__codelineno-3-363) ], [](#__codelineno-3-364) } [](#__codelineno-3-365) ], [](#__codelineno-3-366) model=model, [](#__codelineno-3-367) max_completion_tokens=max_completion_tokens, [](#__codelineno-3-368) ) [](#__codelineno-3-369) [](#__codelineno-3-370) result = chat_completion_from_base64.choices[0].message.content [](#__codelineno-3-371) print("Chat completion output from input audio:\n", result) [](#__codelineno-3-372) [](#__codelineno-3-373)[](#__codelineno-3-374)example_function_map = { [](#__codelineno-3-375) "text-only": run_text_only, [](#__codelineno-3-376) "single-image": run_single_image, [](#__codelineno-3-377) "multi-image": run_multi_image, [](#__codelineno-3-378) "multi-audio": run_multi_audio, [](#__codelineno-3-379) "video": run_video, [](#__codelineno-3-380) "audio": run_audio, [](#__codelineno-3-381)} [](#__codelineno-3-382) [](#__codelineno-3-383)[](#__codelineno-3-384)def parse_args(): [](#__codelineno-3-385) parser = FlexibleArgumentParser( [](#__codelineno-3-386) description="Demo on using OpenAI client for online serving with " [](#__codelineno-3-387) "multimodal language models served with vLLM." [](#__codelineno-3-388) ) [](#__codelineno-3-389) parser.add_argument( [](#__codelineno-3-390) "--chat-type", [](#__codelineno-3-391) "-c", [](#__codelineno-3-392) type=str, [](#__codelineno-3-393) default="single-image", [](#__codelineno-3-394) choices=list(example_function_map.keys()), [](#__codelineno-3-395) help="Conversation type with multimodal data.", [](#__codelineno-3-396) ) [](#__codelineno-3-397) parser.add_argument( [](#__codelineno-3-398) "--max-completion-tokens", [](#__codelineno-3-399) "-n", [](#__codelineno-3-400) type=int, [](#__codelineno-3-401) default=128, [](#__codelineno-3-402) help="Maximum number of tokens to generate for each completion.", [](#__codelineno-3-403) ) [](#__codelineno-3-404) return parser.parse_args() [](#__codelineno-3-405) [](#__codelineno-3-406)[](#__codelineno-3-407)def main(args) -> None: [](#__codelineno-3-408) chat_type = args.chat_type [](#__codelineno-3-409) model = client.models.list().data[0].id [](#__codelineno-3-410) example_function_map[chat_type](model, args.max_completion_tokens) [](#__codelineno-3-411) [](#__codelineno-3-412)[](#__codelineno-3-413)if __name__ == "__main__": [](#__codelineno-3-414) args = parse_args() [](#__codelineno-3-415) main(args)`` ## Qwen2 5 Omni - Readme[¶](#qwen2-5-omni-readme "Permanent link") ` [](#__codelineno-4-1)# Qwen2.5-Omni Offline Inference Examples [](#__codelineno-4-2)[](#__codelineno-4-3)This folder provides several example scripts on how to inference Qwen2.5-Omni offline. [](#__codelineno-4-4)[](#__codelineno-4-5)## Thinker Only [](#__codelineno-4-6)[](#__codelineno-4-7)```bash [](#__codelineno-4-8)# Audio + image + video [](#__codelineno-4-9)python examples/generate/multimodal/qwen2_5_omni/only_thinker.py \ [](#__codelineno-4-10) -q mixed_modalities [](#__codelineno-4-11)[](#__codelineno-4-12)# Read vision and audio inputs from a single video file [](#__codelineno-4-13)python examples/generate/multimodal/qwen2_5_omni/only_thinker.py \ [](#__codelineno-4-14) -q use_audio_in_video [](#__codelineno-4-15)[](#__codelineno-4-16)# Multiple audios [](#__codelineno-4-17)python examples/generate/multimodal/qwen2_5_omni/only_thinker.py \ [](#__codelineno-4-18) -q multi_audios [](#__codelineno-4-19)``` [](#__codelineno-4-20)[](#__codelineno-4-21)This script will run the thinker part of Qwen2.5-Omni, and generate text response. [](#__codelineno-4-22)[](#__codelineno-4-23)You can also test Qwen2.5-Omni on a single modality: [](#__codelineno-4-24)[](#__codelineno-4-25)```bash [](#__codelineno-4-26)# Process audio inputs [](#__codelineno-4-27)python examples/generate/multimodal/audio_language_offline.py \ [](#__codelineno-4-28) --model-type qwen2_5_omni [](#__codelineno-4-29)[](#__codelineno-4-30)# Process image inputs [](#__codelineno-4-31)python examples/generate/multimodal/vision_language_offline.py \ [](#__codelineno-4-32) --modality image \ [](#__codelineno-4-33) --model-type qwen2_5_omni [](#__codelineno-4-34)[](#__codelineno-4-35)# Process video inputs [](#__codelineno-4-36)python examples/generate/multimodal/vision_language_offline.py \ [](#__codelineno-4-37) --modality video \ [](#__codelineno-4-38) --model-type qwen2_5_omni [](#__codelineno-4-39)``` ` ## Qwen2 5 Omni - Only Thinker[¶](#qwen2-5-omni-only-thinker "Permanent link") ``[](#__codelineno-5-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-5-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-5-3)""" [](#__codelineno-5-4)This example shows how to use vLLM for running offline inference [](#__codelineno-5-5)with the correct prompt format on Qwen2.5-Omni (thinker only). [](#__codelineno-5-6)""" [](#__codelineno-5-7)[](#__codelineno-5-8)from typing import NamedTuple [](#__codelineno-5-9)[](#__codelineno-5-10)from vllm import LLM, SamplingParams [](#__codelineno-5-11)from vllm.assets.audio import AudioAsset [](#__codelineno-5-12)from vllm.assets.image import ImageAsset [](#__codelineno-5-13)from vllm.assets.video import VideoAsset [](#__codelineno-5-14)from vllm.multimodal.image import convert_image_mode [](#__codelineno-5-15)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-5-16) [](#__codelineno-5-17)[](#__codelineno-5-18)class QueryResult(NamedTuple): [](#__codelineno-5-19) inputs: dict [](#__codelineno-5-20) limit_mm_per_prompt: dict[str, int] [](#__codelineno-5-21) [](#__codelineno-5-22)[](#__codelineno-5-23)# NOTE: The default `max_num_seqs` and `max_model_len` may result in OOM on [](#__codelineno-5-24)# lower-end GPUs. [](#__codelineno-5-25)# Unless specified, these settings have been tested to work on a single L4. [](#__codelineno-5-26)[](#__codelineno-5-27)default_system = ( [](#__codelineno-5-28) "You are Qwen, a virtual human developed by the Qwen Team, Alibaba " [](#__codelineno-5-29) "Group, capable of perceiving auditory and visual inputs, as well as " [](#__codelineno-5-30) "generating text and speech." [](#__codelineno-5-31)) [](#__codelineno-5-32) [](#__codelineno-5-33)[](#__codelineno-5-34)def get_mixed_modalities_query() -> QueryResult: [](#__codelineno-5-35) question = ( [](#__codelineno-5-36) "What is recited in the audio? " [](#__codelineno-5-37) "What is the content of this image? Why is this video funny?" [](#__codelineno-5-38) ) [](#__codelineno-5-39) prompt = ( [](#__codelineno-5-40) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-5-41) "<|im_start|>user\n<|audio_bos|><|AUDIO|><|audio_eos|>" [](#__codelineno-5-42) "<|vision_bos|><|IMAGE|><|vision_eos|>" [](#__codelineno-5-43) "<|vision_bos|><|VIDEO|><|vision_eos|>" [](#__codelineno-5-44) f"{question}<|im_end|>\n" [](#__codelineno-5-45) f"<|im_start|>assistant\n" [](#__codelineno-5-46) ) [](#__codelineno-5-47) return QueryResult( [](#__codelineno-5-48) inputs={ [](#__codelineno-5-49) "prompt": prompt, [](#__codelineno-5-50) "multi_modal_data": { [](#__codelineno-5-51) "audio": AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-5-52) "image": convert_image_mode( [](#__codelineno-5-53) ImageAsset("cherry_blossom").pil_image, "RGB" [](#__codelineno-5-54) ), [](#__codelineno-5-55) "video": VideoAsset(name="baby_reading", num_frames=16).np_ndarrays, [](#__codelineno-5-56) }, [](#__codelineno-5-57) }, [](#__codelineno-5-58) limit_mm_per_prompt={"audio": 1, "image": 1, "video": 1}, [](#__codelineno-5-59) ) [](#__codelineno-5-60) [](#__codelineno-5-61)[](#__codelineno-5-62)def get_use_audio_in_video_query() -> QueryResult: [](#__codelineno-5-63) question = ( [](#__codelineno-5-64) "Describe the content of the video, then convert what the baby say into text." [](#__codelineno-5-65) ) [](#__codelineno-5-66) prompt = ( [](#__codelineno-5-67) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-5-68) "<|im_start|>user\n<|vision_bos|><|VIDEO|><|vision_eos|>" [](#__codelineno-5-69) f"{question}<|im_end|>\n" [](#__codelineno-5-70) f"<|im_start|>assistant\n" [](#__codelineno-5-71) ) [](#__codelineno-5-72) asset = VideoAsset(name="baby_reading", num_frames=16) [](#__codelineno-5-73) audio = asset.get_audio(sampling_rate=16000) [](#__codelineno-5-74) [](#__codelineno-5-75) return QueryResult( [](#__codelineno-5-76) inputs={ [](#__codelineno-5-77) "prompt": prompt, [](#__codelineno-5-78) "multi_modal_data": { [](#__codelineno-5-79) "video": asset.np_ndarrays, [](#__codelineno-5-80) "audio": audio, [](#__codelineno-5-81) }, [](#__codelineno-5-82) "mm_processor_kwargs": { [](#__codelineno-5-83) "use_audio_in_video": True, [](#__codelineno-5-84) }, [](#__codelineno-5-85) }, [](#__codelineno-5-86) limit_mm_per_prompt={"audio": 1, "video": 1}, [](#__codelineno-5-87) ) [](#__codelineno-5-88) [](#__codelineno-5-89)[](#__codelineno-5-90)def get_multi_audios_query() -> QueryResult: [](#__codelineno-5-91) question = "Are these two audio clips the same?" [](#__codelineno-5-92) prompt = ( [](#__codelineno-5-93) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-5-94) "<|im_start|>user\n<|audio_bos|><|AUDIO|><|audio_eos|>" [](#__codelineno-5-95) "<|audio_bos|><|AUDIO|><|audio_eos|>" [](#__codelineno-5-96) f"{question}<|im_end|>\n" [](#__codelineno-5-97) f"<|im_start|>assistant\n" [](#__codelineno-5-98) ) [](#__codelineno-5-99) return QueryResult( [](#__codelineno-5-100) inputs={ [](#__codelineno-5-101) "prompt": prompt, [](#__codelineno-5-102) "multi_modal_data": { [](#__codelineno-5-103) "audio": [ [](#__codelineno-5-104) AudioAsset("winning_call").audio_and_sample_rate, [](#__codelineno-5-105) AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-5-106) ], [](#__codelineno-5-107) }, [](#__codelineno-5-108) }, [](#__codelineno-5-109) limit_mm_per_prompt={ [](#__codelineno-5-110) "audio": 2, [](#__codelineno-5-111) }, [](#__codelineno-5-112) ) [](#__codelineno-5-113) [](#__codelineno-5-114)[](#__codelineno-5-115)def get_multi_images_query() -> QueryResult: [](#__codelineno-5-116) question = "What are the differences between these two images?" [](#__codelineno-5-117) prompt = ( [](#__codelineno-5-118) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-5-119) "<|im_start|>user\n<|vision_bos|><|IMAGE|><|vision_eos|>" [](#__codelineno-5-120) "<|vision_bos|><|IMAGE|><|vision_eos|>" [](#__codelineno-5-121) f"{question}<|im_end|>\n" [](#__codelineno-5-122) f"<|im_start|>assistant\n" [](#__codelineno-5-123) ) [](#__codelineno-5-124) return QueryResult( [](#__codelineno-5-125) inputs={ [](#__codelineno-5-126) "prompt": prompt, [](#__codelineno-5-127) "multi_modal_data": { [](#__codelineno-5-128) "image": [ [](#__codelineno-5-129) convert_image_mode(ImageAsset("cherry_blossom").pil_image, "RGB"), [](#__codelineno-5-130) convert_image_mode(ImageAsset("stop_sign").pil_image, "RGB"), [](#__codelineno-5-131) ], [](#__codelineno-5-132) }, [](#__codelineno-5-133) }, [](#__codelineno-5-134) limit_mm_per_prompt={ [](#__codelineno-5-135) "image": 2, [](#__codelineno-5-136) }, [](#__codelineno-5-137) ) [](#__codelineno-5-138) [](#__codelineno-5-139)[](#__codelineno-5-140)query_map = { [](#__codelineno-5-141) "mixed_modalities": get_mixed_modalities_query, [](#__codelineno-5-142) "use_audio_in_video": get_use_audio_in_video_query, [](#__codelineno-5-143) "multi_audios": get_multi_audios_query, [](#__codelineno-5-144) "multi_images": get_multi_images_query, [](#__codelineno-5-145)} [](#__codelineno-5-146) [](#__codelineno-5-147)[](#__codelineno-5-148)def main(args): [](#__codelineno-5-149) model_name = "Qwen/Qwen2.5-Omni-7B" [](#__codelineno-5-150) query_result = query_map[args.query_type]() [](#__codelineno-5-151) [](#__codelineno-5-152) llm = LLM( [](#__codelineno-5-153) model=model_name, [](#__codelineno-5-154) max_model_len=5632, [](#__codelineno-5-155) max_num_seqs=5, [](#__codelineno-5-156) limit_mm_per_prompt=query_result.limit_mm_per_prompt, [](#__codelineno-5-157) seed=args.seed, [](#__codelineno-5-158) ) [](#__codelineno-5-159) [](#__codelineno-5-160) # We set temperature to 0.2 so that outputs can be different [](#__codelineno-5-161) # even when all prompts are identical when running batch inference. [](#__codelineno-5-162) sampling_params = SamplingParams(temperature=0.2, max_tokens=64) [](#__codelineno-5-163) [](#__codelineno-5-164) outputs = llm.generate(query_result.inputs, sampling_params=sampling_params) [](#__codelineno-5-165) [](#__codelineno-5-166) for o in outputs: [](#__codelineno-5-167) generated_text = o.outputs[0].text [](#__codelineno-5-168) print(generated_text) [](#__codelineno-5-169) [](#__codelineno-5-170)[](#__codelineno-5-171)def parse_args(): [](#__codelineno-5-172) parser = FlexibleArgumentParser( [](#__codelineno-5-173) description="Demo on using vLLM for offline inference with " [](#__codelineno-5-174) "audio language models" [](#__codelineno-5-175) ) [](#__codelineno-5-176) parser.add_argument( [](#__codelineno-5-177) "--query-type", [](#__codelineno-5-178) "-q", [](#__codelineno-5-179) type=str, [](#__codelineno-5-180) default="mixed_modalities", [](#__codelineno-5-181) choices=query_map.keys(), [](#__codelineno-5-182) help="Query type.", [](#__codelineno-5-183) ) [](#__codelineno-5-184) parser.add_argument( [](#__codelineno-5-185) "--seed", [](#__codelineno-5-186) type=int, [](#__codelineno-5-187) default=0, [](#__codelineno-5-188) help="Set the seed when initializing `vllm.LLM`.", [](#__codelineno-5-189) ) [](#__codelineno-5-190) [](#__codelineno-5-191) return parser.parse_args() [](#__codelineno-5-192) [](#__codelineno-5-193)[](#__codelineno-5-194)if __name__ == "__main__": [](#__codelineno-5-195) args = parse_args() [](#__codelineno-5-196) main(args)`` ## Qwen3 Omni - Only Thinker[¶](#qwen3-omni-only-thinker "Permanent link") ``[](#__codelineno-6-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-6-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-6-3)""" [](#__codelineno-6-4)This example shows how to use vLLM for running offline inference [](#__codelineno-6-5)with the correct prompt format on Qwen3-Omni (thinker only). [](#__codelineno-6-6)""" [](#__codelineno-6-7)[](#__codelineno-6-8)from typing import NamedTuple [](#__codelineno-6-9)[](#__codelineno-6-10)from vllm import LLM, SamplingParams [](#__codelineno-6-11)from vllm.assets.audio import AudioAsset [](#__codelineno-6-12)from vllm.assets.image import ImageAsset [](#__codelineno-6-13)from vllm.assets.video import VideoAsset [](#__codelineno-6-14)from vllm.multimodal.image import convert_image_mode [](#__codelineno-6-15)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-6-16) [](#__codelineno-6-17)[](#__codelineno-6-18)class QueryResult(NamedTuple): [](#__codelineno-6-19) inputs: dict [](#__codelineno-6-20) limit_mm_per_prompt: dict[str, int] [](#__codelineno-6-21) [](#__codelineno-6-22)[](#__codelineno-6-23)# NOTE: The default `max_num_seqs` and `max_model_len` may result in OOM on [](#__codelineno-6-24)# lower-end GPUs. [](#__codelineno-6-25)# Unless specified, these settings have been tested to work on a single L4. [](#__codelineno-6-26)[](#__codelineno-6-27)default_system = ( [](#__codelineno-6-28) "You are Qwen, a virtual human developed by the Qwen Team, Alibaba " [](#__codelineno-6-29) "Group, capable of perceiving auditory and visual inputs, as well as " [](#__codelineno-6-30) "generating text and speech." [](#__codelineno-6-31)) [](#__codelineno-6-32) [](#__codelineno-6-33)[](#__codelineno-6-34)def get_mixed_modalities_query() -> QueryResult: [](#__codelineno-6-35) question = ( [](#__codelineno-6-36) "What is recited in the audio? " [](#__codelineno-6-37) "What is the content of this image? Why is this video funny?" [](#__codelineno-6-38) ) [](#__codelineno-6-39) prompt = ( [](#__codelineno-6-40) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-6-41) "<|im_start|>user\n<|audio_start|><|audio_pad|><|audio_end|>" [](#__codelineno-6-42) "<|vision_start|><|image_pad|><|vision_end|>" [](#__codelineno-6-43) "<|vision_start|><|video_pad|><|vision_end|>" [](#__codelineno-6-44) f"{question}<|im_end|>\n" [](#__codelineno-6-45) f"<|im_start|>assistant\n" [](#__codelineno-6-46) ) [](#__codelineno-6-47) return QueryResult( [](#__codelineno-6-48) inputs={ [](#__codelineno-6-49) "prompt": prompt, [](#__codelineno-6-50) "multi_modal_data": { [](#__codelineno-6-51) "audio": AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-6-52) "image": convert_image_mode( [](#__codelineno-6-53) ImageAsset("cherry_blossom").pil_image, "RGB" [](#__codelineno-6-54) ), [](#__codelineno-6-55) "video": VideoAsset(name="baby_reading", num_frames=16).np_ndarrays, [](#__codelineno-6-56) }, [](#__codelineno-6-57) }, [](#__codelineno-6-58) limit_mm_per_prompt={"audio": 1, "image": 1, "video": 1}, [](#__codelineno-6-59) ) [](#__codelineno-6-60) [](#__codelineno-6-61)[](#__codelineno-6-62)def get_use_audio_in_video_query() -> QueryResult: [](#__codelineno-6-63) question = ( [](#__codelineno-6-64) "Describe the content of the video in details, then convert what the " [](#__codelineno-6-65) "baby say into text." [](#__codelineno-6-66) ) [](#__codelineno-6-67) prompt = ( [](#__codelineno-6-68) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-6-69) "<|im_start|>user\n<|vision_start|><|video_pad|><|vision_end|>" [](#__codelineno-6-70) f"{question}<|im_end|>\n" [](#__codelineno-6-71) f"<|im_start|>assistant\n" [](#__codelineno-6-72) ) [](#__codelineno-6-73) asset = VideoAsset(name="baby_reading", num_frames=16) [](#__codelineno-6-74) audio = asset.get_audio(sampling_rate=16000) [](#__codelineno-6-75) return QueryResult( [](#__codelineno-6-76) inputs={ [](#__codelineno-6-77) "prompt": prompt, [](#__codelineno-6-78) "multi_modal_data": { [](#__codelineno-6-79) "video": asset.np_ndarrays, [](#__codelineno-6-80) "audio": audio, [](#__codelineno-6-81) }, [](#__codelineno-6-82) "mm_processor_kwargs": { [](#__codelineno-6-83) "use_audio_in_video": True, [](#__codelineno-6-84) }, [](#__codelineno-6-85) }, [](#__codelineno-6-86) limit_mm_per_prompt={"audio": 1, "video": 1}, [](#__codelineno-6-87) ) [](#__codelineno-6-88) [](#__codelineno-6-89)[](#__codelineno-6-90)def get_multi_audios_query() -> QueryResult: [](#__codelineno-6-91) question = "Are these two audio clips the same?" [](#__codelineno-6-92) prompt = ( [](#__codelineno-6-93) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-6-94) "<|im_start|>user\n<|audio_start|><|audio_pad|><|audio_end|>" [](#__codelineno-6-95) "<|audio_start|><|audio_pad|><|audio_end|>" [](#__codelineno-6-96) f"{question}<|im_end|>\n" [](#__codelineno-6-97) f"<|im_start|>assistant\n" [](#__codelineno-6-98) ) [](#__codelineno-6-99) return QueryResult( [](#__codelineno-6-100) inputs={ [](#__codelineno-6-101) "prompt": prompt, [](#__codelineno-6-102) "multi_modal_data": { [](#__codelineno-6-103) "audio": [ [](#__codelineno-6-104) AudioAsset("winning_call").audio_and_sample_rate, [](#__codelineno-6-105) AudioAsset("mary_had_lamb").audio_and_sample_rate, [](#__codelineno-6-106) ], [](#__codelineno-6-107) }, [](#__codelineno-6-108) }, [](#__codelineno-6-109) limit_mm_per_prompt={ [](#__codelineno-6-110) "audio": 2, [](#__codelineno-6-111) }, [](#__codelineno-6-112) ) [](#__codelineno-6-113) [](#__codelineno-6-114)[](#__codelineno-6-115)def get_multi_images_query() -> QueryResult: [](#__codelineno-6-116) question = "What are the differences between these two images?" [](#__codelineno-6-117) prompt = ( [](#__codelineno-6-118) f"<|im_start|>system\n{default_system}<|im_end|>\n" [](#__codelineno-6-119) "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>" [](#__codelineno-6-120) "<|vision_start|><|image_pad|><|vision_end|>" [](#__codelineno-6-121) f"{question}<|im_end|>\n" [](#__codelineno-6-122) f"<|im_start|>assistant\n" [](#__codelineno-6-123) ) [](#__codelineno-6-124) return QueryResult( [](#__codelineno-6-125) inputs={ [](#__codelineno-6-126) "prompt": prompt, [](#__codelineno-6-127) "multi_modal_data": { [](#__codelineno-6-128) "image": [ [](#__codelineno-6-129) convert_image_mode(ImageAsset("cherry_blossom").pil_image, "RGB"), [](#__codelineno-6-130) convert_image_mode(ImageAsset("stop_sign").pil_image, "RGB"), [](#__codelineno-6-131) ], [](#__codelineno-6-132) }, [](#__codelineno-6-133) }, [](#__codelineno-6-134) limit_mm_per_prompt={ [](#__codelineno-6-135) "image": 2, [](#__codelineno-6-136) }, [](#__codelineno-6-137) ) [](#__codelineno-6-138) [](#__codelineno-6-139)[](#__codelineno-6-140)query_map = { [](#__codelineno-6-141) "mixed_modalities": get_mixed_modalities_query, [](#__codelineno-6-142) "use_audio_in_video": get_use_audio_in_video_query, [](#__codelineno-6-143) "multi_audios": get_multi_audios_query, [](#__codelineno-6-144) "multi_images": get_multi_images_query, [](#__codelineno-6-145)} [](#__codelineno-6-146) [](#__codelineno-6-147)[](#__codelineno-6-148)def main(args): [](#__codelineno-6-149) model_name = args.model [](#__codelineno-6-150) query_result = query_map[args.query_type]() [](#__codelineno-6-151) [](#__codelineno-6-152) llm = LLM( [](#__codelineno-6-153) model=model_name, [](#__codelineno-6-154) max_model_len=args.max_model_len, [](#__codelineno-6-155) max_num_seqs=5, [](#__codelineno-6-156) limit_mm_per_prompt=query_result.limit_mm_per_prompt, [](#__codelineno-6-157) seed=args.seed, [](#__codelineno-6-158) tensor_parallel_size=args.tensor_parallel_size, [](#__codelineno-6-159) gpu_memory_utilization=args.gpu_memory_utilization, [](#__codelineno-6-160) ) [](#__codelineno-6-161) [](#__codelineno-6-162) # We set temperature to 0.2 so that outputs can be different [](#__codelineno-6-163) # even when all prompts are identical when running batch inference. [](#__codelineno-6-164) sampling_params = SamplingParams(temperature=0.2, max_tokens=256) [](#__codelineno-6-165) [](#__codelineno-6-166) outputs = llm.generate(query_result.inputs, sampling_params=sampling_params) [](#__codelineno-6-167) [](#__codelineno-6-168) for o in outputs: [](#__codelineno-6-169) generated_text = o.outputs[0].text [](#__codelineno-6-170) print(generated_text) [](#__codelineno-6-171) [](#__codelineno-6-172)[](#__codelineno-6-173)def parse_args(): [](#__codelineno-6-174) parser = FlexibleArgumentParser( [](#__codelineno-6-175) description="Demo on using vLLM for offline inference with " [](#__codelineno-6-176) "audio language models" [](#__codelineno-6-177) ) [](#__codelineno-6-178) parser.add_argument( [](#__codelineno-6-179) "--query-type", [](#__codelineno-6-180) "-q", [](#__codelineno-6-181) type=str, [](#__codelineno-6-182) default="mixed_modalities", [](#__codelineno-6-183) choices=query_map.keys(), [](#__codelineno-6-184) help="Query type.", [](#__codelineno-6-185) ) [](#__codelineno-6-186) parser.add_argument( [](#__codelineno-6-187) "--seed", [](#__codelineno-6-188) type=int, [](#__codelineno-6-189) default=0, [](#__codelineno-6-190) help="Set the seed when initializing `vllm.LLM`.", [](#__codelineno-6-191) ) [](#__codelineno-6-192) parser.add_argument( [](#__codelineno-6-193) "--model", [](#__codelineno-6-194) type=str, [](#__codelineno-6-195) default="Qwen/Qwen3-Omni-30B-A3B-Instruct", [](#__codelineno-6-196) help="Model name or path.", [](#__codelineno-6-197) ) [](#__codelineno-6-198) parser.add_argument( [](#__codelineno-6-199) "--tensor-parallel-size", [](#__codelineno-6-200) "-tp", [](#__codelineno-6-201) type=int, [](#__codelineno-6-202) default=1, [](#__codelineno-6-203) help="Tensor parallel size for distributed inference.", [](#__codelineno-6-204) ) [](#__codelineno-6-205) parser.add_argument( [](#__codelineno-6-206) "--gpu-memory-utilization", [](#__codelineno-6-207) type=float, [](#__codelineno-6-208) default=0.9, [](#__codelineno-6-209) help="GPU memory utilization (0.0 to 1.0).", [](#__codelineno-6-210) ) [](#__codelineno-6-211) parser.add_argument( [](#__codelineno-6-212) "--max-model-len", [](#__codelineno-6-213) type=int, [](#__codelineno-6-214) default=12800, [](#__codelineno-6-215) help="Maximum model context length.", [](#__codelineno-6-216) ) [](#__codelineno-6-217) [](#__codelineno-6-218) return parser.parse_args() [](#__codelineno-6-219) [](#__codelineno-6-220)[](#__codelineno-6-221)if __name__ == "__main__": [](#__codelineno-6-222) args = parse_args() [](#__codelineno-6-223) main(args)`` ## Vision Language Multi Image Offline[¶](#vision-language-multi-image-offline "Permanent link") ``[](#__codelineno-7-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-7-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-7-3)""" [](#__codelineno-7-4)This example shows how to use vLLM for running offline inference with [](#__codelineno-7-5)multi-image input on vision language models for text generation, [](#__codelineno-7-6)using the chat template defined by the model. [](#__codelineno-7-7)""" [](#__codelineno-7-8)[](#__codelineno-7-9)import os [](#__codelineno-7-10)from argparse import Namespace [](#__codelineno-7-11)from typing import NamedTuple [](#__codelineno-7-12)[](#__codelineno-7-13)from huggingface_hub import snapshot_download [](#__codelineno-7-14)from PIL.Image import Image [](#__codelineno-7-15)from transformers import AutoProcessor, AutoTokenizer [](#__codelineno-7-16)[](#__codelineno-7-17)from vllm import LLM, EngineArgs, SamplingParams [](#__codelineno-7-18)from vllm.lora.request import LoRARequest [](#__codelineno-7-19)from vllm.multimodal.utils import fetch_image [](#__codelineno-7-20)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-7-21)[](#__codelineno-7-22)QUESTION = "What is the content of each image?" [](#__codelineno-7-23)IMAGE_URLS = [ [](#__codelineno-7-24) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg", [](#__codelineno-7-25) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg", [](#__codelineno-7-26) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/flycatcher.jpeg", [](#__codelineno-7-27) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/somefish.jpg", [](#__codelineno-7-28) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/starfish.jpg", [](#__codelineno-7-29) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/snail.jpg", [](#__codelineno-7-30) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/thistle.jpg", [](#__codelineno-7-31) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/husky.jpg", [](#__codelineno-7-32) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/orangetabbycat.jpg", [](#__codelineno-7-33) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/guineapig.jpg", [](#__codelineno-7-34) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/rabbit.jpg", [](#__codelineno-7-35) "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/horsepony.jpg", [](#__codelineno-7-36)] [](#__codelineno-7-37) [](#__codelineno-7-38)[](#__codelineno-7-39)class ModelRequestData(NamedTuple): [](#__codelineno-7-40) engine_args: EngineArgs [](#__codelineno-7-41) prompt: str [](#__codelineno-7-42) image_data: list[Image] [](#__codelineno-7-43) stop_token_ids: list[int] | None = None [](#__codelineno-7-44) chat_template: str | None = None [](#__codelineno-7-45) lora_requests: list[LoRARequest] | None = None [](#__codelineno-7-46) sampling_params: SamplingParams | None = None [](#__codelineno-7-47) [](#__codelineno-7-48)[](#__codelineno-7-49)# NOTE: The default `max_num_seqs` and `max_model_len` may result in OOM on [](#__codelineno-7-50)# lower-end GPUs. [](#__codelineno-7-51)# Unless specified, these settings have been tested to work on a single L4. [](#__codelineno-7-52) [](#__codelineno-7-53)[](#__codelineno-7-54)def load_aria(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-55) model_name = "rhymes-ai/Aria" [](#__codelineno-7-56) engine_args = EngineArgs( [](#__codelineno-7-57) model=model_name, [](#__codelineno-7-58) tokenizer_mode="slow", [](#__codelineno-7-59) trust_remote_code=True, [](#__codelineno-7-60) dtype="bfloat16", [](#__codelineno-7-61) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-62) ) [](#__codelineno-7-63) placeholders = "<|img|>\n" * len(image_urls) [](#__codelineno-7-64) prompt = ( [](#__codelineno-7-65) f"<|im_start|>user\n{placeholders}{question}<|im_end|>\n<|im_start|>assistant\n" [](#__codelineno-7-66) ) [](#__codelineno-7-67) stop_token_ids = [93532, 93653, 944, 93421, 1019, 93653, 93519] [](#__codelineno-7-68) [](#__codelineno-7-69) return ModelRequestData( [](#__codelineno-7-70) engine_args=engine_args, [](#__codelineno-7-71) prompt=prompt, [](#__codelineno-7-72) stop_token_ids=stop_token_ids, [](#__codelineno-7-73) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-74) ) [](#__codelineno-7-75) [](#__codelineno-7-76)[](#__codelineno-7-77)def load_aya_vision(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-78) model_name = "CohereLabs/aya-vision-8b" [](#__codelineno-7-79) [](#__codelineno-7-80) engine_args = EngineArgs( [](#__codelineno-7-81) model=model_name, [](#__codelineno-7-82) max_num_seqs=2, [](#__codelineno-7-83) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-84) ) [](#__codelineno-7-85) [](#__codelineno-7-86) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-87) messages = [ [](#__codelineno-7-88) { [](#__codelineno-7-89) "role": "user", [](#__codelineno-7-90) "content": [ [](#__codelineno-7-91) *placeholders, [](#__codelineno-7-92) {"type": "text", "text": question}, [](#__codelineno-7-93) ], [](#__codelineno-7-94) } [](#__codelineno-7-95) ] [](#__codelineno-7-96) [](#__codelineno-7-97) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-98) [](#__codelineno-7-99) prompt = processor.apply_chat_template( [](#__codelineno-7-100) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-101) ) [](#__codelineno-7-102) [](#__codelineno-7-103) return ModelRequestData( [](#__codelineno-7-104) engine_args=engine_args, [](#__codelineno-7-105) prompt=prompt, [](#__codelineno-7-106) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-107) ) [](#__codelineno-7-108) [](#__codelineno-7-109)[](#__codelineno-7-110)def load_bee(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-111) model_name = "Open-Bee/Bee-8B-RL" [](#__codelineno-7-112) [](#__codelineno-7-113) engine_args = EngineArgs( [](#__codelineno-7-114) model=model_name, [](#__codelineno-7-115) max_model_len=16384, [](#__codelineno-7-116) max_num_seqs=16, [](#__codelineno-7-117) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-118) trust_remote_code=True, [](#__codelineno-7-119) ) [](#__codelineno-7-120) [](#__codelineno-7-121) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-122) messages = [ [](#__codelineno-7-123) { [](#__codelineno-7-124) "role": "user", [](#__codelineno-7-125) "content": [ [](#__codelineno-7-126) *placeholders, [](#__codelineno-7-127) {"type": "text", "text": question}, [](#__codelineno-7-128) ], [](#__codelineno-7-129) } [](#__codelineno-7-130) ] [](#__codelineno-7-131) [](#__codelineno-7-132) processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-133) [](#__codelineno-7-134) prompt = processor.apply_chat_template( [](#__codelineno-7-135) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-136) ) [](#__codelineno-7-137) [](#__codelineno-7-138) return ModelRequestData( [](#__codelineno-7-139) engine_args=engine_args, [](#__codelineno-7-140) prompt=prompt, [](#__codelineno-7-141) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-142) ) [](#__codelineno-7-143) [](#__codelineno-7-144)[](#__codelineno-7-145)def load_command_a_vision(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-146) model_name = "CohereLabs/command-a-vision-07-2025" [](#__codelineno-7-147) [](#__codelineno-7-148) # NOTE: This model is 122B parameters and requires tensor parallelism [](#__codelineno-7-149) # Recommended to use tp=4 on H100 GPUs [](#__codelineno-7-150) engine_args = EngineArgs( [](#__codelineno-7-151) model=model_name, [](#__codelineno-7-152) max_model_len=32768, [](#__codelineno-7-153) tensor_parallel_size=4, [](#__codelineno-7-154) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-155) ) [](#__codelineno-7-156) [](#__codelineno-7-157) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-158) messages = [ [](#__codelineno-7-159) { [](#__codelineno-7-160) "role": "user", [](#__codelineno-7-161) "content": [ [](#__codelineno-7-162) *placeholders, [](#__codelineno-7-163) {"type": "text", "text": question}, [](#__codelineno-7-164) ], [](#__codelineno-7-165) } [](#__codelineno-7-166) ] [](#__codelineno-7-167) [](#__codelineno-7-168) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-169) [](#__codelineno-7-170) prompt = processor.apply_chat_template( [](#__codelineno-7-171) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-172) ) [](#__codelineno-7-173) [](#__codelineno-7-174) return ModelRequestData( [](#__codelineno-7-175) engine_args=engine_args, [](#__codelineno-7-176) prompt=prompt, [](#__codelineno-7-177) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-178) ) [](#__codelineno-7-179) [](#__codelineno-7-180)[](#__codelineno-7-181)def load_deepseek_vl2(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-182) model_name = "deepseek-ai/deepseek-vl2-tiny" [](#__codelineno-7-183) [](#__codelineno-7-184) engine_args = EngineArgs( [](#__codelineno-7-185) model=model_name, [](#__codelineno-7-186) max_model_len=4096, [](#__codelineno-7-187) max_num_seqs=2, [](#__codelineno-7-188) hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]}, [](#__codelineno-7-189) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-190) ) [](#__codelineno-7-191) [](#__codelineno-7-192) placeholder = "".join( [](#__codelineno-7-193) f"image_{i}:\n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-194) ) [](#__codelineno-7-195) prompt = f"<|User|>: {placeholder}{question}\n\n<|Assistant|>:" [](#__codelineno-7-196) [](#__codelineno-7-197) return ModelRequestData( [](#__codelineno-7-198) engine_args=engine_args, [](#__codelineno-7-199) prompt=prompt, [](#__codelineno-7-200) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-201) ) [](#__codelineno-7-202) [](#__codelineno-7-203)[](#__codelineno-7-204)def load_deepseek_ocr(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-205) from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor [](#__codelineno-7-206) [](#__codelineno-7-207) model_name = "deepseek-ai/DeepSeek-OCR" [](#__codelineno-7-208) [](#__codelineno-7-209) engine_args = EngineArgs( [](#__codelineno-7-210) model=model_name, [](#__codelineno-7-211) max_num_seqs=2, [](#__codelineno-7-212) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-213) logits_processors=[NGramPerReqLogitsProcessor], [](#__codelineno-7-214) ) [](#__codelineno-7-215) [](#__codelineno-7-216) placeholder = "\n" * len(image_urls) [](#__codelineno-7-217) prompt = placeholder + question [](#__codelineno-7-218) [](#__codelineno-7-219) # The following sampling params config is taken from [](#__codelineno-7-220) # the official Deepseek-OCR inference example. [](#__codelineno-7-221) # (IMPORTANT) Use the custom logits processor and avoid skipping [](#__codelineno-7-222) # special tokens for this model for the optimal OCR performance. [](#__codelineno-7-223) sampling_params = SamplingParams( [](#__codelineno-7-224) temperature=0.0, [](#__codelineno-7-225) max_tokens=8192, [](#__codelineno-7-226) # ngram logit processor args [](#__codelineno-7-227) extra_args=dict( [](#__codelineno-7-228) ngram_size=30, [](#__codelineno-7-229) window_size=90, [](#__codelineno-7-230) # whitelist: , [](#__codelineno-7-231) whitelist_token_ids={128821, 128822}, [](#__codelineno-7-232) ), [](#__codelineno-7-233) skip_special_tokens=False, [](#__codelineno-7-234) ) [](#__codelineno-7-235) [](#__codelineno-7-236) return ModelRequestData( [](#__codelineno-7-237) engine_args=engine_args, [](#__codelineno-7-238) prompt=prompt, [](#__codelineno-7-239) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-240) sampling_params=sampling_params, [](#__codelineno-7-241) ) [](#__codelineno-7-242) [](#__codelineno-7-243)[](#__codelineno-7-244)# exaone4_5 [](#__codelineno-7-245)def load_exaone4_5(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-246) model_name = "LGAI-EXAONE/EXAONE-4.5-33B" [](#__codelineno-7-247) [](#__codelineno-7-248) engine_args = EngineArgs( [](#__codelineno-7-249) model=model_name, [](#__codelineno-7-250) max_model_len=8192, [](#__codelineno-7-251) max_num_seqs=2, [](#__codelineno-7-252) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-253) ) [](#__codelineno-7-254) [](#__codelineno-7-255) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-256) messages = [ [](#__codelineno-7-257) { [](#__codelineno-7-258) "role": "user", [](#__codelineno-7-259) "content": [ [](#__codelineno-7-260) *placeholders, [](#__codelineno-7-261) {"type": "text", "text": question}, [](#__codelineno-7-262) ], [](#__codelineno-7-263) } [](#__codelineno-7-264) ] [](#__codelineno-7-265) [](#__codelineno-7-266) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-267) [](#__codelineno-7-268) prompt = processor.apply_chat_template( [](#__codelineno-7-269) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-270) ) [](#__codelineno-7-271) [](#__codelineno-7-272) return ModelRequestData( [](#__codelineno-7-273) engine_args=engine_args, [](#__codelineno-7-274) prompt=prompt, [](#__codelineno-7-275) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-276) ) [](#__codelineno-7-277) [](#__codelineno-7-278)[](#__codelineno-7-279)def load_gemma3(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-280) model_name = "google/gemma-3-4b-it" [](#__codelineno-7-281) [](#__codelineno-7-282) engine_args = EngineArgs( [](#__codelineno-7-283) model=model_name, [](#__codelineno-7-284) max_model_len=8192, [](#__codelineno-7-285) max_num_seqs=2, [](#__codelineno-7-286) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-287) ) [](#__codelineno-7-288) [](#__codelineno-7-289) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-290) messages = [ [](#__codelineno-7-291) { [](#__codelineno-7-292) "role": "user", [](#__codelineno-7-293) "content": [ [](#__codelineno-7-294) *placeholders, [](#__codelineno-7-295) {"type": "text", "text": question}, [](#__codelineno-7-296) ], [](#__codelineno-7-297) } [](#__codelineno-7-298) ] [](#__codelineno-7-299) [](#__codelineno-7-300) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-301) [](#__codelineno-7-302) prompt = processor.apply_chat_template( [](#__codelineno-7-303) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-304) ) [](#__codelineno-7-305) [](#__codelineno-7-306) return ModelRequestData( [](#__codelineno-7-307) engine_args=engine_args, [](#__codelineno-7-308) prompt=prompt, [](#__codelineno-7-309) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-310) ) [](#__codelineno-7-311) [](#__codelineno-7-312)[](#__codelineno-7-313)def load_granite4_vision(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-314) model_name = "ibm-granite/granite-vision-4.1-4b" [](#__codelineno-7-315) engine_args = EngineArgs( [](#__codelineno-7-316) model=model_name, [](#__codelineno-7-317) max_model_len=4096, [](#__codelineno-7-318) max_num_seqs=16, [](#__codelineno-7-319) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-320) ) [](#__codelineno-7-321) [](#__codelineno-7-322) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-323) messages = [ [](#__codelineno-7-324) { [](#__codelineno-7-325) "role": "user", [](#__codelineno-7-326) "content": [ [](#__codelineno-7-327) *placeholders, [](#__codelineno-7-328) {"type": "text", "text": question}, [](#__codelineno-7-329) ], [](#__codelineno-7-330) } [](#__codelineno-7-331) ] [](#__codelineno-7-332) [](#__codelineno-7-333) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-334) prompt = processor.apply_chat_template( [](#__codelineno-7-335) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-336) ) [](#__codelineno-7-337) [](#__codelineno-7-338) return ModelRequestData( [](#__codelineno-7-339) engine_args=engine_args, [](#__codelineno-7-340) prompt=prompt, [](#__codelineno-7-341) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-342) ) [](#__codelineno-7-343) [](#__codelineno-7-344)[](#__codelineno-7-345)def load_h2ovl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-346) model_name = "h2oai/h2ovl-mississippi-800m" [](#__codelineno-7-347) [](#__codelineno-7-348) engine_args = EngineArgs( [](#__codelineno-7-349) model=model_name, [](#__codelineno-7-350) trust_remote_code=True, [](#__codelineno-7-351) max_model_len=8192, [](#__codelineno-7-352) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-353) mm_processor_kwargs={"max_dynamic_patch": 4}, [](#__codelineno-7-354) ) [](#__codelineno-7-355) [](#__codelineno-7-356) placeholders = "\n".join( [](#__codelineno-7-357) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-358) ) [](#__codelineno-7-359) messages = [{"role": "user", "content": f"{placeholders}\n{question}"}] [](#__codelineno-7-360) [](#__codelineno-7-361) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-362) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-363) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-364) ) [](#__codelineno-7-365) [](#__codelineno-7-366) # Stop tokens for H2OVL-Mississippi [](#__codelineno-7-367) # https://huggingface.co/h2oai/h2ovl-mississippi-800m [](#__codelineno-7-368) stop_token_ids = [tokenizer.eos_token_id] [](#__codelineno-7-369) [](#__codelineno-7-370) return ModelRequestData( [](#__codelineno-7-371) engine_args=engine_args, [](#__codelineno-7-372) prompt=prompt, [](#__codelineno-7-373) stop_token_ids=stop_token_ids, [](#__codelineno-7-374) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-375) ) [](#__codelineno-7-376) [](#__codelineno-7-377)[](#__codelineno-7-378)# HunyuanOCR [](#__codelineno-7-379)def load_hunyuan_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-380) model_name = "tencent/HunyuanOCR" [](#__codelineno-7-381) [](#__codelineno-7-382) engine_args = EngineArgs( [](#__codelineno-7-383) model=model_name, [](#__codelineno-7-384) max_model_len=8192, [](#__codelineno-7-385) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-386) ) [](#__codelineno-7-387) [](#__codelineno-7-388) placeholder = ( [](#__codelineno-7-389) "<｜hy_place▁holder▁no▁100｜><｜hy_place▁holder▁no▁102｜><｜hy_place▁holder▁no▁101｜>" # noqa: E501 [](#__codelineno-7-390) ) * len(image_urls) [](#__codelineno-7-391) prompt = f"<｜hy_begin▁of▁sentence｜>{placeholder}{question}<｜hy_User｜>" [](#__codelineno-7-392) [](#__codelineno-7-393) return ModelRequestData( [](#__codelineno-7-394) engine_args=engine_args, [](#__codelineno-7-395) prompt=prompt, [](#__codelineno-7-396) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-397) ) [](#__codelineno-7-398) [](#__codelineno-7-399)[](#__codelineno-7-400)def load_hyperclovax_seed_vision( [](#__codelineno-7-401) question: str, image_urls: list[str] [](#__codelineno-7-402)) -> ModelRequestData: [](#__codelineno-7-403) model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B" [](#__codelineno-7-404) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-405) [](#__codelineno-7-406) engine_args = EngineArgs( [](#__codelineno-7-407) model=model_name, [](#__codelineno-7-408) trust_remote_code=True, [](#__codelineno-7-409) max_model_len=16384, [](#__codelineno-7-410) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-411) ) [](#__codelineno-7-412) [](#__codelineno-7-413) message = {"role": "user", "content": list()} [](#__codelineno-7-414) for _image_url in image_urls: [](#__codelineno-7-415) message["content"].append( [](#__codelineno-7-416) { [](#__codelineno-7-417) "type": "image", [](#__codelineno-7-418) "image": _image_url, [](#__codelineno-7-419) "ocr": "", [](#__codelineno-7-420) "lens_keywords": "", [](#__codelineno-7-421) "lens_local_keywords": "", [](#__codelineno-7-422) } [](#__codelineno-7-423) ) [](#__codelineno-7-424) message["content"].append( [](#__codelineno-7-425) { [](#__codelineno-7-426) "type": "text", [](#__codelineno-7-427) "text": question, [](#__codelineno-7-428) } [](#__codelineno-7-429) ) [](#__codelineno-7-430) [](#__codelineno-7-431) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-432) [ [](#__codelineno-7-433) message, [](#__codelineno-7-434) ], [](#__codelineno-7-435) tokenize=False, [](#__codelineno-7-436) add_generation_prompt=True, [](#__codelineno-7-437) ) [](#__codelineno-7-438) [](#__codelineno-7-439) return ModelRequestData( [](#__codelineno-7-440) engine_args=engine_args, [](#__codelineno-7-441) prompt=prompt, [](#__codelineno-7-442) stop_token_ids=None, [](#__codelineno-7-443) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-444) ) [](#__codelineno-7-445) [](#__codelineno-7-446)[](#__codelineno-7-447)def load_idefics3(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-448) model_name = "HuggingFaceM4/Idefics3-8B-Llama3" [](#__codelineno-7-449) [](#__codelineno-7-450) # The configuration below has been confirmed to launch on a single L40 GPU. [](#__codelineno-7-451) engine_args = EngineArgs( [](#__codelineno-7-452) model=model_name, [](#__codelineno-7-453) max_model_len=8192, [](#__codelineno-7-454) max_num_seqs=16, [](#__codelineno-7-455) enforce_eager=True, [](#__codelineno-7-456) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-457) # if you are running out of memory, you can reduce the "longest_edge". [](#__codelineno-7-458) # see: https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3#model-optimizations [](#__codelineno-7-459) mm_processor_kwargs={ [](#__codelineno-7-460) "size": {"longest_edge": 2 * 364}, [](#__codelineno-7-461) }, [](#__codelineno-7-462) ) [](#__codelineno-7-463) [](#__codelineno-7-464) placeholders = "\n".join( [](#__codelineno-7-465) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-466) ) [](#__codelineno-7-467) prompt = f"<|begin_of_text|>User:{placeholders}\n{question}\nAssistant:" # noqa: E501 [](#__codelineno-7-468) return ModelRequestData( [](#__codelineno-7-469) engine_args=engine_args, [](#__codelineno-7-470) prompt=prompt, [](#__codelineno-7-471) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-472) ) [](#__codelineno-7-473) [](#__codelineno-7-474)[](#__codelineno-7-475)def load_interns1(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-476) model_name = "internlm/Intern-S1-mini" [](#__codelineno-7-477) [](#__codelineno-7-478) engine_args = EngineArgs( [](#__codelineno-7-479) model=model_name, [](#__codelineno-7-480) trust_remote_code=True, [](#__codelineno-7-481) max_model_len=4096, [](#__codelineno-7-482) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-483) ) [](#__codelineno-7-484) [](#__codelineno-7-485) placeholders = "\n".join( [](#__codelineno-7-486) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-487) ) [](#__codelineno-7-488) messages = [{"role": "user", "content": f"{placeholders}\n{question}"}] [](#__codelineno-7-489) [](#__codelineno-7-490) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-491) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-492) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-493) ) [](#__codelineno-7-494) [](#__codelineno-7-495) return ModelRequestData( [](#__codelineno-7-496) engine_args=engine_args, [](#__codelineno-7-497) prompt=prompt, [](#__codelineno-7-498) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-499) ) [](#__codelineno-7-500) [](#__codelineno-7-501)[](#__codelineno-7-502)def load_internvl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-503) model_name = "OpenGVLab/InternVL2-2B" [](#__codelineno-7-504) [](#__codelineno-7-505) engine_args = EngineArgs( [](#__codelineno-7-506) model=model_name, [](#__codelineno-7-507) trust_remote_code=True, [](#__codelineno-7-508) max_model_len=4096, [](#__codelineno-7-509) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-510) mm_processor_kwargs={"max_dynamic_patch": 4}, [](#__codelineno-7-511) ) [](#__codelineno-7-512) [](#__codelineno-7-513) placeholders = "\n".join( [](#__codelineno-7-514) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-515) ) [](#__codelineno-7-516) messages = [{"role": "user", "content": f"{placeholders}\n{question}"}] [](#__codelineno-7-517) [](#__codelineno-7-518) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-519) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-520) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-521) ) [](#__codelineno-7-522) [](#__codelineno-7-523) # Stop tokens for InternVL [](#__codelineno-7-524) # models variants may have different stop tokens [](#__codelineno-7-525) # please refer to the model card for the correct "stop words": [](#__codelineno-7-526) # https://huggingface.co/OpenGVLab/InternVL2-2B/blob/main/conversation.py [](#__codelineno-7-527) stop_tokens = ["<|endoftext|>", "<|im_start|>", "<|im_end|>", "<|end|>"] [](#__codelineno-7-528) stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] [](#__codelineno-7-529) [](#__codelineno-7-530) return ModelRequestData( [](#__codelineno-7-531) engine_args=engine_args, [](#__codelineno-7-532) prompt=prompt, [](#__codelineno-7-533) stop_token_ids=stop_token_ids, [](#__codelineno-7-534) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-535) ) [](#__codelineno-7-536) [](#__codelineno-7-537)[](#__codelineno-7-538)def load_keye_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-539) model_name = "Kwai-Keye/Keye-VL-8B-Preview" [](#__codelineno-7-540) [](#__codelineno-7-541) engine_args = EngineArgs( [](#__codelineno-7-542) model=model_name, [](#__codelineno-7-543) trust_remote_code=True, [](#__codelineno-7-544) max_model_len=8192, [](#__codelineno-7-545) max_num_seqs=5, [](#__codelineno-7-546) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-547) ) [](#__codelineno-7-548) [](#__codelineno-7-549) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-550) messages = [ [](#__codelineno-7-551) { [](#__codelineno-7-552) "role": "user", [](#__codelineno-7-553) "content": [ [](#__codelineno-7-554) *placeholders, [](#__codelineno-7-555) {"type": "text", "text": question}, [](#__codelineno-7-556) ], [](#__codelineno-7-557) }, [](#__codelineno-7-558) ] [](#__codelineno-7-559) [](#__codelineno-7-560) processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-561) [](#__codelineno-7-562) prompt = processor.apply_chat_template( [](#__codelineno-7-563) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-564) ) [](#__codelineno-7-565) [](#__codelineno-7-566) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-567) [](#__codelineno-7-568) return ModelRequestData( [](#__codelineno-7-569) engine_args=engine_args, [](#__codelineno-7-570) prompt=prompt, [](#__codelineno-7-571) image_data=image_data, [](#__codelineno-7-572) ) [](#__codelineno-7-573) [](#__codelineno-7-574)[](#__codelineno-7-575)def load_keye_vl1_5(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-576) model_name = "Kwai-Keye/Keye-VL-1_5-8B" [](#__codelineno-7-577) [](#__codelineno-7-578) engine_args = EngineArgs( [](#__codelineno-7-579) model=model_name, [](#__codelineno-7-580) trust_remote_code=True, [](#__codelineno-7-581) max_model_len=32768, [](#__codelineno-7-582) max_num_seqs=5, [](#__codelineno-7-583) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-584) ) [](#__codelineno-7-585) [](#__codelineno-7-586) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-587) messages = [ [](#__codelineno-7-588) { [](#__codelineno-7-589) "role": "user", [](#__codelineno-7-590) "content": [ [](#__codelineno-7-591) *placeholders, [](#__codelineno-7-592) {"type": "text", "text": question}, [](#__codelineno-7-593) ], [](#__codelineno-7-594) }, [](#__codelineno-7-595) ] [](#__codelineno-7-596) [](#__codelineno-7-597) processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-598) [](#__codelineno-7-599) prompt = processor.apply_chat_template( [](#__codelineno-7-600) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-601) ) [](#__codelineno-7-602) [](#__codelineno-7-603) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-604) [](#__codelineno-7-605) return ModelRequestData( [](#__codelineno-7-606) engine_args=engine_args, [](#__codelineno-7-607) prompt=prompt, [](#__codelineno-7-608) image_data=image_data, [](#__codelineno-7-609) ) [](#__codelineno-7-610) [](#__codelineno-7-611)[](#__codelineno-7-612)def load_kimi_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-613) model_name = "moonshotai/Kimi-VL-A3B-Instruct" [](#__codelineno-7-614) [](#__codelineno-7-615) engine_args = EngineArgs( [](#__codelineno-7-616) model=model_name, [](#__codelineno-7-617) trust_remote_code=True, [](#__codelineno-7-618) max_model_len=4096, [](#__codelineno-7-619) max_num_seqs=4, [](#__codelineno-7-620) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-621) ) [](#__codelineno-7-622) [](#__codelineno-7-623) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-624) messages = [ [](#__codelineno-7-625) { [](#__codelineno-7-626) "role": "user", [](#__codelineno-7-627) "content": [ [](#__codelineno-7-628) *placeholders, [](#__codelineno-7-629) {"type": "text", "text": question}, [](#__codelineno-7-630) ], [](#__codelineno-7-631) } [](#__codelineno-7-632) ] [](#__codelineno-7-633) [](#__codelineno-7-634) processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-635) [](#__codelineno-7-636) prompt = processor.apply_chat_template( [](#__codelineno-7-637) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-638) ) [](#__codelineno-7-639) [](#__codelineno-7-640) return ModelRequestData( [](#__codelineno-7-641) engine_args=engine_args, [](#__codelineno-7-642) prompt=prompt, [](#__codelineno-7-643) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-644) ) [](#__codelineno-7-645) [](#__codelineno-7-646)[](#__codelineno-7-647)def load_llama4(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-648) model_name = "meta-llama/Llama-4-Scout-17B-16E-Instruct" [](#__codelineno-7-649) [](#__codelineno-7-650) engine_args = EngineArgs( [](#__codelineno-7-651) model=model_name, [](#__codelineno-7-652) max_model_len=131072, [](#__codelineno-7-653) tensor_parallel_size=8, [](#__codelineno-7-654) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-655) ) [](#__codelineno-7-656) [](#__codelineno-7-657) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-658) messages = [ [](#__codelineno-7-659) { [](#__codelineno-7-660) "role": "user", [](#__codelineno-7-661) "content": [ [](#__codelineno-7-662) *placeholders, [](#__codelineno-7-663) {"type": "text", "text": question}, [](#__codelineno-7-664) ], [](#__codelineno-7-665) } [](#__codelineno-7-666) ] [](#__codelineno-7-667) [](#__codelineno-7-668) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-669) [](#__codelineno-7-670) prompt = processor.apply_chat_template( [](#__codelineno-7-671) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-672) ) [](#__codelineno-7-673) [](#__codelineno-7-674) return ModelRequestData( [](#__codelineno-7-675) engine_args=engine_args, [](#__codelineno-7-676) prompt=prompt, [](#__codelineno-7-677) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-678) ) [](#__codelineno-7-679) [](#__codelineno-7-680)[](#__codelineno-7-681)def load_llava(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-682) # NOTE: CAUTION! Original Llava models wasn't really trained on multi-image inputs, [](#__codelineno-7-683) # it will generate poor response for multi-image inputs! [](#__codelineno-7-684) model_name = "llava-hf/llava-1.5-7b-hf" [](#__codelineno-7-685) engine_args = EngineArgs( [](#__codelineno-7-686) model=model_name, [](#__codelineno-7-687) max_num_seqs=16, [](#__codelineno-7-688) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-689) ) [](#__codelineno-7-690) [](#__codelineno-7-691) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-692) messages = [ [](#__codelineno-7-693) { [](#__codelineno-7-694) "role": "user", [](#__codelineno-7-695) "content": [ [](#__codelineno-7-696) *placeholders, [](#__codelineno-7-697) {"type": "text", "text": question}, [](#__codelineno-7-698) ], [](#__codelineno-7-699) } [](#__codelineno-7-700) ] [](#__codelineno-7-701) [](#__codelineno-7-702) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-703) [](#__codelineno-7-704) prompt = processor.apply_chat_template( [](#__codelineno-7-705) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-706) ) [](#__codelineno-7-707) [](#__codelineno-7-708) return ModelRequestData( [](#__codelineno-7-709) engine_args=engine_args, [](#__codelineno-7-710) prompt=prompt, [](#__codelineno-7-711) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-712) ) [](#__codelineno-7-713) [](#__codelineno-7-714)[](#__codelineno-7-715)def load_llava_next(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-716) model_name = "llava-hf/llava-v1.6-mistral-7b-hf" [](#__codelineno-7-717) engine_args = EngineArgs( [](#__codelineno-7-718) model=model_name, [](#__codelineno-7-719) max_model_len=8192, [](#__codelineno-7-720) max_num_seqs=16, [](#__codelineno-7-721) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-722) ) [](#__codelineno-7-723) [](#__codelineno-7-724) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-725) messages = [ [](#__codelineno-7-726) { [](#__codelineno-7-727) "role": "user", [](#__codelineno-7-728) "content": [ [](#__codelineno-7-729) *placeholders, [](#__codelineno-7-730) {"type": "text", "text": question}, [](#__codelineno-7-731) ], [](#__codelineno-7-732) } [](#__codelineno-7-733) ] [](#__codelineno-7-734) [](#__codelineno-7-735) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-736) [](#__codelineno-7-737) prompt = processor.apply_chat_template( [](#__codelineno-7-738) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-739) ) [](#__codelineno-7-740) [](#__codelineno-7-741) return ModelRequestData( [](#__codelineno-7-742) engine_args=engine_args, [](#__codelineno-7-743) prompt=prompt, [](#__codelineno-7-744) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-745) ) [](#__codelineno-7-746) [](#__codelineno-7-747)[](#__codelineno-7-748)def load_llava_onevision(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-749) model_name = "llava-hf/llava-onevision-qwen2-7b-ov-hf" [](#__codelineno-7-750) engine_args = EngineArgs( [](#__codelineno-7-751) model=model_name, [](#__codelineno-7-752) max_model_len=16384, [](#__codelineno-7-753) max_num_seqs=16, [](#__codelineno-7-754) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-755) ) [](#__codelineno-7-756) [](#__codelineno-7-757) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-758) messages = [ [](#__codelineno-7-759) { [](#__codelineno-7-760) "role": "user", [](#__codelineno-7-761) "content": [ [](#__codelineno-7-762) *placeholders, [](#__codelineno-7-763) {"type": "text", "text": question}, [](#__codelineno-7-764) ], [](#__codelineno-7-765) } [](#__codelineno-7-766) ] [](#__codelineno-7-767) [](#__codelineno-7-768) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-769) [](#__codelineno-7-770) prompt = processor.apply_chat_template( [](#__codelineno-7-771) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-772) ) [](#__codelineno-7-773) [](#__codelineno-7-774) return ModelRequestData( [](#__codelineno-7-775) engine_args=engine_args, [](#__codelineno-7-776) prompt=prompt, [](#__codelineno-7-777) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-778) ) [](#__codelineno-7-779) [](#__codelineno-7-780)[](#__codelineno-7-781)def load_mistral3(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-782) model_name = "mistralai/Mistral-Small-3.1-24B-Instruct-2503" [](#__codelineno-7-783) [](#__codelineno-7-784) # Adjust this as necessary to fit in GPU [](#__codelineno-7-785) engine_args = EngineArgs( [](#__codelineno-7-786) model=model_name, [](#__codelineno-7-787) max_model_len=8192, [](#__codelineno-7-788) max_num_seqs=2, [](#__codelineno-7-789) tensor_parallel_size=2, [](#__codelineno-7-790) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-791) ignore_patterns=["consolidated.safetensors"], [](#__codelineno-7-792) ) [](#__codelineno-7-793) [](#__codelineno-7-794) placeholders = "[IMG]" * len(image_urls) [](#__codelineno-7-795) prompt = f"[INST]{question}\n{placeholders}[/INST]" [](#__codelineno-7-796) [](#__codelineno-7-797) return ModelRequestData( [](#__codelineno-7-798) engine_args=engine_args, [](#__codelineno-7-799) prompt=prompt, [](#__codelineno-7-800) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-801) ) [](#__codelineno-7-802) [](#__codelineno-7-803)[](#__codelineno-7-804)def load_nvlm_d(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-805) model_name = "nvidia/NVLM-D-72B" [](#__codelineno-7-806) [](#__codelineno-7-807) # Adjust this as necessary to fit in GPU [](#__codelineno-7-808) engine_args = EngineArgs( [](#__codelineno-7-809) model=model_name, [](#__codelineno-7-810) trust_remote_code=True, [](#__codelineno-7-811) max_model_len=8192, [](#__codelineno-7-812) tensor_parallel_size=4, [](#__codelineno-7-813) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-814) mm_processor_kwargs={"max_dynamic_patch": 4}, [](#__codelineno-7-815) ) [](#__codelineno-7-816) [](#__codelineno-7-817) placeholders = "\n".join( [](#__codelineno-7-818) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-819) ) [](#__codelineno-7-820) messages = [{"role": "user", "content": f"{placeholders}\n{question}"}] [](#__codelineno-7-821) [](#__codelineno-7-822) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-823) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-824) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-825) ) [](#__codelineno-7-826) [](#__codelineno-7-827) return ModelRequestData( [](#__codelineno-7-828) engine_args=engine_args, [](#__codelineno-7-829) prompt=prompt, [](#__codelineno-7-830) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-831) ) [](#__codelineno-7-832) [](#__codelineno-7-833)[](#__codelineno-7-834)# OpenPangu [](#__codelineno-7-835)def load_openpangu_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-836) model_name = "FreedomIntelligence/openPangu-VL-7B" [](#__codelineno-7-837) [](#__codelineno-7-838) engine_args = EngineArgs( [](#__codelineno-7-839) model=model_name, [](#__codelineno-7-840) trust_remote_code=True, [](#__codelineno-7-841) max_model_len=8192, [](#__codelineno-7-842) max_num_seqs=2, [](#__codelineno-7-843) enforce_eager=True, [](#__codelineno-7-844) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-845) ) [](#__codelineno-7-846) [](#__codelineno-7-847) placeholders = "[unused18][unused19][unused20]" * len(image_urls) [](#__codelineno-7-848) prompt = ( [](#__codelineno-7-849) f"[unused9]系统：[unused10][unused9]用户：{question}{placeholders}" [](#__codelineno-7-850) "[unused10][unused9]助手：" [](#__codelineno-7-851) ) [](#__codelineno-7-852) [](#__codelineno-7-853) return ModelRequestData( [](#__codelineno-7-854) engine_args=engine_args, [](#__codelineno-7-855) prompt=prompt, [](#__codelineno-7-856) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-857) ) [](#__codelineno-7-858) [](#__codelineno-7-859)[](#__codelineno-7-860)# Ovis [](#__codelineno-7-861)def load_ovis(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-862) model_name = "AIDC-AI/Ovis2-1B" [](#__codelineno-7-863) [](#__codelineno-7-864) engine_args = EngineArgs( [](#__codelineno-7-865) model=model_name, [](#__codelineno-7-866) max_model_len=8192, [](#__codelineno-7-867) max_num_seqs=2, [](#__codelineno-7-868) trust_remote_code=True, [](#__codelineno-7-869) dtype="half", [](#__codelineno-7-870) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-871) ) [](#__codelineno-7-872) [](#__codelineno-7-873) placeholders = "\n".join( [](#__codelineno-7-874) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-875) ) [](#__codelineno-7-876) messages = [{"role": "user", "content": f"{placeholders}\n{question}"}] [](#__codelineno-7-877) [](#__codelineno-7-878) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-879) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-880) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-881) ) [](#__codelineno-7-882) [](#__codelineno-7-883) return ModelRequestData( [](#__codelineno-7-884) engine_args=engine_args, [](#__codelineno-7-885) prompt=prompt, [](#__codelineno-7-886) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-887) ) [](#__codelineno-7-888) [](#__codelineno-7-889)[](#__codelineno-7-890)# ovis2_5 [](#__codelineno-7-891)def load_ovis2_5(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-892) model_name = "AIDC-AI/Ovis2.5-2B" [](#__codelineno-7-893) [](#__codelineno-7-894) engine_args = EngineArgs( [](#__codelineno-7-895) model=model_name, [](#__codelineno-7-896) max_model_len=8192, [](#__codelineno-7-897) max_num_seqs=2, [](#__codelineno-7-898) trust_remote_code=True, [](#__codelineno-7-899) dtype="half", [](#__codelineno-7-900) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-901) ) [](#__codelineno-7-902) [](#__codelineno-7-903) placeholders = "\n".join( [](#__codelineno-7-904) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-905) ) [](#__codelineno-7-906) prompt = ( [](#__codelineno-7-907) f"<|im_start|>user\n\n{placeholders}\n{question}<|im_end|>\n" [](#__codelineno-7-908) "<|im_start|>assistant\n" [](#__codelineno-7-909) ) [](#__codelineno-7-910) [](#__codelineno-7-911) return ModelRequestData( [](#__codelineno-7-912) engine_args=engine_args, [](#__codelineno-7-913) prompt=prompt, [](#__codelineno-7-914) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-915) ) [](#__codelineno-7-916) [](#__codelineno-7-917)[](#__codelineno-7-918)def load_paddleocr_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-919) model_name = "PaddlePaddle/PaddleOCR-VL" [](#__codelineno-7-920) [](#__codelineno-7-921) engine_args = EngineArgs( [](#__codelineno-7-922) model=model_name, [](#__codelineno-7-923) trust_remote_code=True, [](#__codelineno-7-924) max_model_len=8192, [](#__codelineno-7-925) max_num_seqs=2, [](#__codelineno-7-926) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-927) ) [](#__codelineno-7-928) [](#__codelineno-7-929) placeholders = "<|IMAGE_START|><|IMAGE_PLACEHOLDER|><|IMAGE_END|>" * len(image_urls) [](#__codelineno-7-930) prompt = f"<|begin_of_sentence|>User: {question}{placeholders}\nAssistant: " [](#__codelineno-7-931) [](#__codelineno-7-932) return ModelRequestData( [](#__codelineno-7-933) engine_args=engine_args, [](#__codelineno-7-934) prompt=prompt, [](#__codelineno-7-935) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-936) ) [](#__codelineno-7-937) [](#__codelineno-7-938)[](#__codelineno-7-939)def load_pixtral_hf(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-940) model_name = "mistral-community/pixtral-12b" [](#__codelineno-7-941) [](#__codelineno-7-942) # Adjust this as necessary to fit in GPU [](#__codelineno-7-943) engine_args = EngineArgs( [](#__codelineno-7-944) model=model_name, [](#__codelineno-7-945) max_model_len=8192, [](#__codelineno-7-946) max_num_seqs=2, [](#__codelineno-7-947) tensor_parallel_size=2, [](#__codelineno-7-948) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-949) ) [](#__codelineno-7-950) [](#__codelineno-7-951) placeholders = "[IMG]" * len(image_urls) [](#__codelineno-7-952) prompt = f"[INST]{question}\n{placeholders}[/INST]" [](#__codelineno-7-953) [](#__codelineno-7-954) return ModelRequestData( [](#__codelineno-7-955) engine_args=engine_args, [](#__codelineno-7-956) prompt=prompt, [](#__codelineno-7-957) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-958) ) [](#__codelineno-7-959) [](#__codelineno-7-960)[](#__codelineno-7-961)def load_phi3v(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-962) # num_crops is an override kwarg to the multimodal image processor; [](#__codelineno-7-963) # For some models, e.g., Phi-3.5-vision-instruct, it is recommended [](#__codelineno-7-964) # to use 16 for single frame scenarios, and 4 for multi-frame. [](#__codelineno-7-965) # [](#__codelineno-7-966) # Generally speaking, a larger value for num_crops results in more [](#__codelineno-7-967) # tokens per image instance, because it may scale the image more in [](#__codelineno-7-968) # the image preprocessing. Some references in the model docs and the [](#__codelineno-7-969) # formula for image tokens after the preprocessing [](#__codelineno-7-970) # transform can be found below. [](#__codelineno-7-971) # [](#__codelineno-7-972) # https://huggingface.co/microsoft/Phi-3.5-vision-instruct#loading-the-model-locally [](#__codelineno-7-973) # https://huggingface.co/microsoft/Phi-3.5-vision-instruct/blob/main/processing_phi3_v.py#L194 [](#__codelineno-7-974) engine_args = EngineArgs( [](#__codelineno-7-975) model="microsoft/Phi-3.5-vision-instruct", [](#__codelineno-7-976) trust_remote_code=True, [](#__codelineno-7-977) max_model_len=4096, [](#__codelineno-7-978) max_num_seqs=2, [](#__codelineno-7-979) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-980) mm_processor_kwargs={"num_crops": 4}, [](#__codelineno-7-981) ) [](#__codelineno-7-982) placeholders = "\n".join( [](#__codelineno-7-983) f"<|image_{i}|>" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-984) ) [](#__codelineno-7-985) prompt = f"<|user|>\n{placeholders}\n{question}<|end|>\n<|assistant|>\n" [](#__codelineno-7-986) [](#__codelineno-7-987) return ModelRequestData( [](#__codelineno-7-988) engine_args=engine_args, [](#__codelineno-7-989) prompt=prompt, [](#__codelineno-7-990) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-991) ) [](#__codelineno-7-992) [](#__codelineno-7-993)[](#__codelineno-7-994)def load_phi4mm(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-995) """ [](#__codelineno-7-996) Phi-4-multimodal-instruct supports both image and audio inputs. Here, we [](#__codelineno-7-997) show how to process multi images inputs. [](#__codelineno-7-998) """ [](#__codelineno-7-999) [](#__codelineno-7-1000) model_path = snapshot_download("microsoft/Phi-4-multimodal-instruct") [](#__codelineno-7-1001) # Since the vision-lora and speech-lora co-exist with the base model, [](#__codelineno-7-1002) # we have to manually specify the path of the lora weights. [](#__codelineno-7-1003) vision_lora_path = os.path.join(model_path, "vision-lora") [](#__codelineno-7-1004) engine_args = EngineArgs( [](#__codelineno-7-1005) model=model_path, [](#__codelineno-7-1006) trust_remote_code=True, [](#__codelineno-7-1007) max_model_len=4096, [](#__codelineno-7-1008) max_num_seqs=2, [](#__codelineno-7-1009) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1010) enable_lora=True, [](#__codelineno-7-1011) max_lora_rank=320, [](#__codelineno-7-1012) # Note - mm_processor_kwargs can also be passed to generate/chat calls [](#__codelineno-7-1013) mm_processor_kwargs={"dynamic_hd": 4}, [](#__codelineno-7-1014) ) [](#__codelineno-7-1015) [](#__codelineno-7-1016) placeholders = "".join(f"<|image_{i}|>" for i, _ in enumerate(image_urls, start=1)) [](#__codelineno-7-1017) prompt = f"<|user|>{placeholders}{question}<|end|><|assistant|>" [](#__codelineno-7-1018) [](#__codelineno-7-1019) return ModelRequestData( [](#__codelineno-7-1020) engine_args=engine_args, [](#__codelineno-7-1021) prompt=prompt, [](#__codelineno-7-1022) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-1023) lora_requests=[LoRARequest("vision", 1, vision_lora_path)], [](#__codelineno-7-1024) ) [](#__codelineno-7-1025) [](#__codelineno-7-1026)[](#__codelineno-7-1027)def load_phi4siglip(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1028) model_name = "microsoft/Phi-4-reasoning-vision-15B" [](#__codelineno-7-1029) placeholders = "\n".join("" for _ in image_urls) [](#__codelineno-7-1030) prompt = f"<|user|>\n{placeholders}\n{question}<|end|>\n<|assistant|>\n" [](#__codelineno-7-1031) engine_args = EngineArgs( [](#__codelineno-7-1032) model=model_name, [](#__codelineno-7-1033) trust_remote_code=True, [](#__codelineno-7-1034) max_model_len=8192, [](#__codelineno-7-1035) max_num_seqs=2, [](#__codelineno-7-1036) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1037) ) [](#__codelineno-7-1038) return ModelRequestData( [](#__codelineno-7-1039) engine_args=engine_args, [](#__codelineno-7-1040) prompt=prompt, [](#__codelineno-7-1041) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-1042) ) [](#__codelineno-7-1043) [](#__codelineno-7-1044)[](#__codelineno-7-1045)def load_qwen_vl_chat(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1046) model_name = "Qwen/Qwen-VL-Chat" [](#__codelineno-7-1047) engine_args = EngineArgs( [](#__codelineno-7-1048) model=model_name, [](#__codelineno-7-1049) trust_remote_code=True, [](#__codelineno-7-1050) max_model_len=1024, [](#__codelineno-7-1051) max_num_seqs=2, [](#__codelineno-7-1052) hf_overrides={"architectures": ["QwenVLForConditionalGeneration"]}, [](#__codelineno-7-1053) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1054) ) [](#__codelineno-7-1055) placeholders = "".join( [](#__codelineno-7-1056) f"Picture {i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-1057) ) [](#__codelineno-7-1058) [](#__codelineno-7-1059) # This model does not have a chat_template attribute on its tokenizer, [](#__codelineno-7-1060) # so we need to explicitly pass it. We use ChatML since it's used in the [](#__codelineno-7-1061) # generation utils of the model: [](#__codelineno-7-1062) # https://huggingface.co/Qwen/Qwen-VL-Chat/blob/main/qwen_generation_utils.py#L265 [](#__codelineno-7-1063) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-1064) [](#__codelineno-7-1065) # Copied from: https://huggingface.co/docs/transformers/main/en/chat_templating [](#__codelineno-7-1066) chat_template = "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}" # noqa: E501 [](#__codelineno-7-1067) [](#__codelineno-7-1068) messages = [{"role": "user", "content": f"{placeholders}\n{question}"}] [](#__codelineno-7-1069) prompt = tokenizer.apply_chat_template( [](#__codelineno-7-1070) messages, [](#__codelineno-7-1071) tokenize=False, [](#__codelineno-7-1072) add_generation_prompt=True, [](#__codelineno-7-1073) chat_template=chat_template, [](#__codelineno-7-1074) ) [](#__codelineno-7-1075) [](#__codelineno-7-1076) stop_tokens = ["<|endoftext|>", "<|im_start|>", "<|im_end|>"] [](#__codelineno-7-1077) stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] [](#__codelineno-7-1078) [](#__codelineno-7-1079) return ModelRequestData( [](#__codelineno-7-1080) engine_args=engine_args, [](#__codelineno-7-1081) prompt=prompt, [](#__codelineno-7-1082) stop_token_ids=stop_token_ids, [](#__codelineno-7-1083) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-1084) chat_template=chat_template, [](#__codelineno-7-1085) ) [](#__codelineno-7-1086) [](#__codelineno-7-1087)[](#__codelineno-7-1088)def load_qwen2_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1089) try: [](#__codelineno-7-1090) from qwen_vl_utils import smart_resize [](#__codelineno-7-1091) except ModuleNotFoundError: [](#__codelineno-7-1092) print( [](#__codelineno-7-1093) "WARNING: `qwen-vl-utils` not installed, input images will not " [](#__codelineno-7-1094) "be automatically resized. You can enable this functionality by " [](#__codelineno-7-1095) "`pip install qwen-vl-utils`." [](#__codelineno-7-1096) ) [](#__codelineno-7-1097) smart_resize = None [](#__codelineno-7-1098) [](#__codelineno-7-1099) model_name = "Qwen/Qwen2-VL-7B-Instruct" [](#__codelineno-7-1100) [](#__codelineno-7-1101) # Tested on L40 [](#__codelineno-7-1102) engine_args = EngineArgs( [](#__codelineno-7-1103) model=model_name, [](#__codelineno-7-1104) max_model_len=32768 if smart_resize is None else 4096, [](#__codelineno-7-1105) max_num_seqs=5, [](#__codelineno-7-1106) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1107) ) [](#__codelineno-7-1108) [](#__codelineno-7-1109) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1110) messages = [ [](#__codelineno-7-1111) {"role": "system", "content": "You are a helpful assistant."}, [](#__codelineno-7-1112) { [](#__codelineno-7-1113) "role": "user", [](#__codelineno-7-1114) "content": [ [](#__codelineno-7-1115) *placeholders, [](#__codelineno-7-1116) {"type": "text", "text": question}, [](#__codelineno-7-1117) ], [](#__codelineno-7-1118) }, [](#__codelineno-7-1119) ] [](#__codelineno-7-1120) [](#__codelineno-7-1121) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-1122) [](#__codelineno-7-1123) prompt = processor.apply_chat_template( [](#__codelineno-7-1124) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1125) ) [](#__codelineno-7-1126) [](#__codelineno-7-1127) if smart_resize is None: [](#__codelineno-7-1128) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1129) else: [](#__codelineno-7-1130) [](#__codelineno-7-1131) def post_process_image(image: Image) -> Image: [](#__codelineno-7-1132) width, height = image.size [](#__codelineno-7-1133) resized_height, resized_width = smart_resize( [](#__codelineno-7-1134) height, width, max_pixels=1024 * 28 * 28 [](#__codelineno-7-1135) ) [](#__codelineno-7-1136) return image.resize((resized_width, resized_height)) [](#__codelineno-7-1137) [](#__codelineno-7-1138) image_data = [post_process_image(fetch_image(url)) for url in image_urls] [](#__codelineno-7-1139) [](#__codelineno-7-1140) return ModelRequestData( [](#__codelineno-7-1141) engine_args=engine_args, [](#__codelineno-7-1142) prompt=prompt, [](#__codelineno-7-1143) image_data=image_data, [](#__codelineno-7-1144) ) [](#__codelineno-7-1145) [](#__codelineno-7-1146)[](#__codelineno-7-1147)def load_qwen2_5_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1148) try: [](#__codelineno-7-1149) from qwen_vl_utils import smart_resize [](#__codelineno-7-1150) except ModuleNotFoundError: [](#__codelineno-7-1151) print( [](#__codelineno-7-1152) "WARNING: `qwen-vl-utils` not installed, input images will not " [](#__codelineno-7-1153) "be automatically resized. You can enable this functionality by " [](#__codelineno-7-1154) "`pip install qwen-vl-utils`." [](#__codelineno-7-1155) ) [](#__codelineno-7-1156) smart_resize = None [](#__codelineno-7-1157) [](#__codelineno-7-1158) model_name = "Qwen/Qwen2.5-VL-3B-Instruct" [](#__codelineno-7-1159) [](#__codelineno-7-1160) engine_args = EngineArgs( [](#__codelineno-7-1161) model=model_name, [](#__codelineno-7-1162) max_model_len=32768 if smart_resize is None else 4096, [](#__codelineno-7-1163) max_num_seqs=5, [](#__codelineno-7-1164) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1165) ) [](#__codelineno-7-1166) [](#__codelineno-7-1167) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1168) messages = [ [](#__codelineno-7-1169) {"role": "system", "content": "You are a helpful assistant."}, [](#__codelineno-7-1170) { [](#__codelineno-7-1171) "role": "user", [](#__codelineno-7-1172) "content": [ [](#__codelineno-7-1173) *placeholders, [](#__codelineno-7-1174) {"type": "text", "text": question}, [](#__codelineno-7-1175) ], [](#__codelineno-7-1176) }, [](#__codelineno-7-1177) ] [](#__codelineno-7-1178) [](#__codelineno-7-1179) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-1180) [](#__codelineno-7-1181) prompt = processor.apply_chat_template( [](#__codelineno-7-1182) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1183) ) [](#__codelineno-7-1184) [](#__codelineno-7-1185) if smart_resize is None: [](#__codelineno-7-1186) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1187) else: [](#__codelineno-7-1188) [](#__codelineno-7-1189) def post_process_image(image: Image) -> Image: [](#__codelineno-7-1190) width, height = image.size [](#__codelineno-7-1191) resized_height, resized_width = smart_resize( [](#__codelineno-7-1192) height, width, max_pixels=1024 * 28 * 28 [](#__codelineno-7-1193) ) [](#__codelineno-7-1194) return image.resize((resized_width, resized_height)) [](#__codelineno-7-1195) [](#__codelineno-7-1196) image_data = [post_process_image(fetch_image(url)) for url in image_urls] [](#__codelineno-7-1197) [](#__codelineno-7-1198) return ModelRequestData( [](#__codelineno-7-1199) engine_args=engine_args, [](#__codelineno-7-1200) prompt=prompt, [](#__codelineno-7-1201) image_data=image_data, [](#__codelineno-7-1202) ) [](#__codelineno-7-1203) [](#__codelineno-7-1204)[](#__codelineno-7-1205)def load_r_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1206) model_name = "YannQi/R-4B" [](#__codelineno-7-1207) engine_args = EngineArgs( [](#__codelineno-7-1208) model=model_name, [](#__codelineno-7-1209) max_model_len=16384, [](#__codelineno-7-1210) max_num_seqs=16, [](#__codelineno-7-1211) trust_remote_code=True, [](#__codelineno-7-1212) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1213) ) [](#__codelineno-7-1214) [](#__codelineno-7-1215) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1216) messages = [ [](#__codelineno-7-1217) { [](#__codelineno-7-1218) "role": "user", [](#__codelineno-7-1219) "content": [ [](#__codelineno-7-1220) *placeholders, [](#__codelineno-7-1221) {"type": "text", "text": question}, [](#__codelineno-7-1222) ], [](#__codelineno-7-1223) } [](#__codelineno-7-1224) ] [](#__codelineno-7-1225) [](#__codelineno-7-1226) processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-7-1227) [](#__codelineno-7-1228) prompt = processor.apply_chat_template( [](#__codelineno-7-1229) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1230) ) [](#__codelineno-7-1231) [](#__codelineno-7-1232) return ModelRequestData( [](#__codelineno-7-1233) engine_args=engine_args, [](#__codelineno-7-1234) prompt=prompt, [](#__codelineno-7-1235) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-1236) ) [](#__codelineno-7-1237) [](#__codelineno-7-1238)[](#__codelineno-7-1239)def load_smolvlm(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1240) model_name = "HuggingFaceTB/SmolVLM2-2.2B-Instruct" [](#__codelineno-7-1241) [](#__codelineno-7-1242) # The configuration below has been confirmed to launch on a single L40 GPU. [](#__codelineno-7-1243) engine_args = EngineArgs( [](#__codelineno-7-1244) model=model_name, [](#__codelineno-7-1245) max_model_len=8192, [](#__codelineno-7-1246) max_num_seqs=16, [](#__codelineno-7-1247) enforce_eager=True, [](#__codelineno-7-1248) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1249) mm_processor_kwargs={ [](#__codelineno-7-1250) "max_image_size": {"longest_edge": 384}, [](#__codelineno-7-1251) }, [](#__codelineno-7-1252) ) [](#__codelineno-7-1253) [](#__codelineno-7-1254) placeholders = "\n".join( [](#__codelineno-7-1255) f"Image-{i}: \n" for i, _ in enumerate(image_urls, start=1) [](#__codelineno-7-1256) ) [](#__codelineno-7-1257) prompt = ( [](#__codelineno-7-1258) f"<|im_start|>User:{placeholders}\n{question}\nAssistant:" # noqa: E501 [](#__codelineno-7-1259) ) [](#__codelineno-7-1260) return ModelRequestData( [](#__codelineno-7-1261) engine_args=engine_args, [](#__codelineno-7-1262) prompt=prompt, [](#__codelineno-7-1263) image_data=[fetch_image(url) for url in image_urls], [](#__codelineno-7-1264) ) [](#__codelineno-7-1265) [](#__codelineno-7-1266)[](#__codelineno-7-1267)def load_step3(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1268) model_name = "stepfun-ai/step3-fp8" [](#__codelineno-7-1269) [](#__codelineno-7-1270) # NOTE: Below are verified configurations for step3-fp8 [](#__codelineno-7-1271) # on 8xH100 GPUs. [](#__codelineno-7-1272) engine_args = EngineArgs( [](#__codelineno-7-1273) model=model_name, [](#__codelineno-7-1274) max_num_batched_tokens=4096, [](#__codelineno-7-1275) gpu_memory_utilization=0.85, [](#__codelineno-7-1276) tensor_parallel_size=8, [](#__codelineno-7-1277) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1278) reasoning_parser="step3", [](#__codelineno-7-1279) ) [](#__codelineno-7-1280) [](#__codelineno-7-1281) prompt = ( [](#__codelineno-7-1282) "<｜begin▁of▁sentence｜> You are a helpful assistant. <|BOT|>user\n " [](#__codelineno-7-1283) f"{'' * len(image_urls)}{question} <|EOT|><|BOT|" [](#__codelineno-7-1284) ">assistant\n\n" [](#__codelineno-7-1285) ) [](#__codelineno-7-1286) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1287) [](#__codelineno-7-1288) return ModelRequestData( [](#__codelineno-7-1289) engine_args=engine_args, [](#__codelineno-7-1290) prompt=prompt, [](#__codelineno-7-1291) image_data=image_data, [](#__codelineno-7-1292) ) [](#__codelineno-7-1293) [](#__codelineno-7-1294)[](#__codelineno-7-1295)def load_step_vl(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1296) model_name = "stepfun-ai/Step3-VL-10B" [](#__codelineno-7-1297) [](#__codelineno-7-1298) engine_args = EngineArgs( [](#__codelineno-7-1299) model=model_name, [](#__codelineno-7-1300) max_num_batched_tokens=4096, [](#__codelineno-7-1301) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1302) hf_overrides={"vision_config": {"enable_patch": False}}, [](#__codelineno-7-1303) trust_remote_code=True, [](#__codelineno-7-1304) reasoning_parser="deepseek_r1", [](#__codelineno-7-1305) ) [](#__codelineno-7-1306) [](#__codelineno-7-1307) prompt = ( [](#__codelineno-7-1308) "<｜begin▁of▁sentence｜> You are a helpful assistant.<|BOT|>user\n " [](#__codelineno-7-1309) f"{'' * len(image_urls)}{question}<|EOT|><|BOT|>" [](#__codelineno-7-1310) "assistant\n\n" [](#__codelineno-7-1311) ) [](#__codelineno-7-1312) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1313) [](#__codelineno-7-1314) return ModelRequestData( [](#__codelineno-7-1315) engine_args=engine_args, [](#__codelineno-7-1316) prompt=prompt, [](#__codelineno-7-1317) image_data=image_data, [](#__codelineno-7-1318) ) [](#__codelineno-7-1319) [](#__codelineno-7-1320)[](#__codelineno-7-1321)def load_tarsier(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1322) model_name = "omni-research/Tarsier-7b" [](#__codelineno-7-1323) [](#__codelineno-7-1324) engine_args = EngineArgs( [](#__codelineno-7-1325) model=model_name, [](#__codelineno-7-1326) trust_remote_code=True, [](#__codelineno-7-1327) max_model_len=4096, [](#__codelineno-7-1328) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1329) ) [](#__codelineno-7-1330) [](#__codelineno-7-1331) prompt = f"USER: {'' * len(image_urls)}\n{question}\n ASSISTANT:" [](#__codelineno-7-1332) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1333) [](#__codelineno-7-1334) return ModelRequestData( [](#__codelineno-7-1335) engine_args=engine_args, [](#__codelineno-7-1336) prompt=prompt, [](#__codelineno-7-1337) image_data=image_data, [](#__codelineno-7-1338) ) [](#__codelineno-7-1339) [](#__codelineno-7-1340)[](#__codelineno-7-1341)def load_tarsier2(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1342) model_name = "omni-research/Tarsier2-Recap-7b" [](#__codelineno-7-1343) [](#__codelineno-7-1344) engine_args = EngineArgs( [](#__codelineno-7-1345) model=model_name, [](#__codelineno-7-1346) trust_remote_code=True, [](#__codelineno-7-1347) max_model_len=32768, [](#__codelineno-7-1348) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1349) hf_overrides={ [](#__codelineno-7-1350) "architectures": ["Tarsier2ForConditionalGeneration"], [](#__codelineno-7-1351) "model_type": "tarsier2", [](#__codelineno-7-1352) }, [](#__codelineno-7-1353) ) [](#__codelineno-7-1354) [](#__codelineno-7-1355) prompt = ( [](#__codelineno-7-1356) "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" [](#__codelineno-7-1357) f"<|im_start|>user\n<|vision_start|>{'<|image_pad|>' * len(image_urls)}" [](#__codelineno-7-1358) f"<|vision_end|>{question}<|im_end|>\n" [](#__codelineno-7-1359) "<|im_start|>assistant\n" [](#__codelineno-7-1360) ) [](#__codelineno-7-1361) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1362) [](#__codelineno-7-1363) return ModelRequestData( [](#__codelineno-7-1364) engine_args=engine_args, [](#__codelineno-7-1365) prompt=prompt, [](#__codelineno-7-1366) image_data=image_data, [](#__codelineno-7-1367) ) [](#__codelineno-7-1368) [](#__codelineno-7-1369)[](#__codelineno-7-1370)# GLM-4.1V [](#__codelineno-7-1371)def load_glm4_1v(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1372) model_name = "zai-org/GLM-4.1V-9B-Thinking" [](#__codelineno-7-1373) [](#__codelineno-7-1374) engine_args = EngineArgs( [](#__codelineno-7-1375) model=model_name, [](#__codelineno-7-1376) max_model_len=45082, [](#__codelineno-7-1377) max_num_seqs=2, [](#__codelineno-7-1378) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1379) enforce_eager=True, [](#__codelineno-7-1380) ) [](#__codelineno-7-1381) [](#__codelineno-7-1382) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1383) messages = [ [](#__codelineno-7-1384) { [](#__codelineno-7-1385) "role": "user", [](#__codelineno-7-1386) "content": [ [](#__codelineno-7-1387) *placeholders, [](#__codelineno-7-1388) {"type": "text", "text": question}, [](#__codelineno-7-1389) ], [](#__codelineno-7-1390) } [](#__codelineno-7-1391) ] [](#__codelineno-7-1392) [](#__codelineno-7-1393) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-1394) prompt = processor.apply_chat_template( [](#__codelineno-7-1395) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1396) ) [](#__codelineno-7-1397) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1398) [](#__codelineno-7-1399) return ModelRequestData( [](#__codelineno-7-1400) engine_args=engine_args, [](#__codelineno-7-1401) prompt=prompt, [](#__codelineno-7-1402) image_data=image_data, [](#__codelineno-7-1403) ) [](#__codelineno-7-1404) [](#__codelineno-7-1405)[](#__codelineno-7-1406)# GLM-4.5V [](#__codelineno-7-1407)def load_glm4_5v(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1408) model_name = "zai-org/GLM-4.5V" [](#__codelineno-7-1409) [](#__codelineno-7-1410) engine_args = EngineArgs( [](#__codelineno-7-1411) model=model_name, [](#__codelineno-7-1412) max_model_len=32768, [](#__codelineno-7-1413) max_num_seqs=2, [](#__codelineno-7-1414) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1415) enforce_eager=True, [](#__codelineno-7-1416) tensor_parallel_size=4, [](#__codelineno-7-1417) ) [](#__codelineno-7-1418) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1419) messages = [ [](#__codelineno-7-1420) { [](#__codelineno-7-1421) "role": "user", [](#__codelineno-7-1422) "content": [ [](#__codelineno-7-1423) *placeholders, [](#__codelineno-7-1424) {"type": "text", "text": question}, [](#__codelineno-7-1425) ], [](#__codelineno-7-1426) } [](#__codelineno-7-1427) ] [](#__codelineno-7-1428) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-1429) prompt = processor.apply_chat_template( [](#__codelineno-7-1430) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1431) ) [](#__codelineno-7-1432) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1433) [](#__codelineno-7-1434) return ModelRequestData( [](#__codelineno-7-1435) engine_args=engine_args, [](#__codelineno-7-1436) prompt=prompt, [](#__codelineno-7-1437) image_data=image_data, [](#__codelineno-7-1438) ) [](#__codelineno-7-1439) [](#__codelineno-7-1440)[](#__codelineno-7-1441)# GLM-4.5V-FP8 [](#__codelineno-7-1442)def load_glm4_5v_fp8(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1443) model_name = "zai-org/GLM-4.5V-FP8" [](#__codelineno-7-1444) [](#__codelineno-7-1445) engine_args = EngineArgs( [](#__codelineno-7-1446) model=model_name, [](#__codelineno-7-1447) max_model_len=32768, [](#__codelineno-7-1448) max_num_seqs=2, [](#__codelineno-7-1449) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1450) enforce_eager=True, [](#__codelineno-7-1451) tensor_parallel_size=4, [](#__codelineno-7-1452) ) [](#__codelineno-7-1453) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1454) messages = [ [](#__codelineno-7-1455) { [](#__codelineno-7-1456) "role": "user", [](#__codelineno-7-1457) "content": [ [](#__codelineno-7-1458) *placeholders, [](#__codelineno-7-1459) {"type": "text", "text": question}, [](#__codelineno-7-1460) ], [](#__codelineno-7-1461) } [](#__codelineno-7-1462) ] [](#__codelineno-7-1463) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-1464) prompt = processor.apply_chat_template( [](#__codelineno-7-1465) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1466) ) [](#__codelineno-7-1467) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1468) [](#__codelineno-7-1469) return ModelRequestData( [](#__codelineno-7-1470) engine_args=engine_args, [](#__codelineno-7-1471) prompt=prompt, [](#__codelineno-7-1472) image_data=image_data, [](#__codelineno-7-1473) ) [](#__codelineno-7-1474) [](#__codelineno-7-1475)[](#__codelineno-7-1476)def load_molmo2(question: str, image_urls: list[str]) -> ModelRequestData: [](#__codelineno-7-1477) model_name = "allenai/Molmo2-8B" [](#__codelineno-7-1478) [](#__codelineno-7-1479) engine_args = EngineArgs( [](#__codelineno-7-1480) model=model_name, [](#__codelineno-7-1481) trust_remote_code=True, [](#__codelineno-7-1482) dtype="bfloat16", [](#__codelineno-7-1483) limit_mm_per_prompt={"image": len(image_urls)}, [](#__codelineno-7-1484) max_num_batched_tokens=36864, [](#__codelineno-7-1485) ) [](#__codelineno-7-1486) [](#__codelineno-7-1487) placeholders = [{"type": "image", "image": url} for url in image_urls] [](#__codelineno-7-1488) messages = [ [](#__codelineno-7-1489) { [](#__codelineno-7-1490) "role": "user", [](#__codelineno-7-1491) "content": [ [](#__codelineno-7-1492) *placeholders, [](#__codelineno-7-1493) {"type": "text", "text": question}, [](#__codelineno-7-1494) ], [](#__codelineno-7-1495) }, [](#__codelineno-7-1496) ] [](#__codelineno-7-1497) [](#__codelineno-7-1498) processor = AutoProcessor.from_pretrained(model_name) [](#__codelineno-7-1499) [](#__codelineno-7-1500) prompt = processor.apply_chat_template( [](#__codelineno-7-1501) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-7-1502) ) [](#__codelineno-7-1503) [](#__codelineno-7-1504) image_data = [fetch_image(url) for url in image_urls] [](#__codelineno-7-1505) [](#__codelineno-7-1506) return ModelRequestData( [](#__codelineno-7-1507) engine_args=engine_args, [](#__codelineno-7-1508) prompt=prompt, [](#__codelineno-7-1509) image_data=image_data, [](#__codelineno-7-1510) ) [](#__codelineno-7-1511) [](#__codelineno-7-1512)[](#__codelineno-7-1513)model_example_map = { [](#__codelineno-7-1514) "aria": load_aria, [](#__codelineno-7-1515) "aya_vision": load_aya_vision, [](#__codelineno-7-1516) "bee": load_bee, [](#__codelineno-7-1517) "command_a_vision": load_command_a_vision, [](#__codelineno-7-1518) "deepseek_vl_v2": load_deepseek_vl2, [](#__codelineno-7-1519) "deepseek_ocr": load_deepseek_ocr, [](#__codelineno-7-1520) "exaone4_5": load_exaone4_5, [](#__codelineno-7-1521) "gemma3": load_gemma3, [](#__codelineno-7-1522) "granite4_vision": load_granite4_vision, [](#__codelineno-7-1523) "h2ovl_chat": load_h2ovl, [](#__codelineno-7-1524) "hunyuan_vl": load_hunyuan_vl, [](#__codelineno-7-1525) "hyperclovax_seed_vision": load_hyperclovax_seed_vision, [](#__codelineno-7-1526) "idefics3": load_idefics3, [](#__codelineno-7-1527) "interns1": load_interns1, [](#__codelineno-7-1528) "internvl_chat": load_internvl, [](#__codelineno-7-1529) "keye_vl": load_keye_vl, [](#__codelineno-7-1530) "keye_vl1_5": load_keye_vl1_5, [](#__codelineno-7-1531) "kimi_vl": load_kimi_vl, [](#__codelineno-7-1532) "llama4": load_llama4, [](#__codelineno-7-1533) "llava": load_llava, [](#__codelineno-7-1534) "llava-next": load_llava_next, [](#__codelineno-7-1535) "llava-onevision": load_llava_onevision, [](#__codelineno-7-1536) "mistral3": load_mistral3, [](#__codelineno-7-1537) "molmo2": load_molmo2, [](#__codelineno-7-1538) "NVLM_D": load_nvlm_d, [](#__codelineno-7-1539) "openpangu_vl": load_openpangu_vl, [](#__codelineno-7-1540) "ovis": load_ovis, [](#__codelineno-7-1541) "ovis2_5": load_ovis2_5, [](#__codelineno-7-1542) "paddleocr_vl": load_paddleocr_vl, [](#__codelineno-7-1543) "phi3_v": load_phi3v, [](#__codelineno-7-1544) "phi4_mm": load_phi4mm, [](#__codelineno-7-1545) "phi4_siglip": load_phi4siglip, [](#__codelineno-7-1546) "pixtral_hf": load_pixtral_hf, [](#__codelineno-7-1547) "qwen_vl_chat": load_qwen_vl_chat, [](#__codelineno-7-1548) "qwen2_vl": load_qwen2_vl, [](#__codelineno-7-1549) "qwen2_5_vl": load_qwen2_5_vl, [](#__codelineno-7-1550) "rvl": load_r_vl, [](#__codelineno-7-1551) "smolvlm": load_smolvlm, [](#__codelineno-7-1552) "step3": load_step3, [](#__codelineno-7-1553) "stepvl": load_step_vl, [](#__codelineno-7-1554) "tarsier": load_tarsier, [](#__codelineno-7-1555) "tarsier2": load_tarsier2, [](#__codelineno-7-1556) "glm4_1v": load_glm4_1v, [](#__codelineno-7-1557) "glm4_5v": load_glm4_5v, [](#__codelineno-7-1558) "glm4_5v_fp8": load_glm4_5v_fp8, [](#__codelineno-7-1559)} [](#__codelineno-7-1560) [](#__codelineno-7-1561)[](#__codelineno-7-1562)def run_generate( [](#__codelineno-7-1563) model, [](#__codelineno-7-1564) question: str, [](#__codelineno-7-1565) image_urls: list[str], [](#__codelineno-7-1566) seed: int, [](#__codelineno-7-1567) tensor_parallel_size: int | None, [](#__codelineno-7-1568)): [](#__codelineno-7-1569) req_data = model_example_map[model](question, image_urls) [](#__codelineno-7-1570) [](#__codelineno-7-1571) engine_args = req_data.engine_args [](#__codelineno-7-1572) engine_args.seed = seed [](#__codelineno-7-1573) if tensor_parallel_size is not None: [](#__codelineno-7-1574) engine_args.tensor_parallel_size = tensor_parallel_size [](#__codelineno-7-1575) llm = LLM.from_engine_args(engine_args) [](#__codelineno-7-1576) [](#__codelineno-7-1577) sampling_params = SamplingParams( [](#__codelineno-7-1578) temperature=0.0, max_tokens=256, stop_token_ids=req_data.stop_token_ids [](#__codelineno-7-1579) ) [](#__codelineno-7-1580) [](#__codelineno-7-1581) outputs = llm.generate( [](#__codelineno-7-1582) { [](#__codelineno-7-1583) "prompt": req_data.prompt, [](#__codelineno-7-1584) "multi_modal_data": {"image": req_data.image_data}, [](#__codelineno-7-1585) }, [](#__codelineno-7-1586) sampling_params=sampling_params, [](#__codelineno-7-1587) lora_request=req_data.lora_requests, [](#__codelineno-7-1588) ) [](#__codelineno-7-1589) [](#__codelineno-7-1590) print("-" * 50) [](#__codelineno-7-1591) for o in outputs: [](#__codelineno-7-1592) generated_text = o.outputs[0].text [](#__codelineno-7-1593) print(generated_text) [](#__codelineno-7-1594) print("-" * 50) [](#__codelineno-7-1595) [](#__codelineno-7-1596)[](#__codelineno-7-1597)def run_chat( [](#__codelineno-7-1598) model: str, [](#__codelineno-7-1599) question: str, [](#__codelineno-7-1600) image_urls: list[str], [](#__codelineno-7-1601) seed: int, [](#__codelineno-7-1602) tensor_parallel_size: int | None, [](#__codelineno-7-1603)): [](#__codelineno-7-1604) req_data = model_example_map[model](question, image_urls) [](#__codelineno-7-1605) [](#__codelineno-7-1606) # Disable other modalities to save memory [](#__codelineno-7-1607) default_limits = {"image": 0, "video": 0, "audio": 0} [](#__codelineno-7-1608) req_data.engine_args.limit_mm_per_prompt = default_limits | dict( [](#__codelineno-7-1609) req_data.engine_args.limit_mm_per_prompt or {} [](#__codelineno-7-1610) ) [](#__codelineno-7-1611) [](#__codelineno-7-1612) engine_args = req_data.engine_args [](#__codelineno-7-1613) engine_args.seed = seed [](#__codelineno-7-1614) if tensor_parallel_size is not None: [](#__codelineno-7-1615) engine_args.tensor_parallel_size = tensor_parallel_size [](#__codelineno-7-1616) llm = LLM.from_engine_args(engine_args) [](#__codelineno-7-1617) [](#__codelineno-7-1618) sampling_params = ( [](#__codelineno-7-1619) SamplingParams( [](#__codelineno-7-1620) temperature=0.0, max_tokens=256, stop_token_ids=req_data.stop_token_ids [](#__codelineno-7-1621) ) [](#__codelineno-7-1622) if req_data.sampling_params is None [](#__codelineno-7-1623) else req_data.sampling_params [](#__codelineno-7-1624) ) [](#__codelineno-7-1625) outputs = llm.chat( [](#__codelineno-7-1626) [ [](#__codelineno-7-1627) { [](#__codelineno-7-1628) "role": "user", [](#__codelineno-7-1629) "content": [ [](#__codelineno-7-1630) { [](#__codelineno-7-1631) "type": "text", [](#__codelineno-7-1632) "text": question, [](#__codelineno-7-1633) }, [](#__codelineno-7-1634) *( [](#__codelineno-7-1635) { [](#__codelineno-7-1636) "type": "image_url", [](#__codelineno-7-1637) "image_url": {"url": image_url}, [](#__codelineno-7-1638) } [](#__codelineno-7-1639) for image_url in image_urls [](#__codelineno-7-1640) ), [](#__codelineno-7-1641) ], [](#__codelineno-7-1642) } [](#__codelineno-7-1643) ], [](#__codelineno-7-1644) sampling_params=sampling_params, [](#__codelineno-7-1645) chat_template=req_data.chat_template, [](#__codelineno-7-1646) lora_request=req_data.lora_requests, [](#__codelineno-7-1647) ) [](#__codelineno-7-1648) [](#__codelineno-7-1649) print("-" * 50) [](#__codelineno-7-1650) for o in outputs: [](#__codelineno-7-1651) generated_text = o.outputs[0].text [](#__codelineno-7-1652) print(generated_text) [](#__codelineno-7-1653) print("-" * 50) [](#__codelineno-7-1654) [](#__codelineno-7-1655)[](#__codelineno-7-1656)def parse_args(): [](#__codelineno-7-1657) parser = FlexibleArgumentParser( [](#__codelineno-7-1658) description="Demo on using vLLM for offline inference with " [](#__codelineno-7-1659) "vision language models that support multi-image input for text " [](#__codelineno-7-1660) "generation" [](#__codelineno-7-1661) ) [](#__codelineno-7-1662) parser.add_argument( [](#__codelineno-7-1663) "--model-type", [](#__codelineno-7-1664) "-m", [](#__codelineno-7-1665) type=str, [](#__codelineno-7-1666) default="phi3_v", [](#__codelineno-7-1667) choices=model_example_map.keys(), [](#__codelineno-7-1668) help='Huggingface "model_type".', [](#__codelineno-7-1669) ) [](#__codelineno-7-1670) parser.add_argument( [](#__codelineno-7-1671) "--method", [](#__codelineno-7-1672) type=str, [](#__codelineno-7-1673) default="generate", [](#__codelineno-7-1674) choices=["generate", "chat"], [](#__codelineno-7-1675) help="The method to run in `vllm.LLM`.", [](#__codelineno-7-1676) ) [](#__codelineno-7-1677) parser.add_argument( [](#__codelineno-7-1678) "--seed", [](#__codelineno-7-1679) type=int, [](#__codelineno-7-1680) default=0, [](#__codelineno-7-1681) help="Set the seed when initializing `vllm.LLM`.", [](#__codelineno-7-1682) ) [](#__codelineno-7-1683) parser.add_argument( [](#__codelineno-7-1684) "--num-images", [](#__codelineno-7-1685) "-n", [](#__codelineno-7-1686) type=int, [](#__codelineno-7-1687) choices=list(range(1, len(IMAGE_URLS) + 1)), # the max number of images [](#__codelineno-7-1688) default=2, [](#__codelineno-7-1689) help="Number of images to use for the demo.", [](#__codelineno-7-1690) ) [](#__codelineno-7-1691) parser.add_argument( [](#__codelineno-7-1692) "--tensor-parallel-size", [](#__codelineno-7-1693) "-tp", [](#__codelineno-7-1694) type=int, [](#__codelineno-7-1695) default=None, [](#__codelineno-7-1696) help="Tensor parallel size to override the model's default setting. ", [](#__codelineno-7-1697) ) [](#__codelineno-7-1698) return parser.parse_args() [](#__codelineno-7-1699) [](#__codelineno-7-1700)[](#__codelineno-7-1701)def main(args: Namespace): [](#__codelineno-7-1702) model = args.model_type [](#__codelineno-7-1703) method = args.method [](#__codelineno-7-1704) seed = args.seed [](#__codelineno-7-1705) tensor_parallel_size = args.tensor_parallel_size [](#__codelineno-7-1706) [](#__codelineno-7-1707) if tensor_parallel_size is not None and tensor_parallel_size < 1: [](#__codelineno-7-1708) raise ValueError( [](#__codelineno-7-1709) f"tensor_parallel_size must be a positive integer, " [](#__codelineno-7-1710) f"got {tensor_parallel_size}" [](#__codelineno-7-1711) ) [](#__codelineno-7-1712) [](#__codelineno-7-1713) image_urls = IMAGE_URLS[: args.num_images] [](#__codelineno-7-1714) [](#__codelineno-7-1715) if method == "generate": [](#__codelineno-7-1716) run_generate(model, QUESTION, image_urls, seed, tensor_parallel_size) [](#__codelineno-7-1717) elif method == "chat": [](#__codelineno-7-1718) run_chat(model, QUESTION, image_urls, seed, tensor_parallel_size) [](#__codelineno-7-1719) else: [](#__codelineno-7-1720) raise ValueError(f"Invalid method: {method}") [](#__codelineno-7-1721) [](#__codelineno-7-1722)[](#__codelineno-7-1723)if __name__ == "__main__": [](#__codelineno-7-1724) args = parse_args() [](#__codelineno-7-1725) main(args)`` ## Vision Language Offline[¶](#vision-language-offline "Permanent link") ``[](#__codelineno-8-1)# SPDX-License-Identifier: Apache-2.0 [](#__codelineno-8-2)# SPDX-FileCopyrightText: Copyright contributors to the vLLM project [](#__codelineno-8-3)""" [](#__codelineno-8-4)This example shows how to use vLLM for running offline inference with [](#__codelineno-8-5)the correct prompt format on vision language models for text generation. [](#__codelineno-8-6)[](#__codelineno-8-7)For most models, the prompt format should follow corresponding examples [](#__codelineno-8-8)on HuggingFace model repository. [](#__codelineno-8-9)""" [](#__codelineno-8-10)[](#__codelineno-8-11)import os [](#__codelineno-8-12)import random [](#__codelineno-8-13)from contextlib import contextmanager [](#__codelineno-8-14)from typing import NamedTuple [](#__codelineno-8-15)[](#__codelineno-8-16)from huggingface_hub import snapshot_download [](#__codelineno-8-17)from transformers import AutoProcessor, AutoTokenizer [](#__codelineno-8-18)[](#__codelineno-8-19)from vllm import LLM, EngineArgs, SamplingParams [](#__codelineno-8-20)from vllm.assets.image import ImageAsset [](#__codelineno-8-21)from vllm.assets.video import VideoAsset [](#__codelineno-8-22)from vllm.lora.request import LoRARequest [](#__codelineno-8-23)from vllm.multimodal.image import convert_image_mode [](#__codelineno-8-24)from vllm.utils.argparse_utils import FlexibleArgumentParser [](#__codelineno-8-25) [](#__codelineno-8-26)[](#__codelineno-8-27)class ModelRequestData(NamedTuple): [](#__codelineno-8-28) engine_args: EngineArgs [](#__codelineno-8-29) prompts: list[str] [](#__codelineno-8-30) stop_token_ids: list[int] | None = None [](#__codelineno-8-31) lora_requests: list[LoRARequest] | None = None [](#__codelineno-8-32) sampling_params: list[SamplingParams] | None = None [](#__codelineno-8-33) [](#__codelineno-8-34)[](#__codelineno-8-35)# NOTE: The default `max_num_seqs` and `max_model_len` may result in OOM on [](#__codelineno-8-36)# lower-end GPUs. [](#__codelineno-8-37)# Unless specified, these settings have been tested to work on a single L4. [](#__codelineno-8-38) [](#__codelineno-8-39)[](#__codelineno-8-40)# Aria [](#__codelineno-8-41)def run_aria(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-42) assert modality == "image" [](#__codelineno-8-43) model_name = "rhymes-ai/Aria" [](#__codelineno-8-44) [](#__codelineno-8-45) # NOTE: Need L40 (or equivalent) to avoid OOM [](#__codelineno-8-46) engine_args = EngineArgs( [](#__codelineno-8-47) model=model_name, [](#__codelineno-8-48) max_model_len=4096, [](#__codelineno-8-49) max_num_seqs=2, [](#__codelineno-8-50) dtype="bfloat16", [](#__codelineno-8-51) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-52) ) [](#__codelineno-8-53) [](#__codelineno-8-54) prompts = [ [](#__codelineno-8-55) ( [](#__codelineno-8-56) f"<|im_start|>user\n<|img|>{question}" [](#__codelineno-8-57) "<|im_end|>\n<|im_start|>assistant\n" [](#__codelineno-8-58) ) [](#__codelineno-8-59) for question in questions [](#__codelineno-8-60) ] [](#__codelineno-8-61) [](#__codelineno-8-62) stop_token_ids = [93532, 93653, 944, 93421, 1019, 93653, 93519] [](#__codelineno-8-63) [](#__codelineno-8-64) return ModelRequestData( [](#__codelineno-8-65) engine_args=engine_args, [](#__codelineno-8-66) prompts=prompts, [](#__codelineno-8-67) stop_token_ids=stop_token_ids, [](#__codelineno-8-68) ) [](#__codelineno-8-69) [](#__codelineno-8-70)[](#__codelineno-8-71)# Aya Vision [](#__codelineno-8-72)def run_aya_vision(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-73) assert modality == "image" [](#__codelineno-8-74) model_name = "CohereLabs/aya-vision-8b" [](#__codelineno-8-75) [](#__codelineno-8-76) engine_args = EngineArgs( [](#__codelineno-8-77) model=model_name, [](#__codelineno-8-78) max_model_len=2048, [](#__codelineno-8-79) max_num_seqs=2, [](#__codelineno-8-80) mm_processor_kwargs={"crop_to_patches": True}, [](#__codelineno-8-81) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-82) ) [](#__codelineno-8-83) prompts = [ [](#__codelineno-8-84) f"<|START_OF_TURN_TOKEN|><|USER_TOKEN|>{question}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>" [](#__codelineno-8-85) for question in questions [](#__codelineno-8-86) ] [](#__codelineno-8-87) return ModelRequestData( [](#__codelineno-8-88) engine_args=engine_args, [](#__codelineno-8-89) prompts=prompts, [](#__codelineno-8-90) ) [](#__codelineno-8-91) [](#__codelineno-8-92)[](#__codelineno-8-93)# Bee-8B [](#__codelineno-8-94)def run_bee(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-95) assert modality == "image" [](#__codelineno-8-96) model_name = "Open-Bee/Bee-8B-RL" [](#__codelineno-8-97) [](#__codelineno-8-98) prompts = [ [](#__codelineno-8-99) ( [](#__codelineno-8-100) f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" [](#__codelineno-8-101) f"<|im_start|>user\n\n{question}<|im_end|>" [](#__codelineno-8-102) f"<|im_start|>assistant\n\n" [](#__codelineno-8-103) ) [](#__codelineno-8-104) for question in questions [](#__codelineno-8-105) ] [](#__codelineno-8-106) [](#__codelineno-8-107) engine_args = EngineArgs( [](#__codelineno-8-108) model=model_name, [](#__codelineno-8-109) max_model_len=16384, [](#__codelineno-8-110) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-111) trust_remote_code=True, [](#__codelineno-8-112) ) [](#__codelineno-8-113) [](#__codelineno-8-114) return ModelRequestData( [](#__codelineno-8-115) engine_args=engine_args, [](#__codelineno-8-116) prompts=prompts, [](#__codelineno-8-117) ) [](#__codelineno-8-118) [](#__codelineno-8-119)[](#__codelineno-8-120)def run_bagel(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-121) assert modality == "image" [](#__codelineno-8-122) model_name = "ByteDance-Seed/BAGEL-7B-MoT" [](#__codelineno-8-123) [](#__codelineno-8-124) engine_args = EngineArgs( [](#__codelineno-8-125) model=model_name, [](#__codelineno-8-126) trust_remote_code=True, [](#__codelineno-8-127) max_model_len=8192, [](#__codelineno-8-128) max_num_seqs=2, [](#__codelineno-8-129) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-130) ) [](#__codelineno-8-131) [](#__codelineno-8-132) prompts = [ [](#__codelineno-8-133) ( [](#__codelineno-8-134) f"<|im_start|>user\n<|image_pad|>\n{question}<|im_end|>\n" [](#__codelineno-8-135) f"<|im_start|>assistant\n" [](#__codelineno-8-136) ) [](#__codelineno-8-137) for question in questions [](#__codelineno-8-138) ] [](#__codelineno-8-139) [](#__codelineno-8-140) return ModelRequestData( [](#__codelineno-8-141) engine_args=engine_args, [](#__codelineno-8-142) prompts=prompts, [](#__codelineno-8-143) ) [](#__codelineno-8-144) [](#__codelineno-8-145)[](#__codelineno-8-146)# BLIP-2 [](#__codelineno-8-147)def run_blip2(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-148) assert modality == "image" [](#__codelineno-8-149) [](#__codelineno-8-150) # BLIP-2 prompt format is inaccurate on HuggingFace model repository. [](#__codelineno-8-151) # See https://huggingface.co/Salesforce/blip2-opt-2.7b/discussions/15#64ff02f3f8cf9e4f5b038262 #noqa [](#__codelineno-8-152) prompts = [f"Question: {question} Answer:" for question in questions] [](#__codelineno-8-153) engine_args = EngineArgs( [](#__codelineno-8-154) model="Salesforce/blip2-opt-2.7b", [](#__codelineno-8-155) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-156) ) [](#__codelineno-8-157) [](#__codelineno-8-158) return ModelRequestData( [](#__codelineno-8-159) engine_args=engine_args, [](#__codelineno-8-160) prompts=prompts, [](#__codelineno-8-161) ) [](#__codelineno-8-162) [](#__codelineno-8-163)[](#__codelineno-8-164)# Chameleon [](#__codelineno-8-165)def run_chameleon(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-166) assert modality == "image" [](#__codelineno-8-167) [](#__codelineno-8-168) prompts = [f"{question}" for question in questions] [](#__codelineno-8-169) engine_args = EngineArgs( [](#__codelineno-8-170) model="facebook/chameleon-7b", [](#__codelineno-8-171) max_model_len=4096, [](#__codelineno-8-172) max_num_seqs=2, [](#__codelineno-8-173) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-174) ) [](#__codelineno-8-175) [](#__codelineno-8-176) return ModelRequestData( [](#__codelineno-8-177) engine_args=engine_args, [](#__codelineno-8-178) prompts=prompts, [](#__codelineno-8-179) ) [](#__codelineno-8-180) [](#__codelineno-8-181)[](#__codelineno-8-182)# Cheers [](#__codelineno-8-183)def run_cheers(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-184) assert modality == "image" [](#__codelineno-8-185) model_name = "ai9stars/Cheers" [](#__codelineno-8-186) [](#__codelineno-8-187) engine_args = EngineArgs( [](#__codelineno-8-188) model=model_name, [](#__codelineno-8-189) trust_remote_code=True, [](#__codelineno-8-190) max_model_len=4096, [](#__codelineno-8-191) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-192) ) [](#__codelineno-8-193) [](#__codelineno-8-194) prompts = [ [](#__codelineno-8-195) ( [](#__codelineno-8-196) f"<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n" [](#__codelineno-8-197) f"<|im_start|>user\n<|image_pad|>{question}<|im_end|>\n" [](#__codelineno-8-198) f"<|im_start|>assistant\n" [](#__codelineno-8-199) ) [](#__codelineno-8-200) for question in questions [](#__codelineno-8-201) ] [](#__codelineno-8-202) [](#__codelineno-8-203) return ModelRequestData( [](#__codelineno-8-204) engine_args=engine_args, [](#__codelineno-8-205) prompts=prompts, [](#__codelineno-8-206) ) [](#__codelineno-8-207) [](#__codelineno-8-208)[](#__codelineno-8-209)def run_command_a_vision(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-210) assert modality == "image" [](#__codelineno-8-211) [](#__codelineno-8-212) model_name = "CohereLabs/command-a-vision-07-2025" [](#__codelineno-8-213) [](#__codelineno-8-214) engine_args = EngineArgs( [](#__codelineno-8-215) model=model_name, [](#__codelineno-8-216) max_model_len=32768, [](#__codelineno-8-217) tensor_parallel_size=4, [](#__codelineno-8-218) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-219) ) [](#__codelineno-8-220) [](#__codelineno-8-221) prompts = [ [](#__codelineno-8-222) f"<|START_OF_TURN_TOKEN|><|USER_TOKEN|><|IMG_PATCH|>{question}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>" [](#__codelineno-8-223) for question in questions [](#__codelineno-8-224) ] [](#__codelineno-8-225) [](#__codelineno-8-226) return ModelRequestData( [](#__codelineno-8-227) engine_args=engine_args, [](#__codelineno-8-228) prompts=prompts, [](#__codelineno-8-229) ) [](#__codelineno-8-230) [](#__codelineno-8-231)[](#__codelineno-8-232)# Deepseek-VL2 [](#__codelineno-8-233)def run_deepseek_vl2(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-234) assert modality == "image" [](#__codelineno-8-235) [](#__codelineno-8-236) model_name = "deepseek-ai/deepseek-vl2-tiny" [](#__codelineno-8-237) [](#__codelineno-8-238) engine_args = EngineArgs( [](#__codelineno-8-239) model=model_name, [](#__codelineno-8-240) max_model_len=4096, [](#__codelineno-8-241) max_num_seqs=2, [](#__codelineno-8-242) hf_overrides={"architectures": ["DeepseekVLV2ForCausalLM"]}, [](#__codelineno-8-243) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-244) ) [](#__codelineno-8-245) [](#__codelineno-8-246) prompts = [ [](#__codelineno-8-247) f"<|User|>: \n{question}\n\n<|Assistant|>:" for question in questions [](#__codelineno-8-248) ] [](#__codelineno-8-249) [](#__codelineno-8-250) return ModelRequestData( [](#__codelineno-8-251) engine_args=engine_args, [](#__codelineno-8-252) prompts=prompts, [](#__codelineno-8-253) ) [](#__codelineno-8-254) [](#__codelineno-8-255)[](#__codelineno-8-256)def run_deepseek_ocr(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-257) from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor [](#__codelineno-8-258) [](#__codelineno-8-259) assert modality == "image" [](#__codelineno-8-260) [](#__codelineno-8-261) model_name = "deepseek-ai/DeepSeek-OCR" [](#__codelineno-8-262) [](#__codelineno-8-263) engine_args = EngineArgs( [](#__codelineno-8-264) model=model_name, [](#__codelineno-8-265) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-266) logits_processors=[NGramPerReqLogitsProcessor], [](#__codelineno-8-267) ) [](#__codelineno-8-268) [](#__codelineno-8-269) # deepseek-ocr use plain prompt template [](#__codelineno-8-270) prompts = [f"\n{question}" for question in questions] [](#__codelineno-8-271) [](#__codelineno-8-272) # The following sampling params config is taken from [](#__codelineno-8-273) # the official Deepseek-OCR inference example. [](#__codelineno-8-274) # (IMPORTANT) Use the custom logits processor and avoid skipping [](#__codelineno-8-275) # special tokens for this model for the optimal OCR performance. [](#__codelineno-8-276) sampling_params = [ [](#__codelineno-8-277) SamplingParams( [](#__codelineno-8-278) temperature=0.0, [](#__codelineno-8-279) max_tokens=8192, [](#__codelineno-8-280) # ngram logit processor args [](#__codelineno-8-281) extra_args=dict( [](#__codelineno-8-282) ngram_size=30, [](#__codelineno-8-283) window_size=90, [](#__codelineno-8-284) # whitelist: , [](#__codelineno-8-285) whitelist_token_ids={128821, 128822}, [](#__codelineno-8-286) ), [](#__codelineno-8-287) skip_special_tokens=False, [](#__codelineno-8-288) ) [](#__codelineno-8-289) for _ in questions [](#__codelineno-8-290) ] [](#__codelineno-8-291) [](#__codelineno-8-292) return ModelRequestData( [](#__codelineno-8-293) engine_args=engine_args, [](#__codelineno-8-294) prompts=prompts, [](#__codelineno-8-295) sampling_params=sampling_params, [](#__codelineno-8-296) ) [](#__codelineno-8-297) [](#__codelineno-8-298)[](#__codelineno-8-299)def run_deepseek_ocr2(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-300) from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor [](#__codelineno-8-301) [](#__codelineno-8-302) assert modality == "image" [](#__codelineno-8-303) [](#__codelineno-8-304) model_name = "deepseek-ai/DeepSeek-OCR-2" [](#__codelineno-8-305) [](#__codelineno-8-306) engine_args = EngineArgs( [](#__codelineno-8-307) model=model_name, [](#__codelineno-8-308) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-309) logits_processors=[NGramPerReqLogitsProcessor], [](#__codelineno-8-310) ) [](#__codelineno-8-311) [](#__codelineno-8-312) # deepseek-ocr use plain prompt template [](#__codelineno-8-313) prompts = [f"\n{question}" for question in questions] [](#__codelineno-8-314) [](#__codelineno-8-315) # The following sampling params config is taken from [](#__codelineno-8-316) # the official Deepseek-OCR inference example. [](#__codelineno-8-317) # (IMPORTANT) Use the custom logits processor and avoid skipping [](#__codelineno-8-318) # special tokens for this model for the optimal OCR performance. [](#__codelineno-8-319) sampling_params = [ [](#__codelineno-8-320) SamplingParams( [](#__codelineno-8-321) temperature=0.0, [](#__codelineno-8-322) max_tokens=8192, [](#__codelineno-8-323) # ngram logit processor args [](#__codelineno-8-324) extra_args=dict( [](#__codelineno-8-325) ngram_size=30, [](#__codelineno-8-326) window_size=90, [](#__codelineno-8-327) # whitelist: , [](#__codelineno-8-328) whitelist_token_ids={128821, 128822}, [](#__codelineno-8-329) ), [](#__codelineno-8-330) skip_special_tokens=False, [](#__codelineno-8-331) ) [](#__codelineno-8-332) for _ in questions [](#__codelineno-8-333) ] [](#__codelineno-8-334) [](#__codelineno-8-335) return ModelRequestData( [](#__codelineno-8-336) engine_args=engine_args, [](#__codelineno-8-337) prompts=prompts, [](#__codelineno-8-338) sampling_params=sampling_params, [](#__codelineno-8-339) ) [](#__codelineno-8-340) [](#__codelineno-8-341)[](#__codelineno-8-342)# Dots-OCR [](#__codelineno-8-343)def run_dots_ocr(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-344) assert modality == "image" [](#__codelineno-8-345) [](#__codelineno-8-346) prompts = [f"<|img|><|imgpad|><|endofimg|>{question}" for question in questions] [](#__codelineno-8-347) engine_args = EngineArgs( [](#__codelineno-8-348) model="rednote-hilab/dots.ocr", [](#__codelineno-8-349) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-350) trust_remote_code=True, [](#__codelineno-8-351) ) [](#__codelineno-8-352) [](#__codelineno-8-353) return ModelRequestData( [](#__codelineno-8-354) engine_args=engine_args, [](#__codelineno-8-355) prompts=prompts, [](#__codelineno-8-356) ) [](#__codelineno-8-357) [](#__codelineno-8-358)[](#__codelineno-8-359)# Eagle2.5-VL [](#__codelineno-8-360)def run_eagle2_5(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-361) assert modality == "image" [](#__codelineno-8-362) [](#__codelineno-8-363) model_name = "nvidia/Eagle2.5-8B" [](#__codelineno-8-364) [](#__codelineno-8-365) engine_args = EngineArgs( [](#__codelineno-8-366) model=model_name, [](#__codelineno-8-367) max_model_len=4096, [](#__codelineno-8-368) max_num_seqs=2, [](#__codelineno-8-369) trust_remote_code=True, [](#__codelineno-8-370) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-371) ) [](#__codelineno-8-372) [](#__codelineno-8-373) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-8-374) messages = [ [](#__codelineno-8-375) [{"role": "user", "content": f"\n{question}"}] for question in questions [](#__codelineno-8-376) ] [](#__codelineno-8-377) prompts = tokenizer.apply_chat_template( [](#__codelineno-8-378) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-8-379) ) [](#__codelineno-8-380) [](#__codelineno-8-381) # Stop tokens for Eagle2.5 (Qwen2 based) [](#__codelineno-8-382) stop_tokens = ["<|endoftext|>", "<|im_start|>", "<|im_end|>"] [](#__codelineno-8-383) stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens] [](#__codelineno-8-384) stop_token_ids = [token_id for token_id in stop_token_ids if token_id is not None] [](#__codelineno-8-385) [](#__codelineno-8-386) return ModelRequestData( [](#__codelineno-8-387) engine_args=engine_args, [](#__codelineno-8-388) prompts=prompts, [](#__codelineno-8-389) stop_token_ids=stop_token_ids, [](#__codelineno-8-390) ) [](#__codelineno-8-391) [](#__codelineno-8-392)[](#__codelineno-8-393)# Ernie4.5-VL [](#__codelineno-8-394)def run_ernie45_vl(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-395) model_name = "baidu/ERNIE-4.5-VL-28B-A3B-PT" [](#__codelineno-8-396) [](#__codelineno-8-397) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-398) engine_args = EngineArgs( [](#__codelineno-8-399) model=model_name, [](#__codelineno-8-400) max_model_len=4096, [](#__codelineno-8-401) max_num_seqs=5, [](#__codelineno-8-402) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-403) trust_remote_code=True, [](#__codelineno-8-404) ) [](#__codelineno-8-405) [](#__codelineno-8-406) image_placeholder = "Picture 1:<|IMAGE_START|><|image@placeholder|><|IMAGE_END|>" [](#__codelineno-8-407) video_placeholder = "Video 1:<|VIDEO_START|><|video@placeholder|><|VIDEO_END|>" [](#__codelineno-8-408) [](#__codelineno-8-409) if modality == "image": [](#__codelineno-8-410) placeholder = image_placeholder [](#__codelineno-8-411) elif modality == "video": [](#__codelineno-8-412) placeholder = video_placeholder [](#__codelineno-8-413) elif modality == "image+video": [](#__codelineno-8-414) placeholder = image_placeholder + video_placeholder [](#__codelineno-8-415) [](#__codelineno-8-416) prompts = [ [](#__codelineno-8-417) ( [](#__codelineno-8-418) f"<|begin_of_sentence|>User: {question}{placeholder}\n" [](#__codelineno-8-419) "Assistant: " [](#__codelineno-8-420) ) [](#__codelineno-8-421) for question in questions [](#__codelineno-8-422) ] [](#__codelineno-8-423) [](#__codelineno-8-424) return ModelRequestData( [](#__codelineno-8-425) engine_args=engine_args, [](#__codelineno-8-426) prompts=prompts, [](#__codelineno-8-427) ) [](#__codelineno-8-428) [](#__codelineno-8-429)[](#__codelineno-8-430)# EXAONE-4.5 [](#__codelineno-8-431)def run_exaone4_5(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-432) model_name = "LGAI-EXAONE/EXAONE-4.5-33B" [](#__codelineno-8-433) [](#__codelineno-8-434) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-435) engine_args = EngineArgs( [](#__codelineno-8-436) model=model_name, [](#__codelineno-8-437) max_model_len=4096, [](#__codelineno-8-438) max_num_seqs=5, [](#__codelineno-8-439) mm_processor_kwargs={ [](#__codelineno-8-440) "min_pixels": 28 * 28, [](#__codelineno-8-441) "max_pixels": 1280 * 28 * 28, [](#__codelineno-8-442) "fps": 1, [](#__codelineno-8-443) }, [](#__codelineno-8-444) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-445) ) [](#__codelineno-8-446) [](#__codelineno-8-447) image_placeholder = "<|image_pad|>" [](#__codelineno-8-448) video_placeholder = "<|video_pad|>" [](#__codelineno-8-449) [](#__codelineno-8-450) if modality == "image": [](#__codelineno-8-451) placeholder = image_placeholder [](#__codelineno-8-452) elif modality == "video": [](#__codelineno-8-453) placeholder = video_placeholder [](#__codelineno-8-454) elif modality == "image+video": [](#__codelineno-8-455) placeholder = image_placeholder + video_placeholder [](#__codelineno-8-456) [](#__codelineno-8-457) prompts = [ [](#__codelineno-8-458) ( [](#__codelineno-8-459) "<|system|>\nYou are a helpful assistant.<|endofturn|>\n" [](#__codelineno-8-460) f"<|user|>\n{placeholder}" [](#__codelineno-8-461) f"{question}<|endofturn|>\n" [](#__codelineno-8-462) "<|assistant|>\n" [](#__codelineno-8-463) ) [](#__codelineno-8-464) for question in questions [](#__codelineno-8-465) ] [](#__codelineno-8-466) [](#__codelineno-8-467) return ModelRequestData( [](#__codelineno-8-468) engine_args=engine_args, [](#__codelineno-8-469) prompts=prompts, [](#__codelineno-8-470) ) [](#__codelineno-8-471) [](#__codelineno-8-472)[](#__codelineno-8-473)# Fuyu [](#__codelineno-8-474)def run_fuyu(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-475) assert modality == "image" [](#__codelineno-8-476) [](#__codelineno-8-477) prompts = [f"{question}\n" for question in questions] [](#__codelineno-8-478) engine_args = EngineArgs( [](#__codelineno-8-479) model="adept/fuyu-8b", [](#__codelineno-8-480) max_model_len=2048, [](#__codelineno-8-481) max_num_seqs=2, [](#__codelineno-8-482) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-483) ) [](#__codelineno-8-484) [](#__codelineno-8-485) return ModelRequestData( [](#__codelineno-8-486) engine_args=engine_args, [](#__codelineno-8-487) prompts=prompts, [](#__codelineno-8-488) ) [](#__codelineno-8-489) [](#__codelineno-8-490)[](#__codelineno-8-491)# Gemma 3 [](#__codelineno-8-492)def run_gemma3(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-493) assert modality == "image" [](#__codelineno-8-494) model_name = "google/gemma-3-4b-it" [](#__codelineno-8-495) [](#__codelineno-8-496) engine_args = EngineArgs( [](#__codelineno-8-497) model=model_name, [](#__codelineno-8-498) max_model_len=2048, [](#__codelineno-8-499) max_num_seqs=2, [](#__codelineno-8-500) mm_processor_kwargs={"do_pan_and_scan": True}, [](#__codelineno-8-501) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-502) ) [](#__codelineno-8-503) [](#__codelineno-8-504) prompts = [ [](#__codelineno-8-505) ( [](#__codelineno-8-506) "user\n" [](#__codelineno-8-507) f"{question}\n" [](#__codelineno-8-508) "model\n" [](#__codelineno-8-509) ) [](#__codelineno-8-510) for question in questions [](#__codelineno-8-511) ] [](#__codelineno-8-512) return ModelRequestData( [](#__codelineno-8-513) engine_args=engine_args, [](#__codelineno-8-514) prompts=prompts, [](#__codelineno-8-515) ) [](#__codelineno-8-516) [](#__codelineno-8-517)[](#__codelineno-8-518)# Gemma3N [](#__codelineno-8-519)def run_gemma3n(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-520) assert modality == "image" [](#__codelineno-8-521) model_name = "google/gemma-3n-E2B-it" [](#__codelineno-8-522) [](#__codelineno-8-523) engine_args = EngineArgs( [](#__codelineno-8-524) model=model_name, [](#__codelineno-8-525) max_model_len=2048, [](#__codelineno-8-526) max_num_seqs=2, [](#__codelineno-8-527) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-528) enforce_eager=True, [](#__codelineno-8-529) ) [](#__codelineno-8-530) [](#__codelineno-8-531) prompts = [ [](#__codelineno-8-532) ( [](#__codelineno-8-533) "user\n" [](#__codelineno-8-534) f"{question}\n" [](#__codelineno-8-535) "model\n" [](#__codelineno-8-536) ) [](#__codelineno-8-537) for question in questions [](#__codelineno-8-538) ] [](#__codelineno-8-539) return ModelRequestData( [](#__codelineno-8-540) engine_args=engine_args, [](#__codelineno-8-541) prompts=prompts, [](#__codelineno-8-542) ) [](#__codelineno-8-543) [](#__codelineno-8-544)[](#__codelineno-8-545)# GLM-4v [](#__codelineno-8-546)def run_glm4v(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-547) assert modality == "image" [](#__codelineno-8-548) model_name = "zai-org/glm-4v-9b" [](#__codelineno-8-549) [](#__codelineno-8-550) engine_args = EngineArgs( [](#__codelineno-8-551) model=model_name, [](#__codelineno-8-552) max_model_len=2048, [](#__codelineno-8-553) max_num_seqs=2, [](#__codelineno-8-554) trust_remote_code=True, [](#__codelineno-8-555) enforce_eager=True, [](#__codelineno-8-556) hf_overrides={"architectures": ["GLM4VForCausalLM"]}, [](#__codelineno-8-557) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-558) ) [](#__codelineno-8-559) [](#__codelineno-8-560) prompts = [ [](#__codelineno-8-561) ( [](#__codelineno-8-562) "<|user|>\n<|begin_of_image|><|endoftext|><|end_of_image|>" [](#__codelineno-8-563) f"{question}<|assistant|>" [](#__codelineno-8-564) ) [](#__codelineno-8-565) for question in questions [](#__codelineno-8-566) ] [](#__codelineno-8-567) [](#__codelineno-8-568) stop_token_ids = [151329, 151336, 151338] [](#__codelineno-8-569) [](#__codelineno-8-570) return ModelRequestData( [](#__codelineno-8-571) engine_args=engine_args, [](#__codelineno-8-572) prompts=prompts, [](#__codelineno-8-573) stop_token_ids=stop_token_ids, [](#__codelineno-8-574) ) [](#__codelineno-8-575) [](#__codelineno-8-576)[](#__codelineno-8-577)# GLM-4.1V [](#__codelineno-8-578)def run_glm4_1v(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-579) model_name = "zai-org/GLM-4.1V-9B-Thinking" [](#__codelineno-8-580) [](#__codelineno-8-581) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-582) engine_args = EngineArgs( [](#__codelineno-8-583) model=model_name, [](#__codelineno-8-584) max_model_len=4096, [](#__codelineno-8-585) max_num_seqs=2, [](#__codelineno-8-586) mm_processor_kwargs={ [](#__codelineno-8-587) "size": {"shortest_edge": 12544, "longest_edge": 47040000}, [](#__codelineno-8-588) "fps": 1, [](#__codelineno-8-589) }, [](#__codelineno-8-590) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-591) enforce_eager=True, [](#__codelineno-8-592) ) [](#__codelineno-8-593) [](#__codelineno-8-594) image_placeholder = "<|begin_of_image|><|image|><|end_of_image|>" [](#__codelineno-8-595) video_placeholder = "<|begin_of_video|><|video|><|end_of_video|>" [](#__codelineno-8-596) [](#__codelineno-8-597) if modality == "image": [](#__codelineno-8-598) placeholder = image_placeholder [](#__codelineno-8-599) elif modality == "video": [](#__codelineno-8-600) placeholder = video_placeholder [](#__codelineno-8-601) elif modality == "image+video": [](#__codelineno-8-602) placeholder = image_placeholder + video_placeholder [](#__codelineno-8-603) [](#__codelineno-8-604) prompts = [ [](#__codelineno-8-605) ( [](#__codelineno-8-606) "[gMASK]<|system|>\nYou are a helpful assistant.<|user|>\n" [](#__codelineno-8-607) f"{placeholder}" [](#__codelineno-8-608) f"{question}<|assistant|>assistant\n" [](#__codelineno-8-609) ) [](#__codelineno-8-610) for question in questions [](#__codelineno-8-611) ] [](#__codelineno-8-612) [](#__codelineno-8-613) return ModelRequestData( [](#__codelineno-8-614) engine_args=engine_args, [](#__codelineno-8-615) prompts=prompts, [](#__codelineno-8-616) ) [](#__codelineno-8-617) [](#__codelineno-8-618)[](#__codelineno-8-619)# GLM-4.5V [](#__codelineno-8-620)def run_glm4_5v(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-621) model_name = "zai-org/GLM-4.5V" [](#__codelineno-8-622) [](#__codelineno-8-623) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-624) engine_args = EngineArgs( [](#__codelineno-8-625) model=model_name, [](#__codelineno-8-626) max_model_len=4096, [](#__codelineno-8-627) max_num_seqs=2, [](#__codelineno-8-628) mm_processor_kwargs={ [](#__codelineno-8-629) "size": {"shortest_edge": 12544, "longest_edge": 47040000}, [](#__codelineno-8-630) "fps": 1, [](#__codelineno-8-631) }, [](#__codelineno-8-632) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-633) enforce_eager=True, [](#__codelineno-8-634) tensor_parallel_size=4, [](#__codelineno-8-635) ) [](#__codelineno-8-636) [](#__codelineno-8-637) image_placeholder = "<|begin_of_image|><|image|><|end_of_image|>" [](#__codelineno-8-638) video_placeholder = "<|begin_of_video|><|video|><|end_of_video|>" [](#__codelineno-8-639) [](#__codelineno-8-640) if modality == "image": [](#__codelineno-8-641) placeholder = image_placeholder [](#__codelineno-8-642) elif modality == "video": [](#__codelineno-8-643) placeholder = video_placeholder [](#__codelineno-8-644) elif modality == "image+video": [](#__codelineno-8-645) placeholder = image_placeholder + video_placeholder [](#__codelineno-8-646) [](#__codelineno-8-647) prompts = [ [](#__codelineno-8-648) ( [](#__codelineno-8-649) "[gMASK]<|system|>\nYou are a helpful assistant.<|user|>\n" [](#__codelineno-8-650) f"{placeholder}" [](#__codelineno-8-651) f"{question}<|assistant|>assistant\n" [](#__codelineno-8-652) ) [](#__codelineno-8-653) for question in questions [](#__codelineno-8-654) ] [](#__codelineno-8-655) [](#__codelineno-8-656) return ModelRequestData( [](#__codelineno-8-657) engine_args=engine_args, [](#__codelineno-8-658) prompts=prompts, [](#__codelineno-8-659) ) [](#__codelineno-8-660) [](#__codelineno-8-661)[](#__codelineno-8-662)# GLM-4.5V-FP8 [](#__codelineno-8-663)def run_glm4_5v_fp8(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-664) model_name = "zai-org/GLM-4.5V-FP8" [](#__codelineno-8-665) [](#__codelineno-8-666) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-667) engine_args = EngineArgs( [](#__codelineno-8-668) model=model_name, [](#__codelineno-8-669) max_model_len=4096, [](#__codelineno-8-670) max_num_seqs=2, [](#__codelineno-8-671) mm_processor_kwargs={ [](#__codelineno-8-672) "size": {"shortest_edge": 12544, "longest_edge": 47040000}, [](#__codelineno-8-673) "fps": 1, [](#__codelineno-8-674) }, [](#__codelineno-8-675) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-676) enforce_eager=True, [](#__codelineno-8-677) tensor_parallel_size=4, [](#__codelineno-8-678) ) [](#__codelineno-8-679) [](#__codelineno-8-680) image_placeholder = "<|begin_of_image|><|image|><|end_of_image|>" [](#__codelineno-8-681) video_placeholder = "<|begin_of_video|><|video|><|end_of_video|>" [](#__codelineno-8-682) [](#__codelineno-8-683) if modality == "image": [](#__codelineno-8-684) placeholder = image_placeholder [](#__codelineno-8-685) elif modality == "video": [](#__codelineno-8-686) placeholder = video_placeholder [](#__codelineno-8-687) elif modality == "image+video": [](#__codelineno-8-688) placeholder = image_placeholder + video_placeholder [](#__codelineno-8-689) [](#__codelineno-8-690) prompts = [ [](#__codelineno-8-691) ( [](#__codelineno-8-692) "[gMASK]<|system|>\nYou are a helpful assistant.<|user|>\n" [](#__codelineno-8-693) f"{placeholder}" [](#__codelineno-8-694) f"{question}<|assistant|>assistant\n" [](#__codelineno-8-695) ) [](#__codelineno-8-696) for question in questions [](#__codelineno-8-697) ] [](#__codelineno-8-698) [](#__codelineno-8-699) return ModelRequestData( [](#__codelineno-8-700) engine_args=engine_args, [](#__codelineno-8-701) prompts=prompts, [](#__codelineno-8-702) ) [](#__codelineno-8-703) [](#__codelineno-8-704)[](#__codelineno-8-705)# GLM-OCR [](#__codelineno-8-706)def run_glm_ocr(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-707) model_name = "zai-org/GLM-OCR" [](#__codelineno-8-708) [](#__codelineno-8-709) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-710) engine_args = EngineArgs( [](#__codelineno-8-711) model=model_name, [](#__codelineno-8-712) max_model_len=4096, [](#__codelineno-8-713) max_num_seqs=2, [](#__codelineno-8-714) mm_processor_kwargs={ [](#__codelineno-8-715) "size": {"shortest_edge": 12544, "longest_edge": 47040000}, [](#__codelineno-8-716) "fps": 1, [](#__codelineno-8-717) }, [](#__codelineno-8-718) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-719) enforce_eager=True, [](#__codelineno-8-720) ) [](#__codelineno-8-721) [](#__codelineno-8-722) image_placeholder = "<|begin_of_image|><|image|><|end_of_image|>" [](#__codelineno-8-723) video_placeholder = "<|begin_of_video|><|video|><|end_of_video|>" [](#__codelineno-8-724) [](#__codelineno-8-725) if modality == "image": [](#__codelineno-8-726) placeholder = image_placeholder [](#__codelineno-8-727) elif modality == "video": [](#__codelineno-8-728) placeholder = video_placeholder [](#__codelineno-8-729) elif modality == "image+video": [](#__codelineno-8-730) placeholder = image_placeholder + video_placeholder [](#__codelineno-8-731) [](#__codelineno-8-732) prompts = [ [](#__codelineno-8-733) ( [](#__codelineno-8-734) "[gMASK]<|system|>\nYou are a helpful assistant.<|user|>\n" [](#__codelineno-8-735) f"{placeholder}" [](#__codelineno-8-736) f"{question}<|assistant|>assistant\n" [](#__codelineno-8-737) ) [](#__codelineno-8-738) for question in questions [](#__codelineno-8-739) ] [](#__codelineno-8-740) [](#__codelineno-8-741) return ModelRequestData( [](#__codelineno-8-742) engine_args=engine_args, [](#__codelineno-8-743) prompts=prompts, [](#__codelineno-8-744) ) [](#__codelineno-8-745) [](#__codelineno-8-746)[](#__codelineno-8-747)# H2OVL-Mississippi [](#__codelineno-8-748)def run_h2ovl(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-749) assert modality == "image" [](#__codelineno-8-750) [](#__codelineno-8-751) model_name = "h2oai/h2ovl-mississippi-800m" [](#__codelineno-8-752) [](#__codelineno-8-753) engine_args = EngineArgs( [](#__codelineno-8-754) model=model_name, [](#__codelineno-8-755) trust_remote_code=True, [](#__codelineno-8-756) max_model_len=8192, [](#__codelineno-8-757) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-758) ) [](#__codelineno-8-759) [](#__codelineno-8-760) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-8-761) messages = [ [](#__codelineno-8-762) [{"role": "user", "content": f"\n{question}"}] for question in questions [](#__codelineno-8-763) ] [](#__codelineno-8-764) prompts = tokenizer.apply_chat_template( [](#__codelineno-8-765) messages, tokenize=False, add_generation_prompt=True [](#__codelineno-8-766) ) [](#__codelineno-8-767) [](#__codelineno-8-768) # Stop tokens for H2OVL-Mississippi [](#__codelineno-8-769) # https://huggingface.co/h2oai/h2ovl-mississippi-800m [](#__codelineno-8-770) stop_token_ids = [tokenizer.eos_token_id] [](#__codelineno-8-771) [](#__codelineno-8-772) return ModelRequestData( [](#__codelineno-8-773) engine_args=engine_args, [](#__codelineno-8-774) prompts=prompts, [](#__codelineno-8-775) stop_token_ids=stop_token_ids, [](#__codelineno-8-776) ) [](#__codelineno-8-777) [](#__codelineno-8-778)[](#__codelineno-8-779)# HunyuanOCR [](#__codelineno-8-780)def run_hunyuan_vl(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-781) assert modality == "image" [](#__codelineno-8-782) [](#__codelineno-8-783) model_name = "tencent/HunyuanOCR" [](#__codelineno-8-784) [](#__codelineno-8-785) engine_args = EngineArgs( [](#__codelineno-8-786) model=model_name, [](#__codelineno-8-787) max_model_len=8192, [](#__codelineno-8-788) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-789) ) [](#__codelineno-8-790) [](#__codelineno-8-791) placeholder = "<｜hy_place▁holder▁no▁100｜><｜hy_place▁holder▁no▁102｜><｜hy_place▁holder▁no▁101｜>" # noqa: E501 [](#__codelineno-8-792) prompts = [ [](#__codelineno-8-793) f"<｜hy_begin▁of▁sentence｜>{placeholder}{question}<｜hy_User｜>" [](#__codelineno-8-794) for question in questions [](#__codelineno-8-795) ] [](#__codelineno-8-796) [](#__codelineno-8-797) return ModelRequestData( [](#__codelineno-8-798) engine_args=engine_args, [](#__codelineno-8-799) prompts=prompts, [](#__codelineno-8-800) stop_token_ids=None, [](#__codelineno-8-801) ) [](#__codelineno-8-802) [](#__codelineno-8-803)[](#__codelineno-8-804)# naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B [](#__codelineno-8-805)def run_hyperclovax_seed_vision( [](#__codelineno-8-806) questions: list[str], modality: str [](#__codelineno-8-807)) -> ModelRequestData: [](#__codelineno-8-808) model_name = "naver-hyperclovax/HyperCLOVAX-SEED-Vision-Instruct-3B" [](#__codelineno-8-809) tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) [](#__codelineno-8-810) [](#__codelineno-8-811) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-812) engine_args = EngineArgs( [](#__codelineno-8-813) model=model_name, [](#__codelineno-8-814) trust_remote_code=True, [](#__codelineno-8-815) max_model_len=16384 if modality in ("video", "image+video") else 8192, [](#__codelineno-8-816) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-817) ) [](#__codelineno-8-818) [](#__codelineno-8-819) messages = list() [](#__codelineno-8-820) for question in questions: [](#__codelineno-8-821) if modality == "image": [](#__codelineno-8-822) """ [](#__codelineno-8-823) ocr: List the words in the image in raster order. [](#__codelineno-8-824) Even if the word order feels unnatural for reading, [](#__codelineno-8-825) the model will handle it as long as it follows raster order. [](#__codelineno-8-826) e.g. "Naver, CLOVA, bigshane" [](#__codelineno-8-827) lens_keywords: List the entity names in the image. [](#__codelineno-8-828) e.g. "iPhone" [](#__codelineno-8-829) lens_local_keywords: List the entity names with quads in the image. [](#__codelineno-8-830) e.g. "[0.07, 0.21, 0.92, 0.90] iPhone" [](#__codelineno-8-831) """ [](#__codelineno-8-832) messages.append( [](#__codelineno-8-833) [ [](#__codelineno-8-834) { [](#__codelineno-8-835) "role": "user", [](#__codelineno-8-836) "content": [ [](#__codelineno-8-837) { [](#__codelineno-8-838) "type": "image", [](#__codelineno-8-839) "ocr": "", [](#__codelineno-8-840) "lens_keywords": "", [](#__codelineno-8-841) "lens_local_keywords": "", [](#__codelineno-8-842) }, [](#__codelineno-8-843) { [](#__codelineno-8-844) "type": "text", [](#__codelineno-8-845) "text": question, [](#__codelineno-8-846) }, [](#__codelineno-8-847) ], [](#__codelineno-8-848) } [](#__codelineno-8-849) ] [](#__codelineno-8-850) ) [](#__codelineno-8-851) elif modality == "video": [](#__codelineno-8-852) messages.append( [](#__codelineno-8-853) [ [](#__codelineno-8-854) { [](#__codelineno-8-855) "role": "user", [](#__codelineno-8-856) "content": [ [](#__codelineno-8-857) { [](#__codelineno-8-858) "type": "video", [](#__codelineno-8-859) }, [](#__codelineno-8-860) { [](#__codelineno-8-861) "type": "text", [](#__codelineno-8-862) "text": question, [](#__codelineno-8-863) }, [](#__codelineno-8-864) ], [](#__codelineno-8-865) } [](#__codelineno-8-866) ] [](#__codelineno-8-867) ) [](#__codelineno-8-868) elif modality == "image+video": [](#__codelineno-8-869) messages.append( [](#__codelineno-8-870) [ [](#__codelineno-8-871) { [](#__codelineno-8-872) "role": "user", [](#__codelineno-8-873) "content": [ [](#__codelineno-8-874) { [](#__codelineno-8-875) "type": "image", [](#__codelineno-8-876) "ocr": "", [](#__codelineno-8-877) "lens_keywords": "", [](#__codelineno-8-878) "lens_local_keywords": "", [](#__codelineno-8-879) }, [](#__codelineno-8-880) { [](#__codelineno-8-881) "type": "video", [](#__codelineno-8-882) }, [](#__codelineno-8-883) { [](#__codelineno-8-884) "type": "text", [](#__codelineno-8-885) "text": question, [](#__codelineno-8-886) }, [](#__codelineno-8-887) ], [](#__codelineno-8-888) } [](#__codelineno-8-889) ] [](#__codelineno-8-890) ) [](#__codelineno-8-891) else: [](#__codelineno-8-892) raise ValueError(f"Unsupported modality: {modality}") [](#__codelineno-8-893) [](#__codelineno-8-894) prompts = tokenizer.apply_chat_template( [](#__codelineno-8-895) messages, [](#__codelineno-8-896) tokenize=False, [](#__codelineno-8-897) add_generation_prompt=True, [](#__codelineno-8-898) ) [](#__codelineno-8-899) [](#__codelineno-8-900) return ModelRequestData( [](#__codelineno-8-901) engine_args=engine_args, [](#__codelineno-8-902) prompts=prompts, [](#__codelineno-8-903) stop_token_ids=None, [](#__codelineno-8-904) ) [](#__codelineno-8-905) [](#__codelineno-8-906)[](#__codelineno-8-907)# Idefics3-8B-Llama3 [](#__codelineno-8-908)def run_idefics3(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-909) assert modality == "image" [](#__codelineno-8-910) model_name = "HuggingFaceM4/Idefics3-8B-Llama3" [](#__codelineno-8-911) [](#__codelineno-8-912) engine_args = EngineArgs( [](#__codelineno-8-913) model=model_name, [](#__codelineno-8-914) max_model_len=8192, [](#__codelineno-8-915) max_num_seqs=2, [](#__codelineno-8-916) enforce_eager=True, [](#__codelineno-8-917) # if you are running out of memory, you can reduce the "longest_edge". [](#__codelineno-8-918) # see: https://huggingface.co/HuggingFaceM4/Idefics3-8B-Llama3#model-optimizations [](#__codelineno-8-919) mm_processor_kwargs={ [](#__codelineno-8-920) "size": {"longest_edge": 3 * 364}, [](#__codelineno-8-921) }, [](#__codelineno-8-922) limit_mm_per_prompt={modality: 1}, [](#__codelineno-8-923) ) [](#__codelineno-8-924) prompts = [ [](#__codelineno-8-925) (f"<|begin_of_text|>User:{question}\nAssistant:") [](#__codelineno-8-926) for question in questions [](#__codelineno-8-927) ] [](#__codelineno-8-928) [](#__codelineno-8-929) return ModelRequestData( [](#__codelineno-8-930) engine_args=engine_args, [](#__codelineno-8-931) prompts=prompts, [](#__codelineno-8-932) ) [](#__codelineno-8-933) [](#__codelineno-8-934)[](#__codelineno-8-935)# Intern-S1 [](#__codelineno-8-936)def run_interns1(questions: list[str], modality: str) -> ModelRequestData: [](#__codelineno-8-937) model_name = "internlm/Intern-S1-mini" [](#__codelineno-8-938) [](#__codelineno-8-939) mm_limit = {"image": 1, "video": 1} if modality == "image+video" else {modality: 1} [](#__codelineno-8-940) engine_args = EngineArgs( [](#__codelineno-8-941) model=model_name, [](#__codelineno-8-942) trust_remote_code=True, [](#__codelineno-8-943) max_model_len=8192, [](#__codelineno-8-944) max_num_seqs=2, [](#__codelineno-8-945) limit_mm_per_prompt=mm_limit, [](#__codelineno-8-946) enforce_eager=True, [](#__codelineno-8-947) ) [](#__codelineno-8-948) [](#__codelineno-8-949) image_placeholder = "" [](#__codelineno-8-950) video_placeholder = "