GPU Optimization for LLM Inference on AMD MI300X

The master switch. When set to 1, vLLM replaces its default Python-backed attention layers with AITER's pre-compiled C++ kernel library. This covers the full attention pathway, prefill, decode, and the KV-cache interaction layer, all rewritten with CDNA3-specific tile sizes, warp occupancy targets, and HBM3 prefetch hints.

Without this flag, vLLM falls back to ROCm's generic attention path, which is functionally correct but ignores the MI300X's memory topology. The AITER kernels exploit the fact that HBM3 has lower latency per access than HBM2e, they structure their memory access patterns specifically for that characteristic, squeezing out bandwidth that generic kernels leave unused.

Disables AITER's multi-head attention (MHA) kernel specifically. This is deliberately set to 0, an intentional carve-out within the broader AITER suite. The MHA kernel in AITER is still maturing: while it performs well on standard attention shapes, edge cases around grouped-query attention (GQA) configurations like those used by GPT-OSS 120B can produce correctness regressions or stability issues on certain sequence lengths.

With VLLM_ROCM_USE_AITER_MHA=0, the broader AITER suite (VLLM_ROCM_USE_AITER=1) still handles all other operations, GEMM, layer norms, activation functions, the KV-cache read/write path, but MHA falls back to the stable FlashAttention-for-ROCm implementation.

Configures the all-reduce quantisation scheme used during tensor-parallel communication. Setting this to INT4 means intermediate activation tensors are quantised to 4-bit integers before they are broadcast across GPU communication links during all-reduce operations.

Even with --tensor-parallel-size 1 (a single GPU), internal pipeline stages and certain fused kernels still perform reduce operations across compute units. INT4 quantisation here halves the data volume relative to INT8, and quarters it relative to FP16, directly reducing memory bandwidth pressure on HBM3 during these reductions. The MI300X's HBM3 is fast, but memory bandwidth is always the bottleneck in autoregressive decode; every byte saved translates to measurable throughput.

Forces kernel arguments to be passed via device memory rather than host memory. When a GPU kernel is dispatched, its arguments, tensor pointers, shape metadata, stride information, need to reach the GPU. The default HIP path copies these from host (CPU) memory. Setting this flag tells the HIP runtime to pre-stage kernel arguments directly in device memory, eliminating a PCIe round-trip on every kernel launch.

This is a micro-optimisation that looks small on any single kernel invocation — microseconds at most. It compounds, however, during high-concurrency inference where vLLM dispatches hundreds of kernels per decode step across attention, GEMM, and KV-cache operations. On the MI300X's CDNA3 architecture, where the CPU-to-GPU argument path is a well-documented latency source, this flag removes a systemic overhead that otherwise accrues quietly at scale.

Runs the entire model on a single GPU. Tensor parallelism splits weight matrices across multiple GPUs, with each GPU owning a shard and all-reducing activations at every layer boundary. With --tensor-parallel-size 1, no splitting occurs, the full 120B parameter model runs as a single monolithic graph on one MI300X.

This is only possible because the MI300X has 192 GB of HBM3. A BF16 GPT-OSS 120B model occupies roughly 240 GB at full precision, but with --kv-cache-dtype fp8 and --gpu-memory-utilization 0.92, the combined weight + KV-cache footprint is managed within budget. Without tensor parallelism, there is zero all-reduce communication overhead, which is the dominant latency cost in multi-GPU setups.

Caps the maximum sequence length the server will accept. This covers the combined length of input prompt tokens plus generated output tokens. A request with a 7000-token prompt and a 1000-token output would exceed this limit and be rejected.

The KV-cache size is derived directly from this value: vLLM pre-allocates KV-cache blocks sized for up to max-model-len tokens per sequence. A lower limit means smaller KV-cache blocks, which means more sequences can share the same HBM pool simultaneously. Setting this at 8192 rather than the model's native 128K context window is a deliberate trade, you sacrifice very long context in exchange for higher concurrency and a more predictable memory budget.

Allows vLLM to execute custom Python code bundled with the model repository. Some models ship with non-standard architectures, custom attention implementations, or tokeniser plugins that are not part of the stock Hugging Face Transformers library. GPT-OSS 120B uses a custom modelling file that must be imported at load time.

Without this flag, vLLM will refuse to load any model that requires importing code not already installed in the Python environment. The flag is an explicit acknowledgement that you have reviewed the model's repository and accept its code as safe to run.

Sets the compute precision for model weights and activations to BF16. BFloat16 uses the same 8-bit exponent range as FP32 (preserving dynamic range) but truncates the mantissa to 7 bits rather than FP16's 10. For transformer inference, this is almost universally the right choice: it avoids the numeric instability of FP16 at large activation magnitudes while using half the memory of FP32.

The MI300X's CDNA3 architecture has native BF16 matrix cores, the hardware executes BF16 GEMM at full TFLOPS rate without any software emulation. Running at FP32 would halve throughput (each operation twice the data) and double the activation memory footprint. Running at FP16 risks overflow in large models with extreme activation norms, which manifests as NaN in generated text.

Stores the key and value tensors in the KV-cache at FP8 precision instead of BF16. The KV-cache is the memory buffer that holds the attention keys and values for every token in every active sequence. During autoregressive decode, every new token must read the entire KV history, making this buffer the single most bandwidth-intensive structure in LLM inference.

Quantising it to FP8 halves its memory footprint compared to BF16. On a 120B model with 96 layers and 128 KV-heads, a sequence at 8192 tokens generates a KV-cache of roughly 8 GB at BF16, and 4 GB at FP8. For 512 concurrent sequences that difference is the gap between server capacity and an OOM crash. The FP8 read path is also faster: fewer bytes transferred from HBM means lower latency per decode step, which directly improves TPOT.

Reserves 92% of GPU HBM for vLLM, weights, KV-cache, and activations combined. The remaining 8% (roughly 15 GB on a 192 GB card) is left unallocated to accommodate ROCm runtime overhead, AITER kernel workspace buffers, and activation peaks during large prefills.

vLLM uses this fraction to compute how many PagedAttention KV-cache blocks it can allocate after the model weights are loaded. A higher value gives more room for KV-cache blocks (more concurrent sequences); a lower value is safer on systems where other processes share the GPU. At 0.92, the server is aggressive but stable — 0.95+ on MI300X with AITER active can trigger OOM during concurrent large prefills because AITER's workspace allocations spike temporarily.

Disables chunked prefill, forces each prefill to run as a single, uninterrupted operation. Chunked prefill is a feature that breaks long prompt processing into smaller chunks so decode steps can be interleaved during prefill, reducing time-to-first-token for concurrent requests.

On NVIDIA hardware with CUDA graph capture, chunked prefill provides a good latency-throughput balance. On the MI300X with AITER kernels, the chunking boundary introduces a kernel launch flush that breaks the AITER pipeline's internal batching optimisation. The net effect is worse throughput than simply running prefill to completion. The AITER prefill kernels are tuned for full-sequence operation, they achieve better HBM utilisation when given the entire prompt at once.

Sets the maximum number of sequences the engine will process simultaneously in a single forward pass. This is the batch size ceiling for the continuous batching scheduler. When more than 512 requests are queued, additional requests wait in the admission queue until active sequences complete.

Higher values maximise GPU utilisation, more sequences per step means the matrix multiplications in each transformer layer operate on taller weight matrices, achieving better arithmetic intensity (FLOP/byte). Lower values reduce per-step latency (each step processes fewer tokens) but leave the GPU underutilised. At 512, a 120B model on MI300X typically achieves 85–92% MFU during steady-state decode, which is near-optimal for this hardware.

Binds the OpenAI-compatible HTTP server to port 8080. vLLM implements the full OpenAI REST API spec, /v1/chat/completions, /v1/completions, /v1/models, so any client that targets the OpenAI API works against this server without modification. Just swap the base URL from api.openai.com to localhost:8080.

Port 8080 is the convention for this stack (the MI300X cloud environments typically expose this port externally). In a Kubernetes deployment behind a LoadBalancer service, this maps to port 80 or 443 at the service level.