Deploy and Scale vLLM Clusters on Azure Kubernetes Service

Production LLM serving requires balancing compute cost and latency. Raw PyTorch deployments suffer memory fragmentation from KV caching. vLLM uses PagedAttention to allocate KV cache in virtual blocks, preventing fragmentation. This setup is key to implementing robust AI infrastructure engineering patterns for production workloads.

Single VM limits scaling. Traffic spikes need fast replica spin up. AKS scales these workloads. This guide covers setting up vLLM on AKS, persistent caching, queue-based scaling, and Prometheus monitoring.

Designing the GPU Node Pool

Select Azure VM size. LLM serving is memory-bound. GPU memory (VRAM) must hold model weights and KV cache.

Calculate required VRAM. Formula: V_min = (P * 2 * 1.2) / T. V_min is minimum VRAM. P is parameter count in billions. 2 represents bytes per parameter (FP16). 1.2 adds 20 percent overhead for activation memory. T is tensor parallel size (number of GPUs).

Models up to 13B parameters (like Llama-3-8B) run on single NVIDIA A100 (80GB) or L4 (24GB). For production deployments, optimizing hardware utilization is critical, similar to how providers run compact AI models with tighter latency budgets to avoid GPU excess. Larger models (like Llama-3-70B) require tensor parallelism across multiple GPUs on same node.

Azure GPU VM sizes:

• Standard_NC24ads_A100_v4: 1x NVIDIA A100 80GB GPU. Best for small to medium models.
• Standard_NC96ads_A100_v4: 4x NVIDIA A100 80GB GPUs. Best for medium models with tensor parallelism.
• Standard_ND96asr_v4: 8x NVIDIA A100 80GB GPUs. Best for large models requiring multi-GPU tensor parallelism.
• Standard_NG24_v6: 1x NVIDIA L4 24GB GPU. Cost-effective for small models.

Provision GPU node pool in existing AKS cluster.

1. Create a resource group in your target Azure region.
2. Create an AKS cluster with system node pool for control plane workloads.
3. Enable the Azure GPU toolchain and NVIDIA device plugin on the cluster.
4. Add a dedicated GPU node pool with specific taints and labels.

Run commands:

az group create -name myResourceGroup -location eastus
 
az aks create \
    -resource-group myResourceGroup \
    -name myAKSCluster \
    -node-count 2 \
    -generate-ssh-keys
 
az aks nodepool add \
    -resource-group myResourceGroup \
    -cluster-name myAKSCluster \
    -name gpunode \
    -node-vm-size Standard_NC24ads_A100_v4 \
    -node-count 1 \
    -aks-custom-headers EnableGPUDedicatedVHD=true \
    -labels sku=gpu app=vllm \
    -node-taints sku=gpu:NoSchedule

Taint sku=gpu:NoSchedule prevents standard CPU workloads on GPU nodes. Custom header EnableGPUDedicatedVHD=true installs NVIDIA drivers automatically.

Model Caching with Azure Blob Storage

Do not download weights from Hugging Face on pod restart. 70B model is over 130GB. Downloading over public internet causes startup delay and rate limits.

Download weights once. Store in Azure Blob Storage container. Mount container to pods using Blob storage CSI driver.

Create storage account and container:

az storage account create \
    -name modelcachestorage \
    -resource-group myResourceGroup \
    -location eastus \
    -sku Standard_LRS
 
az storage container create \
    -name models \
    -account-name modelcachestorage

Install Blob storage CSI driver on AKS:

az aks update \
    -enable-blob-driver \
    -name myAKSCluster \
    -resource-group myResourceGroup

Upload model weights to container.

1. Install the Azure CLI and the storage-preview extension.
2. Retrieve the storage account connection string.
3. Download the Llama-3-8B-Instruct model weights from Hugging Face using the huggingface-cli tool.
4. Upload the model directory to the Azure Blob Storage container using the Azure CLI.

Run upload command:

az storage blob upload-batch \
    -destination models/Meta-Llama-3-8B-Instruct \
    -source ./Meta-Llama-3-8B-Instruct \
    -account-name modelcachestorage

Define StorageClass for Blob CSI driver:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: azureblob-nfs
provisioner: blob.csi.azure.com
parameters:
  protocol: nfs
mountOptions:
  - noatime
  - nodiratime

Define Persistent Volume (PV) using blob.csi.azure.com driver:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: model-cache-pv
spec:
  capacity:
    storage: 500Gi
  accessModes:
    - ReadOnlyMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: azureblob-nfs
  csi:
    driver: blob.csi.azure.com
    volumeHandle: modelcachestorage_models
    volumeAttributes:
      containerName: models
      storageAccountName: modelcachestorage

Define Persistent Volume Claim (PVC):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache-pvc
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 500Gi
  volumeName: model-cache-pv
  storageClassName: azureblob-nfs

Apply manifests. Pods read weights from Azure Blob Storage at local network speed.

The vLLM Deployment Configuration

Configure shared memory (/dev/shm). PyTorch uses shared memory for inter-process communication in tensor parallelism. Without it, container crashes with bus errors.

For multi-GPU setups, configure -tensor-parallel-size argument. Match argument value to number of requested GPUs in resources section.

Create vllm-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
  labels:
    app: vllm-llama3-8b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama3-8b
  template:
    metadata:
      labels:
        app: vllm-llama3-8b
    spec:
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      nodeSelector:
        sku: "gpu"
      containers:
      - name: vllm-server
        image: vllm/vllm-openai:latest
        command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
        args:
        - "-model"
        - "/mnt/models/Meta-Llama-3-8B-Instruct"
        - "-port"
        - "8000"
        - "-gpu-memory-utilization"
        - "0.90"
        - "-max-model-len"
        - "4096"
        - "-tensor-parallel-size"
        - "1"
        - "-block-size"
        - "16"
        - "-swap-space"
        - "4"
        resources:
          limits:
            nvidia.com/gpu: "1"
            memory: 64Gi
            cpu: "8"
          requests:
            nvidia.com/gpu: "1"
            memory: 32Gi
            cpu: "4"
        ports:
        - containerPort: 8000
          name: http
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
        - name: model-volume
          mountPath: /mnt/models
          readOnly: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 120
          periodSeconds: 5
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 16Gi
      - name: model-volume
        persistentVolumeClaim:
          claimName: model-cache-pvc

Parameter -gpu-memory-utilization set to 0.90 reserves 90% VRAM for KV cache and weights. Remaining 10% is for PyTorch overhead. Adjust if OOM occurs.

Parameter -block-size set to 16 defines token block size for PagedAttention.

Parameter -swap-space set to 4 allocates 4GB CPU memory for swapping KV cache blocks when GPU memory runs out.

Apply deployment and expose via Service:

apiVersion: v1
kind: Service
metadata:
  name: vllm-service
  labels:
    app: vllm-llama3-8b
spec:
  ports:
  - port: 8000
    targetPort: 8000
    name: http
  selector:
    app: vllm-llama3-8b
  type: ClusterIP

Optimizing Engine Parameters for Production

Fine-tune engine parameters to maximize throughput.

• -max-num-batched-tokens: Maximum number of tokens batched in one iteration. Set to match context window size. High value increases throughput but uses more VRAM, directly impacting context window replication and token consumption costs.
• -max-num-seqs: Maximum concurrent sequences per iteration. Default is 256. Reduce if OOM errors occur during high concurrency.
• -kv-cache-dtype: Data type for KV cache storage. Set to fp8 to reduce KV cache memory footprint by 50 percent. Allows larger batch sizes.
• -device: Target execution device. Set to cuda for NVIDIA GPUs.

Configuring Pod Disruption Budgets and Node Affinity

Prevent node drains from taking down all replicas. Define PodDisruptionBudget (PDB):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: vllm-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: vllm-llama3-8b

Use node affinity to ensure pods run on correct GPU nodes. Add topology spread constraints to distribute pods across availability zones:

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: vllm-llama3-8b

Scaling with KEDA and Prometheus Metrics

Standard autoscaling uses CPU or memory. This fails for LLM serving. vLLM pre-allocates GPU memory on startup, keeping usage near 90%. CPU usage is poor indicator due to asynchronous engine idle states.

Scale based on concurrency. Use number of requests waiting in queue. High queue means replicas cannot process fast enough, increasing latency.

KEDA scales workloads using Prometheus metrics.

Install KEDA:

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda -namespace keda -create-namespace

Deploy KEDA ScaledObject. Monitors vllm:num_requests_waiting. Scales when average queue exceeds 5 requests per pod:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-autoscaler
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama3-8b
  minReplicaCount: 1
  maxReplicaCount: 5
  cooldownPeriod: 300
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 10
            periodSeconds: 60
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-k8s.monitoring.svc.cluster.local:9090
      metricName: vllm_num_requests_waiting
      query: sum(vllm:num_requests_waiting)
      threshold: '5'

scaleDown uses 300-second stabilization window. Prevents cluster thrashing during bursty traffic.

Setting Up Observability

vLLM exposes Prometheus metrics on port 8000 at /metrics.

Configure ServiceMonitor to scrape metrics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: vllm-llama3-8b
  endpoints:
  - port: http
    interval: 10s
    path: /metrics

Build Grafana dashboard using key metrics:

• vllm:num_requests_waiting: Queued requests. Value above zero requires more replicas.
• vllm:gpu_cache_usage_factor: KV cache usage percentage. Value at 1.0 causes request preemption and latency spikes.
• vllm:num_requests_running: Concurrent active requests.
• vllm:request_prompt_tokens_count and vllm:request_generation_tokens_count: Input and output token throughput.

Calculate prompt throughput:

rate(vllm:request_prompt_tokens_sum[1m])

Calculate generation throughput:

rate(vllm:request_generation_tokens_sum[1m])

Ratio of prompt to generation tokens helps optimize batching. Long prompts may require adjusting -max-num-seqs for larger batch sizes.

Verifying the Deployment

Verify autoscaling and serving. Send test request from inside cluster:

kubectl run curl-test -image=curlimages/curl -i -tty -rm - \
  curl -X POST http://vllm-service:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/mnt/models/Meta-Llama-3-8B-Instruct",
    "prompt": "Explain Kubernetes in three sentences.",
    "max_tokens": 100,
    "temperature": 0
  }'

Response contains generated text. Test autoscaler using hey or k6 for concurrent requests. Monitor pods with kubectl get pods -w. KEDA triggers GPU node creation as queue grows.