The Workload Variant Autoscaler (WVA) is a Kubernetes-based global autoscaler for inference model servers serving LLMs. WVA works alongside the standard Kubernetes HPA and external autoscalers like KEDA to drive the scale subresource of inference deployments. The high-level details of the algorithms are documented here. It determines optimal replica counts for a given request traffic load by considering constraints such as GPU availability, energy budget, and performance budget (latency/throughput).
WVA introduces the concept of variants — multiple model servers in an InferencePool that all serve the same base model but differ in hardware configuration (e.g., GPU type), serving configuration (e.g., tensor parallelism, max batch size, quantization), or both.
Use cases include:
- P/D disaggregation: prefill is one variant, decode is another — variant = role in a disaggregated pipeline.
- batch-gateway: variants distinguish batch vs. interactive workloads sharing the same pool.
- Autoscaler: a costed serving configuration the autoscaler chooses among.
- Intelligent Autoscaling: Optimizes replica count by observing the current state of the system
- Cost Optimization: Minimizes infrastructure costs by picking the correct accelerator variant
See the architecture and autoscaling design docs for high-level algorithm details.
See the docs directory for design docs, developer guide, and more.
Prerequisites: deploy llm-d infrastructure (model servers) and create an HPA or KEDA object targeting each deployment.
WVA then:
- Continuously monitors request rates and server performance via Prometheus metrics
- Capacity model obtains KV cache utilization and queue depth to determine desired replica counts
- Actuator emits optimization metrics to Prometheus
- External autoscaler (
HPA/KEDA) reads the metrics and scales the deployment accordingly
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llama-8b-autoscaler
namespace: llm-inference
annotations:
llm-d.ai/managed: "true" # Opt-in to WVA management
llm-d.ai/variant-cost: "10.0" # Optional, defaults to "10.0"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llama-8b
# minReplicas: 0 # scale to zero - alpha feature
maxReplicas: 2
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 10
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 10
periodSeconds: 15
metrics:
- type: External
external:
metric:
name: wva_desired_replicas
selector:
matchLabels:
variant_name: llama-8b
exported_namespace: llm-inference
target:
type: AverageValue
averageValue: "1"More examples in config/samples/hpa/ and config/samples/keda/.
We welcome contributions! See CONTRIBUTING.md for guidelines.
Join the llm-d autoscaling community meetings to get involved.
Apache 2.0 - see LICENSE for details.