Scale-in your workload with Vertical Pod Autoscaler

Introduction

The Vertical Pod Autoscaler (VPA) frees you from the need to manually set up CPU and memory requests and limits for your containers. It automatically adjusts these values based on historical usage analysis to "right-size" your pods.

Unlike the Horizontal Pod Autoscaler (HPA), which scales out by adding more pod replicas, VPA scales up/down by allocating more or less resources to existing pods. It helps you to optimize resources usage, reduce costs, and help to absorb spikes in traffic.

How it works

VPA consists of three main components:

1. Recommender: Monitors resource utilization and events (like OOM kills) to calculate recommended values.
2. Updater: Checks if a pod's current resources deviate significantly from recommendations and decides if it should be updated.
3. Admission Controller: A webhook that intercepts pod creation requests to apply the recommended resources to new pods.

graph TD
    subgraph Control Plane
    M[Metrics Server or Prometheus] -->|Usage Data| R[Recommender]
    R -->|Calculates & Stores| VPA_Obj[VPA Object]
    end

    subgraph Nodes
    P[Pod]
    end

    VPA_Obj -->|Reads Recommendation| U[Updater]
    U -->|Evicts if needed| P

    AC[Admission Controller] -->|Intercepts Creation| P_New[New Pod]
    VPA_Obj -.->|Applies Recommendations| AC
    AC -->|Mutates| P_New

Installation

VPA is not part of the standard Kubernetes control plane and needs to be installed separately. We can install it using a Helm chart:

1
2

helm repo add fairwinds-stable https://charts.fairwinds.com/stable
helm install my-vpa fairwinds-stable/vpa --version 4.10.1

Customization

Here is a values.yaml file to customize the installation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

recommender:
  enabled: true
  extraArgs:
    v: "4"
    # define minimum recommended resources
    pod-recommendation-min-cpu-millicores: 50
    pod-recommendation-min-memory-mb: 32
    # Optional: Prometheus service 
    storage: "prometheus"
    prometheus-address: "http://prometheus.monitoring.svc"
    prometheus-cadvisor-job-name: "kubelet"
    container-pod-name-label: pod
    container-name-label: container
    metric-for-pod-labels: kube_pod_labels{job="kube-state-metrics"}[8d]
    pod-namespace-label: namespace
    pod-name-label: pod
  # Add more replicas for high availability
  replicaCount: 1
  resources:
    requests:
      cpu: 50m
      memory: 600Mi
    limits:
      cpu: 200m
      memory: 600Mi
  # Enable pod monitoring if you want to use pod monitors to track resource usage (requires Prometheus)
  podMonitor:
    enabled: true

updater:
  enabled: true
  extraArgs:
    # If you run Kubernetes >= 1.33, you can use InPlaceOrRecreate to update resources without restarting the pod
    "feature-gates": "InPlaceOrRecreate=true"
    # Allow VPA to update pods with less than 1 replica (warning: will produce downtime on pod restart)
    # So you should be ok with this behavior
    "min-replicas": 1
  resources:
    requests:
      cpu: 50m
      memory: 500Mi
    limits:
      cpu: 200m
      memory: 500Mi
  # Enable pod monitoring if you want to use pod monitors to track resource usage (requires Prometheus)
  podMonitor:
    enabled: true

admissionController:
  enabled: true
  extraArgs:
    # If you run Kubernetes >= 1.33, you can use InPlaceOrRecreate to update resources without restarting the pod
    "feature-gates": "InPlaceOrRecreate=true"
  registerWebhook: true
  resources:
    requests:
      cpu: 50m
      memory: 500Mi
    limits:
      cpu: 200m
      memory: 500Mi

By default, the VPA will use metrics-server to collect resource usage data and define recommendations.

Configuration

Let's look at a standard VPA configuration targeting a Deployment.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  # The target workload to scale (Deployment, StatefulSet, etc.)
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  # How VPA applies changes: "Off", "Initial", "Recreate", "InPlaceOrRecreate, or "Auto"
  updatePolicy:
    updateMode: "Off"
  # Fine-grained control over resources
  resourcePolicy:
    containerPolicies:
      - containerName: '*' # Apply to all containers inside the pod
        # Lower bound: VPA will never recommend less than this
        minAllowed:
          cpu: 100m
          memory: 250Mi
        # Upper bound: VPA will never recommend more than this
        maxAllowed:
          cpu: 1
          memory: 1Gi
        # Which resources to manage
        controlledResources: ["cpu", "memory"]

Then wait a few seconds, and describe the VPA, you'll see in the coming minutes the recommendations. Example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

$ kubectl describe vpa -n cert-manager cert-manager-vpa

Name:         cert-manager-vpa
Namespace:    cert-manager
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: cert-manager-config
              meta.helm.sh/release-namespace: cert-manager
API Version:  autoscaling.k8s.io/v1
Kind:         VerticalPodAutoscaler
...
Spec:
  Resource Policy:
    Container Policies:
      Container Name:  *
      Max Allowed:
        Cpu:     500m
        Memory:  300Mi
      Min Allowed:
        Cpu:     10m
        Memory:  50Mi
  Target Ref:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         cert-manager
  Update Policy:
    Update Mode:  InPlaceOrRecreate
Status:
...
  Recommendation:
    Container Recommendations:
      Container Name:  cert-manager-controller
      Lower Bound:
        Cpu:     50m
        Memory:  50Mi
      Target:
        Cpu:     50m
        Memory:  50Mi
      Uncapped Target:
        Cpu:     50m
        Memory:  49566436
      Upper Bound:
        Cpu:     241m
        Memory:  300Mi
Events:          <none>

Update Modes

The updateMode controls how VPA applies changes to your pods.

In-Place Pod Resizing

Traditionally, changing a Pod's resources required a restart. However, with the InPlacePodVerticalScaling feature, Kubernetes can resize pods without restarting the containers.

When using updateMode: "InPlaceOrRecreate", VPA leverages this capability.

We can control the restart behavior using the resizePolicy in your Pod spec (inside a Deployment, StatefulSet, etc.):

1
2
3
4
5
6
7
8

spec:
  containers:
  - name: my-container
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired  # Default behavior
    - resourceName: memory
      restartPolicy: RestartContainer # Force restart for memory changes

Workflow: From Observation to Update

sequenceDiagram
    participant Dev as Developer
    participant K8s as Kubernetes
    participant VPA as VPA Recommender

    Dev->>K8s: 1. Deploy VPA (Mode: Off)
    loop Data Collection
        K8s->>VPA: Pod Usage Metrics
        VPA->>VPA: Calculate Recommendations
    end

    Dev->>VPA: 2. Check Recommendations
    VPA-->>Dev: Suggest: 500m CPU, 1Gi Mem

    Dev->>K8s: 3. Apply Update (Mode: Auto)
    K8s->>K8s: Recreate Pods with New Resources

1. Deploy in Off Mode: Create the VPA object with updateMode: "Off".
2. Wait: Let the application run under realistic load for at least 24 hours (or a full traffic cycle).

3. Inspect Recommendations:

   1	kubectl describe vpa my-app-vpa

   Look for the Recommendation section:
   • Target: The ideal value.
   • Lower Bound: The minimum value VPA is confident in.
   • Upper Bound: The maximum value VPA is confident in.

   You can also use Prometheus/Grafana to visualize the recommendations and make informed decisions:

   4. Apply: * Manually update your Deployment manifests with the Target values. * OR change updateMode to Recreate / InPlaceOrRecreate to let VPA handle it.

Best Practices

• PDBs: Ensure you have Pod Disruption Budgets configured. VPA respects them, so it won't evict too many pods at once during updates.
• OOM Handling: VPA reacts to OOM events. If a pod gets OOMKilled, VPA will increase the memory recommendation to prevent it from happening again.
• Golden Metrics: Use VPA recommendations as a guide to set baseline requests/limits in your GitOps manifests, even if you don't use auto-updates.