Scaling Kubernetes with Prometheus Custom Metrics

The Limitation of Standard Autoscaling

In the early days of Kubernetes adoption, most teams rely on the default Horizontal Pod Autoscaler (HPA) behavior: scaling based on CPU and Memory utilization. While this works for generic services, it is often a lagging indicator of actual load. For example, a Java application might consume high memory due to heap settings even when idle, or a Node.js event loop might be saturated while CPU usage remains low. To build truly resilient and responsive systems at SiberFX, we recommend moving toward custom metrics that reflect the actual work your application is performing.

Understanding the Metrics Pipeline

Kubernetes does not natively know about your application-level metrics like 'active orders' or 'request latency.' It relies on the Custom Metrics API. To bridge the gap between your monitoring stack and the Kubernetes control plane, we use a service called the Prometheus Adapter. This adapter queries Prometheus, translates the data into a format Kubernetes understands, and serves it via the custom.metrics.k8s.io endpoint.

Step 1: Exposing Application Metrics

Before scaling, your application must expose metrics in a Prometheus-compatible format. If you are using a Python Flask app with the prometheus_client library, you might expose a counter for total requests:

from prometheus_client import Counter, generate_latest

REQUEST_COUNT = Counter('http_requests_total', 'Total HTTP Requests', ['method', 'endpoint'])

@app.route('/api/data')
def get_data():
    REQUEST_COUNT.labels(method='GET', endpoint='/api/data').inc()
    return {"status": "success"}

Prometheus will scrape this endpoint and store a time series of your request counts.

Step 2: Configuring the Prometheus Adapter

The Prometheus Adapter is the most critical piece of the puzzle. It requires a configuration file (usually passed via a ConfigMap or Helm values) that defines how to transform a Prometheus query into a Kubernetes metric. Below is an example configuration for the http_requests_total metric, transformed into a 'rate' (requests per second):

rules:
- seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: {resource: "namespace"}
      pod: {resource: "pod"}
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

Breaking Down the Config:

• seriesQuery: This tells the adapter which Prometheus series to look for.
• resources: This maps Prometheus labels (like pod) to Kubernetes API resources. This is how the HPA knows which pod the metric belongs to.
• name: We use a regex to rename http_requests_total to http_requests_per_second for clarity in the HPA.
• metricsQuery: This is the actual PromQL executed. The <<.Series>> and <<.LabelMatchers>> are templates that the adapter fills in dynamically.

Step 3: Defining the Horizontal Pod Autoscaler

Once the adapter is running and the metric is available in the API (verify with kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1), you can create an HPA that targets this specific metric. Unlike a CPU-based HPA, we use the Pods metric type.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 50

In this configuration, Kubernetes will try to maintain an average of 50 requests per second per pod. If your app is receiving 500 requests per second, the HPA will scale the deployment up to 10 replicas.

Advanced Use Case: Scaling by Message Queue Depth

Custom metrics aren't limited to pod-level data. Sometimes you need to scale based on an external source, like a RabbitMQ or SQS queue depth. In this case, you use an External metric type. The logic remains the same: Prometheus scrapes the RabbitMQ exporter, the Adapter exposes the metric, and the HPA consumes it.

metrics:
- type: External
  external:
    metric:
      name: rabbitmq_queue_messages_ready
      selector:
        matchLabels:
          queue: "order-processing"
    target:
      type: AverageValue
      averageValue: 100

This is particularly powerful for background workers. It ensures that if your 'order-processing' queue spikes, you spin up more workers immediately rather than waiting for those workers to hit 90% CPU usage.

Best Practices and Common Pitfalls

1. Cooldown and Stabilization

Scaling up is usually aggressive, but scaling down should be conservative to avoid 'flapping' (rapidly adding and removing pods). Use the behavior field in the HPA spec to define stabilization windows. A 300-second (5-minute) stabilization window for scale-down is a standard starting point.

2. Aligning Scrape Intervals

If Prometheus scrapes your app every 30 seconds, but your HPA checks every 15 seconds, the HPA will see stale data. Ensure your Prometheus scrape interval, the Adapter's cache duration, and the HPA's evaluation period are somewhat aligned to prevent erratic scaling decisions.

3. Handling Zero Traffic

If your metric disappears when there is zero traffic (e.g., no requests means no time series), the HPA might default to its minimum replica count or, in some misconfigured cases, stay at its last known value. Use PromQL functions like absent() or or vector(0) in your adapter query to ensure a value is always returned.

Conclusion

Moving beyond CPU and Memory for autoscaling is a significant step in operational maturity. By integrating Prometheus with the Kubernetes Custom Metrics API, you can ensure your infrastructure reacts to user behavior and business demand in real-time. Whether it's HTTP request rates, database connection counts, or queue depths, the Prometheus Adapter provides the flexibility needed to build a truly self-healing and efficient platform at SiberFX.