A production baseline for Kubernetes workloads

Why a baseline

Kubernetes runs almost anything with almost no configuration, and that is the problem: the defaults are chosen for getting started, not for staying up. The settings below are cheap to add on day one and expensive to discover during an outage. They apply to stateless Deployments; stateful workloads need more.

Requests and limits

The scheduler places pods by their requests, and the Horizontal Pod Autoscaler measures utilisation against them. A pod without requests is placed blindly and cannot be autoscaled on CPU.

Memory cannot be compressed: a container that exceeds its memory limit is killed. Set the memory limit equal or close to the request, so a pod is never scheduled on memory it cannot keep. CPU is different — a container above its CPU limit is throttled, which shows up as latency. Many teams set CPU requests and leave CPU limits off for latency-sensitive services; whichever you choose, measure throttling.

Three probes, three jobs

• Readiness decides whether the pod receives traffic. Be careful with shared dependencies: if every pod fails readiness at once, the service is left with no endpoints at all.
• Liveness decides whether the container is restarted. Keep it to the process itself: if it checks the database, a database outage restarts every pod at once.
• Startup holds the other probes back while a slow application boots, instead of inflating the liveness delay.

Surviving node drains and zone failures

Node upgrades and autoscaling evict pods all the time. A PodDisruptionBudget limits how many replicas a voluntary disruption may take down at once, and topology spread constraints keep replicas in different zones, so one zone failure does not take the whole service with it.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: api }

Graceful shutdown

When a pod is deleted, its removal from the service endpoints and the shutdown of its containers (the preStop hook, then SIGTERM) start at the same time. An application that exits immediately drops requests that are still being routed to it. Add a short preStop pause so load balancers catch up, handle SIGTERM by finishing in-flight work, and keep terminationGracePeriodSeconds longer than the pause plus the drain.

Security context and images

Run as a non-root user, drop all Linux capabilities, block privilege escalation and mount the root filesystem read-only; the restricted Pod Security Standard enforces most of this per namespace. Reference images by digest instead of a mutable tag, so what was tested is exactly what runs.

Putting it together

A Deployment with the baseline applied. The values are placeholders: size requests from measured usage, not from this example.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  selector:
    matchLabels: { app: api }
  template:
    metadata:
      labels: { app: api }
    spec:
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels: { app: api }
      securityContext:
        runAsNonRoot: true
        seccompProfile: { type: RuntimeDefault }
      containers:
        - name: api
          image: registry.example.com/api@sha256:<digest>
          resources:
            requests: { cpu: 250m, memory: 256Mi }
            limits: { memory: 256Mi }
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          startupProbe:
            httpGet: { path: /live, port: 8080 }
            periodSeconds: 5
            failureThreshold: 30
          lifecycle:
            preStop:
              sleep: { seconds: 5 }   # native sleep action, Kubernetes 1.30+
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }