Error budgets and burn-rate alerts: paging for the right reasons

Start from what users experience

Most noisy paging comes from alerting on causes — CPU, memory, a restarted pod — instead of on what users feel. A service-level indicator (SLI) measures the latter directly: the share of valid requests that were good. Good can mean a response without a server error, a response faster than a threshold, or both.

A service-level objective (SLO) sets a target for that ratio over a window, for example 99.9% of requests succeed over 30 days. The gap to 100% is the error budget: here 0.1% of requests, or about 43 minutes of complete outage in the month.

Why threshold alerts fail

An alert such as “error rate above 1% for 5 minutes” is too sensitive and too slow at the same time. It pages for a short blip that used a sliver of the budget, and it never fires for a 0.5% error rate that quietly burns the whole budget in a week.

Burn rate fixes this. It is the speed at which the budget is consumed, relative to the SLO: a burn rate of 1 spends exactly the budget over the window; a burn rate of 14.4 spends a 30-day budget in a little over two days.

Multiwindow, multi-burn-rate alerts

The approach described in Google's SRE Workbook pairs each burn-rate threshold with a long window, which proves the problem is significant, and a short window, which proves it is still happening. For a 99.9% SLO over 30 days, a common starting point is:

• Page: burn rate above 14.4 over 1 hour and over 5 minutes — 2% of the budget spent in one hour.
• Page: burn rate above 6 over 6 hours and over 30 minutes — 5% of the budget spent.
• Ticket: burn rate above 1 over 3 days and over 6 hours — 10% of the budget spent.

# Fast-burn page for a 99.9% availability SLO (error budget = 0.001)
(
  sum(rate(http_requests_total{job="api", code=~"5.."}[1h]))
    / sum(rate(http_requests_total{job="api"}[1h]))
) > (14.4 * 0.001)
and
(
  sum(rate(http_requests_total{job="api", code=~"5.."}[5m]))
    / sum(rate(http_requests_total{job="api"}[5m]))
) > (14.4 * 0.001)

The short window matters as much as the long one: once the fix lands, the 5-minute ratio drops and the alert resolves, instead of firing for the rest of the hour.

Details that decide whether it works

• Recording rules: precompute the error ratio for each window, so alert expressions stay cheap and readable.
• Low traffic: with a few requests per minute, one failure is a large ratio. Require a minimum number of requests, or measure with synthetic probes.
• Latency SLOs: count requests slower than the threshold as bad events, with histogram buckets that include the threshold value.
• Exclusions: decide in advance whether client errors and health checks count, and write it next to the SLO.

What the budget is for

The budget is a decision tool, not only a pager. While it is healthy, the team can ship faster and take on riskier changes. When it is exhausted, reliability work moves ahead of features until the trend recovers. Agree on that policy with the product owners before the first breach, not during it.