What Is an SLO? A Practical Guide for Platform Teams

What is an SLO? A service level objective is a target for how reliable one specific user-facing behavior must be, measured over a fixed window. "99.9% of checkout requests succeed in under 800 ms, over 30 days" is an SLO. "The site should be fast" is a wish. The number matters less than what it controls: the SLO decides when you keep shipping and when you stop.

It is 03:12. PagerDuty says checkout latency is "elevated". You open the dashboard and see a p99 line that has wandered up for forty minutes. Nobody can tell you whether that is a problem, because nobody ever wrote down how much latency is too much. The argument that follows, on a call, with one eye on Slack, is the cost of not having an SLO.

SLI, SLO, SLA: three words, three owners

People use these interchangeably. They are not the same thing, and mixing them up is how teams end up with a contract they cannot meet.

• SLI (service level indicator) is the measurement. A ratio of good events to valid events: requests that returned 2xx in under 800 ms, divided by all requests that should have.

• SLO (service level objective) is the target for that ratio over a window. 99.9% over 30 days.

• SLA (service level agreement) is the contract with a customer, with money attached if you miss. It is always looser than the SLO. If your SLA says 99.9% and your SLO says 99.9%, you have no room to notice a problem before you owe credits.

The Google SRE book chapter on service level objectives draws the same line: the SLO is an internal target that sits tighter than anything you promise externally. Engineers own the SLI and SLO. Product and legal own the SLA.

Skip the instinct to add a fourth word for every metric you already collect. CPU, memory and queue depth are useful for debugging. They are not SLIs, because no customer experiences them directly.

What a good SLI looks like in practice

A good SLI is a ratio, it is measured as close to the user as you can get, and it has a clear definition of "good".

For a request-driven service, the default is two SLIs: availability (non-5xx responses over all valid requests) and latency (requests under a threshold over all valid requests). Use a threshold, not an average. An average hides the tail, and the tail is where the support tickets come from.

Measure at the load balancer or the edge, not inside the service. A service that crashes on startup reports a perfect error rate for the requests it never received. Server-side metrics are blind to that failure, and the user sees it as an outage.

Be explicit about what is excluded. Health checks, synthetic probes from your own monitoring, and 4xx responses caused by bad client input usually should not count against you. Write the exclusions down next to the SLO. When someone argues about the number during an incident review, the argument should be about the definition, not about what the dashboard happens to show.

The arithmetic nobody does before picking nines

Every extra nine cuts your allowed downtime by a factor of ten. This is the table to keep in the doc.

• 99% over 30 days: 1% error budget, 7.2 hours of allowed full outage.

• 99.9%: 0.1% budget, 43.2 minutes.

• 99.95%: 0.05% budget, 21.6 minutes.

• 99.99%: 0.01% budget, 4.32 minutes.

Read the last line again. At 99.99%, a single bad deploy that takes five minutes to detect and revert has already blown the month. Humans cannot respond to a page, open a laptop and revert in under four minutes, which means that target is only reachable with automation: an automated gate that stops the rollout before the budget is gone.

So pick the number the service actually needs. Most internal APIs live happily at 99.9%. A payments path may justify more. A batch job that runs nightly may be fine at 99%. Choosing 99.99% because it sounds serious is how a team signs up for a pager rotation it cannot staff.

The error budget is the point, not the SLO

An SLO of 99.9% means you are allowed to fail 0.1% of the time. That 0.1% is your error budget, and it is the most useful idea in the whole discipline.

The budget turns an argument into a number. Product wants to ship the risky migration. The SRE says it is risky. Without a budget, that is an opinion against an opinion, and seniority wins. With a budget, the question becomes: how much is left, and how much does this change plausibly burn?

Two rules make it work.

1. Budget remaining: ship. You are allowed to take risk, so take it. A team that never spends its budget is over-investing in reliability the users did not ask for.

2. Budget exhausted: freeze. Feature work pauses until reliability work brings the SLI back above the target. This only holds if leadership agreed to it before the first incident, not during it.

If the second rule is not agreed in advance, you do not have an SLO. You have a dashboard with a green and a red zone.

Alerting on burn rate, not on the SLI itself

Here is where most first attempts fall over. Someone sets an alert when availability drops below 99.9%. The alert fires after the damage is done, or it flaps on every blip, or both.

The better signal is burn rate: how fast you are consuming the budget relative to the pace that would exactly exhaust it at the end of the window. A burn rate of 1 means you will finish the month with zero budget left. A burn rate of 14.4 means you are on track to burn the whole month's budget in about two days.

The Google SRE Workbook on alerting on SLOs recommends pairing a long and a short window so the alert is both significant and still happening. For a 30-day SLO, two pages are a good starting point:

• Burn rate 14.4 over 1 hour, confirmed over 5 minutes: consumes 2% of the monthly budget. Page someone.

• Burn rate 6 over 6 hours, confirmed over 30 minutes: consumes 5% of the monthly budget. Page someone.

The arithmetic is plain. A 30-day window is 720 hours. Burning at 14.4x for one hour uses 14.4 / 720 = 2% of the budget. The short window stops the alert from firing an hour after the problem already ended.

Here is the first rule as a Prometheus alert, assuming you record an error ratio per window:

groups:
  - name: checkout-slo-burn
    rules:
      - alert: CheckoutErrorBudgetFastBurn
        expr: |
          job:checkout_error_ratio:rate1h > (14.4 * 0.001)
          and
          job:checkout_error_ratio:rate5m > (14.4 * 0.001)
        labels:
          severity: page
        annotations:
          summary: "Checkout is burning 2% of its 30-day error budget per hour"

The 0.001 is the allowed error ratio for a 99.9% SLO. Change the target and that constant moves with it. Keep it in one place, or the alert and the SLO will drift apart without anyone noticing.

Slower burns do not need a page. A burn rate of 1 sustained over three days belongs in a ticket for the next working morning. Waking someone for a slow leak is how on-call teams learn to ignore the pager.

Where to compute the SLI without buying a second platform

You do not need a dedicated SLO product to start. You need a good-events counter, a total-events counter, and a place to store them for 30 days.

If your metrics already live in a hosted platform, the SLO feature is usually a thin layer over data you pay for already.

Datadog covers SLOs natively, with burn-rate alerts, and it is the easy path if your traces and metrics already live there. The bill is the catch: custom metrics and high-cardinality tags multiply fast, so check the price of the SLI series before you commit.

If you run Prometheus, Grafana covers the same ground through recording rules and its SLO tooling, at the price of more setup and more of your own time.

The honest comparison is time to first useful alert against cost at scale. Datadog wins the first. Grafana with Prometheus usually wins the second. Pick based on which one your team is short of.

Connecting the SLO to the rollout, which is the part most teams skip

An SLO that only produces a report is half used. The step that changes behavior is wiring the budget into the deploy path, so the release stops itself.

A progressive rollout with an SLO gate works like this. Ship to 1% of traffic behind a flag. Watch the SLI for that cohort against the baseline. If the burn rate for the canary cohort passes your threshold, revert automatically. If it holds, step to 5%, then 25%, then 100%.

That is a different thing from a dashboard someone is supposed to be watching at 03:12. At 03:12, you do not want to think. You want a button, or better, no button at all.

This is where feature flags earn their keep, because a flag turns a revert into a config change that takes seconds rather than a redeploy that takes ten minutes.

LaunchDarkly and similar platforms give you the percentage rollout and the kill switch. What most of them leave to you is the decision: which SLI to read, what threshold counts as bad, and who gets to override. That is the gap an automated pilot with auto-rollback on SLO is meant to close, and it is the angle we take at upstreamapi. LaunchDarkly wins on SDK breadth and enterprise controls, and we say so.

Here is the shape of the gate in TypeScript. The decision logic is the part worth writing yourself, whichever flag system you use:

type Cohort = { good: number; total: number };

const SLO_TARGET = 0.999;
const BUDGET = 1 - SLO_TARGET; // 0.001
const PAGE_BURN_RATE = 14.4;

function burnRate(c: Cohort): number {
  if (c.total === 0) return 0;
  const errorRatio = 1 - c.good / c.total;
  return errorRatio / BUDGET;
}

export function rolloutDecision(canary: Cohort): "advance" | "hold" | "revert" {
  if (canary.total < 500) return "hold"; // not enough traffic to judge
  return burnRate(canary) > PAGE_BURN_RATE ? "revert" : "advance";
}

Notice the hold branch. A canary at 1% of a quiet service may see forty requests in ten minutes, and one error out of forty is a 2.5% error ratio. Without a minimum sample, the gate reverts on noise and your team learns to distrust it.

Mistakes we keep seeing at 80-engineer teams

The same few failures show up across teams, and almost none of them are technical.

Too many SLOs. Twenty objectives per service means nobody remembers any of them. Two or three per user journey is enough.

SLOs nobody agreed to. An engineer writes them alone over a weekend. Product has never seen them, so the budget policy collapses the first time it says "no" to a launch.

SLOs set at current performance. If the service runs at 99.97% today, setting the target at 99.97% leaves no budget and guarantees a miss. Set it where users start to be unhappy, which is almost always lower.

Treating the SLO as permanent. Review the number every quarter. If you never spend the budget, tighten the target or ship faster. If you blow it every month, either fix the system or admit the target was a fantasy.

Counting what is easy rather than what is felt. A 200 response with an empty cart is a success in your logs and a failure for the customer. Define "good" from the user's side.

Who should write the first one this week

You do not need a program. Pick the one journey that would make the CEO call someone if it broke. Define one availability SLI and one latency SLI for it. Set the target at 99.9% for the first quarter, write the freeze policy in two sentences, and get product to reply "agreed" in writing.

Then add a burn-rate alert and a place to see the remaining budget where the whole team can see it.

If you need a lightweight place to start before committing to a full platform, Better Stack handles uptime monitoring and incident paging with a much smaller setup, though it will not give you the budget math for free.

The first version will be wrong. That is acceptable. What matters is that the next incident review starts from a number you wrote down beforehand rather than a feeling.

So what does your 03:12 look like now?

Take the last incident your team had. Was there a number written down that said whether it mattered? If the honest answer is no, that is your first SLO, and the post-mortem already told you what to measure.