What Is Observability? A Working Definition for SREs

What is observability? It is the ability to explain why your system misbehaves using only the telemetry it already emits, including for failures nobody predicted. Monitoring tells you that checkout latency crossed 800ms. Observability lets you ask which tenant, which dependency and which deploy caused it, without shipping new code at 3am to find out.

Here is the scene that makes the difference concrete. The alert fires: p99 on the payments API is up from 180ms to 1.4s. The dashboard is green on CPU, memory and error rate. Every pre-built panel says the service is fine, and every customer says it is not. What you do in the next ten minutes is the whole subject of this article.

Why does the definition matter before the tooling?

The term comes from control theory: a system is observable if you can infer its internal state from its external outputs. Software borrowed it, and then vendors stretched it until it meant "a dashboard product". Ignore that stretch.

The working definition for an engineering team is narrower and more useful. A system is observable when you can answer a new question about its behavior, in production, without deploying new instrumentation. The word that matters is "new". A threshold alert answers a question somebody wrote down last quarter. An observable system answers the one you are asking right now.

This is not a purity test. Nobody has a perfectly observable system, and the goal is not a score. The goal is a shorter path from "something is wrong" to "this is the change that caused it". Mean time to detect and mean time to understand are different numbers, and most teams only track the first.

How is observability different from monitoring?

Monitoring is a set of checks against failure modes you already know. CPU above 90% for five minutes, error rate above 1%, queue depth above 10,000. It is cheap, it is reliable, and you still need it. Observability does not replace it.

The difference shows up when the failure is novel. Monitoring is built from your past incidents. Observability is built for the next one, which by definition you have not seen.

A useful test: look at your last three incidents and ask how the root cause was found.

• If someone read a pre-built dashboard and the answer was on it, that was monitoring doing its job.

• If someone grepped logs across six hosts, guessed a hypothesis, and then added a log line and redeployed to confirm it, that was a gap in observability.

• If the real answer came from a customer's screenshot, you have both problems.

Skip the debate about whether the two are rivals. They are layers. Monitoring is the smoke detector. Observability is the ability to walk the building afterwards and work out which circuit started the fire.

What are the three signals, and where does each one fail?

Logs, metrics and traces are the standard trio. Each one answers a different question, and each one has a failure mode that vendor guides tend to skip.

Metrics are numeric measurements aggregated over time: request rate, error rate, latency percentiles, saturation. They are cheap to store and fast to query, which is why alerting lives on them. The failure mode is aggregation. A p99 of 1.4s tells you that something is slow. It cannot tell you that every slow request belongs to one customer on one shard.

Logs are timestamped records of discrete events. They carry the most detail and the least structure unless you force structure on them. The failure mode is volume and cost. Unstructured logs are searchable only by people who already know what string to look for, which makes them a monitoring tool wearing a debugging costume.

Traces follow a single request across service boundaries, with a span for each hop and its timing. They are the only signal that shows you the shape of a request: this call waited 900ms on a downstream dependency that retried three times. The failure mode is coverage. One service that drops the trace context breaks the chain, and a trace with a hole in it is worse than useless because it looks authoritative.

Having all three is not the same as being observable. The point is correlation. You need to go from a metric spike to the traces that contributed to it, and from a trace to the logs for that exact request. If the three live in separate tools with no shared identifier, you have three monitoring systems and an engineer acting as the join.

Why does cardinality decide whether you can debug anything?

Cardinality is the number of distinct values a field can take. status_code has maybe twenty. customer_id has hundreds of thousands. request_id has one per request.

The fields you most need during an incident are the high-cardinality ones. "Which customer is affected" is a customer_id question. "Which build is serving the slow requests" is a deploy_version question. If your storage model punishes high-cardinality fields, either by pricing them per unique series or by silently dropping them, you will be unable to ask the question that cracks the incident.

This is the real architectural line between a metrics-only setup and an observable one. Pre-aggregated metrics throw away the dimensions you will want later. Wide, structured events keep them, and you slice after the fact.

Here is what a useful event looks like at the edge of a service. One structured record per request, with enough context to slice by anything you will later care about:

logger.info("request.completed", {
  trace_id: span.traceId,
  route: "/v1/payments",
  customer_id: ctx.customerId,
  deploy_version: process.env.BUILD_SHA,
  flag_variant: flags.get("new-ledger-path"),
  db_ms: timings.db,
  upstream_ms: timings.upstream,
  status: res.statusCode,
  duration_ms: elapsed,
});

Note the feature flag variant in there. Half of the incidents we see in published post-mortems start with a change, and a flag evaluation is a change. If the variant is not on the event, you cannot ask whether the slow requests are the ones on the new path.

What does an incident look like with and without it?

Take the payments scenario again. Same alert, same 1.4s p99.

Without observability, the loop is: open the service dashboard, find nothing, open the database dashboard, find nothing, ask in the incident channel whether anything deployed. Someone remembers a config change. Someone else disagrees. Forty minutes in, you revert the last deploy to see if that helps. It does not, because the cause was a flag rollout to a subset of tenants.

With observability, the loop is: group slow requests by flag_variant, see that 100% of them are on new-ledger-path, group by customer_id, see that three large tenants account for most of the volume, open one trace and see the ledger call fan out into 40 sequential queries. Total time: a few minutes. You turn off the flag, not the deploy.

Notice what the second loop did not need. It did not need a prepared dashboard. It did not need anyone to predict "flag variant" as an interesting dimension in advance. It needed the data to be there, wide and queryable.

That is the Honeycomb argument in one paragraph, and it holds up. What it does not say is that this is free.

What does it cost, and who should skip the expensive version?

Telemetry is a data volume problem before it is a tooling problem. According to Grafana Labs' 2025 observability survey of 1,255 practitioners, the average organization runs eight different observability tools, and cost is the most-cited criterion when choosing new ones. Those two facts are connected. Teams accrete tools because each one solves a slice, then pay for overlapping ingestion in all of them.

Practical rules that hold at most team sizes:

• Instrument with OpenTelemetry, not a proprietary agent. The collector gives you one place to sample, filter and route. Switching backends becomes a config change rather than a re-instrumentation project.

• Sample traces deliberately. Keep every error and every slow request, and sample the healthy fast ones. Head-based sampling at a flat 1% will discard the exact trace you need.

• Put structure on logs at write time. Parsing free text at query time is the expensive way to get fields.

• Set a retention policy per signal. Metrics for a year, wide events for two weeks, debug logs for three days. Nobody has ever needed last quarter's debug logs.

Skip the full-platform approach if you run a handful of services and a single on-call rotation. A small team gets most of the benefit from structured logs, RED metrics and traces on the two or three hot paths. The expensive, do-everything contract pays off when you have enough services that nobody holds the whole topology in their head, which in our experience is somewhere past 30 or 40 services.

Which tools are worth looking at?

Tool choice depends on how much you want to run yourself, what your cardinality looks like, and what your bill can absorb. A short, opinionated tour:

Datadog is the broadest commercial option: metrics, logs, traces and profiling under one roof, with strong correlation between them. The price of that integration is a bill that scales with hosts, custom metrics and ingested log volume. Worth it when you want one vendor and have the budget to match. Check the custom-metrics line item before you commit.

Grafana is the default choice when you want open components you can self-host or buy as a managed stack. It pairs well with Prometheus-style metrics and OpenTelemetry pipelines. The trade is operational: you own more of the assembly, and the sprawl figure from the survey above applies to you too.

SigNoz is an open source, OpenTelemetry-native option that puts logs, metrics and traces in one datastore. That matters for the correlation point earlier, since a shared store makes the metric-to-trace-to-log jump a query rather than a tab switch. Reasonable for teams that want to self-host without stitching four projects together.

Better Stack combines uptime monitoring, incident alerting and log management at a price aimed at smaller teams. It is a sensible first step if you are moving from "we find out from customers" to something structured, and less suited to a large service mesh with heavy trace requirements.

How do you start without boiling the ocean?

Do not begin with a tool evaluation. Begin with the last three incidents and one honest question: what did we wish we could have asked?

A sequence that works for a team of 50 to 200 engineers:

1. Pick one critical request path and instrument it end to end with OpenTelemetry. Checkout, login, whatever pages you at 3am.

2. Emit one wide structured event per request at the service edge, with customer, version, flag variants and downstream timings.

3. Define one SLO on that path and alert on burn rate, not on raw thresholds. Burn-rate alerts page you for user impact, and the telemetry from step 2 tells you why.

4. Run a game day. Inject a failure you have not alerted on and see how long it takes someone who did not write the code to find it. That number is your baseline.

5. Expand outward one path at a time, only after the first path has paid for itself in a real incident.

Observability gets bought as a platform and adopted as a habit. The habit is asking "what would I need to know to explain this?" at code review, before the incident rather than during it.

Look at your last incident review. Find the sentence that says "we then added logging to confirm". Every one of those is a question your system could not answer in production. How many do you have on file, and which of them would still be unanswerable today?