The observability baseline every production service needs

Many teams have plenty of telemetry and still find out about outages from customers. Dashboards nobody opens, hundreds of alerts nobody trusts, logs that cannot be searched across services. More data is rarely the fix. What helps is a small, consistent baseline that every service meets, built around what users actually experience.

Start from the user: SLIs and SLOs

Pick a few service level indicators (SLIs) that describe the user's experience — for most request-driven services, the share of requests that succeed and the share that are fast enough. Set an objective (SLO) for each, such as 99.9% of requests succeeding over 30 days. The gap between the objective and 100% is your error budget: the amount of unreliability you can spend on shipping changes before reliability work takes priority.

SLOs turn "is it broken?" from a debate into a number, and they give product and engineering a shared way to decide when to slow down.

Metrics: the same few signals everywhere

Every service should expose the same core metrics with the same names, so that one dashboard template works for all of them. For request-driven services that means the RED method — rate, errors and duration. For the resources underneath, such as nodes, queues and databases, the USE method — utilization, saturation and errors. Together they cover the four golden signals from Google's SRE book: latency, traffic, errors and saturation.

• Record latency as a histogram, not an average; averages hide the slow requests users complain about.
• Keep label cardinality under control. A label holding user IDs or full URLs can multiply your time series into the millions and take Prometheus down with it.
• Label by route template (/orders/:id), status class and version — the version label is what lets you compare a canary against the stable release.

Traces: follow a request across services

In a system with more than a handful of services, the question during an incident is usually "where is the time going?". Distributed tracing answers it. OpenTelemetry is now the default choice: vendor-neutral SDKs for most languages, automatic instrumentation for common frameworks, and W3C trace context propagation so a trace survives the trip across services and queues.

Send telemetry through an OpenTelemetry Collector rather than straight from applications to a vendor. The collector lets you sample, filter and route data — for example, keep every trace that contains an error and only a fraction of the successful ones — and switch backends without touching application code.

Logs: structured and correlated

• Log as structured JSON with consistent field names, not free-form text.
• Include the trace ID and span ID in every log line, so you can jump from a slow trace to its logs and back.
• Log events and decisions, not every function call; high-volume debug logging is expensive and rarely read.
• Never log secrets, tokens or personal data. Redact at the source rather than hoping to filter later.

Alert on symptoms, not causes

Page a human only when users are affected or are about to be. High CPU is not an incident if latency is fine; a disk that fills in four hours is. The most reliable way to alert on symptoms is on SLO burn rate: how fast you are spending the error budget. Multi-window burn-rate alerts — a fast one that pages for a sudden outage, a slow one that opens a ticket for a gradual degradation — catch real problems with far fewer false alarms than static thresholds.

Every alert that pages should link to a runbook that says what to check first. If nobody can write that runbook, the alert is probably not actionable and should not page.

Make it the default

A baseline only works if every service gets it without each team rebuilding it. Put instrumentation in the shared service template, dashboards and SLO definitions in code next to the service, and a check in CI that a new service exposes the standard metrics. Then the baseline is something you get by starting a service, not a project each team schedules.

Why it pays off twice

Good observability shortens incidents, but it also makes delivery safer. Automated canary analysis is only as good as the metrics it reads, and teams that can see the impact of a change within minutes are the ones that deploy many times a day. The baseline is the prerequisite for both.

We set up observability baselines with Prometheus, Grafana and OpenTelemetry as part of our DevOps and platform engineering work, including on a self-service CI/CD platform for 60 engineers. If your team is drowning in alerts or blind during incidents, get in touch.