Where did the time go?
Traces
A trace follows one request across every service it touches. Each hop is a span with a start, a duration, a parent and a bag of attributes. Traces turn a distributed system back into a story you can read.
A field guide to observability
Your system is
already talking.
Learn to listen.
Traces, metrics and logs, explained plainly. How OpenTelemetry made them a shared language — and how to pick the backend that stores them without regretting it a year later.
trace_id 4bf92f3577b34da6a3ce929d0e0e4736412 ms · 8 spans · 6 services
0ms100ms200ms300ms400ms
span POST acme-pay/charges
duration 104 msstatus=ERROR
- http.response.status_code
- 503
- error.type
- ServiceUnavailable
↑ a real-shaped trace. Hover a span — one retry hid a 503.
01The signals
Four signals, each answering a different question.
Is something wrong right now?
Metrics
Metrics are numbers aggregated over time — request rate, error ratio, p99 latency, queue depth. They are cheap to store and fast to query, which makes them ideal for dashboards and alerts.
What exactly happened?
Logs
Logs are timestamped records of discrete events. Structured logs carrying a trace_id let you jump from a slow span straight to the lines that explain it.
Which line of code is hot?
Profiles
Continuous profiling samples CPU and memory in production. It is the newest signal, and OpenTelemetry is adding it so profiles can be linked to the spans that caused them.
02Principles
01
Monitoring tells you that. Observability tells you why.
Monitoring checks for failures you predicted. Observability is the ability to ask new questions about failures you never imagined — without shipping new code to answer them.
02
Context is the real product.
A metric spike, a slow trace and an error log are three views of one incident. Shared identifiers — trace IDs, resource attributes, consistent names — are what let you move between them.
03
Cardinality is a feature, and a cost.
The fields that make debugging possible — user ID, build SHA, tenant — are exactly the ones that make storage expensive. Picking a backend is largely picking how you pay for them.
chapter 2
OpenTelemetry, end to end
API, SDK, Collector and OTLP — what each piece does, and how data flows from a line of code to a dashboard.
read the chapter →
chapter 3
Choosing a backend
Twelve open-source and commercial backends compared by signal, storage model and how they accept OpenTelemetry data.
open the directory →