Monitoring, Logs and Traces: Find Problems Step by Step

A user says the website is slow. CPU usage alone may not explain why. You need several kinds of information to understand the request.

Observability means being able to learn about a system from the signals it produces. Start with a useful question, not a dashboard full of every number.

Three useful signals

• Metrics: Numbers over time, such as request counts and duration.
• Logs: Records of events, such as one failed request.
• Traces: Connected steps from one request as it moves through services.

Prometheus stores and queries metrics. Grafana can show data from supported sources. OpenTelemetry helps collect and export telemetry; it is not itself a complete long-term storage and dashboard system.

Use the demo app

The app in the Docker lesson already has /metrics. It counts responses by a small status label and writes a JSON log with a request ID.

Run it and request /health, /missing and /fail. The last path creates a simulated 500 response for this lab. Open /metrics to see the counts.

This simple metric includes health checks and is not a complete measure of real user success. Decide which requests belong in a production service-level measure.

Add Prometheus

Save this as prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - /etc/prometheus/alerts.yml
scrape_configs:
  - job_name: demo-web
    static_configs:
      - targets: ["web:8080"]

Add this compose.observability.yaml beside the base compose.yaml from the local-development lesson:

services:
  prometheus:
    image: prom/prometheus:v3.15.1
    ports:
      - "127.0.0.1:9091:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./alerts.yml:/etc/prometheus/alerts.yml:ro

Also save the alert rules below as alerts.yml. Then run:

docker compose -f compose.yaml -f compose.observability.yaml up --build

Open the app on port 8085 and Prometheus on http://127.0.0.1:9091. The target web:8080 is the app’s address inside the Compose network. It is not your laptop’s loopback address.

Read two queries

sum(rate(demo_http_requests_total[5m]))

sum(rate(demo_http_requests_total{status=~"5.."}[5m]))
/
sum(rate(demo_http_requests_total[5m]))

The first is requests per second over a five-minute range. The second is the share of measured requests returning 5xx errors. It is a ratio, not a percentage until you multiply by 100.

At least two samples and some traffic are needed for useful rates. With no requests, the ratio can be undefined. Do not label missing data as proof that everything is healthy.

Make an alert useful

groups:
  - name: demo-web
    rules:
      - alert: DemoHighErrorRatio
        expr: sum(rate(demo_http_requests_total{status=~"5.."}[5m])) / sum(rate(demo_http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: More than five percent of measured demo requests are failing

The rule waits for a high error share to last five minutes. It shows alert state in Prometheus. It does not send a message by itself; notification needs routing, for example through Alertmanager.

A real alert needs an owner, user impact, a clear next check and a link to a runbook. High CPU is not always a page-worthy incident.

Connect a log to a request

The demo returns X-Request-ID and writes the same ID to its log. Use that ID to find one request. In a distributed app, trace context can connect several services.

Do not use a user ID or full URL as an unbounded metric label. Too many distinct label values can make a metrics system expensive. Keep private data out of logs.

A small OpenTelemetry trace

In a separate environment, install opentelemetry-api and opentelemetry-sdk. Save and run this trace_demo.py:

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor

provider = TracerProvider(resource=Resource.create({"service.name": "lecturre-lab"}))
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
with tracer.start_as_current_span("prepare-lesson") as span:
    span.set_attribute("lesson.name", "containers")
    with tracer.start_as_current_span("read-demo-data"):
        print("Loaded local lesson")
provider.shutdown()

It prints a parent span and a child span to the console. A span records one operation. This example does not send data to a hosted service.

A practical investigation

1. Check which users and endpoints are affected.
2. Look for a change in error share and response duration.
3. Find a failing request ID in logs.
4. Use a trace to see where time was spent.
5. Compare the time with releases or infrastructure changes.
6. Make one controlled change and check user impact again.

Practice

Create a few simulated failures. Explain the difference between a counter, a rate and a ratio. Write a two-line alert message that tells the next person what to check.

Official references and further reading

Reviewed on 25 October 2026. Tools change, so check the official documentation for the version you use.

• OpenTelemetry signals
• OpenTelemetry Python instrumentation
• Prometheus query basics
• Prometheus alerting overview

DevOps learning roadmap · All DevOps & Cloud tutorials