Error Handling and Logging Patterns for Production Systems

Previously in this course, we discussed designing for failure and implemented circuit breakers to keep our services stable. While those patterns prevent outages, they don't tell you why an error occurred or what state the system was in when it happened. This lesson adds the visibility layer: how to implement structured logging and centralized error reporting so you can debug production issues with confidence.

From Text Logs to Structured Data

In local development, we often rely on console.log or print() to output human-readable strings. In production, these are useless. If you have 50 instances of a service, you cannot grep across hundreds of gigabytes of flat text files.

Structured logging converts your logs into machine-readable formats, typically JSON. Instead of a message like "User 123 failed to update profile", you output an object:

JSON
{
  "timestamp": "2023-10-27T10:00:00Z",
  "level": "error",
  "event": "profile_update_failed",
  "user_id": 123,
  "error_code": "DB_TIMEOUT",
  "latency_ms": 1500
}

By using structured fields, your log aggregation tool (like ELK, Datadog, or Grafana Loki) can index user_id and error_code, allowing you to filter by specific users or aggregate statistics on which errors happen most frequently.

Designing Centralized Error Workflows

Centralization is the process of shipping logs from individual hosts to a single searchable repository. Without this, your logs disappear the moment a container restarts or a server is terminated.

A robust error reporting workflow should look like this:

1. Instrument: The application logs structured data.
2. Collect: A sidecar or agent (like Fluentd or Promtail) tails these logs.
3. Aggregate: Logs are pushed to a central store.
4. Notify: Errors above a certain severity trigger alerts in Slack or PagerDuty.

For deeper insights into this architecture, you might find Observability and Logging: Mastering MLOps Production Telemetry helpful, as it covers the specific technical requirements for keeping track of production telemetry.

Worked Example: Implementing Structured Logging

Let’s refine our system design project by implementing a basic structured logger in a Python-based service.

PYTHON
import json
import time
import uuid

def log_event(level, message, **kwargs):
    entry = {
        "timestamp": time.time(),
        "level": level,
        "message": message,
        "correlation_id": getattr(context, CE9178">'request_id', CE9178">'none')
    }
    entry.update(kwargs)
    print(json.dumps(entry))

# Usage in a service
def update_user_profile(user_id, data):
    request_id = str(uuid.uuid4())
    try:
        # Simulate DB operation
        raise ValueError("Database connection lost")
    except Exception as e:
        log_event("error", "profile_update_failed", 
                  user_id=user_id, 
                  error=str(e),
                  correlation_id=request_id)

In a real system, you would attach a correlation_id to every request. This allows you to trace a single user's path across multiple services—a concept we'll expand upon in Distributed Tracing Basics. If you are building plugins or specialized services, Advanced Error Handling: Building Production-Grade WordPress Plugins offers a good look at how to handle fatal crashes outside the standard application flow.

Hands-on Exercise

1. Choose one service in your running design project.
2. Replace all print statements with a JSON-formatted logger.
3. Add at least three context fields to your logs (e.g., user_id, request_id, service_name).
4. Goal: Ensure that if you search for {"level": "error"}, you can see the specific user_id involved in the failure.

Common Pitfalls

• Logging Sensitive Data: Never include PII (Personally Identifiable Information) like passwords, credit card numbers, or email addresses in your logs. These logs often get stored in third-party systems where you lose control over data privacy.
• Logging Too Much: Logging every single function call will overwhelm your storage and compute costs. Log events (the start/end of a process, errors, state transitions) rather than implementation details.
• Blocking I/O: If your logger performs a network request to ship the log synchronously, you will introduce latency into your critical path. Always log to stdout/stderr and let a background agent handle the shipping.

FAQ

Q: Should I use a logging library or just json.dumps? A: Use a library (like structlog for Python or Winston for Node.js). They handle edge cases like circular references in objects and automatic timestamp generation.

Q: Does centralized logging add latency? A: If done correctly with a sidecar process, the performance impact is negligible.

Q: How do I handle logs when the network is down? A: Most robust log agents have local disk buffering. If the central server is unreachable, they store logs on the local disk until connectivity is restored.

Recap

We've moved beyond simple debugging by adopting structured logging and centralized aggregation. By treating logs as searchable data, we can now move from reactive "firefighting" to proactive system monitoring. Your design document should now include a section on your observability stack, detailing where logs are stored and which specific events trigger alerts.

Up next: Securing Communication with HTTPS/TLS