Monitoring System Health: KPIs, Dashboards, and Health Checks

Previously in this course, we explored error handling and logging patterns for production systems. While logs provide the "why" behind an event, monitoring provides the "what" and "how much." In this lesson, we move from passive observation to active system health management.

Monitoring isn't just about watching a screen; it's about building a feedback loop that tells you if your system is doing what it was designed to do.

Defining Key Performance Indicators (KPIs)

Before you can monitor, you must define what "healthy" looks like for your architecture. We use Key Performance Indicators (KPIs)—quantifiable metrics that track the success of your services.

For a standard web-based service, focus on the "Four Golden Signals":

1. Latency: The time it takes to service a request (e.g., p95 response time).
2. Traffic: A measure of how much demand is being placed on your system (e.g., requests per second).
3. Errors: The rate of requests that fail (e.g., 5xx status codes).
4. Saturation: How "full" your service is (e.g., CPU/Memory utilization or queue depth).

If you are monitoring specialized components, remember that specific tools provide deeper insights, such as monitoring Redis performance using the INFO command to catch memory spikes before they crash your cache layer.

Setting Up Health Check Endpoints

A health check is a dedicated API endpoint that returns the status of your service. Load balancers and orchestrators (like Kubernetes) ping these endpoints to determine if a service node is ready to receive traffic.

A robust health check should be "deep," meaning it verifies not just that the process is running, but that its critical dependencies (like the database) are reachable.

Worked Example: Basic Health Check

Here is a simple implementation in Node.js/Express:

JAVASCRIPT
const express = require(CE9178">'express');
const app = express();

// A simple liveness probe: is the process running?
app.get(CE9178">'/health/live', (req, res) => {
  res.status(200).send(CE9178">'OK');
});

// A deep readiness probe: can we talk to the database?
app.get(CE9178">'/health/ready', async (req, res) => {
  try {
    await db.ping(); // Check DB connectivity
    res.status(200).json({ status: CE9178">'UP', database: CE9178">'connected' });
  } catch (err) {
    res.status(503).json({ status: CE9178">'DOWN', database: CE9178">'disconnected' });
  }
});

When the /health/ready check returns a 503, your load balancer should stop sending traffic to that instance immediately.

Creating a Monitoring Dashboard

Once you have metrics flowing, you need a way to visualize them. A monitoring dashboard should be the "single pane of glass" for your system's state.

When building dashboards, apply the Rule of Three:

• The Big Picture: High-level status (System UP/DOWN).
• The Trend: Graphs showing latency and traffic over the last hour.
• The Detail: Resource saturation (CPU, RAM).

Whether you use Cloud-native tools like creating CloudWatch Dashboards or third-party solutions like Grafana, keep your most critical KPIs at the top-left of the screen, where the eye naturally lands first.

Hands-On Exercise

1. Add a /health/ready endpoint to your current project.
2. Ensure it performs a check on at least one external dependency (e.g., your database or cache).
3. Create a simple text-based "dashboard" (or a mock-up image) that lists your chosen KPIs (Latency, Traffic, Errors, Saturation) and write down the threshold at which each would trigger an alert.

Common Pitfalls

• The "Flapping" Health Check: If your deep health check is too sensitive, a temporary network blip might cause your system to remove all healthy nodes from the rotation. Ensure your checks have a timeout and perhaps a "grace period."
• Metric Overload: Monitoring too many things leads to alert fatigue. Focus on the Four Golden Signals first; add granular metrics only when you need to debug a specific issue.
• Ignoring Saturation: Many engineers monitor Errors and Latency but forget to monitor CPU or Memory until the system is already crashing. Always track how much "headroom" your resources have left.

FAQ

• How often should I poll my health checks? Every 5-10 seconds is standard for most web services to balance responsiveness with overhead.
• What is the difference between "Live" and "Ready"? "Live" means the process is alive (process monitor). "Ready" means the service is fully initialized and ready to serve traffic (dependency check).
• Should dashboards be public? Never. Dashboards leak operational details that can aid an attacker in mapping your system architecture.

Recap

Monitoring is the eyes and ears of your architecture. By implementing deep health checks, you protect your users from failing nodes. By defining clear KPIs and visualizing them on a dashboard, you transform raw data into a reliable map of your system's health.

Up next: Distributed Tracing Basics