Monitoring Cluster Events: A Guide to Kubernetes Visibility

Previously in this course, we explored defining Ingress rules to manage external traffic. While routing is critical for connectivity, knowing why a resource isn't behaving as expected is equally vital. This lesson adds a layer of observability to your workflow by teaching you how to use events to track cluster state changes and debug issues.

Understanding Kubernetes Events

In Kubernetes, an "Event" is a record of a state change or an important occurrence within the cluster. Unlike application logs—which are generated by your code—events are generated by the Kubernetes control plane. They tell the "story" of your resources: when a Pod was scheduled, why a container crashed, or when a node encountered a disk pressure issue.

Think of events as the "audit trail" of your cluster's heartbeat. When you ask, "Why isn't my application running?", the answer is almost always hiding in the event stream.

Retrieving and Filtering Events

To view events, we use the kubectl get events command. By default, this lists events from the current namespace.

Bash
# List events in the current namespace
kubectl get events

The output gives you several key pieces of information:

• LAST SEEN: How long ago the event occurred.
• TYPE: Usually "Normal" (for routine operations) or "Warning" (for issues requiring attention).
• REASON: A short, machine-readable string explaining the event (e.g., FailedScheduling, Pulling, Started).
• OBJECT: The resource involved.
• MESSAGE: A human-readable description of what happened.

Filtering for Clarity

In a busy cluster, kubectl get events can be overwhelming. You can filter events to find what matters:

1. Sort by time:

   Bash
   kubectl get events --sort-by='.lastTimestamp'
   

2. Filter by namespace:

   Bash
   kubectl get events -n my-app
   

3. Focus on Warnings: Events are often buried in noise. If you are troubleshooting Pod crashes, look specifically for warnings:

   Bash
   kubectl get events --field-selector type=Warning
   

Interpreting Common Warning Events

As you gain experience, you'll recognize the "usual suspects" in the event stream. Understanding these allows you to jump directly to the fix.

Hands-on Exercise: Triggering and Inspecting Events

Let's see this in action by intentionally breaking a deployment.

1. Create a failing pod: Create a file named broken-pod.yaml with an invalid image name:

   YAML
   apiVersion: v1
   kind: Pod
   metadata:
     name: broken-pod
   spec:
     containers:
     - name: nginx
       image: nginx:invalid-tag-12345
   

2. Apply the manifest: kubectl apply -f broken-pod.yaml.
3. Observe the events: Run kubectl get events --sort-by='.lastTimestamp'.
4. Identify the failure: You will see a Warning event with the reason Failed. This is the cluster telling you it cannot pull the image.
5. Clean up: Delete the pod with kubectl delete pod broken-pod.

Common Pitfalls

• Event TTL: Kubernetes events are ephemeral. They are stored in etcd and usually purged after one hour. If you need to audit events for long-term compliance, you must export them to a logging stack (like ELK or Datadog).
• Ignoring "Normal" events: Sometimes the most important information is in the "Normal" category—like seeing a Pod being deleted and recreated unexpectedly, which might indicate an unstable controller.
• Namespace Confusion: Forgetting to specify -n is the most common reason users think they have no events. If your resource is in a custom namespace, always include the flag.

FAQ

Q: Are events the same as application logs? A: No. Events come from the Kubernetes API and track system-level activity. Application logs come from the code running inside your container. You should use both for full strategic logging and observability.

Q: Can I see events for a specific resource? A: Yes, use kubectl describe <resource-type> <resource-name>. The bottom section of the output is a dedicated event list for that specific object.

Recap

Monitoring events is your first line of defense in cluster debugging. By using kubectl get events, filtering by Warning types, and checking resource descriptions, you can quickly bridge the gap between "something is wrong" and "I know exactly what to fix."

Up next: We will learn how to use kubectl top to monitor resource consumption, allowing you to identify resource-heavy containers before they trigger further cluster warnings.