Analyzing Container Logs: A Guide to kubectl logs

Previously in this course, we explored Troubleshooting Pod Crashes: Solving CrashLoopBackOff Errors to identify why your containers stop unexpectedly. While events tell you that something went wrong, logs tell you what actually happened inside the code.

In Kubernetes, logs are the primary way to inspect the internal state of your application. Since containers are ephemeral, you cannot simply SSH into them to view a log file; instead, we use the kubectl logs command to pull the stdout and stderr streams directly from the container runtime.

Fetching Logs with kubectl logs

The kubectl logs command is your primary tool for examining container behavior. When you run this command, Kubernetes reaches out to the Kubelet on the node where your Pod is running, retrieves the current log buffer, and displays it in your terminal.

To get the logs for a specific Pod, run:

Bash
kubectl logs <pod-name>

If your Pod has multiple containers (a common pattern for logging sidecars or proxies), you must specify the container name:

Bash
kubectl logs <pod-name> -c <container-name>

Following Live Log Streams

Often, you need to see what your application is doing in real-time as you trigger requests or debug an active process. Similar to the Linux tail -f command, you can use the -f (follow) flag:

Bash
kubectl logs -f <pod-name>

This keeps the connection open and streams new log entries to your console until you terminate the command with Ctrl+C. This is invaluable when you are testing connectivity or Kubernetes Port-Forward: Debugging Pods Locally to verify that your service is processing incoming requests.

Accessing Logs from Previous Instances

One of the most common scenarios in debugging is a Pod that restarts. If a container crashes, the logs from the crashed instance are lost unless you specifically request them. Use the --previous flag to fetch the logs from the container instance that existed just before the current one:

Bash
kubectl logs <pod-name> --previous

This is the "aha!" moment for many engineers. If your application crashes due to a database connection timeout or an unhandled exception, the exit logs are usually trapped in that previous instance.

Hands-on Exercise: Inspecting Application Output

Let’s put this into practice with a simple demo Pod.

1. Deploy a logger pod: Create a file named logger-pod.yaml:

   YAML
   apiVersion: v1
   kind: Pod
   metadata:
     name: log-demo
   spec:
     containers:
     - name: busybox
       image: busybox
       command: ["/bin/sh", "-c", "while true; do echo 'Logging heartbeat...'; sleep 5; done"]
   

   Apply it: kubectl apply -f logger-pod.yaml

2. View the logs: Run kubectl logs log-demo. You will see the output of the echo command.

3. Follow the logs: Run kubectl logs -f log-demo. Watch as new lines appear every five seconds.

4. Clean up: Delete the pod: kubectl delete pod log-demo.

Common Pitfalls

• Log Rotation: Kubernetes does not provide infinite log storage. If your application logs an excessive amount of data, the logs will be rotated, and older entries will be purged. Use a centralized logging stack (like ELK or Loki) for long-term storage.
• Missing Output: If your application writes to a custom file (e.g., /var/log/app.log) instead of stdout or stderr, kubectl logs will return nothing. Always configure your application to write to standard output.
• Permissions: If you are restricted by RBAC (Role-Based Access Control), you might not have permission to view logs. Ensure your ServiceAccount has the get and list permissions for the pods/log resource.

FAQ

Q: Can I filter logs by time? A: Yes, use the --since flag (e.g., --since=1h for the last hour) or --since-time for specific timestamps.

Q: Why do my logs show stderr and stdout mixed together? A: Kubernetes interleaves both streams into the same output buffer. If you need to separate them, it is best to handle this at the application level or through a logging agent.

Q: Does kubectl logs store logs on my local machine? A: No, it only displays the stream. If you need to keep them, redirect the output to a file: kubectl logs <pod-name> > my-app.log.

Recap

Logging is the heartbeat of debugging. We’ve covered how to use kubectl logs to inspect current output, -f to monitor live behavior, and --previous to diagnose crashes. These commands provide immediate visibility into your application's health, allowing you to move from guessing to knowing exactly why a container is behaving in a certain way.

Up next: Debugging ImagePullErrors and resolving container startup failures.