Showing posts with label Kubernetes. Show all posts
Showing posts with label Kubernetes. Show all posts

Kubernetes Troubleshooting 2026: Fixing CrashLoopBackOff and OOMKilled Errors



The 2026 Kubernetes Survival Guide: Debugging "Silent" Pod Failures in an AI-Driven World

In 2026, your pods aren't just crashing because of bad code; they are often crashing because an AI-orchestrator misconfigured a resource limit or an Agent pushed a breaking schema change.

If you see a pod stuck in CrashLoopBackOff or OOMKilled, don't panic. Here is the professional 2026 workflow to fix it.

1. The "Describe" Command (Your First Line of Defense)

Before you check logs, check the Events. Most "silent" failures (like mounting a missing secret) won't even show up in application logs.

kubectl describe pod <pod-name>

What to look for: Scroll to the bottom under Events. Look for "FailedMount," "FailedScheduling," or "Back-off restarting failed container."

2. Hunting the "Exit Code 137" (The Memory Killer)

If your pod was running and suddenly vanished, check the Last State.

If you see Exit Code 137, it means the Linux OOM (Out of Memory) killer stepped in.

  • The 2026 Fix: Check if your AI Agent set the limits too low. In 2026, we recommend using a Vertical Pod Autoscaler (VPA) to let Kubernetes "right-size" the memory for you automatically.

3. The "Previous" Log Trick

When a pod is in a crash loop, kubectl logs often shows nothing because the container is currently dead. You need to see why the last one died.

kubectl logs <pod-name> --previous

This is the single most forgotten command by junior DevOps engineers, but it's the only way to see the stack trace of a crashed container.

4. Debugging "ImagePullBackOff" in 2026

Since we are using more private registries and AI-generated images, this error is common.

  • The Quick Fix: Run kubectl get events -n <namespace> --sort-by='.lastTimestamp'.

  • Often, the issue is a typo in the imagePullSecrets or the AI agent tried to pull a v2.0 tag that hasn't finished its CI/CD build yet.

Kubernetes Zero Trust: How to Secure Your Cluster with Network Policies



Stop the Lateral Movement: Zero Trust Security in Kubernetes 

By default, Kubernetes is an "open house"—any Pod can talk to any other Pod, even across different namespaces. If a hacker compromises your frontend web server, they can move laterally to your database and steal your data.

In this guide, we’ll implement a Default Deny strategy, ensuring that only authorized traffic can move through your cluster.

1. The Concept: "Default Deny"

Think of your cluster like a hotel. In a default setup, every guest has a master key to every room. In a Zero Trust setup, every door is locked by default, and you only get a key to the specific room you need.

2. Step 1: Lock Everything Down

We start by creating a policy that drops all ingress (incoming) and egress (outgoing) traffic for a specific namespace. This is your "Base Security."

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {} # Selects all pods in the namespace
policyTypes:
- Ingress
- Egress

3. Step 2: Open "Micro-Segments"

Now that everything is locked, we selectively open "holes" in the firewall. For example, let's allow the API Gateway to talk to the Order Service.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-gateway-to-orders
namespace: production
spec:
podSelector:
matchLabels:
app: order-service
ingress:
- from:
- podSelector:
matchLabels:
app: api-gateway

How to Fix Kubernetes CrashLoopBackOff: A Practical Guide



It’s the most famous (and frustrating) status in the Kubernetes world. You run kubectl get pods, and there it is: 0/1 CrashLoopBackOff.

Despite the scary name, CrashLoopBackOff isn’t actually the error—it’s Kubernetes telling you: "I tried to start your app, it died, I waited, and I’m about to try again."

Here is the "Triple Post" finale to get your cluster healthy before the weekend.


1. The "First 3" Commands

Before you start guessing, run these three commands in order. They tell you 90% of what you need to know.

CommandWhy run it?
kubectl describe pod <name>Look at the Events section at the bottom. It often says why it failed (e.g., OOMKilled).
kubectl logs <name> --previousCrucial. This shows the logs from the failed instance before it restarted.
kubectl get events --sort-by=.metadata.creationTimestampShows a timeline of cluster-wide issues (like Node pressure).

2. The Usual Suspects

If the logs are empty (a common headache!), the issue is likely happening before the app even starts.

  • OOMKilled: Your container exceeded its memory limit.

    • Fix: Increase resources.limits.memory.

  • Config Errors: You referenced a Secret or ConfigMap that doesn't exist, or has a typo.

    • Fix: Check the describe pod output for "MountVolume.SetUp failed".

  • Permissions: Your app is trying to write to a directory it doesn't own (standard in hardened images).

    • Fix: Check your securityContext or Dockerfile USER permissions.

  • Liveness Probe Failure: Your app is actually running fine, but the probe is checking the wrong port.

    • Fix: Double-check livenessProbe.httpGet.port.


3. The Pro-Tip: The "Sleeper" Debug

If you still can't find the bug because the container crashes too fast to inspect, override the entrypoint.

Update your deployment YAML to just run a sleep loop:

command: ["/bin/sh", "-c", "while true; do sleep 30; done;"]

Now the pod will stay "Running," and you can kubectl exec -it <pod> -- /bin/sh to poke around the environment manually!



Navigating Data Flow in Kubernetes: Unraveling Ingress and Egress Concepts


What is Ingress and Egress? An Introduction:

In the ever-evolving landscape of information technology and container orchestration, terms like "ingress" and "egress" are integral to understanding how data traverses within Kubernetes clusters. As organizations increasingly adopt containerized applications, the proper management of ingress and egress points becomes crucial for ensuring secure and efficient communication between microservices. In this article, we will explore the significance of ingress and egress within the context of Kubernetes, shedding light on their roles in facilitating seamless data flow.

Defining Ingress and Egress in general:

  1. Ingress: 

    Ingress refers to the entry point of data into a network or system. It is the pathway through which external data or traffic enters a local network. This can include data from the internet, other networks, or external devices. Ingress points are typically managed and monitored to control the type and volume of incoming data, ensuring network security and optimal performance.

  2. Egress: Conversely, egress is the exit point for data leaving a network. It represents the pathway through which data flows out of a system to external destinations. Egress points are strategically managed to regulate the outbound traffic, preventing unauthorized access and safeguarding sensitive information from leaving the network without proper authorization.


Defining Ingress and Egress in Kubernetes:


  1. Ingress in Kubernetes:

    In the Kubernetes ecosystem, ingress refers to the entry point for external traffic into the cluster. It serves as a way to manage external access to services within the cluster, acting as a traffic controller. Ingress resources allow users to define routing rules, hostnames, and TLS settings, directing incoming requests to the appropriate services.

  2. Egress in Kubernetes: Egress, on the other hand, involves the outbound traffic from pods within the cluster to external services or destinations. Managing egress in Kubernetes is crucial for controlling which external resources a pod can access and ensuring that communication adheres to security and compliance standards.

Importance of Ingress and Egress in Kubernetes:

  1. Service Discovery: Ingress resources enable service discovery by providing a standardized way to route external traffic to services within the cluster. This simplifies the process of exposing and accessing services, enhancing the overall scalability and flexibility of Kubernetes applications.
  2. Security Policies: Ingress controllers, such as Nginx Ingress or Traefik, allow for the implementation of security policies at the entry point of the cluster. This includes SSL/TLS termination, rate limiting, and web application firewall capabilities, bolstering the security posture of the entire Kubernetes deployment.
  3. Egress Control: Kubernetes Network Policies can be leveraged to enforce egress controls, specifying which pods are allowed to communicate with external resources and under what conditions. This ensures that only authorized communication occurs, mitigating potential security risks.

Practical Applications in Kubernetes:

  1. Ingress Controllers: Deploying and configuring Ingress controllers play a pivotal role in managing external access to services. These controllers are responsible for processing and implementing the rules defined in Ingress resources, directing traffic to the appropriate services within the cluster.
  2. Egress Policies: Utilizing Kubernetes Network Policies allows organizations to define fine-grained controls over egress traffic. This is particularly important in scenarios where strict compliance requirements or data sovereignty regulations need to be adhered to.
  3. API Gateway Integration:                                                                                                         Ingress points can be integrated with API gateways to manage external access to microservices, enabling features like authentication, rate limiting, and request transformation. This ensures a secure and streamlined interaction between external clients and services within the Kubernetes cluster.                                                                                                     

Conclusion:


Ingress and egress play pivotal roles in shaping the data flow within Kubernetes clusters. As organizations embrace container orchestration and microservices architectures, understanding and effectively managing these entry and exit points are essential for building resilient, secure, and scalable applications. By leveraging the capabilities provided by Kubernetes Ingress and implementing robust egress controls, organizations can navigate the complexities of modern application deployment with confidence.

The Power Duo: Kubernetes and Microservices




In the realm of modern software development, two buzzwords have taken center stage: Kubernetes and microservices. These two technologies have revolutionized how applications are built, deployed and managed in a new era of scalability, flexibility, and efficiency.

Microservices, an architectural style, breaks down applications into smaller, loosely-coupled services that can be developed, deployed and scaled independently. This approach promotes agility and accelerates development, allowing teams to focus on specific functionalities. However, managing these services manually can be complex and resource-intensive.


Enter Kubernetes, an open-source container orchestration platform. Kubernetes automates the deployment, scaling and management of containerized applications, making it a perfect match for microservices. It provides tools for automating updates, load balancing and fault tolerance, ensuring seamless operation across a dynamic environment.


Together, Kubernetes and microservices offer several benefits. They enable organizations to swiftly respond to market demands by deploying updates to individual services without disrupting the entire application. Autoscaling ensures optimal resource utilization, and inherent fault tolerance enhances reliability.


In conclusion, the synergy between Kubernetes and microservices has reshaped the software development landscape. Organizations embracing this duo can innovate faster, deliver robust applications, and effectively navigate the complexities of modern IT infrastructure.


How to Expose a Kubernetes Pod to a Specific Port for Running an application



If you are running an application on Kubernetes, you may want to expose a specific port to a pod so that you can access it outside world. Kubernetes provides several ways to do this and we are going to use one of the method.

Step 1: Let's create an HTML application 

I am going to create an EMI calculator using HTML and Javascript and save it as index.html

<!DOCTYPE html>
<html>
<head>
<title>EMI Calculator</title>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<style>
form {
display: flex;
flex-direction: column;
align-items: center;
margin-top: 50px;
}
input[type="number"], select {
padding: 10px;
margin-bottom: 20px;
width: 300px;
border-radius: 5px;
border: none;
box-shadow: 0 0 10px rgba(0, 0, 0, 0.1);
}
input[type="submit"] {
padding: 10px;
width: 200px;
background-color: #4CAF50;
color: white;
border: none;
border-radius: 5px;
cursor: pointer;
}
input[type="submit"]:hover {
background-color: #3e8e41;
}
</style>
</head>
<body>
<h1>EMI Calculator</h1>
<form onsubmit="calculateEMI(); return false;">
<label for="principal">Loan Amount:</label>
<input type="number" id="principal" name="principal"
placeholder="Enter loan amount in INR" required>

<label for="interest">Interest Rate:</label>
<input type="number" id="interest" name="interest"
placeholder="Enter interest rate in %" required>

<label for="tenure">Loan Tenure:</label>
<select id="tenure" name="tenure" required>
<option value="">--Select Loan Tenure--</option>
<option value="12">1 Year</option>
<option value="24">2 Years</option>
<option value="36">3 Years</option>
<option value="48">4 Years</option>
<option value="60">5 Years</option>
</select>

<input type="submit" value="Calculate EMI">
</form>

<div id="result"></div>

<script>
function calculateEMI() {
// Get input values
let principal = document.getElementById('principal').value;
let interest = document.getElementById('interest').value;
let tenure = document.getElementById('tenure').value;

// Calculate EMI
let monthlyInterest = interest / 1200; // 12 months * 100%
let monthlyPayment =
(principal * monthlyInterest) / (1 - (1 / Math.pow(1 + monthlyInterest, tenure)
));
let totalPayment = monthlyPayment * tenure;

// Display result
document.getElementById('result').innerHTML = `
<h2>EMI Calculation Result</h2>
<p>Loan Amount: INR ${principal}</p>
<p>Interest Rate: ${interest}%</p>
<p>Loan Tenure: ${tenure} months</p>
<p>Monthly EMI: INR ${monthlyPayment.toFixed(2)}</p>
<p>Total Payment: INR ${totalPayment.toFixed(2)}</p>
`;
}
</script>
</body>
</html>

Now your html application is ready. 

Step 2: Dockerize your application

let's create a Dockerfile with below command inside it and name it Dockerfile in the same location.

FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html

Here we are using nginx server where our index file/EMI calculator will be hosted.

Step 3: Build an image for your application

Use below command to build an image

docker build -t emi .

here -t is called as tag and emi is tag name.

. is current directory. So the docker build command will look for Dockerfile in current directory.

=> [internal] load build definition from Dockerfile
=> => transferring dockerfile: 201B
=> [internal] load .dockerignore
=> => transferring context: 2B
=> [internal] load metadata for docker.io/library/nginx:alpine
=> [internal] load build context
=> => transferring context: 79B
=> [1/2] FROM docker.io/library/nginx:alpine
=> CACHED [2/2] COPY index.html /usr/share/nginx/html/index.html

You would see output like this above.

Now if you check if the image has been created with below command.

docker images

You would see the tag name as emi in your result.

Step 4: Create a deployment

Since we have already created an image, now it's time to create a deployment using the same image.

apiVersion: v1
kind: Pod
metadata:
name: emi
namespace: default
spec:
containers:
- name: emi
image: emi:latest
imagePullPolicy: Never
restartPolicy: Never

save it as deployment.yaml

Now run the below command to create a deployment:

kubectl apply -f deployment.yaml

Once command is completed. Let's verify it by kubectl get pod command like below.

kubectl get pods
NAME READY STATUS RESTARTS AGE
emi 1/1 Running 0 7s

Step 5: Access it via browser

Since we have already created our application want to access it via browser. We may need to use port forwarder. It is for TCP connections only.

kubectl port-forward emi 8087:80

Once command has been completed. Let's access it via localhost:8087 in the browser.


Finally we have created an application, dockerized it and running it on a Pod and able to access via browser. That was it about the spinning up an application on Pod.

Note: If you think this helped you and you want to learn more stuff on devops, then I would recommend joining the Kodecloud devops course and go for the complete certification path by clicking this link

Automate PII Redaction: Building Privacy-First Data Pipelines in Python

  The "Privacy-First" Pipeline: How to Auto-Redact PII with Python As a Data Engineer, you are the gatekeeper. If an email address...