Diagnosing Node OOM

How to find why a node ran out of memory and which workload is responsible. Uses standard tooling (kubectl, kernel logs, metrics) - nothing Kubius-specific.

The node Diagnostics tab surfaces most of these signals automatically - a verdict, OOM evidence (SystemOOM / pressure / evictions / NotReady flaps), the memory budget, a per-container Top Memory Consumers list (with a red N OOM badge from the kubelet’s cAdvisor container_oom_events_total, which names kernel-OOM victims that pod status alone misses), and a merged OOMKilled containers history. That picture is snapshotted to disk per node, so it survives the node dying - when a node goes NotReady and its live metrics vanish, the tab falls back to the last snapshot with an “as of …” banner. The node Logs tab (when the node serves logs) is also available. This guide is the full manual procedure, including how to read the authoritative kernel kill record that the API can’t expose.

Applies to AWS (EKS), Google Cloud (GKE), and self-managed Kubernetes. The kubectl steps are identical everywhere - only host access (Step 2) and a couple of OS quirks differ by provider, called out where they matter.

Container OOMKill vs Node OOM

Two different events get called “OOM”, with different causes and fixes:

	Container OOMKill	Node / system OOM
Trigger	A container exceeded its own memory limit (cgroup)	The node’s physical RAM was exhausted
Killer	cgroup memory controller	The kernel OOM killer (global)
Who dies	Just that container, restarted in place	Any process the kernel picks, often innocent neighbours
Evidence	lastState.terminated.reason: OOMKilled, rising restarts	SystemOOM event, dmesg “Killed process”, MemoryPressure=True, evictions, flapping NotReady

The most common cause of node OOM is a workload with no memory limit (unbounded), or a node whose pods’ limits are overcommitted versus its allocatable memory, so total usage climbs past capacity.

Step 1 - Confirm it’s node-level OOM

# Conditions - look for MemoryPressure=True and Ready flapping
kubectl describe node {{node}} | sed -n '/Conditions:/,/Addresses:/p'

# Events on the node (SystemOOM, evictions, NodeNotReady)
kubectl get events -A --field-selector involvedObject.name={{node}} --sort-by=.lastTimestamp

# Pods the kubelet evicted under pressure
kubectl get pods -A --field-selector spec.nodeName={{node}} -o wide | grep -i evicted

SystemOOM events or MemoryPressure=True confirm node-level OOM. Many Evicted pods mean the kubelet hit its eviction thresholds - pressure building, often just before a hard kernel OOM.

Step 2 - Read the kernel kill record (the smoking gun)

This is the authoritative record of what the kernel killed. You need a shell on the node host. kubectl debug node works without an SSH key - it runs a throwaway privileged pod on the node:

# 1. Point kubectl at the right cluster
kubectl config use-context {{context}}

# 2. Open a root shell on the node
kubectl debug node/{{node}} -it --image=busybox --profile=general

# 3. Inside that shell, switch to the host filesystem
chroot /host

# 4. Read the OOM kill record
dmesg -T | grep -i 'killed process'

busybox’s own dmesg is stripped down - chroot /host first so you get the host’s full dmesg. If dmesg -T errors, drop the -T.

kubectl debug node is provider-agnostic and works on EKS, GKE, and self-managed clusters - except GKE Autopilot, where node access is blocked entirely (see below).

If dmesg fails with “Operation not permitted”

Many managed node OSes - Amazon Linux 2023 (EKS) and Container-Optimized OS (GKE) included - set kernel.dmesg_restrict=1, so reading the ring buffer needs the CAP_SYSLOG capability, which --profile=general does not grant. Use one of these instead.

A. Read it from the journal (works under --profile=general). journald captures the kernel log to files, so this avoids the restricted syscall:

chroot /host
journalctl -k --since "-2h" | grep -iE 'out of memory|killed process'

B. Re-run the debug pod privileged so it has CAP_SYSLOG:

kubectl debug node/{{node}} -it --image=busybox --profile=sysadmin
chroot /host
dmesg -T | grep -i 'killed process'

If your kubectl is too old for --profile=sysadmin, drop the restriction in a privileged shell first: sysctl -w kernel.dmesg_restrict=0, then dmesg.

Host access by provider

If kubectl debug node is blocked (Pod Security policies) or the node is unreachable, get onto the host directly. You’re root there, so dmesg/ journalctl -k work without the capability dance.

AWS · EKS - SSM Session Manager (no SSH key needed):

# instance id = the i-… at the end of: kubectl get node {{node}} -o jsonpath='{.spec.providerID}'
aws ssm start-session --target <instance-id> --region <region>
sudo journalctl -k --since "-2h" | grep -iE 'out of memory|killed process'

Google Cloud · GKE - SSH to the node VM (the VM name and zone come from the providerID gce://<project>/<zone>/<instance>):

gcloud compute ssh <instance> --zone <zone>
sudo journalctl -k --since "-2h" | grep -iE 'out of memory|killed process'

GKE Autopilot: there is no node access - you can’t open a host shell or run dmesg. Rely on the API signals instead: Step 1 (events / conditions), Step 3 (budget) and Step 4 (consumers), plus SystemOOM events on the node.

Self-managed / other - plain SSH:

ssh <user>@<node-ip>
sudo journalctl -k --since "-2h" | grep -iE 'out of memory|killed process'   # or: sudo dmesg -T | grep …

A kill line looks like:

Out of memory: Killed process 12345 (java) total-vm:9871234kB, anon-rss:8123456kB ...

It names the process killed and its RSS (anon-rss). The lines just above it rank every process by memory; container processes are grouped under their pod’s cgroup (/kubepods/.../pod<uid>/...), which maps the offender back to a pod.

Cleanup - kubectl debug node does not auto-delete the debugger pod:

kubectl delete pod <node-debugger-pod-name>

If the node is flapping NotReady, the debug pod may not start (its kubelet is unhealthy). Fall back to your provider’s host access (above), which reaches the host directly without going through the kubelet.

Also useful - the kubelet’s own view:

journalctl -u kubelet --since "-1h" | grep -iE 'oom|evict|memory'

Step 3 - Assess the node’s memory budget

kubectl top node {{node}}                                   # live usage (needs metrics-server)
kubectl describe node {{node}} | sed -n '/Allocated resources:/,/Events:/p'

kubectl describe node prints Requests and Limits as a % of allocatable. If Limits > 100% the node is oversubscribed - if pods approach their limits together it will OOM. Low/zero Requests with high real usage means the scheduler overpacked the node.

Step 4 - Rank the memory consumers

# Live per-pod memory, highest first
kubectl top pod -A --field-selector spec.nodeName={{node}} --sort-by=memory

# Requests/limits per pod (find the unbounded ones)
kubectl get pods -A --field-selector spec.nodeName={{node}} \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,\
MEM_REQ:.spec.containers[*].resources.requests.memory,\
MEM_LIM:.spec.containers[*].resources.limits.memory'

• High in top with no MEM_LIM → unbounded, can grow until the node dies. Prime suspect.
• Usage near/over its limit → will be container-OOMKilled; adds churn.
• Many pods each over their request → cumulative overcommit.

Step 5 - Check container OOMKills & restarts

kubectl get pods -A --field-selector spec.nodeName={{node}} -o wide   # high/climbing restarts

kubectl get pod <pod> -n <ns> \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{": "}{.lastState.terminated.reason}{"\n"}{end}'

Step 6 - Leak vs under-provisioning

Look at the usage trend (metrics history, Prometheus container_memory_working_set_bytes, or repeated kubectl top):

• Climbs without plateau → memory leak; raising the limit only delays the kill. Fix the app.
• Stable but above request/limit → under-provisioned; raise to fit observed peak.

Step 7 - Root cause and fix

Finding	Cause	Fix
Top consumer has no memory limit	Unbounded workload	Add resources.limits.memory from observed peak
Usage pinned at limit, repeated OOMKills	Limit too low or leak	Raise limit / fix leak
describe node Limits > 100%	Node oversubscribed	Lower limits, add nodes, spread pods
Low/absent requests, node overpacked	Bad scheduling inputs	Set realistic memory requests
Node too small for its load	Capacity	Larger instance / more nodes

A container with no memory limit has no ceiling and can consume all remaining RAM, forcing the kernel to kill other pods. Setting requests (scheduling + QoS) and a limit (the ceiling) caps the offender and makes it - not its neighbours - the first eviction candidate.

See Right-Sizing - it samples real usage and recommends requests/limits, anchoring the memory request to steady-state (p95) and the limit above peak so you cap a workload without re-introducing OOMKills.

Step 8 - Verify the fix

kubectl top node {{node}}                                          # headroom restored
kubectl describe node {{node}} | grep -A6 'Allocated resources'    # Limits back under 100%
kubectl get events -A --field-selector involvedObject.name={{node}} --sort-by=.lastTimestamp

No new SystemOOM/eviction events, MemoryPressure=False, and the top consumers sitting comfortably below their limits.

TL;DR

1. Confirm node OOM: SystemOOM event / MemoryPressure=True / dmesg OOM line.
2. Read dmesg on the node - it names the killed process and its RSS.
3. kubectl describe node - Limits > 100% of allocatable means oversubscribed.
4. kubectl top pod --field-selector spec.nodeName={{node}} --sort-by=memory - rank consumers.
5. The ones with no memory limit or usage near limit are your rogue workloads.
6. Fix: set/raise the offender’s memory limit and a realistic request; fix leaks; rebalance or scale.