Optimisation & nodes

Draining & deleting a node

Sometimes you need to retire a node - a failing instance, an autoscaler quirk that left a node stuck, a manual decommission. Other times you just want to move workloads off a node temporarily - to rebalance a hotspot, do maintenance, or test scheduler decisions - without actually removing the node from the cluster. Kubius’s drain workflow handles both, with live per-pod feedback so you can see exactly what’s happening at each step.

This page covers the three node-retirement actions on the Actions menu of a node detail view:

Drain Node (Keep)… - cordon and evict, then stop. Node stays in the cluster, unschedulable. Reversible by uncordoning.
Drain & Delete Node… - cordon, evict, and then delete the Node from the API server.
Force Delete (skip drain) - straight delete with no graceful eviction. Escape hatch for stuck nodes.

Which one do I want?

You want to…	Use
Move workloads off a hotspot, then put the node back in service	Drain Node (Keep)… then Uncordon
Temporarily drain for OS patching / kernel upgrade / kubelet restart	Drain Node (Keep)…, leave cordoned until done, then Uncordon
Retire a healthy node, evict workloads gracefully	Drain & Delete Node…
Move workloads off a node before terminating its underlying VM	Drain & Delete Node… (then terminate the VM in your cloud console)
Force a stuck `NotReady` node out of the API server	Force Delete (skip drain)
Remove a Node entry the cloud provider already cleaned up	Force Delete (skip drain)

The drain stage is identical for both Keep and Delete modes - same per-pod eviction logic, same PDB respect, same terminationGracePeriodSeconds wait. Only the final step differs (uncordon-or-leave vs delete). Force delete bypasses drain entirely.

What you’ll see

Open a node → Actions ▸ Drain & Delete Node…. A sheet appears with four stages.

1. Plan

Up-front, Kubius classifies every pod on the node:

Evictable - workload pods that will be evicted through the Eviction API
DaemonSet (skipped) - the DaemonSet controller would just recreate the pod on this same node (which we’re about to cordon), so eviction is pointless
Mirror (skipped) - static pods managed directly by the kubelet (control-plane components on self-managed clusters). They aren’t evictable via the API
Already terminating (skipped) - pods that already have a deletionTimestamp
emptyDir (skipped by default) - pods using emptyDir volumes. emptyDir is local scratch space, so evicting destroys whatever’s in it. Default off matches kubectl drain without --delete-emptydir-data. Toggle Also evict pods using emptyDir volumes to include them

Coloured tags at the top of the card summarise the breakdown so you know what you’re committing to before you click anything.

2. Draining

Click Cordon & Drain and Kubius will:

Cordon the node (PATCH /api/v1/nodes/{name} with spec.unschedulable: true) so the scheduler stops placing new pods here
Evict every eligible pod in parallel via the policy/v1 Eviction subresource (POST /api/v1/namespaces/{ns}/pods/{name}/eviction) - this is the same path kubectl drain uses

Each pod’s row updates live:

Icon	State	What it means
ⓘ Pending	Hasn’t started yet	Will be picked up shortly
⟳ Evicting…	Eviction in progress	Covers both the API call and the kubelet termination window (SIGTERM + `terminationGracePeriodSeconds`)
⏳ PDB blocked · retry N	Eviction returned HTTP 429	A PodDisruptionBudget would be violated. Kubius backs off (exponential, capped at 15s) and retries - gives up after 120s per pod with the K8s status message extracted from the response
✓ Evicted	Pod is truly gone	The watch has confirmed the pod is no longer on this node - not just that the eviction API ACK’d
⚠︎ Failed: {message}	Terminal error	Whatever the API server said
⊖ Skipped	Won’t be evicted	DaemonSet / mirror / emptyDir / already-terminating - see the Plan stage above

A progress bar at the top shows N / total so you can see overall progress without scanning the list.

Why “Evicting…” stays on screen so long. The Eviction API returns 200 OK as soon as the eviction is accepted, not when the pod is actually gone. Behind the scenes the kubelet still has to SIGTERM the container and wait terminationGracePeriodSeconds before removing the pod. Kubius keeps watching the live cluster state and only flips the row to Evicted when the pod actually disappears - so the Delete Node button can’t enable prematurely.

3. Ready

Once every evictable pod is in a terminal state (Evicted, Failed, or Skipped), the footer flips to the terminal action - different depending on which menu item you started from.

Drain & Delete mode: a single Delete Node button. This is enforced - there’s no way to delete the node while a pod is still mid-termination.

Drain Node (Keep) mode: two buttons -

Uncordon - calls PATCH /api/v1/nodes/{name} setting spec.unschedulable: false. The scheduler can place new pods again. The sheet header flips to “Done - node is back in service” and the left button becomes Done.
Leave Cordoned - closes the sheet, node stays unschedulable. Use this if you intend to do further work on the node (a kernel upgrade, manual debugging, anything that’s not safe with new pods landing). To put it back in service later, open the sheet again and Uncordon, or use kubectl uncordon <node>.

4. Deleting / Uncordoning

Click whichever applies. Kubius issues the API call and dismisses the sheet (plus the parent detail panel, on a successful Delete). If the call fails, the error stays on the sheet so you can read it.

Cancelling mid-drain

The Cancel button stops issuing new evictions and dismisses the sheet. In-flight evictions complete on the cluster side - once an Eviction API call has been accepted, kubelet will see it through. Kubius just stops driving the workflow.

The node stays cordoned. This is intentional - leaving a half-drained node schedulable would risk new pods landing on a host you intended to retire. If you change your mind and want the node back in service, uncordon it from kubectl uncordon <node> or use a future Kubius uncordon action.

Managed clusters - important caveat

On GKE and EKS, deleting the Node object only removes the API server’s view of the node. The cloud provider’s node controller will usually recreate the Node entry within seconds unless the underlying VM is also terminated via the provider (GCP / AWS console, gcloud compute instances delete, aws ec2 terminate-instances, autoscaler scale-down, etc).

So on managed clusters, the typical workflow is:

Use Drain & Delete Node… to evict workloads cleanly
Then terminate the underlying VM in your cloud console (or let the autoscaler reap it)

If you just want to force a stuck Node entry out of the API server, Force Delete (skip drain) is the right action - but the cloud controller may recreate it.

Force Delete (skip drain)

The second item on the Actions menu, Force Delete (skip drain), performs a straight DELETE /api/v1/nodes/{name} with gracePeriodSeconds=0. No cordon, no drain, no eviction. Behind a single confirmation.

Use this when:

The node is NotReady and you’ve already established that workloads can’t be recovered from it
The cloud provider has already cleaned up the VM and the Node entry is just a phantom
You’re certain pods will be rescheduled elsewhere by their controllers (Deployments, StatefulSets, etc. handle this fine; standalone pods will be lost)

Pods on a force-deleted node end up “orphaned” briefly until their controllers reschedule them - controllers handle it, but it’s not graceful in the sense that the pods don’t get a SIGTERM.

Permissions

To run the drain workflow you need:

patch nodes (for cordon)
create pods/eviction (for the evictions themselves)
delete nodes (for the final delete)

On most clusters, the cluster-admin role grants all of these. On more locked-down clusters you may need a custom role. RBAC denials are surfaced in the sheet’s per-pod or footer error area as Forbidden (403).

Just want to move one pod, not the whole node?

If a single pod is on the wrong host (noisy neighbour, scheduler placed it sub-optimally, you want to rebalance), use Evict Pod on that pod’s Actions menu instead - same Eviction API, single call, controller-managed pods get rescheduled by the scheduler wherever it picks. See Pod actions for details, including the DaemonSet caveat (kubelet recreates DaemonSet pods on the same node, so eviction doesn’t move them).

Beyond manual drains: continuous rebalancing

Kubius’s drain and per-pod eviction are one-off tools - you decide when. If you want the cluster to keep itself balanced on an ongoing basis, the relevant Kubernetes projects are:

Descheduler (sigs/k8s) - a CronJob/Deployment you install. Periodically evicts pods that violate configurable policies: LowNodeUtilization (consolidate from under-used nodes), HighNodeUtilization (spread from hot ones), RemoveDuplicates (too many of one controller per node), RemovePodsViolatingTopologySpreadConstraint, RemovePodsHavingTooManyRestarts, and more. Respects PDBs. Only evicts - scheduler still picks where things go next.
Karpenter (AWS, increasingly elsewhere) - autoscaler with Consolidation that actively reschedules workloads onto fewer/cheaper nodes by evicting and replacing. Goes further than Cluster Autoscaler, which only adds/removes nodes based on unschedulable pods.
Declarative scheduler hints - topologySpreadConstraints, podAntiAffinity, nodeAffinity, priorityClass + preemption. These shape future placements; combine with a descheduler to clean up existing imbalances.