SundayHK
在Kubernetes中，Pod的Evicted状态表示Pod已经被驱逐，并不再运行在节点上。Pod驱逐主要是由于资源约束，如内存不足或磁盘空间不足。
原理
Kubernetes集群会监控节点资源使用情况，当资源不足时，kubelet会触发Pod驱逐过程。驱逐过程会根据优先级和资源使用情况选择要驱逐的Pod。一旦Pod被驱逐，其状态将变为Evicted，相关事件也会被记录。
原因
内存不足：当节点上的可用内存不足以满足Pod的内存需求时，kubelet会尝试回收内存，如果回收不足，会触发Pod驱逐。
磁盘空间不足：当节点上的磁盘空间不足以满足Pod的存储需求时，kubelet会尝试回收磁盘空间，如果回收不足，会触发Pod驱逐。
[root@harbor yaml]# kubectl get pod
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 3353 (30m ago) 139d
busybox-pod 0/1 Error 15 (6h53m ago) 21h
nfs-pod-test 1/1 Running 0 12d
nginx-774f96d4d9-2f7n2 0/1 Evicted 0 6h46m
nginx-774f96d4d9-2lbtr 0/1 Evicted 0 7h18m
nginx-774f96d4d9-4lpfb 0/1 Evicted 0 28m
nginx-774f96d4d9-4srb2 0/1 Completed 0 7h18m
nginx-774f96d4d9-5r7rb 0/1 Evicted 0 28m
nginx-774f96d4d9-6dbvp 0/1 Evicted 0 6h46m
nginx-774f96d4d9-746dd 0/1 ContainerStatusUnknown 1 8h
nginx-774f96d4d9-bs4pr 0/1 Evicted 0 7h18m
nginx-774f96d4d9-bw6tb 0/1 Evicted 0 6h46m
nginx-774f96d4d9-cbl5v 0/1 Evicted 0 28m
nginx-774f96d4d9-cwq48 0/1 Evicted 0 28m
nginx-774f96d4d9-fd9mg 0/1 Evicted 0 7h18m
nginx-774f96d4d9-fknwl 0/1 Evicted 0 5h4m
nginx-774f96d4d9-h7tjq 0/1 Evicted 0 7h18m
nginx-774f96d4d9-hftqh 0/1 Evicted 0 28m
nginx-774f96d4d9-hfxp6 0/1 Evicted 0 5h4m
nginx-774f96d4d9-hxmrq 0/1 Evicted 0 7h18m
nginx-774f96d4d9-j2c46 0/1 Evicted 0 7h18m
nginx-774f96d4d9-j6glf 0/1 Evicted 0 28m
nginx-774f96d4d9-jlzws 0/1 Evicted 0 5h4m
nginx-774f96d4d9-k9b4w 0/1 Evicted 0 28m
nginx-774f96d4d9-kmb2r 0/1 Completed 0 6h46m
nginx-774f96d4d9-l4lhv 0/1 Evicted 0 7h18m
nginx-774f96d4d9-lfskb 0/1 Evicted 0 28m
nginx-774f96d4d9-lgjlr 0/1 Evicted 0 7h18m
nginx-774f96d4d9-lkv2d 0/1 ContainerStatusUnknown 1 5h4m
nginx-774f96d4d9-lw78n 0/1 Evicted 0 7h18m
nginx-774f96d4d9-lww22 0/1 Completed 0 161m
nginx-774f96d4d9-mndr5 0/1 Evicted 0 6h46m
nginx-774f96d4d9-n9fth 0/1 Evicted 0 7h18m
nginx-774f96d4d9-n9w6b 0/1 Evicted 0 5h4m
nginx-774f96d4d9-ncz8c 0/1 Evicted 0 6h46m
nginx-774f96d4d9-nsxvp 0/1 Evicted 0 28m
nginx-774f96d4d9-pprn7 0/1 Evicted 0 7h18m
nginx-774f96d4d9-pqjvs 0/1 Completed 0 139d
nginx-774f96d4d9-q88pf 0/1 Evicted 0 5h4m
nginx-774f96d4d9-qtxrg 0/1 Evicted 0 28m
nginx-774f96d4d9-qwc8v 0/1 Evicted 0 5h4m
nginx-774f96d4d9-qxfr4 0/1 Evicted 0 5h4m
nginx-774f96d4d9-r8wkc 0/1 Evicted 0 28m
nginx-774f96d4d9-rzh9q 0/1 Evicted 0 5h4m
nginx-774f96d4d9-shb72 0/1 Evicted 0 6h46m
nginx-774f96d4d9-snhm7 0/1 Evicted 0 7h18m
nginx-774f96d4d9-t8npl 0/1 Evicted 0 28m
nginx-774f96d4d9-twqwx 0/1 Evicted 0 5h4m
nginx-774f96d4d9-vklmr 0/1 Evicted 0 28m
nginx-774f96d4d9-w5k2z 0/1 Evicted 0 5h4m
nginx-774f96d4d9-w6rqm 1/1 Running 0 28m
nginx-774f96d4d9-w74x7 0/1 Evicted 0 28m
nginx-774f96d4d9-x75m8 0/1 Evicted 0 28m
nginx-774f96d4d9-zhcd7 1/1 Running 0 139d
nginx-774f96d4d9-zjsmm 0/1 Evicted 0 28m[root@harbor yaml]# kubectl get event
LAST SEEN TYPE REASON OBJECT MESSAGE
48m Normal Pulled pod/busybox Container image "busybox:1.28" already present on machine
48m Normal Created pod/busybox Created container busybox
48m Normal Started pod/busybox Started container busybox
41m Normal NodeHasSufficientMemory node/k8s-node01 Node k8s-node01 status is now: NodeHasSufficientMemory
46m Warning EvictionThresholdMet node/k8s-node01 Attempting to reclaim memory
46m Normal NodeHasInsufficientMemory node/k8s-node01 Node k8s-node01 status is now: NodeHasInsufficientMemory
35m Warning BackOff pod/mysql-deployment-797b5779bb-87mtl Back-off restarting failed container mysql in pod mysql-deployment-797b5779bb-87mtl_default(d5454db8-30e7-4fa0-99f4-d03585546564)
46m Normal Scheduled pod/nginx-774f96d4d9-4lpfb Successfully assigned default/nginx-774f96d4d9-4lpfb to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-4lpfb The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-5r7rb Successfully assigned default/nginx-774f96d4d9-5r7rb to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-5r7rb The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-cbl5v Successfully assigned default/nginx-774f96d4d9-cbl5v to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-cbl5v The node had condition: [MemoryPressure].
46m Warning Evicted pod/nginx-774f96d4d9-cwq48 The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-cwq48 Successfully assigned default/nginx-774f96d4d9-cwq48 to k8s-node01
46m Normal Scheduled pod/nginx-774f96d4d9-hftqh Successfully assigned default/nginx-774f96d4d9-hftqh to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-hftqh The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-j6glf Successfully assigned default/nginx-774f96d4d9-j6glf to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-j6glf The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-k9b4w Successfully assigned default/nginx-774f96d4d9-k9b4w to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-k9b4w The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-lfskb Successfully assigned default/nginx-774f96d4d9-lfskb to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-lfskb The node had condition: [MemoryPressure].
46m Warning Evicted pod/nginx-774f96d4d9-lww22 The node was low on resource: memory. Threshold quantity: 100Mi, available: 176Ki. Container nginx was using 3632Ki, request is 0, has larger consumption of memory.
46m Normal Killing pod/nginx-774f96d4d9-lww22 Stopping container nginx
46m Normal Scheduled pod/nginx-774f96d4d9-nsxvp Successfully assigned default/nginx-774f96d4d9-nsxvp to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-nsxvp The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-qtxrg Successfully assigned default/nginx-774f96d4d9-qtxrg to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-qtxrg The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-r8wkc Successfully assigned default/nginx-774f96d4d9-r8wkc to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-r8wkc The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-t8npl Successfully assigned default/nginx-774f96d4d9-t8npl to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-t8npl The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-vklmr Successfully assigned default/nginx-774f96d4d9-vklmr to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-vklmr The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-w6rqm Successfully assigned default/nginx-774f96d4d9-w6rqm to k8s-node02
46m Normal Pulled pod/nginx-774f96d4d9-w6rqm Container image "nginx:alpine" already present on machine
46m Normal Created pod/nginx-774f96d4d9-w6rqm Created container nginx
46m Normal Started pod/nginx-774f96d4d9-w6rqm Started container nginx
16m Normal Killing pod/nginx-774f96d4d9-w6rqm Stopping container nginx
46m Normal Scheduled pod/nginx-774f96d4d9-w74x7 Successfully assigned default/nginx-774f96d4d9-w74x7 to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-w74x7 The node had condition: [MemoryPressure].
46m Normal Scheduled pod/nginx-774f96d4d9-x75m8 Successfully assigned default/nginx-774f96d4d9-x75m8 to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-x75m8 The node had condition: [MemoryPressure].
16m Normal Killing pod/nginx-774f96d4d9-zhcd7 Stopping container nginx
46m Normal Scheduled pod/nginx-774f96d4d9-zjsmm Successfully assigned default/nginx-774f96d4d9-zjsmm to k8s-node01
46m Warning Evicted pod/nginx-774f96d4d9-zjsmm The node had condition: [MemoryPressure].
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 (combined from similar events): Created pod: nginx-774f96d4d9-w6rqm
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-k9b4w
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-t8npl
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-cbl5v
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-4lpfb
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-cwq48
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-qtxrg
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-5r7rb
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-j6glf
46m Normal SuccessfulCreate replicaset/nginx-774f96d4d9 Created pod: nginx-774f96d4d9-x75m8[root@harbor yaml]# kubectl describe node k8s-node01
Name: k8s-node01
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=k8s-node01
kubernetes.io/os=linux
node.kubernetes.io/node=
Annotations: management.cattle.io/pod-limits: {}
management.cattle.io/pod-requests: {"cpu":"250m","pods":"1"}
node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 192.168.77.94/24
projectcalico.org/IPv4IPIPTunnelAddr: 10.243.85.192
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 12 Dec 2023 13:35:12 -0500
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: k8s-node01
AcquireTime: <unset>
RenewTime: Tue, 30 Jul 2024 11:48:33 -0400
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Tue, 12 Mar 2024 17:01:23 -0400 Tue, 12 Mar 2024 17:01:23 -0400 CalicoIsUp Calico is running on this node
MemoryPressure False Tue, 30 Jul 2024 11:48:30 -0400 Tue, 30 Jul 2024 11:01:27 -0400 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 30 Jul 2024 11:48:30 -0400 Tue, 12 Mar 2024 16:55:37 -0400 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 30 Jul 2024 11:48:30 -0400 Tue, 12 Mar 2024 16:55:37 -0400 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 30 Jul 2024 11:48:30 -0400 Fri, 19 Jul 2024 04:50:42 -0400 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.77.94
Hostname: k8s-node01
Capacity:
cpu: 2
ephemeral-storage: 40383Mi
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3703128Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 38110180085
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3600728Ki
pods: 110
System Info:
Machine ID: a9bd66d853144a08a6d71037ffebc3f3
System UUID: feb32f42-112b-1a87-5952-36ff542fc96a
Boot ID: d530df3e-e939-4eee-897a-147a6b86de5a
Kernel Version: 5.14.0-362.8.1.el9_3.x86_64
OS Image: Rocky Linux 9.3 (Blue Onyx)
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.21
Kubelet Version: v1.26.7
Kube-Proxy Version: v1.26.7
PodCIDR: 172.16.1.0/24
PodCIDRs: 172.16.1.0/24
Non-terminated Pods: (1 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system calico-node-b4wtl 250m (12%) 0 (0%) 0 (0%) 0 (0%) 212d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 250m (12%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning EvictionThresholdMet 52m (x17 over 9h) kubelet Attempting to reclaim memory
Normal NodeHasInsufficientMemory 52m (x17 over 9h) kubelet Node k8s-node01 status is now: NodeHasInsufficientMemory
Normal NodeHasSufficientMemory 47m (x18 over 11d) kubelet Node k8s-node01 status is now: NodeHasSufficientMemory此次故障是 Rocky9 Kubernetes部署MySQL57报错 内存泄漏导致，注意要做好资源限制
