Kubernetes实战(十八)-Pod配置污点和容忍

1 污点

1.1 污点简介

亲和性调度的方式都是站在Pod的角度上，通过在Pod上增加属性来将Pod调度到到指定的节点上，其实也可以站在Node节点的角度上，通过给Node节点设置属性，来决定是否允许Pod调度过来，这就是污点。

Node被设置上污点之后就和Pod存在了一种相斥的关系，进而拒绝Pod调度进来，甚至可以将已经存在的Pod驱逐出去。

污点的格式为 key=value:effect，key和value是污点的标签，effect描述五点多额作用，支持如下三个选项

• PreferNoSchedule：Kubernetes将尽量避免把Pod调度到具有此污点的Node上，除非没有其他节点可调度了
• NoSchedule：Kubernetes将不会把Pod调度到具有该污点的Node上，但不会影响当前Node上已经存在的Pod
• NoExecute：Kubernetes将不会把Pod调度到具有此污点的Node上，同时也会将Node上已经存在的Pod驱逐

1.2?污点命令

# 设置污点

$ kubectl taint nodes node1 key=value:effect

# 去除污点

$ kubectl taint nodes node1 key:effect-

# 去除所有污点

$ kubectl taint nodes node1 key-

1.3?污点案例

1）给node1设置一个污点，尽量不要调度过来pod

[root@master resource_manage]# kubectl taint nodes node1 name=nginx:PreferNoSchedule
node/node1 tainted

2）创建 nginx pod

[root@master resource_manage]# kubectl run nginx --image=nginx:1.17.1 --port=80
pod/nginx created

3）查询pod调度信息

[root@master resource_manage]# kubectl get pod -o wide
NAME ? ?READY ? STATUS ? ?RESTARTS ? AGE ? IP ? ? ? ? ? ?NODE ? ?NOMINATED NODE ? READINESS GATES
nginx ? 1/1 ? ? Running ? 0 ? ? ? ? ?7s ? ?10.244.2.48 ? node2 ? <none> ? ? ? ? ? <none>

可以看到此时直接调度到node2了，不会调度到node1的，当然如果此时node2挂了，只有node1存活时，也会调度过来的。

1.4 查询节点污点

[root@master resource_manage]# kubectl describe node node1
Name:               node1
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=node1
                    kubernetes.io/os=linux
                    nodeenv=test
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"ba:fe:1f:25:fe:26"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.16.41
                    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 14 Mar 2022 14:41:02 +0800
Taints:             name=nginx:PreferNoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  node1
  AcquireTime:     <unset>
  RenewTime:       Sat, 26 Mar 2022 00:00:54 +0800
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Mon, 14 Mar 2022 14:43:39 +0800   Mon, 14 Mar 2022 14:43:39 +0800   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Fri, 25 Mar 2022 23:58:57 +0800   Mon, 14 Mar 2022 14:41:02 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Fri, 25 Mar 2022 23:58:57 +0800   Mon, 14 Mar 2022 14:41:02 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Fri, 25 Mar 2022 23:58:57 +0800   Mon, 14 Mar 2022 14:41:02 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Fri, 25 Mar 2022 23:58:57 +0800   Mon, 14 Mar 2022 14:43:42 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.16.41
  Hostname:    node1
Capacity:
  cpu:                8
  ephemeral-storage:  208357992Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32882960Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  192022725110
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32780560Ki
  pods:               110
System Info:
  Machine ID:                 f9c2b25f57184e06b8855490b4be6013
  System UUID:                d1042642-3933-564f-4f2d-279b5e96cead
  Boot ID:                    8517c1cc-8935-452e-9efb-a34f396b98a5
  Kernel Version:             5.4.179-200.el7.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.9
  Kubelet Version:            v1.21.2
  Kube-Proxy Version:         v1.21.2
PodCIDR:                      10.244.1.0/24
PodCIDRs:                     10.244.1.0/24
Non-terminated Pods:          (4 in total)
  Namespace                   Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                         ------------  ----------  ---------------  -------------  ---
  kube-system                 kube-flannel-ds-gg4jq                        100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      11d
  kube-system                 kube-proxy-tqzjl                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         11d
  kubernetes-dashboard        dashboard-metrics-scraper-c45b7869d-7ll25    0 (0%)        0 (0%)      0 (0%)           0 (0%)         11d
  kubernetes-dashboard        kubernetes-dashboard-79b5779bf4-t28b4        0 (0%)        0 (0%)      0 (0%)           0 (0%)         11d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (1%)  100m (1%)
  memory             50Mi (0%)  50Mi (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-1Gi      0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:              <none>

1.5?删除污点

$ kubectl taint nodes node1 name:PreferNoSchedule-
node/node1 untainted

1.6?为什么创建Pod的时候不会调度到master节点？

通过如下命令可以看到master节点是默认设置了node-role.kubernetes.io/master:NoSchedule类型的污点，因此在创建pod的时候是不会往master节点调度的。

[root@master resource_manage]# kubectl describe nodes master
Name:               master
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=master
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node-role.kubernetes.io/master=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"02:f6:8e:03:60:51"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.16.40
                    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 14 Mar 2022 14:38:03 +0800
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  master
  AcquireTime:     <unset>
  RenewTime:       Sat, 26 Mar 2022 00:05:31 +0800
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Mon, 14 Mar 2022 14:42:58 +0800   Mon, 14 Mar 2022 14:42:58 +0800   FlannelIsUp                  Flannel is running on this node
  MemoryPressure       False   Sat, 26 Mar 2022 00:01:28 +0800   Mon, 14 Mar 2022 14:38:02 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Sat, 26 Mar 2022 00:01:28 +0800   Mon, 14 Mar 2022 14:38:02 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Sat, 26 Mar 2022 00:01:28 +0800   Mon, 14 Mar 2022 14:38:02 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Sat, 26 Mar 2022 00:01:28 +0800   Mon, 14 Mar 2022 14:43:03 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.16.40
  Hostname:    master
Capacity:
  cpu:                8
  ephemeral-storage:  208357992Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32882960Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  192022725110
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             32780560Ki
  pods:               110
System Info:
  Machine ID:                 f9c2b25f57184e06b8855490b4be6013
  System UUID:                c5d32642-f84c-61ef-ac7f-d65ae6880a51
  Boot ID:                    9cbc9b25-2cf2-42d8-aa89-1fdab687c447
  Kernel Version:             5.4.179-200.el7.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.9
  Kubelet Version:            v1.21.2
  Kube-Proxy Version:         v1.21.2
PodCIDR:                      10.244.0.0/24
PodCIDRs:                     10.244.0.0/24
Non-terminated Pods:          (6 in total)
  Namespace                   Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                              ------------  ----------  ---------------  -------------  ---
  kube-system                 etcd-master                       100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         11d
  kube-system                 kube-apiserver-master             250m (3%)     0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 kube-controller-manager-master    200m (2%)     0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 kube-flannel-ds-n76xj             100m (1%)     100m (1%)   50Mi (0%)        50Mi (0%)      11d
  kube-system                 kube-proxy-h27ms                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         11d
  kube-system                 kube-scheduler-master             100m (1%)     0 (0%)      0 (0%)           0 (0%)         11d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                750m (9%)   100m (1%)
  memory             150Mi (0%)  50Mi (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

2?容忍

2.1?容忍简介

当对一个node节点定义了污点，但是又希望某一些pod是可以调度到带有污点的节点上，此时就需要容忍了，污点就是拒绝，容忍就是忽略/允许，Node通过污点拒绝Pod调度上去，Pod通过容忍忽略拒绝，如下：

2.2?容忍实战

1）给node1设置NoSchedule污点

此时为演示，可以先保持只有node1一个节点，将其他节点关闭

[root@master resource_manage]# kubectl taint nodes node1 name=nginx:NoSchedule
node/node1 tainted

2）编辑带有容忍的pod_toleration.yaml文件

apiVersion: v1
kind: Namespace
metadata:
? name: dev

---

apiVersion: v1
kind: Pod
metadata:
? name: nginx-pod
? namespace: dev
spec:
? containers:
? - name: nginx
? ? image: nginx:1.17.1
? tolerations:
? - key: "name"
? ? operator: "Equal"
? ? value: "nginx"
? ? effect: "NoSchedule"

3）创建资源

[root@master resource_manage]# kubectl apply -f pod_toleration.yaml
namespace/dev created
pod/nginx-pod created

4）查看验证

然后通过如下命令查看，可以发现此时还是可以调度到node1节点上的

[root@master resource_manage]# kubectl get pod -n dev -o wide
NAME ? ? ? ?READY ? STATUS ? ?RESTARTS ? AGE ? IP ? ? ? ? ? ?NODE ? ?NOMINATED NODE ? READINESS GATES
nginx-pod ? 1/1 ? ? Running ? 0 ? ? ? ? ?13s ? 10.244.2.49 ? node1 ? <none> ? ? ? ? ? <none>

2.3 容忍配置项说明

通过如下命令可以查看配置项的说明：

[root@master resource_manage]# kubectl explain pod.spec.tolerations
KIND:     Pod
VERSION:  v1

RESOURCE: tolerations <[]Object>

DESCRIPTION:
     If specified, the pod's tolerations.

     The pod this Toleration is attached to tolerates any taint that matches the
     triple <key,value,effect> using the matching operator <operator>.

FIELDS:
   effect       <string>
     Effect indicates the taint effect to match. Empty means match all taint
     effects. When specified, allowed values are NoSchedule, PreferNoSchedule
     and NoExecute.

   key  <string>
     Key is the taint key that the toleration applies to. Empty means match all
     taint keys. If the key is empty, operator must be Exists; this combination
     means to match all values and all keys.

   operator     <string>
     Operator represents a key's relationship to the value. Valid operators are
     Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for
     value, so that a pod can tolerate all taints of a particular category.

   tolerationSeconds    <integer>
     TolerationSeconds represents the period of time the toleration (which must
     be of effect NoExecute, otherwise this field is ignored) tolerates the
     taint. By default, it is not set, which means tolerate the taint forever
     (do not evict). Zero and negative values will be treated as 0 (evict
     immediately) by the system.

   value        <string>
     Value is the taint value the toleration matches to. If the operator is
     Exists, the value should be empty, otherwise just a regular string.