Kubernetes部署ingress
发布时间：2024年01月19日
环境
- RHEL 9.3
- Docker Community 24.0.7
- minikube v1.32.0
部署ingress
可以访问 registry.k8s.io 的环境
通过minikube启用 ingress ：
minikube addons enable ingress
注：接下来在不能访问 registry.k8s.io 的环境里，用该环境作为“标准环境”，以便做对比。
不能访问 registry.k8s.io 的环境
通过minikube启用 ingress 会报错：
$ minikube addons enable ingress
💡 ingress is an addon maintained by Kubernetes. For any concerns contact minikube on GitHub.
You can view the list of minikube maintainers at: https://github.com/kubernetes/minikube/blob/master/OWNERS
? Using image registry.k8s.io/ingress-nginx/controller:v1.9.4
? Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
? Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
🔎 Verifying ingress addon...
? Exiting due to MK_ADDON_ENABLE: enable failed: run callbacks: running callbacks: [waiting for app.kubernetes.io/name=ingress-nginx pods: context deadline exceeded]
╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ Please also attach the following file to the GitHub issue: │
│ - /tmp/minikube_addons_bbcace35d8e0350ae758b20ff8fff60714ee7a80_0.log │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
ingress使用的是 ingress-nginx namespace：
$ kubectl get namespace
NAME STATUS AGE
default Active 42h
ingress-nginx Active 31h
kube-node-lease Active 42h
kube-public Active 42h
kube-system Active 42h
kubernetes-dashboard Active 41h
查看 ingress-nginx namespace：
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-274rs 0/1 ImagePullBackOff 0 51s
pod/ingress-nginx-admission-patch-tg887 0/1 ContainerCreating 0 51s
pod/ingress-nginx-controller-7c6974c4d8-lq5q4 0/1 ContainerCreating 0 51s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.111.104.1 <none> 80:31322/TCP,443:31156/TCP 51s
service/ingress-nginx-controller-admission ClusterIP 10.100.55.232 <none> 443/TCP 51s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 51s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-7c6974c4d8 1 1 0 51s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 0/1 51s 51s
job.batch/ingress-nginx-admission-patch 0/1 51s 51s
查看deployment：
$ kubectl describe deployment ingress-nginx-controller -n ingress-nginx
......
Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
......
NewReplicaSet: ingress-nginx-controller-7c6974c4d8 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 3m22s deployment-controller Scaled up replica set ingress-nginx-controller-7c6974c4d8 to 1
$ kubectl describe replicaset ingress-nginx-controller-7c6974c4d8 -n ingress-nginx
......
Controlled By: Deployment/ingress-nginx-controller
......
Containers:
controller:
Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 11m replicaset-controller Created pod: ingress-nginx-controller-7c6974c4d8-lq5q4
$ kubectl describe pod ingress-nginx-controller-7c6974c4d8-lq5q4 -n ingress-nginx
......
Controlled By: ReplicaSet/ingress-nginx-controller-7c6974c4d8
Containers:
controller:
Container ID:
Image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
......
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-controller-7c6974c4d8-lq5q4 to minikube
Warning FailedMount 65s (x14 over 13m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
查看job：
$ kubectl describe job ingress-nginx-admission-create -n ingress-nginx
......
Containers:
create:
Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 22m job-controller Created pod: ingress-nginx-admission-create-274rs
$ kubectl describe pod ingress-nginx-admission-create-274rs -n ingress-nginx
......
Controlled By: Job/ingress-nginx-admission-create
Containers:
create:
Container ID:
Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-admission-create-274rs to minikube
Normal Pulling 12m (x4 over 16m) kubelet Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
Warning Failed 12m (x4 over 15m) kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 108.177.97.82:443: i/o timeout
Warning Failed 12m (x4 over 15m) kubelet Error: ErrImagePull
Warning Failed 11m (x6 over 15m) kubelet Error: ImagePullBackOff
Normal BackOff 56s (x44 over 15m) kubelet Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
$ kubectl describe job ingress-nginx-admission-patch -n ingress-nginx
......
Containers:
patch:
Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 25m job-controller Created pod: ingress-nginx-admission-patch-tg887
$ kubectl describe pod ingress-nginx-admission-patch-tg887 -n ingress-nginx
......
Controlled By: Job/ingress-nginx-admission-patch
Containers:
patch:
Container ID:
Image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m default-scheduler Successfully assigned ingress-nginx/ingress-nginx-admission-patch-tg887 to minikube
Normal Pulling 14m (x4 over 17m) kubelet Pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
Warning Failed 13m (x4 over 16m) kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 108.177.97.82:443: i/o timeout
Warning Failed 13m (x4 over 16m) kubelet Error: ErrImagePull
Warning Failed 12m (x7 over 16m) kubelet Error: ImagePullBackOff
Normal BackOff 7m43s (x23 over 16m) kubelet Back-off pulling image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"
Warning Failed 2m32s kubelet Failed to pull image "registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": Error response from daemon: Get "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/ingress-nginx/kube-webhook-certgen/manifests/sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": dial tcp 142.251.8.82:443: i/o timeout
所以，归根到底，还是 registry.k8s.io 的网络问题。从 minikube addons enable ingress 的输出可知，使用到的image是：
registry.k8s.io/ingress-nginx/controller:v1.9.4registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
注：从上述 kubectl describe xxx 命令也可以看出，是这两个image pull不下来。
所以，接下来我们用手工方式来部署ingress。
手工部署
在可以访问 registry.k8s.io 的地方，找到上述两个image，pull下来，tag一下，push到DockerHub自己的repository里。
比如：我的image是：
kaiding1/controller:v1.9.4kaiding1/kube-webhook-certgen:v20231011-8b53cabe0
打开浏览器，访问 https://github.com/kubernetes/ingress-nginx ，其中“Supported Versions table”如下：
查看Kubernetes版本：
$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
由对照表可知，k8s v1.28.3支持的Ingress-Nginx版本有v1.9.0到v1.9.5。我们就用v1.9.4，和 registry.k8s.io/ingress-nginx/controller:v1.9.4 保持一致（也和“标准环境”保持一致）。
把 https://github.com/kubernetes/ingress-nginx/blob/controller-v1.9.4/deploy/static/provider/kind/deploy.yaml 下载到本地。
注：有kind、cloud、baremetal等多个版本，我比较了一下，kind里的版本最接近“标准环境”。
然后修改 deploy.yaml ：
- 把：
image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3
替换为：
image: kaiding1/controller:v1.9.4
- 把：
image: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
替换为：
image: kaiding1/kube-webhook-certgen:v20231011-8b53cabe0
注意有两处 kube-webhook-certgen 要替换。
注：删掉了 @sha256:xxxxx ，因为跟原来的digest不一样，如果想加上，需要使用新的digest。
部署该yaml文件：
$ kubectl apply -f deploy.yaml
namespace/ingress-nginx unchanged
serviceaccount/ingress-nginx configured
serviceaccount/ingress-nginx-admission configured
role.rbac.authorization.k8s.io/ingress-nginx configured
role.rbac.authorization.k8s.io/ingress-nginx-admission configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx configured
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission configured
rolebinding.rbac.authorization.k8s.io/ingress-nginx configured
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission configured
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx configured
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission configured
configmap/ingress-nginx-controller configured
service/ingress-nginx-controller configured
service/ingress-nginx-controller-admission configured
deployment.apps/ingress-nginx-controller configured
ingressclass.networking.k8s.io/nginx configured
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-create\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-create\"},\"spec\":{\"containers\":[{\"args\":[\"create\",\"--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc\",\"--namespace=$(POD_NAMESPACE)\",\"--secret-name=ingress-nginx-admission\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"create\",\"securityContext\":{\"allowPrivilegeEscalation\":false}}],\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"fsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"ingress-nginx-admission\"}}}}\n"},"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"template":{"metadata":{"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"$setElementOrder/containers":[{"name":"create"}],"containers":[{"image":"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0","name":"create"}],"nodeSelector":{"minikube.k8s.io/primary":null},"securityContext":{"fsGroup":2000}}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-create", Namespace: "ingress-nginx"
for: "deploy.yaml": error when patching "deploy.yaml": Job.batch "ingress-nginx-admission-create" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-create", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "app.kubernetes.io/part-of":"ingress-nginx", "app.kubernetes.io/version":"1.9.4", "batch.kubernetes.io/controller-uid":"4dc1f43f-04d8-4e43-9469-addcd7a96f8c", "batch.kubernetes.io/job-name":"ingress-nginx-admission-create", "controller-uid":"4dc1f43f-04d8-4e43-9469-addcd7a96f8c", "job-name":"ingress-nginx-admission-create"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"create", Image:"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0", Command:[]string(nil), Args:[]string{"create", "--host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc", "--namespace=$(POD_NAMESPACE)", "--secret-name=ingress-nginx-admission"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00b2e2720)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc00b40c2a0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00d6fdd40), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00b52e480), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch/v1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-patch\",\"namespace\":\"ingress-nginx\"},\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"admission-webhook\",\"app.kubernetes.io/instance\":\"ingress-nginx\",\"app.kubernetes.io/name\":\"ingress-nginx\",\"app.kubernetes.io/part-of\":\"ingress-nginx\",\"app.kubernetes.io/version\":\"1.9.4\"},\"name\":\"ingress-nginx-admission-patch\"},\"spec\":{\"containers\":[{\"args\":[\"patch\",\"--webhook-name=ingress-nginx-admission\",\"--namespace=$(POD_NAMESPACE)\",\"--patch-mutating=false\",\"--secret-name=ingress-nginx-admission\",\"--patch-failure-policy=Fail\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"metadata.namespace\"}}}],\"image\":\"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"patch\",\"securityContext\":{\"allowPrivilegeEscalation\":false}}],\"nodeSelector\":{\"kubernetes.io/os\":\"linux\"},\"restartPolicy\":\"OnFailure\",\"securityContext\":{\"fsGroup\":2000,\"runAsNonRoot\":true,\"runAsUser\":2000},\"serviceAccountName\":\"ingress-nginx-admission\"}}}}\n"},"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"template":{"metadata":{"labels":{"app.kubernetes.io/part-of":"ingress-nginx","app.kubernetes.io/version":"1.9.4"}},"spec":{"$setElementOrder/containers":[{"name":"patch"}],"containers":[{"image":"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0","name":"patch"}],"nodeSelector":{"minikube.k8s.io/primary":null},"securityContext":{"fsGroup":2000}}}}}
to:
Resource: "batch/v1, Resource=jobs", GroupVersionKind: "batch/v1, Kind=Job"
Name: "ingress-nginx-admission-patch", Namespace: "ingress-nginx"
for: "deploy.yaml": error when patching "deploy.yaml": Job.batch "ingress-nginx-admission-patch" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"ingress-nginx-admission-patch", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app.kubernetes.io/component":"admission-webhook", "app.kubernetes.io/instance":"ingress-nginx", "app.kubernetes.io/name":"ingress-nginx", "app.kubernetes.io/part-of":"ingress-nginx", "app.kubernetes.io/version":"1.9.4", "batch.kubernetes.io/controller-uid":"1033bcb2-b2f4-4a6a-b43c-15ad9351d6ec", "batch.kubernetes.io/job-name":"ingress-nginx-admission-patch", "controller-uid":"1033bcb2-b2f4-4a6a-b43c-15ad9351d6ec", "job-name":"ingress-nginx-admission-patch"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"patch", Image:"kaiding1/kube-webhook-certgen:v20231011-8b53cabe0", Command:[]string(nil), Args:[]string{"patch", "--webhook-name=ingress-nginx-admission", "--namespace=$(POD_NAMESPACE)", "--patch-mutating=false", "--secret-name=ingress-nginx-admission", "--patch-failure-policy=Fail"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:"POD_NAMESPACE", Value:"", ValueFrom:(*core.EnvVarSource)(0xc00bb53f80)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil), Claims:[]core.ResourceClaim(nil)}, ResizePolicy:[]core.ContainerResizePolicy(nil), RestartPolicy:(*core.ContainerRestartPolicy)(nil), VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*core.SecurityContext)(0xc00bb5d440), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:"OnFailure", TerminationGracePeriodSeconds:(*int64)(0xc00bb82c00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"ingress-nginx-admission", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc00bb75d40), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", SetHostnameAsFQDN:(*bool)(nil), Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil), OS:(*core.PodOS)(nil), SchedulingGates:[]core.PodSchedulingGate(nil), ResourceClaims:[]core.PodResourceClaim(nil)}}: field is immutable
有些配置跟原来已有的配置冲突，为了简单以前，把原来的namespace整个删掉：
kubectl delete namespace ingress-nginx
所有的deployment、pod都资源都没了，然后再重新部署：
$ kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx unchanged
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission unchanged
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
deployment.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx unchanged
networkpolicy.networking.k8s.io/ingress-nginx-admission created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission configured
再次查看资源：
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-sbrf8 0/1 Completed 0 40s
pod/ingress-nginx-admission-patch-ftgxj 0/1 Completed 1 40s
pod/ingress-nginx-controller-6d5bdfb648-wkrdr 0/1 Pending 0 40s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.103.85.240 <none> 80:30507/TCP,443:32553/TCP 40s
service/ingress-nginx-controller-admission ClusterIP 10.96.189.167 <none> 443/TCP 40s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 0/1 1 0 40s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-6d5bdfb648 1 1 0 40s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 40s
job.batch/ingress-nginx-admission-patch 1/1 5s 40s
可见， /ingress-nginx-controller 的pod没有启起来，处于“Pending”状态。
查看该pod：
$ kubectl describe pod ingress-nginx-controller-6d5bdfb648-wkrdr -n ingress-nginx
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 97s default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
和“标准环境”的 ingress-nginx-controller deployment做对比，发现需要注释掉这一行：
apiVersion: apps/v1
kind: Deployment
......
nodeSelector:
# ingress-ready: "true" # 把这一行注释掉
还有一些别的不同之处，但是貌似不影响，只有这一行是有影响的。
然后再次删掉 ingress-nginx namespace重建：
kubectl delete namespace ingress-nginx
kubectl apply -f deploy.yaml
查看资源：
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-pk7jj 0/1 Completed 0 13s
pod/ingress-nginx-admission-patch-2twvg 0/1 Completed 0 13s
pod/ingress-nginx-controller-68db9c8cbf-zvsmr 1/1 Running 0 13s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.101.105.112 <none> 80:31854/TCP,443:31461/TCP 13s
service/ingress-nginx-controller-admission ClusterIP 10.98.77.135 <none> 443/TCP 13s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 13s
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-68db9c8cbf 1 1 1 13s
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 4s 13s
job.batch/ingress-nginx-admission-patch 1/1 4s 13s
现在所有资源都OK了。
总结
deploy.yaml 需要修改之处：
image: registry.k8s.io/ingress-nginx/controller:v1.9.4@sha256:5b161f051d017e55d358435f295f5e9a297e66158f136321d9b04520ec6c48a3：替换为可访问的imageimage: registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80：替换为可访问的image，注意有两处ingress-ready: "true"：注释掉这一行
测试ingress
创建 ingress-example.yaml 文件如下：
kind: Pod
apiVersion: v1
metadata:
name: foo-app
labels:
app: foo
spec:
containers:
- name: foo-app
image: 'kicbase/echo-server:1.0'
---
kind: Service
apiVersion: v1
metadata:
name: foo-service
spec:
selector:
app: foo
ports:
- port: 8080
---
kind: Pod
apiVersion: v1
metadata:
name: bar-app
labels:
app: bar
spec:
containers:
- name: bar-app
image: 'kicbase/echo-server:1.0'
---
kind: Service
apiVersion: v1
metadata:
name: bar-service
spec:
selector:
app: bar
ports:
- port: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-ingress
spec:
rules:
- http:
paths:
- pathType: Prefix
path: /foo
backend:
service:
name: foo-service
port:
number: 8080
- pathType: Prefix
path: /bar
backend:
service:
name: bar-service
port:
number: 8080
---
部署：
$ kubectl apply -f ingress-example.yaml
pod/foo-app created
service/foo-service created
pod/bar-app created
service/bar-service created
ingress.networking.k8s.io/example-ingress created
（注意：部署完以后查看一下这些pod，有时会遇到pull不下来的情况，报错是“You have reached your pull rate limit”。解决方法是 docker login 。但是今天好像login了也不行……我没有细看这个问题，直接用老办法，把它pull下来然后push到个人的repository了。）
查看ingress：
$ kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress <none> * localhost 80 39s
但是，不能用localhost作为host：
$ curl http://localhost/foo
curl: (7) Failed to connect to localhost port 80: Connection refused
不知道是不是我哪里配置的有问题。
而要用 192.168.49.2 （注： 192.168.49.2 是minikube的internal IP）：
$ curl http://192.168.49.2/foo
Request served by foo-app
HTTP/1.1 GET /foo
Host: 192.168.49.2
Accept: */*
User-Agent: curl/7.76.1
X-Forwarded-For: 192.168.49.1
X-Forwarded-Host: 192.168.49.2
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 192.168.49.1
X-Request-Id: 47a53f8a53335fd2e412c08d52918966
X-Scheme: http
同理：
$ curl http://192.168.49.2/bar
Request served by bar-app
HTTP/1.1 GET /bar
Host: 192.168.49.2
Accept: */*
User-Agent: curl/7.76.1
X-Forwarded-For: 192.168.49.1
X-Forwarded-Host: 192.168.49.2
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: 192.168.49.1
X-Request-Id: c471ed7ba12a154cafe052107d553480
X-Scheme: http
查看 exmaple-ingress ，如下：
$ kubectl describe ingress example-ingress
Name: example-ingress
Labels: <none>
Namespace: default
Address: localhost
Ingress Class: <none>
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
*
/foo foo-service:8080 (10.244.0.34:8080)
/bar bar-service:8080 (10.244.0.35:8080)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 52m (x2 over 53m) nginx-ingress-controller Scheduled for sync
参考
https://minikube.sigs.k8s.io/docs/starthttps://blog.csdn.net/alwaysbefine/article/details/129201448
文章来源:https://blog.csdn.net/duke_ding2/article/details/135642990
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：chenni525@qq.com进行投诉反馈，一经查实，立即删除！
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：chenni525@qq.com进行投诉反馈，一经查实，立即删除！
最新文章
- Python教程
- 深入理解 MySQL 中的 HAVING 关键字和聚合函数
- Qt之QChar编码（1）
- MyBatis入门基础篇
- 用Python脚本实现FFmpeg批量转换
- 基于python舆情分析可视化系统+情感分析+爬虫+机器学习（源码）?
- 栈和队列详解
- 【手把手带你玩转MyBatis】源码篇：探秘工作流程的执行内幕
- 商品推荐系统+可视化+2种协同过滤推荐算法 Django框架 大数据毕业设计（附源码+论文）?
- webpack配置 - gpt问答 待整理
- ARM_Linux中GCC编译器的使用
- 图书管理系统(JSP+java+springmvc+mysql+MyBatis)
- pikachu靶场通关（部分）
- 文章解读与仿真程序复现思路——中国电机工程学报EI\CSCD\北大核心《考虑用户禀赋效应和环保意识不确定性的微电网鲁棒优化调度方法》
- 第6章-用户标签体系与用户画像