K8S Helm 安装ingress-nginx/ingress-nginx

发布时间：2023年12月29日

安装ingress-nginx/ingress-nginx
- 参考：
- https://www.cnblogs.com/syushin/p/15271304.html

添加helm仓库

- [root@k8s-master ~]# helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
- [root@k8s-master ~]# helm repo update

下载安装包
- helm pull ingress-nginx/ingress-nginx
  - 解压
    - tar -zxvf ingress-nginx-4.0.1.tgz
备份并修改 values.yaml 文件
- 修改controller的镜像地址

controller:
  name: /
  enableAnnotationValidations: false
  image:
    ## Keep false as default for now!
    chroot: false
    registry: docker.io
    image: willdockerhub/ingress-nginx-controller
    ## for backwards compatibility consider setting the full image url via the repository value below
    ## use *either* current default registry/image or repository format or installing chart by providing the values.yaml will fail
    ## repository:
    tag: "v1.0.0"
    digest: sha256:0851b34f69f69352bf168e6ccf30e1e20714a264ab1ecd1933e4d8c0fc3215c6
    pullPolicy: IfNotPresent
    # www-data -> uid 101
    runAsUser: 101
    allowPrivilegeEscalation: true

- 修改 hostNetwork 的值为 true：

# is merged hostNetwork: true ## Use host ports 80 and 443

- dnsPolicy的值改为: ClusterFirstWithHostNet

# to keep resolving names inside the k8s network, use ClusterFirstWithHostNet. dnsPolicy: ClusterFirstWithHostNet

- nodeSelector 添加标签: ingress: "true"，用于部署 ingress-controller 到指定节点

 # -- Node labels for controller pod assignment
  ## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
  ##
  nodeSelector:
    kubernetes.io/os: linux
    ingress: "true"

- kind类型更改为：DaemonSet

 # -- Use a `DaemonSet` or `Deployment`
  kind: DaemonSet
  # -- Annotations to be added to the controller Deployment or DaemonSet

- kube-webhook-certgen的镜像地址改为国内仓库地址

patch:
      enabled: true
      image:
        registry: registry.cn-hangzhou.aliyuncs.com
        image: google_containers/kube-webhook-certgen
        ## for backwards compatibility consider setting the full image url via the repository value below
        ## use *either* current default registry/image or repository format or installing chart by providing the values.yaml will fail
        ## repository:
        tag: "v20231011-8b53cabe0"
        #digest: sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80
        digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9
        pullPolicy: IfNotPresent

安装
- 使用当前目录中的values.yaml文件
  - helm install ingress-nginx -n ingress-nginx .
- 指定values.yaml文件，并使用debug参数，方便调试异常
  - helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx -n ingress-nginx --values values.yaml --debug
    - upgrade --install 会导致重新下载包
      - 如果不想重新下载，直接使用install
- 安装成功，并给了一个创建ingress的例子

NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'

An example Ingress that makes use of the controller:
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    name: example
    namespace: foo
  spec:
    ingressClassName: nginx
    rules:
      - host: www.example.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: exampleService
                  port:
                    number: 80
              path: /
    # This section is only required if TLS is to be enabled for the Ingress
    tls:
      - hosts:
        - www.example.com
        secretName: example-tls

If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

处于pending状态的ingress-nginx-controller

[root@k8s-master01 ingress-nginx]# kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.99.163.44    <pending>     80:31268/TCP,443:31052/TCP   4m54s
ingress-nginx-controller-admission   ClusterIP      10.100.131.12   <none>        443/TCP                      4m54s


[root@k8s-master01 ingress-nginx]# kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE   SELECTOR
ingress-nginx-controller   LoadBalancer   10.99.163.44   <pending>     80:31268/TCP,443:31052/TCP   94s   app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

给节点打标签ingress=true
- 需要给节点打上刚刚设置的标签ingress=true，让 Pod 调度到指定的节点，比如调度到 master 节点

# 给master节点打上标签 ingress=ture
[root@k8s-master ingress-nginx]# kubectl label node master1 ingress=true
node/master1 labeled

# k8s默认集群中，出于安全考虑，默认配置下Kubernetes不会将Pod调度到Master节点。测试环境无所谓，所以执行下面命令去除master的污点：
[root@k8s-master ingress-nginx]# kubectl taint node master1 node-role.kubernetes.io/master-

参考：

How do I set up a pod to allow scheduling on the master · Issue #1814 · k3s-io/k3s · GitHub How do I set up a pod to allow scheduling on the master

K8S 将 pod 调度到指定 nodes 上运行 - 知乎 K8S 将 pod 调度到指定 nodes 上运行

[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl taint nodes --all node-role.kubernetes.io/control-plane-
node/master01 untainted
taint "node-role.kubernetes.io/control-plane" not found
taint "node-role.kubernetes.io/control-plane" not found
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
master01   Ready    control-plane   44d   v1.28.2
node02     Ready    <none>          44d   v1.28.2
node03     Ready    <none>          44d   v1.28.2
[root@master01 rocketmq]# kubectl describe nodes | grep Taints
Taints:             <none>
Taints:             <none>
Taints:             <none>
[root@master01 rocketmq]# kubectl get pods -A -owide |grep ingress
ingress-nginx            ingress-nginx-controller-bc96v                           1/1     Running            0                  41s     yourip     master01   <none>           <none>

#将 Master 也当作 Node 使用
kubectl taint node nodename node-role.kubernetes.io/master-

[root@app01 home]# kubectl taint node app01 node-role.kubernetes.io/master-
node/app01 untainted
[root@app01 home]# 

#将 Master 恢复成 Master Only 状态
kubectl taint node nodename node-role.kubernetes.io/master="":NoSchedule
  
[root@app01 rabbitmq]# kubectl taint node app01  node-role.kubernetes.io/master="":NoSchedule
node/app01 tainted
[root@app01 rabbitmq]#

- 本次直接将标签打到了node02上
  - 所以配置的域名要指向node02节点的ip地址
  - ingress-nginx-controller对应的pod会自动部署到node02节点上
配置访问hosts
- 192.168.221.132 harbor.david.org
- 测试
  - 用浏览器访问没问，curl缺少header中的一些参数，所以报308错误

[root@k8s-master01 harbor]# curl harbor.david.org
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

configmap权限异常处理

查看ingress-nginx-controller日志

- [root@k8s-master01 harbor]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l

- - E1122 05:49:57.933496 7 leaderelection.go:329] error initially creating leader election record: configmaps is forbidden: User "system:serviceaccount:ingress-nginx:ingress-nginx" cannot create resource "configmaps" in API group "" in the namespace "ingress-nginx"
    - 需要create和update权限，修改role即可
      - [root@k8s-master01 nginx]# kubectl edit clusterrole -n ingress-nginx ingress-nginx
        添加
        - create
        - update
        保存并退出，再查看ingress-nginx-controller日志
        [root@k8s-master01 harbor]# kubectl logs -n ingress-nginx ingress-nginx-controller-4nl8l
      - configmaps （用于配置文件注入）

rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  - endpoints
  - nodes
  - pods
  - secrets
  - namespaces
  verbs:
  - list
  - watch
  - create
  - update

异常调试
- 安装加上debug参数展示详细安装过程，方便排查错误
- client.go:779: [debug] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
- 提示job有异常，先找到这个job

[root@k8s-master01 ingress-nginx]# kubectl get job -A
NAMESPACE       NAME                             COMPLETIONS   DURATION   AGE
ingress-nginx   ingress-nginx-admission-create   0/1           8s         8s
ingress-nginx   ingress-nginx-admission-patch    0/1           69m        69m


[root@k8s-master01 ingress-nginx]# kubectl get pods -A
ingress-nginx          ingress-nginx-admission-create-r7bps

有个job对应的pod有异常


[root@k8s-master01 ingress-nginx]# kubectl describe pods -n ingress-nginx          ingress-nginx-admission-create-r7bps

 Normal   BackOff    5s (x2 over 28s)   kubelet            Back-off pulling image "registry.aliyuncs.com/google_containers/kube-webhook-certgen:v1.5.1@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80"

提示镜像有异常

Failed to pull image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": rpc error: code = NotFound desc = failed to pull and unpack image "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": failed to resolve reference "registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80": registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen@sha256:a7943503b45d552785aa3b5e457f169a5661fb94d82b8a3373bcd9ebaf9aac80: not found

修改了镜像还不管用，还需要修改sha256

使用ctr下载该镜像

[root@k8s-master01 ingress-nginx]# ctr -n=k8s.io images pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0

找到该镜像

[root@k8s-master01 ingress-nginx]# ctr -n=k8s.io images list

registry.cn-hangzhou.aliyuncs.com/google_containers/kube-webhook-certgen:v20231011-8b53cabe0 application/vnd.docker.distribution.manifest.list.v2+json sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9 22.1 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/s390x io.cri-containerd.image=managed

把sha256字符串加到values.yaml里

digest: sha256:488fc1dcc9269161ac781ffb5df0a9751cb64693bf195fe76e57f211db332dd9

再次安装即可

文章来源:https://blog.csdn.net/alksjdfp32r/article/details/135270898
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：chenni525@qq.com进行投诉反馈，一经查实，立即删除！

K8S Helm 安装ingress-nginx/ingress-nginx

安装ingress-nginx/ingress-nginx

添加helm仓库

下载安装包

备份并修改 values.yaml 文件

安装

配置访问hosts