kubernetes ingress 详解（包含灰度发布/金丝雀部署）

发布时间：2024年01月09日

写在前面：如有问题，以你为准，

目前24年应届生，各位大佬轻喷，部分资料与图片来自网络

内容较长，页面右上角目录方便跳转

ingress 介绍架构

原理

官网资料：Ingress 控制器 | Kubernetes

我们已经知道，Service对集群之外暴露服务的主要方式有两种：NodePort和LoadBalancer，但是这两种方式，都有一定的缺点：
1. NodePort方式的缺点是会占用很多集群机器的端口，那么当集群服务变多的时候，这个缺点就愈发明显。
2. LoadBalancer的缺点是每个Service都需要一个LB，浪费，麻烦，并且需要kubernetes之外的设备的支持。

ingress可以解决的问题

1）动态配置服务

如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作。

2）减少不必要的端口暴露

配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式。

基于这种现状，kubernetes提供了Ingress资源对象，Ingress只需要一个NodePort或者一个LB就可以满足暴露多个Service的需求，实现多个service的反向代理和负载均衡，工作机制大致如下图所示：

实际上，Ingress相当于一个七层的负载均衡器，是kubernetes对反向代理的一个抽象，它的工作原理类似于Nginx，可以理解为Ingress里面建立了诸多映射规则，Ingress Controller通过监听这些配置规则并转化为Nginx的反向代理配置，然后对外提供服务。

Ingress: K8s中的一个抽象资源，给管理员提供一个暴露应用的入口定义方法

Ingress Controller: 根据Ingress生成具体的路由规则，并对Pod负载均衡器

1. Ingress：kubernetes中的一个对象，作用是定义请求如何转发到Service的规则。将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可，每一个ingress就是一个nginx方向代理规则
2. Ingress Controller：具体实现反向代理及负载均衡的程序，对Ingress定义的规则进行解析，根据配置的规则来实现请求转发，实现的方式有很多，比如Nginx，Contour，Haproxy等。将新加入的Ingress转化成Nginx的配置文件并使之生效
Ingress（以Nginx）的工作原理如下：
1. 用户编写Ingress规则，说明那个域名对应kubernetes集群中的那个Service。
2. Ingress控制器动态感知Ingress服务规则的变化，然后生成一段对应的Nginx的反向代理配置。
3. Ingress控制器会将生成的Nginx配置写入到一个运行着的Nginx服务中，并动态更新。
4. 到此为止，其实真正在工作的就是一个Nginx，来实现反向代理，内部配置了用户定义的请求规则。（当然也可以是一个haproxy Controller）

下面图中ingress service 就是ingress资源对象，通过yaml部署，规则动态更新

架构图

流量路径

浏览器(ip:port) --> ServiceNodeport (ingress controller) --> ingress controller pod --> service( web) --> deployment pod (web)

[root@master kube-bench]# kubectl get svc,pod? -n ingress-nginx

NAME???????????????????????????????????????? TYPE??????? CLUSTER-IP?????? EXTERNAL-IP?? PORT(S)????????????????????? AGE

service/ingress-nginx-controller???????????? NodePort??? 10.111.123.202?? <none>??????? 80:32172/TCP,443:31755/TCP?? 256d

service/ingress-nginx-controller-admission?? ClusterIP?? 10.106.250.140?? <none>??????? 443/TCP????????????????????? 256d



NAME?????????????????????????????????????????? READY?? STATUS??? RESTARTS?? AGE

pod/ingress-nginx-controller-cf4967654-f9v7j?? 1/1???? Running?? 0????????? 45h

应用场景

灰度发布/金丝雀（详解在后面）

业务域拆分

证书 CA

性能调优

部署 Ingress Controller（nginx ingress）

基于nginx服务的ingress controller根据不同的开发公司，又分为

k8s社区的ingres-nginx和nginx公司的nginx-ingress

k8s社区提供的ingress，github地址如下：https://github.com/kubernetes/ingress-nginx

官网：https://kubernetes.github.io/ingress-nginx/deploy/

nginx社区提供的ingress，github地址如下：https://github.com/nginxinc/kubernetes-ingress

这次实操使用的是 k8s社区

Ingress-NGINX version	k8s supported version	Alpine Version	Nginx Version
v1.6.4	1.26, 1.25, 1.24, 1.23	3.17.0	1.21.6
v1.5.1	1.25, 1.24, 1.23	3.16.2	1.21.6
v1.4.0	1.25, 1.24, 1.23, 1.22	3.16.2	1.19.10?
v1.3.1	1.24, 1.23, 1.22, 1.21, 1.20	3.16.2	1.19.10?
v1.3.0	1.24, 1.23, 1.22, 1.21, 1.20	3.16.0	1.19.10?
v1.2.1	1.23, 1.22, 1.21, 1.20, 1.19	3.14.6	1.19.10?

部署教程：kubernetes 部署 Ingress Controller（nginx ingress）-CSDN博客

登入 ingress controller 查看nginx配置

原理

https://www.bilibili.com/video/BV1r64y1m72f/?spm_id_from=333.337.search-card.all.click&vd_source=d649e016cd28e49d35879f17f8ba6892

多进程管理，ingress-nginx-controller 是在pod 里面，但是外部还有一个ingress-nginx-controller进程来通过kubernetes api 管理这些pod，并对这些pod写入ingress 配置

reload 流程

在高并发吞吐量的情况下，reload 还是有一些影响的

部署 Service和Dployment

apiVersion: apps/v1

kind: Deployment

metadata:

? name: nginx-deployment

? namespace: study

spec:

? replicas: 3

? selector:

??? matchLabels:

????? app: nginx-pod

? template:

??? metadata:

????? labels:

??????? app: nginx-pod

??? spec:

????? containers:

????? - name: nginx

??????? image: nginx:1.17.1

??????? ports:

??????? - containerPort: 80



---



apiVersion: apps/v1

kind: Deployment

metadata:

? name: tomcat-deployment

? namespace: study

spec:

? replicas: 3

? selector:

??? matchLabels:

????? app: tomcat-pod

? template:

??? metadata:

????? labels:

??????? app: tomcat-pod

??? spec:

????? containers:

????? - name: tomcat

??????? image: tomcat:8.0.52

??????? ports:

??????? - containerPort: 8080



---



apiVersion: v1

kind: Service

metadata:

? name: nginx-service

? namespace: study

spec:

? selector:

??? app: nginx-pod

? clusterIP: None

? type: ClusterIP

? ports:

? - port: 80

??? targetPort: 80



---



apiVersion: v1

kind: Service

metadata:

? name: tomcat-service

? namespace: study

spec:

? selector:

??? app: tomcat-pod

? clusterIP: None

? type: ClusterIP

? ports:

? - port: 8080

??? targetPort: 8080

[root@master k8s]# kubectl apply -f ms.yaml

deployment.apps/nginx-deployment created

deployment.apps/tomcat-deployment created

service/nginx-service created

service/tomcat-service created

[root@master k8s]# kubectl get deploy,svc -n study

NAME??????????????????????????????? READY?? UP-TO-DATE?? AVAILABLE?? AGE

deployment.apps/nginx-deployment??? 3/3???? 3??????????? 3?????????? 69s

deployment.apps/tomcat-deployment?? 3/3???? 3??????????? 3?????????? 69s



NAME???????????????????? TYPE??????? CLUSTER-IP?? EXTERNAL-IP?? PORT(S)??? AGE

service/nginx-service??? ClusterIP?? None???????? <none>??????? 80/TCP???? 69s

service/tomcat-service?? ClusterIP?? None???????? <none>??????? 8080/TCP?? 69s

ingress yaml 详解

注意：

?service,ingress 需要在同一命名空间

?查看 ingressClassName

[root@master k8s]# kubectl get ingressclasses.networking.k8s.io

NAME??? CONTROLLER???????????? PARAMETERS?? AGE

nginx?? k8s.io/ingress-nginx?? <none>?????? 32m

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: ingress-http

? namespace: study

spec:

? ingressClassName: nginx # （1.20开始使用）

# defaultBackend: 处理不匹配的请求的后端,如果没有指定任何规则

? # 则必须指定默认后端如果没有设置默认后端处理请求不匹配任何规则将由入口控制器决定

? # service:

??? # name: nginx-service

??? # port:

???? #? number: 80

# tls: # 指定https 密钥证书

# - hosts: # 指定使用该密钥的规则

? # - nginx.xudaxian.com

? # - tomcat.xudaxian.com

? # secretName: tls-secret # 指定秘钥

???????????

? rules: # 规则指定请求匹配规则，每一个规则都是<[]Object>

? - host: www.snj.com #指定匹配域名，默认端口为80，443

??? http:

????? paths:

????? - path: / # 这里的路径指的host访问的路径，不同路径可以绑定不同的service

??????? pathType: Prefix # 匹配规则，Prefix 前缀匹配 it.nginx.com/* 都可以匹配到

??????? backend: # 后端的service

????????? service:

??????????? name: nginx-service # 绑定 service

??????????? port:

????????????? number: 80 # 指定service 端口

? - host: tomcat.snj.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: tomcat-service

??????????? port:

????????????? number: 8080

???????????????????????????

pathType

Prefix：基于以 / 分隔的 URL 路径前缀匹配。匹配区分大小写，并且对路径中的元素逐个完成。路径元素指的是由 / 分隔符分隔的路径中的标签列表。如果每个 p 都是请求路径 p 的元素前缀，则请求与路径 p 匹配。

Exact：精确匹配 URL 路径，且区分大小写。

ImplementationSpecific：对于这种路径类型，匹配方法取决于 IngressClass。具体实现可以将其作为单独的 pathType 处理或者与 Prefix 或 Exact 类型作相同处理。

[root@master k8s]# kubectl get ing -n study

NAME?????????? CLASS??? HOSTS????????????????????????? ADDRESS?? PORTS?? AGE

ingress-http?? <none>?? nginx.snj.com,tomcat.snj.com???????????? 80????? 12m



[root@master k8s]# kubectl get ingress -n? study -o wide

NAME?????????? CLASS?? HOSTS????????????????????????? ADDRESS?? PORTS?? AGE

ingress-http?? nginx?? nginx.snj.com,tomcat.snj.com???????????? 80????? 28s

[root@master k8s]# kubectl describe ingress -n? study

Name:???????????? ingress-http

Labels:?????????? <none>

Namespace:??????? study

Address:?????????

Ingress Class:??? nginx

Default backend:? <default>

Rules:

? Host??????????? Path? Backends

? ----??????????? ----? --------

? nginx.snj.com??

????????????????? /?? nginx-service:80 (10.244.104.28:80,10.244.166.153:80,10.244.166.155:80)

? tomcat.snj.com?

????????????????? /?? tomcat-service:80 (10.244.104.29:8080,10.244.104.34:8080,10.244.166.154:8080)

Annotations:????? <none>

Events:

? Type??? Reason? Age?? From????????????????????? Message

? ----??? ------? ----? ----????????????????????? -------

? Normal? Sync??? 33s?? nginx-ingress-controller? Scheduled for sync

# 有上面这个Normal 则绑定成功

测试（http实现）

[root@master k8s]# kubectl describe svc -n ingress-nginx ingress-nginx-controller

Name:???????????????????? ingress-nginx-controller

Namespace:??????????????? ingress-nginx

Labels:?????????????????? app.kubernetes.io/component=controller

????????????????????????? app.kubernetes.io/instance=ingress-nginx

????????????????????????? app.kubernetes.io/name=ingress-nginx

????????????????????????? app.kubernetes.io/part-of=ingress-nginx

????????????????????????? app.kubernetes.io/version=1.6.4

Annotations:????????????? <none>

Selector:???????????????? app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx

Type:???????????????????? NodePort

IP Family Policy:???????? SingleStack

IP Families:??????????? ??IPv4

IP:?????????????????????? 10.111.123.202

IPs:????????????????????? 10.111.123.202

Port:???????????????????? http? 80/TCP

TargetPort:?????????????? http/TCP

NodePort:???????????????? http? 32172/TCP

Endpoints:??????????????? 192.168.100.51:80 # 外部访问ingress的IP地址

Port:???????????????????? https? 443/TCP

TargetPort:?????????????? https/TCP

NodePort:???????????????? https? 31755/TCP

Endpoints:??????????????? 192.168.100.51:443 # 外部访问ingress的IP地址

Session Affinity:???????? None

External Traffic Policy:? Local

Events:?????????????????? <none>

出现的问题：集群内部都能也可以从自己的地址访问到ingress，hosts对应写自身地址

因为搭建ingress-nginx的service的时候，会向ipvs写入规则（访问ip为自身地址）

Prot LocalAddress:Port Scheduler Flags

? -> RemoteAddress:Port?????????? Forward Weight ActiveConn InActConn

TCP? 192.168.100.53:31755 rr

? -> 192.168.100.51:443?????????? Masq??? 1????? 0????????? 0????????

TCP? 192.168.100.53:32172 rr

? -> 192.168.100.51:80??????????? Masq??? 1????? 0????????? 0??

但是外部不能通过master物理地址访问到ingress，外部要通过ingress pod 部署到的node的物理地址来访问

写入hosts文件

192.168.100.51 nginx.snj.com

192.168.100.51 tomcat.snj.com

查看 ingress-nginx 开放端口

[root@master k8s]# kubectl get svc? -n ingress-nginx

NAME???????????????????????????????? TYPE??????? CLUSTER-IP????? EXTERNAL-IP?? PORT(S)????????????????????? AGE

ingress-nginx-controller???????????? NodePort??? 10.102.42.91??? <none>??????? 80:32310/TCP,443:30777/TCP?? 4m29s

ingress-nginx-controller-admission?? ClusterIP?? 10.96.129.195?? <none>??????? 443/TCP????????????????????? 4m29s

# http端口为 32310 https端口为 30777

访问

https 实现（ingress 证书）

生成证书：

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/C=CN/ST=BJ/L=BJ/O=nginx/CN=xudaxian.com"

[root@master k8s]# ls | grep tls

tls.crt

tls.key

创建密钥

[root@master k8s]# kubectl create secret tls tls-secret --key tls.key --cert tls.crt

secret/tls-secret created

环境

[root@master k8s]# kubectl get pod,svc,deploy -n study

NAME???????????????????????????????????? READY?? STATUS??? RESTARTS?? AGE

pod/nginx-deployment-6bb9d9f778-6dnhh??? 1/1???? Running?? 0????????? 86m

pod/nginx-deployment-6bb9d9f778-nfzk4??? 1/1???? Running?? 0????????? 86m

pod/nginx-deployment-6bb9d9f778-rl7nr??? 1/1???? Running?? 0????????? 86m

pod/tomcat-deployment-84748d94d5-dgmvh?? 1/1???? Running?? 0????????? 86m

pod/tomcat-deployment-84748d94d5-nczhr?? 1/1???? Running?? 0????????? 86m

pod/tomcat-deployment-84748d94d5-qbp59?? 1/1???? Running?? 0????????? 86m



NAME???????????????????? TYPE??????? CLUSTER-IP?? EXTERNAL-IP?? PORT(S)??? AGE

service/nginx-service??? ClusterIP?? None???????? <none>??????? 80/TCP???? 86m

service/tomcat-service?? ClusterIP?? None???????? <none>??????? 8080/TCP?? 86m



NAME??????????????????????????????? READY?? UP-TO-DATE?? AVAILABLE?? AGE

deployment.apps/nginx-deployment??? 3/3???? 3??????????? 3?????????? 86m

deployment.apps/tomcat-deployment?? 3/3???? 3??????????? 3?????????? 86m

ingress yaml

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: ingress-http

? namespace: study

spec:

? ingressClassName: nginx # （1.20开始使用）





? tls: # 指定https 密钥证书

? - hosts: # 指定使用该密钥的规则

??? - nginx.xudaxian.com

??? - tomcat.xudaxian.com

??? secretName: tls-secret # 指定秘钥???????????

? rules: # 规则指定请求匹配规则，每一个规则都是<[]Object>

? - host: nginx.snj.com #指定匹配域名，默认端口为80，443

??? http:

????? paths:

????? - path: / # 这里的路径指的host访问的路径，不同路径可以绑定不同的service

??????? pathType: Prefix

??????? backend: #后端的service

????????? service:

??????????? name: nginx-service # 绑定 service

??????????? port:

????????????? number: 80 # 指定service 端口

? - host: tomcat.snj.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: tomcat-service

??????????? port:

????????????? number: 8080?????????????

[root@master k8s]# kubectl apply -f ingress.yaml

ingress.networking.k8s.io/ingress-http created

[root@master k8s]# kubectl get ing -n study

NAME?????????? CLASS?? HOSTS??????????????????????? ADDRESS?? PORTS???? AGE

ingress-http?? nginx?? www.snj.com,tomcat.snj.com???????????? 80, 443?? 8s

[root@master k8s]# kubectl describe ing -n study

Name:???????????? ingress-http

Labels:?????????? <none>

Namespace:??????? study

Address:?????????

Ingress Class:??? nginx

Default backend:? <default>

TLS:

? tls-secret terminates nginx.xudaxian.com,tomcat.xudaxian.com

Rules:

? Host??????????? Path? Backends

? ----??????????? ----? --------

? www.snj.com????

????????????????? /?? nginx-service:80 (10.244.104.38:80,10.244.166.165:80,10.244.166.166:80)

? tomcat.snj.com?

????????????????? /?? tomcat-service:8080 (10.244.104.39:8080,10.244.104.40:8080,10.244.166.167:8080)

Annotations:????? <none>

Events:

? Type??? Reason? Age?? From????????????????????? Message

? ----??? ------? ----? ----????????????????????? -------

? Normal? Sync??? 38s?? nginx-ingress-controller? Scheduled for sync



?[root@master k8s]# kubectl get svc -n ingress-nginx

NAME???????????????????????????????? TYPE??????? CLUSTER-IP?????? EXTERNAL-IP?? PORT(S)????????????????????? AGE

ingress-nginx-controller???????????? NodePort??? 10.111.123.202?? <none>??????? 80:32172/TCP,443:31755/TCP?? 92m

ingress-nginx-controller-admission?? ClusterIP?? 10.106.250.140?? <none>??????? 443/TCP????????????????????? 92m

# 访问端口 443:31755/TCP

curl -k https://nginx.snj.com:31755

单域名多路径

访问时跳转指定地址

不使用域名，使用ip

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: ingress-v1

? namespace: study

spec:

? ingressClassName: "nginx"

? rules:



? - http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: v1-service

??????????? port:

????????????? number: 80

[root@master canary]# curl 10.111.123.202

v1-pod

限流操作

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: rate-ingress

? namespace: default

? annotations: # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rate-limiting

??? kubernetes.io/ingress.class: "nginx"

??? nginx.ingress.kubernetes.io/backend-protocol: "HTTP"?

??? nginx.ingress.kubernetes.io/limit-rps: "1" # 限流

spec:

? rules:

? - host: nginx.xudaxian.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: nginx-svc

??????????? port:

????????????? number: 80

基于 Cookie 的会话保持技术

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: rate-ingress

? namespace: default

? annotations: # https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#rate-limiting

??? kubernetes.io/ingress.class: "nginx"

??? nginx.ingress.kubernetes.io/backend-protocol: "HTTP"?

??? nginx.ingress.kubernetes.io/affinity: "cookie"

spec:

? rules:

? - host: nginx.xudaxian.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: nginx-svc

??????????? port:

????????????? number: 80

其他示例

Ingress | Kubernetes

出现问题

IngressClass

kubectl logs -n ingress-nginx ingress-nginx-controller-cf4967654-fkwcl

?# 进行排错

?# 比如出现一下错误

?# W0102 12:51:07.374785?????? 7 controller.go:278] ignoring ingress nginx-ingress in study based on annotation : ingress does not contain a valid IngressClass

?# 说明没有配置 ingress class

?

??

? I0102 13:53:52.597216?????? 7 main.go:100] "successfully validated configuration, accepting" ingress="study/nginx-ingress"

I0102 13:53:52.604268?????? 7 store.go:433] "Found valid IngressClass" ingress="study/nginx-ingress" ingressclass="nginx"

I0102 13:53:52.614011?????? 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"study", Name:"nginx-ingress", UID:"0f72d9c8-4b9c-4dee-afec-e5a7d213323e", APIVersion:"networking.k8s.io/v1", ResourceVersion:"8126946", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

I0102 13:53:53.128640?????? 7 controller.go:188] "Configuration changes detected, backend reload required"

I0102 13:53:53.246681?????? 7 controller.go:205] "Backend successfully reloaded"

I0102 13:53:53.254908?????? 7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-cf4967654-fkwcl", UID:"a50e3220-3928-4fac-bdd6-b5fe5409be00", APIVersion:"v1", ResourceVersion:"8047653", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

I0102 13:54:14.813365?????? 7 status.go:300] "updating Ingress status" namespace="study" ingress="nginx-ingress" currentValue=[] newValue=[{IP:10.111.123.202 Hostname: Ports:[]}]

I0102 13:54:14.817935?????? 7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"study", Name:"nginx-ingress", UID:"0f72d9c8-4b9c-4dee-afec-e5a7d213323e", APIVersion:"networking.k8s.io/v1", ResourceVersion:"8126996", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

[root@master k8s]# kubectl get ingress -n study

NAME??????????? CLASS?? HOSTS?? ADDRESS????????? PORTS?? AGE

nginx-ingress?? nginx?? *?????? 10.111.123.202?? 80????? 65s

master 访问问题

因为master没有ingress-nginx-controller控制器

外部不能通过master物理地址访问到ingress，外部要通过ingress pod 部署到的node的物理地址来访问

灰度发布/蓝绿发布/金丝雀发布

概述

以前使用 Kubernetes 的 Service 配合 Deployment 进行金丝雀的部署，原理如下所示：

但是，也是有缺点的，就是不能自定义灰度逻辑，如：指定用户进行灰度（新功能只让 VIP 先体验一个月，一个月之后再将新功能解锁，让所有用户都体验到）。
Ingress 金丝雀发布的原理如下所示：

使用canary 和 service 来实现

canary? - 监绿和灰度发布而社区版本

service - 蓝绿和灰度发布云端版本（阿里云）

# 两个配置没有一点区别,只需要更换名字即可使用

两种实现一个 header或cookie ，权重

基于Request Headerf的流量切分，适用于灰度发布及AB测试

基于Cookie的流量切分，适用于灰度发布及AB测试

基于权重的流量气氛，适用于蓝绿发布

环境：已经部署了一个old nginx的ingress以及后端的deployment service资源

现在部署完了new nginx 的deployment service资源，要部署一个实现蓝绿发布的ingress（同一个host 域名）

环境部署

deployment service

部署 v1 deployment service 和 v2 deployment service

apiVersion: v1

kind: Namespace

metadata:

? name: study

---

apiVersion: apps/v1

kind: Deployment

metadata:

? name: v1-deployment

? labels:

??? app: v1-deployment

? namespace: study

spec:

? replicas: 2

? selector:

??? matchLabels:

????? app: v1-pod

? template:

??? metadata:

????? name: nginx

????? labels:

??????? app: v1-pod

??? spec:

????? initContainers:

??????? - name: busybox

????????? image: busybox:1.30

????????? imagePullPolicy: IfNotPresent

????????? command: ["/bin/sh","-c","echo v2-pod > /app/index.html;"]

????????? volumeMounts:

??????????? - mountPath: /app

????????????? name: app

????? containers:

??????? - name: nginx

????????? image: nginx:1.17.1

????????? imagePullPolicy: IfNotPresent

????????? ports:

??????????? - containerPort: 80

????????? resources:

??????????? requests:

????????????? cpu: 100m

????????????? memory: 100Mi

??????????? limits:

????????????? cpu: 250m

????????????? memory: 500Mi

????????? volumeMounts:

??????????? - name: app

????????????? mountPath: /usr/share/nginx/html????????????

????? volumes:

??????? - name: app

????????? emptyDir: {}??????

????? restartPolicy: Always

---

apiVersion: v1

kind: Service

metadata:

? name: v1-service

? namespace: study

spec:

? selector:

??? app: v1-pod

? type: ClusterIP

? ports:

? - name: nginx

??? protocol: TCP

??? port: 80

??? targetPort: 80

---

apiVersion: apps/v1

kind: Deployment

metadata:

? name: v2-deployment

? labels:

??? app: v2-deployment

? namespace: study

spec:

? replicas: 3

? selector:

??? matchLabels:

????? app: v2-pod

? template:

??? metadata:

????? name: v2-pod

????? labels:

??????? app: v2-pod

??? spec:

????? initContainers:

??????? - name: busybox

????????? image: busybox:1.30

????????? imagePullPolicy: IfNotPresent

????????? command: ["/bin/sh","-c","echo v2-pod > /app/index.html;"]

????????? volumeMounts:

??????????? - mountPath: /app

????????????? name: app

????? containers:

??????? - name: nginx

????????? image: nginx:1.17.2

????????? imagePullPolicy: IfNotPresent

????????? ports:

??????????? - containerPort: 80

????????? resources:

??????????? requests:

????????????? cpu: 100m

????????????? memory: 100Mi

??????????? limits:

????????????? cpu: 250m

????????????? memory: 500Mi

????????? volumeMounts:

??????????? - name: app

????????????? mountPath: /usr/share/nginx/html

????? volumes:

??????? - name: app

????????? emptyDir: {}

????? restartPolicy: Always

---

apiVersion: v1

kind: Service

metadata:

? name: v2-service

? namespace: study

spec:

? selector:

??? app: v2-pod

? type: ClusterIP

? ports:

? - name: nginx

??? protocol: TCP

??? port: 80

??? targetPort: 80

v1-ingress

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: ingress-v1

? namespace: default

spec:

? ingressClassName: "nginx"

? rules:

? - host: nginx.xudaxian.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: v1-service

??????????? port:

????????????? number: 80

检查

[root@master canary]# kubectl get deploy,svc,pod,ing -n study

NAME??????????????????????????? READY?? UP-TO-DATE?? AVAILABLE?? AGE

deployment.apps/v1-deployment?? 2/2???? 2??????????? 2?????????? 9m1s

deployment.apps/v2-deployment?? 3/3???? 3??????????? 3?????????? 9m1s



NAME???????????????? TYPE??????? CLUSTER-IP????? EXTERNAL-IP?? PORT(S)?? AGE

service/v1-service?? ClusterIP?? 10.100.50.208?? <none>??????? 80/TCP??? 9m1s

service/v2-service?? ClusterIP?? 10.107.230.80?? <none>??????? 80/TCP??? 9m1s



NAME???????????????????????????????? READY?? STATUS??? RESTARTS?? AGE

pod/v1-deployment-7c455c58d8-6njc2?? 1/1???? Running?? 0????????? 9m1s

pod/v1-deployment-7c455c58d8-ngkq2?? 1/1???? Running?? 0????????? 9m1s

pod/v2-deployment-bc9fc6679-6lgjf??? 1/1???? Running?? 0????????? 9m1s

pod/v2-deployment-bc9fc6679-n8b49??? 1/1???? Running?? 0????????? 9m1s

pod/v2-deployment-bc9fc6679-p86ln??? 1/1???? Running?? 0????????? 9m1s



NAME?????????????????????????????????? CLASS?? HOSTS?????????? ADDRESS????????? PORTS?? AGE

ingress.networking.k8s.io/ingress-v1?? nginx?? nginx.snj.com?? 10.111.123.202?? 80????? 83s

[root@ip-15 ~]# curl nginx.snj.com

v1-pod

v2-ingress

基于 Header 的流量切分

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: v2-ingress-canary-header

? namespace: study

? annotations:

??? nginx.ingress.kubernetes.io/canary: "true" # 开启金丝雀

??? nginx.ingress.kubernetes.io/canary-by-header: "Region" # 基于请求头

??? # 如果 请求头 Region = always ，就路由到金丝雀版本；如果 Region = never ，就永远不会路由到金丝雀版本。

??? # canary-by-header-value 默认 是always和 nerver

??? nginx.ingress.kubernetes.io/canary-by-header-value: "sz" # 自定义值

??? # 如果 请求头 Region = sz ，就路由到金丝雀版本；如果 Region != sz ，就永远不会路由到金丝雀版本。

??? # nginx.ingress.kubernetes.io/canary-by-header-pattern: "sh|sz"

??? # 如果 请求头 Region = sh 或 Region = sz ，就路由到金丝雀版本；如果 Region != sz 并且 Region != sz ，就永远不会路由到金丝雀版本。

spec:

? ingressClassName: "nginx"

? rules:

? - host: nginx.snj.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: v2-service

??????????? port:

????????????? number: 80

[root@ip-15 ~]# curl nginx.snj.com

v1-pod

[root@ip-15 ~]# curl -H "Region: sz" nginx.snj.com

v2-pod
# 此时进行测试，如负载等测试，如果没问题即可直接根据下面的完全上线新版本

基于 Cookie 的流量切分

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: ingress-canary-cookie

? namespace: study

? annotations:

??? nginx.ingress.kubernetes.io/canary: "true" # 开启金丝雀

??? nginx.ingress.kubernetes.io/canary-by-cookie: "vip"

??? # 如果 cookie 是 vip = always ，就会路由到到金丝雀版本

??? # 如果 cookie 是 vip = never ，就永远不会路由到金丝雀的版本。

spec:

? ingressClassName: "nginx"

? rules:

? - host: nginx.snj.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: v2-service

??????????? port:

????????????? number: 80

[root@ip-15 ~]# curl nginx.snj.com

v1-pod

[root@ip-15 ~]# curl --cookie "vip=always" nginx.snj.com

v2-pod

# 此时进行测试，如负载等测试，如果没问题即可直接根据下面的完全上线新版本

基于服务权重的流量切分

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

? name: ingress-canary-weight

? namespace: study

? annotations:

??? nginx.ingress.kubernetes.io/canary: "true" # 开启金丝雀

??? nginx.ingress.kubernetes.io/canary-weight: "10" # 基于服务权重

spec:

? ingressClassName: "nginx"

? rules:

? - host: nginx.snj.com

??? http:

????? paths:

????? - path: /

??????? pathType: Prefix

??????? backend:

????????? service:

??????????? name: v2-service

??????????? port:

????????????? number: 80

[root@master canary]# kubectl get deploy,svc,pod,ing -n study

NAME??????????????????????????? READY?? UP-TO-DATE?? AVAILABLE?? AGE

deployment.apps/v1-deployment?? 2/2???? 2??????????? 2?????????? 28m

deployment.apps/v2-deployment?? 3/3???? 3??????????? 3?????????? 28m



NAME???????????????? TYPE??????? CLUSTER-IP?????? EXTERNAL-IP?? PORT(S)?? AGE

service/v1-service?? ClusterIP?? 10.107.153.174?? <none>??????? 80/TCP??? 28m

service/v2-service?? ClusterIP?? 10.97.36.128???? <none>??????? 80/TCP??? 28m



NAME???????????????????????????????? READY?? STATUS??? RESTARTS?? AGE

pod/v1-deployment-7c455c58d8-68k9r?? 1/1???? Running?? 0????????? 28m

pod/v1-deployment-7c455c58d8-kdmgl?? 1/1???? Running?? 0????????? 28m

pod/v2-deployment-568ff74948-cs6dr?? 1/1???? Running?? 0????????? 28m

pod/v2-deployment-568ff74948-ptsdx?? 1/1???? Running?? 0????????? 28m

pod/v2-deployment-568ff74948-rpql8?? 1/1???? Running?? 0????????? 28m



NAME????????????????????????????????????????????? CLASS?? HOSTS?????????? ADDRESS????????? PORTS?? AGE

ingress.networking.k8s.io/ingress-canary-weight?? nginx?? nginx.snj.com?? 10.111.123.202?? 80????? 30s

ingress.networking.k8s.io/ingress-v1????????????? nginx?? nginx.snj.com?? 10.111.123.202?? 80????? 27m



[root@ip-15 ~]# for i in {1..10}; do curl nginx.snj.com; done;

v1-pod

v1-pod

v1-pod

v1-pod

v1-pod

v2-pod

v1-pod

v1-pod

v1-pod

v1-pod

完全上线新版本

改ingress的后端service

[root@master canary]# kubectl edit ingress -n study ingress-v1

ingress.networking.k8s.io/ingress-v1 edited



????? - backend:

????????? service:

??????????? name: v2-service

??????????? port:

????????????? number: 80

改service标签选择器

直接就老版本的service的标签选择器指向新版本的deploy的标签

首先在网络中使用的http协议进行传输，http协议是一个明文传输协议

此时为了安全，我们就需要使用https加密传输协议，但是需要对 ingress 进行证书配置

文章来源:https://blog.csdn.net/weixin_48297181/article/details/135458196
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：chenni525@qq.com进行投诉反馈，一经查实，立即删除！

kubernetes ingress 详解 （包含灰度发布/金丝雀部署）

ingress 介绍 架构

原理

架构图