阿里云ingress配置时间超时的参数

一、背景

在使用阿里云k8s集群的时候，内网API网关，刚开始是用的是Nginx，后面又搭建了ingress。
区别于nginx配置，ingress又该怎么设置参数呢？比如http超时时间等等。

本文会先梳理nginx是如何配置，再对比ingress的配置方式。
示例以超时时间的设置。

二、nginx配置

在k8s部署两个节点的Nginx容器

      containers:
        - env:
            - name: aliyun_logs_nginx-log
              value: /var/log/nginx/*.log
          image: nginx
          imagePullPolicy: Always
          name: xh-nginx
          ports:
            - containerPort: 80
              protocol: TCP
          resources:
            limits:
              cpu: '2'
              memory: 4Gi
            requests:
              cpu: 250m
              memory: 2Gi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /etc/nginx/nginx.conf
              name: nginx
              subPath: nginx.conf
            - mountPath: /etc/nginx/conf.d
              name: nginx-cm
            - mountPath: /var/log/nginx/
              name: volume-k8s-inner-nginx-log
      volumes:
        - configMap:
            defaultMode: 420
            items:
              - key: nginx.conf
                path: nginx.conf
            name: nginx-conf
          name: nginx
        - configMap:
            defaultMode: 420
            name: nginx-cm
          name: nginx-cm
        - hostPath:
            path: /var/log/nginx
            type: Directory
          name: volume-k8s-inner-nginx-log
        - emptyDir: {}
          name: volumn-sls-16578614717160

这里把/etc/nginx/nginx.conf和下面的/etc/nginx/conf.d/*.conf分别挂载到configMap

1、nginx-conf下的新增了一个子项nginx.conf

对应容器里的文件/etc/nginx/nginx.conf

详情见下：

user  nginx;
worker_processes  auto;

worker_cpu_affinity auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

worker_rlimit_nofile 10240;

events {
    use epoll;
    worker_connections  10240;
}


http {
    underscores_in_headers on;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# 传递http header值
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
# 设置log格式
    log_format  access '$proxy_add_x_forwarded_for $time_local $request $request_time "$upstream_response_time" '
                  '$status $body_bytes_sent $host "$http_user_agent" $bytes_sent $request_length "$upstream_addr" ';

    access_log  /var/log/nginx/access.log  access;

    charset  utf-8;

    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
    client_max_body_size 500m;

    sendfile       on;
    tcp_nopush     on;
    tcp_nodelay    on;

    keepalive_timeout  600;
    server {
        listen       80;
        server_name  nginx_status;
        location /ngx_status {
        stub_status;
                          }
            }
    fastcgi_connect_timeout 600;
    fastcgi_send_timeout 600;
    fastcgi_read_timeout 600;
    fastcgi_buffer_size 64k;
    fastcgi_buffers 4 64k;
    fastcgi_busy_buffers_size 128k;
    fastcgi_temp_file_write_size 128k;

    include /etc/nginx/conf.d/*.conf;
    }        

2、nginx-cm

对应容器里的文件/etc/nginx/conf.d/*.conf

下面以常见的用户服务为示例：

upstream user-service-cloud-cluster {
  server 172.16.17.9:8081 weight=50 max_fails=2 fail_timeout=10s;
}
server
{
  listen       80;
  server_name  user.xxx.cloud;
  location / {
     proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
     proxy_pass http://user-service-cloud-cluster;
     proxy_redirect off;
     proxy_set_header Host $host;
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header HTTP_HOST $host;
     proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
     proxy_set_header HTTP_X_FORWARDED_HOST $host;
     proxy_set_header X-Forwarded-Host $host;
     proxy_set_header X-Forwarded-Server $host;
     proxy_set_header X-Forwarded-HTTPS 0;
  }
  access_log  /var/log/nginx/user-service_cloud_access.log  access;
  error_log  /var/log/nginx/user-service_cloud_error.log;
 }

3、小节

当你修改了nginx的配置，别忘记了进入Nginx容器进行reload，以使配置生效。

nginx -s reload

三、ingress配置

除了已知的一些区别，它和Nginx的一个最大不同是，不用手动去reload才能让配置生效。

同样部署两个ingress节点

建议你使用Helm安装ingress，简单方便。具体就不在本文赘述了。

下面再看下它的yaml详情：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-ack-ingress-nginx-v1-controller
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: nginx-ingress
      app.kubernetes.io/name: ack-ingress-nginx-v1
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: nginx-ingress
        app.kubernetes.io/name: ack-ingress-nginx-v1
    spec:
      containers:
        - args:
            - /nginx-ingress-controller
            - >-
              --publish-service=$(POD_NAMESPACE)/nginx-ingress-ack-ingress-nginx-v1-controller-internal
            - '--election-id=ingress-controller-leader-ack-nginx'
            - '--controller-class=k8s.io/ack-ingress-nginx'
            - '--ingress-class=ack-nginx'
            - >-
              --configmap=$(POD_NAMESPACE)/nginx-ingress-ack-ingress-nginx-v1-controller
            - '--validating-webhook=:8443'
            - '--validating-webhook-certificate=/usr/local/certificates/cert'
            - '--validating-webhook-key=/usr/local/certificates/key'
            - '--v=2'
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.namespace
            - name: LD_PRELOAD
              value: /usr/local/lib/libmimalloc.so
          image: >-
            registry-vpc.cn-hangzhou.aliyuncs.com/acs/aliyun-ingress-controller:v1.8.0-aliyun.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          livenessProbe:
            failureThreshold: 5
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          name: controller
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
            - containerPort: 443
              name: https
              protocol: TCP
            - containerPort: 8443
              name: webhook
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - NET_BIND_SERVICE
              drop:
                - ALL
            runAsUser: 101
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /usr/local/certificates/
              name: webhook-cert
              readOnly: true
            - mountPath: /etc/localtime
              name: localtime
              readOnly: true
      dnsPolicy: ClusterFirst
      initContainers:
        - command:
            - /bin/sh
            - '-c'
            - |
              if [ "$POD_IP" != "$HOST_IP" ]; then
              mount -o remount rw /proc/sys
              sysctl -w net.core.somaxconn=65535
              sysctl -w net.ipv4.ip_local_port_range="1024 65535"
              sysctl -w kernel.core_uses_pid=0
              fi
          env:
            - name: POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.hostIP
          image: 'registry-vpc.cn-hangzhou.aliyuncs.com/acs/busybox:v1.29.2'
          imagePullPolicy: IfNotPresent
          name: init-sysctl
          resources: {}
          securityContext:
            capabilities:
              add:
                - SYS_ADMIN
              drop:
                - ALL
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress-ack-ingress-nginx-v1
      serviceAccountName: nginx-ingress-ack-ingress-nginx-v1
      terminationGracePeriodSeconds: 300
      tolerations:
        - effect: NoSchedule
          key: node-role.alibabacloud.com/addon
          operator: Exists
      volumes:
        - name: webhook-cert
          secret:
            defaultMode: 420
            secretName: nginx-ingress-ack-ingress-nginx-v1-admission
        - hostPath:
            path: /etc/localtime
            type: File
          name: localtime

这里使用了一个初始化容器initContainers，它会对系统做一个个性化配置。

sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
sysctl -w kernel.core_uses_pid=0

其次，HOST_IP和POD_IP都从K8s环境变量中读取，因为它们是动态的，非固定不变。

必要的健康检测，配置了livenessProbe和readinessProbe，详情见上。

1、configMap配置

日志格式，见下：

其他的配置这里就不一一列举，总之，它支持你通过变量进行配置就行。

它就对应上文的nginx.conf文件。

2、创建Ingress路由

操作比较简单，下面要切入到本文的重点。

四、Ingress设置超时时间

要说Ingress如何设置超时时间前，先看一看nginx是如何设置。

默认是60秒，现在业务上有需求调整为600秒。
请看下文的具体配置：

1、nginx配置

upstream xxx-cloud-cluster {
  server 172.16.17.6:8080 weight=9 max_fails=2 fail_timeout=10s;
}
server
{
  listen       80;
  server_name  image-xxx.xx.cloud;
  location / {
     proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
     proxy_pass http://xxx-cloud-cluster;
     proxy_redirect off;
     proxy_set_header Host $host;
     # 增加下面三行
     proxy_connect_timeout 600;
     proxy_send_timeout 600;
     proxy_read_timeout 600;
     
     proxy_set_header X-Real-IP $remote_addr;
     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
     proxy_set_header HTTP_HOST $host;
     proxy_set_header HTTP_X_FORWARDED_FOR $remote_addr;
     proxy_set_header HTTP_X_FORWARDED_HOST $host;
     proxy_set_header X-Forwarded-Host $host;
     proxy_set_header X-Forwarded-Server $host;
     proxy_set_header X-Forwarded-HTTPS 0;
  }
  access_log  /var/log/nginx/xxx_access.log  access;
  error_log  /var/log/nginx/xxx_error.log;
 }

2、ingress配置

参数设置通过注解配置：
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;

yaml详情见下：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-connect-timeout: '600'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '600'
  labels:
    ingress-controller: nginx
  name: image-xxx
  namespace: java-service
spec:
  ingressClassName: ack-nginx
  rules:
    - host: image.xxx.cloud
      http:
        paths:
          - backend:
              service:
                name: image-xxx
                port:
                  number: 8080
            path: /
            pathType: ImplementationSpecific

五、总结

这里只是以设置超时时间为例，讲述k8s容器部署的Nginx和ingress，如何设置一定自定义的参数配置。

当然，这里没有讲述怎么安装它们，更多的是梳理了一下如何配置，侧重于使用这块。