IaC基础设施即代码：Terraform 创建ACK集群 与部署应用

发布时间：2024年01月23日

目录

?一、实验

1.环境

2.Terraform 创建网络资源

3. 阿里云给RAM添加权限

4.Terraform 创建 ACK集群

5.在ACK集群中部署应用

6.销毁资源

二、问题

1.Terraform 验证失败

2.Terraform申请资源失败

?

?

?

?一、实验

1.环境

（1）主机

表1-1 主机

主机系统软件工具备注
jia

Windows?

Terraform 1.6.6VS Code、?PowerShell、?Chocolatey?

?

2.Terraform 创建网络资源

（1）查看项目

9ab54cbcdc0a4aed9008074caec794d0.png

（2）网络配置文件

network.tf?

//VPC 专有网络
resource "alicloud_vpc" "vpc" {
  vpc_name   = "k8s_vpc"
  cidr_block = "172.16.0.0/12"
}

//switch 交换机
resource "alicloud_vswitch" "vsw" {
  vpc_id     = alicloud_vpc.vpc.id
  cidr_block = "172.16.0.0/16"
  zone_id    = "cn-hangzhou-j"
}

?

06a5289d858147ab8a2f7c5425ff31d7.png

(3) 版本配置文件

versions.tf

terraform {
  required_providers {
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.214.1"
    }
  }
}

# Configure the Alicloud Provider 默认供应商
provider "alicloud" {
  access_key = var.access_key
  secret_key = var.secret_key
  region     = "cn-hangzhou"
}

49ca244d623c406fb44ad147a60d5e4f.png

（4）变量配置文件

variables.tf

variable "access_key" {
  description = "access_key"

}

variable "secret_key" {
  description = "secret_key"
}

e55d43062af947219e90bad52e375ab0.png

?

(5) 密钥配置文件

terraform.tfvars

?

62801c8bf0df4c3594cfedd4b723a1ea.png

（6）初始化

terraform init

?

d99663c94fc8471aadf9d9183696334c.png

（7）格式化代码

terraform fmt

?

f8ea133b87ed4b669a1b6fb48495d671.png

（8）验证代码

terraform validate

?

8a696b4d4e894d8bbe13e70f21938809.png

（9）计划与预览

 terraform plan

0c2c279e6f5244ab82281506b3025768.png

?

（10）申请资源

terraform apply

a4d668221043449aadf61c97f8ce107e.png

?

（11）登录阿里云系统查看VPC

VPC已新增1个 （cn-hangzhou）

d8e06e53dfe54090b4d34160f39d26fb.png

?

交换机已新增1个 （cn-hangzhou）

?

acaf993301104456b7ab2c59d93b3d8c.png

?

3. 阿里云给RAM添加权限

（1）AliyunCSFullAcess

b305f07f47e746ae81a0826e34a2d019.png

b59bce833dc84be8aa7a428d42687ff5.png

（2）AliyunApiGatewayFullAcess

f4e724593d3d479e9569976d117eae8c.png

918dc98145b34cafb8e751a3e51312af.png

（3）NATGatewayFullAcess

9fafd458eb4240e5a93ea81f98787e1a.png

279074f1916440a99fbb30900d883c2a.png

?

4.Terraform 创建 ACK集群

（1）查看alicloud provider 示例

Terraform Registry

托管版K8S 示例

……

resource "alicloud_cs_managed_kubernetes" "k8s" {
  name         = var.name
  cluster_spec = "ack.pro.small"
  # version can not be defined in variables.tf.
  version            = "1.26.3-aliyun.1"
  worker_vswitch_ids = length(var.vswitch_ids) > 0 ? split(",", join(",", var.vswitch_ids)) : length(var.vswitch_cidrs) < 1 ? [] : split(",", join(",", alicloud_vswitch.vswitches.*.id))
  pod_vswitch_ids    = length(var.terway_vswitch_ids) > 0 ? split(",", join(",", var.terway_vswitch_ids)) : length(var.terway_vswitch_cidrs) < 1 ? [] : split(",", join(",", alicloud_vswitch.terway_vswitches.*.id))
  new_nat_gateway    = true
  node_cidr_mask     = var.node_cidr_mask
  proxy_mode         = var.proxy_mode
  service_cidr       = var.service_cidr

  dynamic "addons" {
    for_each = var.cluster_addons
    content {
      name   = lookup(addons.value, "name", var.cluster_addons)
      config = lookup(addons.value, "config", var.cluster_addons)
    }
  }
}

867214271df148639049e7276f578535.png

(2) 修改主配置文件

main.tf

locals {
  cluster_version = "1.26.3-aliyun.1"
  service_cidr    = "192.168.0.0/16"
  pod_cidr        = "10.212.0.0/16"
}

resource "alicloud_cs_managed_kubernetes" "k8s" {
  name                 = var.cluster_name
  version              = local.cluster_version  
  cluster_spec         = "ack.standard"
  worker_vswitch_ids   = [alicloud_vswitch.vsw.id] 
  new_nat_gateway      = true
  pod_cidr             = local.service_cidr
  service_cidr         = local.pod_cidr
  load_balancer_spec   = "slb.s1.small"
  slb_internet_enabled = true

  dynamic "addons" {
    for_each = var.cluster_addons
    content {
      name   = lookup(addons.value, "name", var.cluster_addons)
      config = lookup(addons.value, "config", var.cluster_addons)
    }
  }
}

resource "alicloud_cs_kubernetes_node_pool" "default" {
  name                 = var.nodepool_name
  cluster_id           = alicloud_cs_managed_kubernetes.k8s.id
  vswitch_ids          = [alicloud_vswitch.vsw.id]
  instance_types       = ["ecs.g6.xlarge"]
  system_disk_category = "cloud_efficiency"
  system_disk_size     = 40
  desired_size         = 1
  password             = "Admin@123"
  runtime_name         = "containerd"
  runtime_version      = "1.6.20"
}

496c30d4126a435fb2d114d8cfc0805a.png

(3) 修改变量配置文件

variables.tf

variable "access_key" {
  description = "access_key"

}

variable "secret_key" {
  description = "secret_key"
}

variable "cluster_name" {
  default = "k8s_cluster_01"
}

variable "nodepool_name" {
  default = "k8s-nodepool"
}

variable "cluster_addons" {
  type = list(object({
    name   = string
    config = string
  }))

  default = [
    {
      "name"   = "flannel",
      "config" = "",
    },
    {
      "name"   = "csi-plugin",
      "config" = "",
    },
    {
      "name"   = "csi-provisioner",
      "config" = "",
    },
    {
      "name"   = "logtail-ds",
      "config" = "{'IngressDashboardEnabled':'true'}",
    },
    {
      "name"   = "nginx-ingress-controller",
      "config" = "{'IngressSlbNetworkType':'internet'}",
    },
    {
      "name"   = "arms-prometheus",
      "config" = "",
    },
    {
      "name"   = "ack-node-problem-detector",
      "config" = "{'sls_project_name':''}",
    }
  ]
}

f9e0deeedcef46dcbf63e79c8510516e.png

(4)??验证代码

terraform validate

?

a015185b62de40649647df5a496982d8.png

(5)?计划与预览

 terraform plan

?

1e50ea9cff4c4e98bf802aa38121bad3.png02288a9be8ff4b65b20428cb443741c2.png12fa61981e6647aebecd0a3d98102d6c.png

(6)申请资源

terraform apply

01a5109d97fd42aaae6c628637cbe848.png1cfb34e02aa04d7fac3defc04e6a9cac.png810e5004b89340f6b2e79a855b2d951e.png

yes ,用时大约6分钟

a2d3c9ba2bb648eca84c2cfcbf82953d.png

?

(7) 登录阿里云系统查看ACK集群

初始化中

0c7a956196ef4e14bebfdb61bf8bf016.png

运行中

005888b3894e47cd9e552167e7785c34.png

（8）查看节点池

节点池

290b8ae7495c41d1964fa65947935080.png

伸缩活动

7ee80fc837b941f9bd232c46da215fd8.png

（9）查看命名空间

8a2ef837e349412abf552aac4a729123.png

（10）查看网络

服务 service

cf08235fc56a4b3fbf3d4d67ca0414a8.png

?

5.在ACK集群中部署应用

（1）查看目录

54da1544775044e1953aae5a83ed794e.png

（2）Terraform模板(docker)

Terraform Registry

USE?PROVIDER

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.25.2"
    }
  }
}

provider "kubernetes" {
  # Configuration options
}

2c0da6a7327e4b7fbe839ac6e1690162.png

(3)下载软件包

https://github.com/hashicorp/terraform-provider-kubernetes/releases

ebdf9996b578423983a7665763011a2b.png

（3）修改K8S集群配置文件

阿里云系统查看连接集群信息

8aabbe2ef4ca4fe29ce876d3cdb46800.png

?

复制上面的连接集群信息到clustera.config

dce09b0204c34f4cbd507e1bc38bb0e4.png

?

（5）修改主配置文件

provider "kubernetes" {
  # Configuration options
  config_path    = "../config/clustera.config"
  config_context = "kubernetes-admin-c718a5ce282f94d539ee5ce1986370194"
  alias          = "clustera"
  insecure       = true
}

resource "kubernetes_namespace" "jenkins" {
  provider = kubernetes.clustera
  metadata {
    name = "devops"
  }
}

3eeee993a8834d13a964fd9f1d01d6c0.png

?

（6）修改版本配置文件

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.25.2"
    }
  }
}

provider "kubernetes" {
  # Configuration options
}

b78fa2155a5045b8afd212627906c50e.png

（7）修改输出配置文件

output "service_name" {
  value = kubernetes_service_v1.jenkins.metadata[0].name

}

bec8e761d0a54774a18777d977835a12.png

（8）修改服务配置文件

jenkins.tf

resource "kubernetes_deployment_v1" "jenkins" {
  provider = kubernetes.clustera
  metadata {
    name = "jenkins"
    labels = {
      app = "jenkins"
    }
    namespace = kubernetes_namespace.jenkins.id
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        app = "jenkins"
      }
    }

    template {
      metadata {
        labels = {
          app = "jenkins"
        }
      }

      spec {
        container {
          image             = "jenkins/jenkins:latest"
          name              = "jenkins"
          image_pull_policy = "IfNotPresent"

          port {
            container_port = 8080
          }

          resources {
            limits = {
              cpu    = "1000m"
              memory = "4096Mi"
            }
            requests = {
              cpu    = "250m"
              memory = "1024Mi"
            }
          }
          #   liveness_probe {
          #     http_get {
          #       path = "/"
          #       port = 8080
          #     }
          #     initial_delay_seconds = 30
          #     period_seconds        = 3
          #   }
        }
      }
    }
  }
}


resource "kubernetes_service_v1" "jenkins" {
  provider = kubernetes.clustera
  metadata {
    name      = "jenkins-service"
    namespace = kubernetes_namespace.jenkins.id
  }
  spec {
    selector = {
      app = kubernetes_deployment_v1.jenkins.metadata[0].labels.app
    }
    port {
      port        = 8080
      target_port = 8080
    }

    type = "ClusterIP"
  }
}


resource "kubernetes_ingress_v1" "jenkins_ingress" {
  provider = kubernetes.clustera
  metadata {
    name      = "jenkins-ingress"
    namespace = kubernetes_namespace.jenkins.id
  }

  spec {
    rule {
      host = "jenkins.maojing.site"
      http {
        path {
          backend {
            service {
              name = kubernetes_service_v1.jenkins.metadata[0].name
              port {
                number = 8080
              }
            }
          }
          path_type = "Prefix"
          path = "/"
        }
      }
    }
  }
}

0c764a459d2e4a5caa95512f43d6654b.png

(9)初始化

terraform init

a8e8dad86e1d4d1995ebf819936cf92c.png

?

(10)格式化代码

terraform fmt

a9d301edc39e4edea7c03e55707ef3fa.png

?

(11)验证代码

terraform validate

80a0ff1a7d484db3afca5121823966e6.png

?

(12)计划与预览

 terraform plan

46e0e19c6eb04472a44ac368269ea357.png59f23b6d3d354a228af227ebabedcd2d.png501aa7a983814358a6e00cb609d2a4cc.pnge51633331f334bd6a64fcd117db8697e.png

(13)?申请资源

terraform apply

14111e7cebd64e2bb6e0d467795f06a3.png

668f719b79c04821aa7a135feec4925a.png

5ce807acff5e40e9b70434cddcb34f37.png

c47ae47f57e44f4db4b3fcac6903f308.png

yes , 4个资源将被添加

f38b01b25a874a8d8cb19344ca164ef8.png

（14）登录阿里云系统查看

命名空间新增1个 devops

e534e2a6cb7b4582afb6bf2c6ac5fe35.png

工作负载（无状态deployment）新增1个jenkins

95d72e5139074ca284b537598946de7e.png

进入jenkins，状态为running

abaed85fe1d54faaa64c5ee07ed4b482.png

服务service

185db580c1454f89845e545eb07bd532.png

service关联路由

57a7eec08076465fb27374c1d8b276be.png

?

（15）修改输出配置文件

outputs.tf,添加如下代码

output "ingress_ip" {
  value = kubernetes_ingress_v1.jenkins_ingress.status[0].load_balancer[0].ingress[0].ip
}

ac0702191acb47e0ad0caee2f8c94475.png

（16）计划与预览

 terraform plan

成功拿到ingress的ip

aee32260f17341068e0ae673e8c90b53.png

?

（17）添加DNS配置文件

dns.tf

#  DNS
resource "alicloud_dns_record" "record" {
  name        = "maojing.site"
  host_record = "jenkins"
  type        = "A"
  value       = kubernetes_ingress_v1.jenkins_ingress.status[0].load_balancer[0].ingress[0].ip
}

5d7a4ce04361414e8af5cc4be5222e82.png

(18) 添加变量配置文件

variables.tf

variable "access_key" {
  description = "access_key"

}

variable "secret_key" {
  description = "secret_key"
}

5850877302ba40c984462ffc8d2f64e6.png

(19) 修改版本配置文件

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.25.2"
    }
    alicloud = {
      source  = "aliyun/alicloud"
      version = "1.214.1"
    }
  }
}

provider "kubernetes" {
  # Configuration options
}

# Configure the Alicloud Provider 默认供应商
provider "alicloud" {
  access_key = var.access_key
  secret_key = var.secret_key
  region     = "cn-hangzhou"
}

1dbbdd06239e42c78be8f803f9b6ac82.png

92d1cad54d0e4565966ade5759dbb4f7.png

（20）初始化

terraform init

0e573712dc0b4d1eace1bc0cac12afa0.png

（21）格式化代码

terraform fmt

8f8a45f0b93143e29304238183cd3243.png

（22）验证代码

terraform validate

e39fd6de02484f4cafd32d34312a68d1.png

（23）计划与预览

 terraform plan

54ebef1848704de090e44227e4f832a0.png

（24）申请资源

terraform apply

5268af8a19944d4bb62842de06745ef3.png

yes

cd219865faa14e618862c8b577d07a35.png

（25）阿里云系统查看

域名解析已新增

ca1f0736fd3645649107d32a446ed29f.png

（26）dig测试DNS

dig jenkins.maojing.site

2be47f1ee1da48c3a7b4eb135fa8dfb5.png

(27) 浏览器测试

显示Jenkins安装界面

6d774444547d4faba7520cf172a56a92.png

(28) 查看集群监控

92fd50d2e4254138b32e6f7f8473787a.pngef9644502c7d4a79bce612e5a482ddaa.png

?

6.销毁资源

（1）销毁服务资源

terraform destroy

2fc0d46bfcc4495ba18d2fa04556d87f.png

b95360b726714f8db446ba8a14c96a3e.png016b65fa035d4e5c9086ee4b31f28f67.png67c32bfc509244a39138352ce4635eb8.png

yes

a2315e97301e48e1923851357fef1105.png

（2）登录阿里云系统

DNS解析已删除

ac3ae6d1caf8427d94c1ebed923894e3.png

devops命名空间已删除

3e113ff8ef7f44d9816672ccb7281dc2.png

?

（3）销毁集群资源

terraform destroy

9ec0ef4bbb8b4719bf43550f1ad43804.png

2f66b1a0b7664a0b8cc8b1afa2d086bb.pngd93f44573a854fa98b82d6f618e1edc1.png17f838a0cfe844f4a52e8e4e1f253e82.png800b50e43e8b43d6a4e20344f7aa699f.png

yes ，用时大约5分钟

e2562b2d1a35470daeccd0d1b05844c1.png

(4)登录阿里云系统查看集群

删除中

8ed3dc23e0244a608ec99859a892e928.png

已删除

b7f0097142664195859c85db2aa9d984.png

二、问题

1.Terraform 验证失败

（1）报错

?
│ Error: "availability_zone": [REMOVED] Field 'availability_zone' has been removed from provider version 1.212.0.
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

7eb41f2eb3d44837a1cc1ab20963b052.png

│ Error: "availability_zone": [REMOVED] Field 'availability_zone' has been removed from provider version 1.212.0.
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

c4f60055b8374e5cb38697f0f24e40fa.png

│ Error: "runtime": [REMOVED] Field 'runtime' has been removed from provider version 1.212.0. Please use resource 'alicloud_cs_kubernetes_node_pool' to manage cluster nodes, by using field 'runtime_name' and 'runtime_version' to replace it.
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

c266f418dc8f4ca9a30879fe8015ab1c.png

（2）原因分析

Terraform Registry

从1.212版本开始，部分关键地段被移除，推荐使用alicloud_cs_kubernetes_node_pool 管理工作节点。

From version 1.212.0, runtime,enable_ssh,rds_instances,exclude_autoscaler_nodes,worker_number,worker_instance_types,password,key_name,kms_encrypted_password,kms_encryption_context,worker_instance_charge_type,worker_period,worker_period_unit,worker_auto_renew,worker_auto_renew_period,worker_disk_category,worker_disk_size,worker_data_disks,node_name_mode,node_port_range,os_type,platform,image_id,cpu_policy,user_data,taints,worker_disk_performance_level,worker_disk_snapshot_policy_id,install_cloud_monitor,kube_config,availability_zone are removed. Please use resource alicloud_cs_kubernetes_node_pool to manage your cluster worker nodes.

e34839b283c64b1297fddf016aa83af2.png

（3）解决方法

修改配置文件。

496c30d4126a435fb2d114d8cfc0805a.png

?

2.Terraform申请资源失败

（1）报错

Error: [ERROR] terraform-provider-alicloud/alicloud/resource_alicloud_cs_kubernetes.go:1230: Resource c28e6d5ac0cf64922a476e6963f1239b8 DescribeNatGateways Failed!!! [SDK alibaba-cloud-sdk-go ERROR]:
│ SDK.ServerError
│ ErrorCode: Forbidden.RAM
│ Recommend: https://api.aliyun.com/troubleshoot?q=Forbidden.RAM&product=Vpc&requestId=0254494A-FE5F-51C9-96DA-394123C37E13
│ RequestId: 0254494A-FE5F-51C9-96DA-394123C37E13
│ Message: User not authorized to operate on the specified resource, or this API doesn't support RAM.
│ RespHeaders: map[Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[*] Connection:[keep-alive] Content-Length:[568] Content-Type:[application/json;charset=utf-8] Date:[Tue, 23 Jan 2024 05:11:28 GMT] Keep-Alive:[timeout=25] X-Acs-Request-Id:[0254494A-FE5F-51C9-96DA-394123C37E13] X-Acs-Trace-Id:[740d51a284c42eb37e67556a9d62faa6]]
│ AccessDeniedDetail: map[AuthPrincipalDisplayName:205814005146961779 AuthPrincipalOwnerId:1889388625243280 AuthPrincipalType:SubUser EncodedDiagnosticMessage:AQEAAAAAZa9KgzAyNTQ0OTRBLUZFNUYtNTFDOS05NkRBLTM5NDEyM0MzN0UxMw==]
│
│   with alicloud_cs_managed_kubernetes.k8s,
│   on main.tf line 7, in resource "alicloud_cs_managed_kubernetes" "k8s":
│    7: resource "alicloud_cs_managed_kubernetes" "k8s" {

da0f1184cbd94abb94a798e05a9dbb8e.png623371129a834af5a4e48d8a7b527ed2.png

（2）原因分析

RAM缺少NATGatewayFullAcess权限

（3）解决方法

RAM添加NATGatewayFullAcess权限。

afcbd6005b6a443db00ca567061355ea.png

重新申请资源

7992822f61d4460199c79d9a99ff5aa8.png39f3413d72e54eb99d0334ac64675ca6.pngcf012b388bce488b8909f7d293995138.png

yes,先删除旧的实例

ecedb8b836304d689833664cbfa94c51.png

开始创建新实例

a2d3c9ba2bb648eca84c2cfcbf82953d.png

?

?

文章来源:https://blog.csdn.net/cronaldo91/article/details/135764873
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。
最新文章