kubernetes存储类迁移-数据拷贝

背景介绍

kubernetes集群最开始使用了nfs作为存储，随着后续使用过程中数据量逐渐增加，nfs存储性能逐步出现不足，现增加了基于csi的分布式块存储后，需要对原有基于nfs存储类下的pv迁移到新的存储类下。

测试环境

• k8s集群版本：1.25.12
• 节点数：4
• 旧存储类：nfs-storage
• 新存储类：rook-ceph-block
• 测试应用：mysql
• 需求：kubectl、具有pv、pvc、pod、sts、deploy控制权限

测试方法

1. 基于旧存储类创建pvc和mysql应用

apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  ports:
    - port: 3306
  selector:
    app: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  storageClassName: nfs-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  selector:
    matchLabels:
      app: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
        - image: mysql:5.7
          name: mysql
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: Passw0rd#1
          ports:
            - containerPort: 3306
              name: mysql
          volumeMounts:
            - name: mysql-persistent-storage
              mountPath: /var/lib/mysql
      volumes:
        - name: mysql-persistent-storage
          persistentVolumeClaim:
            claimName: mysql-pv-claim

2. mysql建库表，插入数据

mysql> create database test;
Query OK, 1 row affected (0.00 sec)

mysql> use test;
Database changed
mysql> create table newtb (id int);
Query OK, 0 rows affected (0.02 sec)

mysql> insert into newtb (id) values (1);
Query OK, 1 row affected (0.00 sec)

mysql> insert into newtb (id) values (2);
Query OK, 1 row affected (0.00 sec)

mysql> insert into newtb (id) values (3);
Query OK, 1 row affected (0.00 sec)

mysql> select id from newtb;
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql> 

3. 关闭mysql服务
4. 基于新存储类创建相同大小的pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: new-mysql-pv-claim
  labels:
    app: mysql
spec:
  storageClassName: rook-ceph-block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

5. 创建同时挂载新旧pvc的迁移任务pod

apiVersion: v1
kind: Pod
metadata:
  name: datacp
spec:
  containers:
    - name: busybox
      image: busybox
      command:
        - 'sleep'
      args:
        - infinity
      volumeMounts:
        - name: old-pvc
          mountPath: /old
        - name: new-pvc
          mountPath: /new
  volumes:
    - name: old-pvc
      persistentVolumeClaim:
        claimName: mysql-pv-claim
    - name: new-pvc
      persistentVolumeClaim:
        claimName: new-mysql-pv-claim

6. 在任务pod中执行数据拷贝

[root@rocky8 examples]# kubectl exec -it datacp -- sh
/ #
/ # ls /old
auto.cnf            client-cert.pem     ib_logfile0         mysql               private_key.pem     server-key.pem
ca-key.pem          client-key.pem      ib_logfile1         mysql.sock          public_key.pem      sys
ca.pem              ib_buffer_pool      ibdata1             performance_schema  server-cert.pem     test
/ # ls /new/
lost+found
/ # rmdir /new/lost+found/
/ # (cd /old; tar -cf - .) | (cd /new; tar -xpf -)
/ # ls -l /old/
total 176196
-rw-r-----    1 999      999             56 Dec 27 07:26 auto.cnf
-rw-------    1 999      999           1680 Dec 27 07:26 ca-key.pem
-rw-r--r--    1 999      999           1112 Dec 27 07:26 ca.pem
-rw-r--r--    1 999      999           1112 Dec 27 07:26 client-cert.pem
-rw-------    1 999      999           1680 Dec 27 07:26 client-key.pem
-rw-r-----    1 999      999            679 Dec 27 07:33 ib_buffer_pool
-rw-r-----    1 999      999       50331648 Dec 27 07:33 ib_logfile0
-rw-r-----    1 999      999       50331648 Dec 27 07:26 ib_logfile1
-rw-r-----    1 999      999       79691776 Dec 27 07:33 ibdata1
drwxr-x---    2 999      999           4096 Dec 27 07:26 mysql
lrwxrwxrwx    1 999      999             27 Dec 27 07:26 mysql.sock -> /var/run/mysqld/mysqld.sock
drwxr-x---    2 999      999           8192 Dec 27 07:26 performance_schema
-rw-------    1 999      999           1676 Dec 27 07:26 private_key.pem
-rw-r--r--    1 999      999            452 Dec 27 07:26 public_key.pem
-rw-r--r--    1 999      999           1112 Dec 27 07:26 server-cert.pem
-rw-------    1 999      999           1676 Dec 27 07:26 server-key.pem
drwxr-x---    2 999      999           8192 Dec 27 07:26 sys
drwxr-x---    2 999      999             54 Dec 27 07:29 test
/ # ls -l /new/
total 176192
-rw-r-----    1 999      999             56 Dec 27 07:26 auto.cnf
-rw-------    1 999      999           1680 Dec 27 07:26 ca-key.pem
-rw-r--r--    1 999      999           1112 Dec 27 07:26 ca.pem
-rw-r--r--    1 999      999           1112 Dec 27 07:26 client-cert.pem
-rw-------    1 999      999           1680 Dec 27 07:26 client-key.pem
-rw-r-----    1 999      999            679 Dec 27 07:33 ib_buffer_pool
-rw-r-----    1 999      999       50331648 Dec 27 07:33 ib_logfile0
-rw-r-----    1 999      999       50331648 Dec 27 07:26 ib_logfile1
-rw-r-----    1 999      999       79691776 Dec 27 07:33 ibdata1
drwxr-x---    2 999      999           4096 Dec 27 07:43 mysql
lrwxrwxrwx    1 root     root            27 Dec 27 07:43 mysql.sock -> /var/run/mysqld/mysqld.sock
drwxr-x---    2 999      999           4096 Dec 27 07:43 performance_schema
-rw-------    1 999      999           1676 Dec 27 07:26 private_key.pem
-rw-r--r--    1 999      999            452 Dec 27 07:26 public_key.pem
-rw-r--r--    1 999      999           1112 Dec 27 07:26 server-cert.pem
-rw-------    1 999      999           1676 Dec 27 07:26 server-key.pem
drwxr-x---    2 999      999          12288 Dec 27 07:43 sys
drwxr-x---    2 999      999           4096 Dec 27 07:43 test
/ # diff /old/ /new/
diff: /old/mysql.sock: No such file or directory
diff: /new/mysql.sock: No such file or directory
Common subdirectories: /old/mysql and /new/mysql
Common subdirectories: /old/performance_schema and /new/performance_schema
Common subdirectories: /old/sys and /new/sys
Common subdirectories: /old/test and /new/test

7. 关闭任务pod
8. 修改新旧pv的删除策略，保证为Retain

[root@rocky8 examples]# kubectl get pvc 
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim       Bound    pvc-05a52d71-4289-4ef3-aa8e-da13c6b9e835   1Gi        RWO            nfs-storage       20m
new-mysql-pv-claim   Bound    pvc-7bf42770-ca3d-433c-9b44-71dd5549563d   1Gi        RWO            rook-ceph-block   12m
root@rocky8 examples]# kubectl describe pv pvc-05a52d71-4289-4ef3-aa8e-da13c6b9e835 | grep Reclaim
Reclaim Policy:  Delete
[root@rocky8 examples]# kubectl describe pv pvc-7bf42770-ca3d-433c-9b44-71dd5549563d | grep Reclaim
Reclaim Policy:  Delete
[root@rocky8 examples]#
[root@rocky8 examples]# kubectl patch pv pvc-05a52d71-4289-4ef3-aa8e-da13c6b9e835 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-05a52d71-4289-4ef3-aa8e-da13c6b9e835 patched
[root@rocky8 examples]# kubectl patch pv pvc-7bf42770-ca3d-433c-9b44-71dd5549563d -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
persistentvolume/pvc-7bf42770-ca3d-433c-9b44-71dd5549563d patched
[root@rocky8 examples]# kubectl describe pv pvc-05a52d71-4289-4ef3-aa8e-da13c6b9e835 | grep Reclaim
Reclaim Policy:  Retain
[root@rocky8 examples]# kubectl describe pv pvc-7bf42770-ca3d-433c-9b44-71dd5549563d | grep Reclaim
Reclaim Policy:  Retain
[root@rocky8 examples]# 

9. 备份旧pvc定义，删除新旧pvc

[root@rocky8 examples]# kubectl get pvc mysql-pv-claim -o yaml >old-pvc.yaml
[root@rocky8 examples]# kubectl delete pvc mysql-pv-claim
persistentvolumeclaim "mysql-pv-claim" deleted
[root@rocky8 examples]# kubectl delete pvc new-mysql-pv-claim
persistentvolumeclaim "new-mysql-pv-claim" deleted
[root@rocky8 examples]#

10. 解绑新pv和pvc的绑定关系

[root@rocky8 examples]# kubectl edit pv pvc-7bf42770-ca3d-433c-9b44-71dd5549563d
# 删除claimRef定义，保存退出
[root@rocky8 examples]# kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                    STORAGECLASS      REASON   AGE
pvc-05a52d71-4289-4ef3-aa8e-da13c6b9e835   1Gi        RWO            Retain           Released    default/mysql-pv-claim   nfs-storage                29m
pvc-7bf42770-ca3d-433c-9b44-71dd5549563d   1Gi        RWO            Retain           Available                            rook-ceph-block            21m
[root@rocky8 examples]#

11. 创建和旧pvc相同的使用新存储类的新pvc

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  storageClassName: rook-ceph-block
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

# 查看新pvc和旧pv的绑定情况
[root@rocky8 examples]# kubectl get pvc 
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-7bf42770-ca3d-433c-9b44-71dd5549563d   1Gi        RWO            rook-ceph-block   55s
[root@rocky8 examples]#

12. 恢复mysql应用，检查数据

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+----------------+
| Tables_in_test |
+----------------+
| newtb          |
+----------------+
1 row in set (0.00 sec)

mysql> select id from newtb;
+------+
| id   |
+------+
|    1 |
|    2 |
|    3 |
+------+
3 rows in set (0.00 sec)

mysql>