kubernetes集群认证（kubernetes生产实践之redis-cluster）

在kubernetes中部署Redis集群面临挑战，因为每个Redis实例都依赖于一个配置文件，该文件可以跟踪其他集群实例及其角色。为此，需要结合使用Kubernetes StatefulSets和PersistentVolumes实现。redis cluster架构示意图：

--- apiVersion:v1 kind:ConfigMap metadata: name:redis-cluster data: update.sh:| #!/bin/sh REDIS_NODES="/data/nodes.conf" sed-i-e"/myself/s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${POD_IP}/"${REDIS_NODES} exec"$@" redis.conf:| bind0.0.0.0 cluster-enabledyes cluster-require-full-coverageno cluster-node-timeout30000 cluster-config-file/data/nodes.conf cluster-migration-barrier1 appendonlyyes protected-modeno --- apiVersion:apps.kruise.io/v1beta1 #apiVersion:apps/v1 kind:StatefulSet metadata: name:redis-cluster spec: serviceName:redis-cluster replicas:6 selector: matchLabels: app:redis-cluster template: metadata: labels: app:redis-cluster spec: containers: -name:redis image:redis:6.2.1-alpine ports: -containerPort:6379 name:client -containerPort:16379 name:gossip command:["/conf/update.sh","redis-server","/conf/redis.conf"] env: -name:POD_IP valueFrom: fieldRef: fieldPath:status.podIP volumeMounts: -name:conf mountPath:/conf readOnly:false -name:data mountPath:/data readOnly:false volumes: -name:conf configMap: name:redis-cluster defaultMode:0755 volumeClaimTemplates: -metadata: name:data spec: accessModes:["ReadWriteOnce"] resources: requests: storage:100Gi storageClassName:rbd

注释：

cluster-migration-barrier那些分配后仍然剩余migrationbarrier个从节点的主节点才会触发节点分配，而不是分配前有migrationbarrier个从节点的主节点就会触发节点分配，默认是1，生产环境建议维持默认值 protected-modeno参数是为了禁止外网访问redis，如果启用了，则只能够通过lookbackip（127.0.0.1）访问Redis，如果外网访问redis，会报出异常。 apiVersion:apps.kruise.io/v1beta1控制器这里使用kruise提供的AdvancedStatefulSet，如果集群没有安装kruise，可以使用apps/v1

[root@qd01-stop-k8s-master001redis]#kubectlapply-finstall-redis.yaml configmap/redis-clustercreated statefulset.apps.kruise.io/redis-clustercreated [root@qd01-stop-k8s-master001redis]#kubectlgetpo-nop NAMEREADYSTATUSRESTARTSAGE redis-cluster-01/1Running03m26s redis-cluster-11/1Running03m14s redis-cluster-21/1Running02m54s redis-cluster-31/1Running02m23s redis-cluster-41/1Running02m14s redis-cluster-51/1Running0114s

--- apiVersion:v1 kind:Service metadata: name:redis-cluster namespace:op spec: type:ClusterIP ports: -port:6379 targetPort:6379 name:client -port:16379 targetPort:16379 name:gossip selector: app:redis-cluster

[root@qd01-stop-k8s-master001redis]#kubectlapply-fredis-svc.yml service/redis-clustercreated [root@qd01-stop-k8s-master001redis]#kubectlgetsvc-nop NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE redis-clusterClusterIP10.97.197.224<none>6379/TCP,16379/TCP9s 测试能后连通 [root@qd01-stop-k8s-master001redis]#telnet10.97.197.2246379 Trying10.97.197.224... Connectedto10.97.197.224. Escapecharacteris'^]'.

执行如下命令，获取到pod IP，然后使用redis-cli --cluster创建集群

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-0--redis-cli--clustercreate--cluster-replicas1$(kubectl-nopgetpods-lapp=redis-cluster-ojsonpath='{range.items[*]}{.status.podIP}:6379{end}') >>>Performinghashslotsallocationon6nodes... Master[0]->Slots0-5460 Master[1]->Slots5461-10922 Master[2]->Slots10923-16383 Addingreplica100.88.43.67:6379to100.64.147.152:6379 Addingreplica100.113.170.5:6379to100.98.174.217:6379 Addingreplica100.64.147.153:6379to100.80.158.227:6379 M:b47b27a3dbddf3fc1370cbe14ae753f4fce20b04100.64.147.152:6379 slots:[0-5460](5461slots)master M:09543217c903350e963fc4fdf4acb73f8a1b7f8b100.98.174.217:6379 slots:[5461-10922](5462slots)master M:5389ace495b68eeac85370d6783648dff68f2fb6100.80.158.227:6379 slots:[10923-16383](5461slots)master S:b1f39714c006ae55b12b18e6537303d7a00e1704100.64.147.153:6379 replicates5389ace495b68eeac85370d6783648dff68f2fb6 S:0113f4668ec2f3ca2e9470c44bd5faab532b0936100.88.43.67:6379 replicatesb47b27a3dbddf3fc1370cbe14ae753f4fce20b04 S:e1e2f18ae66c79f1943390beabb59613abbad38a100.113.170.5:6379 replicates09543217c903350e963fc4fdf4acb73f8a1b7f8b CanIsettheaboveconfiguration?(type'yes'toaccept):yes >>>Nodesconfigurationupdated >>>Assignadifferentconfigepochtoeachnode >>>SendingCLUSTERMEETmessagestojointhecluster Waitingfortheclustertojoin .. >>>PerformingClusterCheck(usingnode100.64.147.152:6379) M:b47b27a3dbddf3fc1370cbe14ae753f4fce20b04100.64.147.152:6379 slots:[0-5460](5461slots)master 1additionalreplica(s) S:0113f4668ec2f3ca2e9470c44bd5faab532b0936100.88.43.67:6379 slots:(0slots)slave replicatesb47b27a3dbddf3fc1370cbe14ae753f4fce20b04 M:09543217c903350e963fc4fdf4acb73f8a1b7f8b100.98.174.217:6379 slots:[5461-10922](5462slots)master 1additionalreplica(s) M:5389ace495b68eeac85370d6783648dff68f2fb6100.80.158.227:6379 slots:[10923-16383](5461slots)master 1additionalreplica(s) S:e1e2f18ae66c79f1943390beabb59613abbad38a100.113.170.5:6379 slots:(0slots)slave replicates09543217c903350e963fc4fdf4acb73f8a1b7f8b S:b1f39714c006ae55b12b18e6537303d7a00e1704100.64.147.153:6379 slots:(0slots)slave replicates5389ace495b68eeac85370d6783648dff68f2fb6 [OK]Allnodesagreeaboutslotsconfiguration. >>>Checkforopenslots... >>>Checkslotscoverage... [OK]All16384slotscovered.

[root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-0--redis-cliclusterinfo cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:6 cluster_my_epoch:1 cluster_stats_messages_ping_sent:178 cluster_stats_messages_pong_sent:181 cluster_stats_messages_sent:359 cluster_stats_messages_ping_received:176 cluster_stats_messages_pong_received:178 cluster_stats_messages_meet_received:5 cluster_stats_messages_received:359 [root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-0--redis-cliclusternodes 0113f4668ec2f3ca2e9470c44bd5faab532b0936100.88.43.67:6379@16379slaveb47b27a3dbddf3fc1370cbe14ae753f4fce20b04016153483111561connected 09543217c903350e963fc4fdf4acb73f8a1b7f8b100.98.174.217:6379@16379master-016153483141622connected5461-10922 b47b27a3dbddf3fc1370cbe14ae753f4fce20b04100.64.147.152:6379@16379myself,master-016153483120001connected0-5460 5389ace495b68eeac85370d6783648dff68f2fb6100.80.158.227:6379@16379master-016153483120003connected10923-16383 e1e2f18ae66c79f1943390beabb59613abbad38a100.113.170.5:6379@16379slave09543217c903350e963fc4fdf4acb73f8a1b7f8b016153483131602connected b1f39714c006ae55b12b18e6537303d7a00e1704100.64.147.153:6379@16379slave5389ace495b68eeac85370d6783648dff68f2fb6016153483121583connected

从输出可以看到，集群总共6个节点，三主三从

1、安装KubeDB获取AppsCode License https://license-issuer.appscode.com/下载KubeDB charts https://github.com/appscode/charts/tree/master/stable/kubedb-community

[root@qd01-stop-k8s-master001kubedb-community]#unzipkubedb-community-v0.16.2.tgz [root@qd01-stop-k8s-master001kubedb-community]#cdkubedb-community [root@qd01-stop-k8s-master001kubedb-community]#ls-al total96 drwxr-xr-x4rootroot158Mar1015:26. drwxr-xr-x3rootroot66Mar1015:24.. -rw-r--r--1rootroot351Feb1609:55Chart.yaml drwxr-xr-x2rootroot28Mar1015:24ci -rw-r--r--1rootroot493Feb1609:55doc.yaml -rw-r--r--1rootroot353Feb1609:55.helmignore -rw-r--r--1rootroot24422Feb1609:55README.md drwxr-xr-x2rootroot4096Mar1015:24templates -rw-r--r--1rootroot47437Feb1609:55values.openapiv3_schema.yaml -rw-r--r--1rootroot5230Feb1609:55values.yaml

修改values.yaml，把License文件放到kubedb-community目录下2、使用helm安装

[root@qd01-stop-k8s-master001kubedb-community]#helminstallkubedb-community--namespacekube-system--set-filelicense=./kubedb-community-license.txt-fvalues.yaml. NAME:kubedb-community LASTDEPLOYED:WedMar1015:38:592021 NAMESPACE:kube-system STATUS:deployed REVISION:1 TESTSUITE:None NOTES: ToverifythatKubeDBhasstarted,run: kubectlgetdeployment--namespacekube-system-l"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community" Nowinstall/upgradeappscode/kubedb-catalogchart. Toinstall,run: helminstallkubedb-catalogappscode/kubedb-catalog--versionv0.16.2--namespacekube-system Toupgrade,run: helmupgradekubedb-catalogappscode/kubedb-catalog--versionv0.16.2--namespacekube-system 执行如下命令查看是否安装完成 [root@qd01-stop-k8s-master001kubedb-community]#kubectlgetdeployment--namespacekube-system-l"app.kubernetes.io/name=kubedb-community,app.kubernetes.io/instance=kubedb-community" NAMEREADYUP-TO-DATEAVAILABLEAGE kubedb-community1/11138s 等待crds注册成功 [root@qd01-stop-k8s-master001kubedb-community]#kubectlgetcrds-lapp.kubernetes.io/name=kubedb-w NAMECREATEDAT elasticsearches.kubedb.com2021-03-10T07:39:42Z elasticsearchversions.catalog.kubedb.com2021-03-10T07:39:45Z etcds.kubedb.com2021-03-10T07:39:42Z etcdversions.catalog.kubedb.com2021-03-10T07:39:45Z memcacheds.kubedb.com2021-03-10T07:39:43Z memcachedversions.catalog.kubedb.com2021-03-10T07:39:45Z mongodbs.kubedb.com2021-03-10T07:39:43Z mongodbversions.catalog.kubedb.com2021-03-10T07:39:45Z mysqls.kubedb.com2021-03-10T07:39:43Z mysqlversions.catalog.kubedb.com2021-03-10T07:39:46Z perconaxtradbs.kubedb.com2021-03-10T07:39:43Z perconaxtradbversions.catalog.kubedb.com2021-03-10T07:39:46Z pgbouncers.kubedb.com2021-03-10T07:39:44Z pgbouncerversions.catalog.kubedb.com2021-03-10T07:39:46Z postgreses.kubedb.com2021-03-10T07:39:44Z postgresversions.catalog.kubedb.com2021-03-10T07:39:46Z proxysqls.kubedb.com2021-03-10T07:39:44Z proxysqlversions.catalog.kubedb.com2021-03-10T07:39:46Z redises.kubedb.com2021-03-10T07:39:45Z redisversions.catalog.kubedb.com2021-03-10T07:39:46Z

3、安装KubeDB Catalog同样，先下载https://github.com/appscode/charts/tree/master/stable/kubedb-catalog

[root@qd01-stop-k8s-master001kubedb-catalog]#tar-zxfkubedb-catalog-v0.16.2.tgz [root@qd01-stop-k8s-master001kubedb-catalog]#cdkubedb-catalog [root@qd01-stop-k8s-master001kubedb-catalog]#ls-al total24 drwxr-xr-x3rootroot148Mar1015:48. drwxr-xr-x3rootroot28Mar1015:48.. -rw-r--r--1rootroot321Jan2620:08Chart.yaml -rw-r--r--1rootroot467Jan2620:08doc.yaml -rw-r--r--1rootroot353Jan2620:08.helmignore -rw-r--r--1rootroot3195Jan2620:08README.md drwxr-xr-x12rootroot188Mar1015:48templates -rw-r--r--1rootroot744Jan2620:08values.openapiv3_schema.yaml -rw-r--r--1rootroot1070Jan2620:08values.yaml [root@qd01-stop-k8s-master001kubedb-catalog]#helminstallkubedb-catalog--namespacekube-system-fvalues.yaml. NAME:kubedb-catalog LASTDEPLOYED:WedMar1015:50:502021 NAMESPACE:kube-system STATUS:deployed REVISION:1 TESTSUITE:None

1、先看下官方给的redis生命周期示意图

kubedb安装redis支持如下特性

FeaturesAvailability Clustering✓ InstantBackup✗ ScheduledBackup✗ PersistentVolume✓ InitializeusingSnapshot✗ InitializeusingScript✗ CustomConfiguration✓ UsingCustomdockerimage✓ BuiltinPrometheusDiscovery✓ UsingPrometheusoperator✓

2、查看支持的版本

[root@qd01-stop-k8s-master001kubedb-catalog]#kubectlgetredisversions NAMEVERSIONDB_IMAGEDEPRECATEDAGE 4.0.114.0.11kubedb/redis:4.0.1115m 4.0.6-v24.0.6kubedb/redis:4.0.6-v215m 5.0.3-v15.0.3kubedb/redis:5.0.3-v115m 6.0.66.0.6kubedb/redis:6.0.615m

3、编辑yaml安装文件可以参照https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/clustering/demo-1.yaml这里选择安装6.0.6这个版本，我的集群storageClassName: "rbd"，请根据实际修改如果想自定义redis.conf,请参考https://github.com/kubedb/docs/blob/v2021.01.26/docs/examples/redis/custom-config/redis-custom.yaml

apiVersion:kubedb.com/v1alpha2 kind:Redis metadata: name:redis-cluster namespace:op spec: version:6.0.6 mode:Cluster cluster: master:3 replicas:1 storageType:Durable storage: resources: requests: storage:1Gi storageClassName:"rbd" accessModes: -ReadWriteOnce

执行安装

[root@qd01-stop-k8s-master001kubedb-community]#kubectlapply-fredis-cluster.yaml redis.kubedb.com/redis-clustercreated 安装完成，可以如下查看 [root@qd01-stop-k8s-master001kubedb-community]#kubectlgetrd,po-nop NAMEVERSIONSTATUSAGE redis.kubedb.com/redis-cluster6.0.6Provisioning6m55s NAMEREADYSTATUSRESTARTSAGE pod/redis-cluster-shard0-01/1Running06m54s pod/redis-cluster-shard0-11/1Running06m18s pod/redis-cluster-shard1-01/1Running05m38s pod/redis-cluster-shard1-11/1Running05m1s pod/redis-cluster-shard2-01/1Running04m30s pod/redis-cluster-shard2-11/1Running04m8s [root@qd01-stop-k8s-master001redis]#kubectlgetsvc-nop NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE redis-clusterClusterIP10.97.197.224<none>6379/TCP5h16m redis-cluster-podsClusterIPNone<none>6379/TCP17m

4、验证集群

[root@qd01-stop-k8s-master001kubedb-community]#kubectlgetpods-nop-ojsonpath='{range.items[*]}{.metadata.name}----------{.status.podIP}:6379{"\t\n"}{end}'|grepredis redis-cluster-shard0-0----------100.64.147.156:6379 redis-cluster-shard0-1----------100.98.174.218:6379 redis-cluster-shard1-0----------100.126.252.204:6379 redis-cluster-shard1-1----------100.113.170.6:6379 redis-cluster-shard2-0----------100.107.55.69:6379 redis-cluster-shard2-1----------100.78.230.4:6379 [root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-shard0-0--redis-cliclusterinfo cluster_state:ok cluster_slots_assigned:16384 cluster_slots_ok:16384 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_known_nodes:6 cluster_size:3 cluster_current_epoch:3 cluster_my_epoch:1 cluster_stats_messages_ping_sent:864 cluster_stats_messages_pong_sent:882 cluster_stats_messages_sent:1746 cluster_stats_messages_ping_received:879 cluster_stats_messages_pong_received:864 cluster_stats_messages_meet_received:3 cluster_stats_messages_received:1746 [root@qd01-stop-k8s-master001redis]#kubectl-nopexec-itredis-cluster-shard0-0--redis-cliclusternodes 1895cb4b9c31b848666c61000e502f55a29a8255100.64.147.155:6379@16379master-016153651620082connected5461-10922 30bdbf2ca37001774498a9b935afbc1cd2ce389c100.126.252.203:6379@16379slave2c06092fafa99e0158e39e6237a04fed25be3550016153651630001connected 9b2cfbd5c1b417121d410141b6da9512ad29ce3c100.78.230.3:6379@16379slavee83446c368839c5fdccf5f70e3b1004eb67cb651016153651635123connected 2c06092fafa99e0158e39e6237a04fed25be3550100.82.197.130:6379@16379myself,master-016153651620001connected0-5460 1379d2b20f26ab13d53068d276ec5d988b7a0273100.64.122.197:6379@16379slave1895cb4b9c31b848666c61000e502f55a29a8255016153651630002connected e83446c368839c5fdccf5f70e3b1004eb67cb651100.107.55.68:6379@16379master-016153651640143connected10923-16383