La fin d’année arrive et c’est souvent l’occasion des renouvellements divers et variés..
Nous allons voir un problème qu’on rencontre parfois après avoir renouveler
le certificat SSL/TLS d’un vCenter sur des clusters kubernetes Tanzu (ou qui utilise la CPI vCenter !)
Aie mon MC : VCenterUnreachable
En me connectant sur ma jumpbox pour lancer une mise à jour d’un cluster de test, je constate un souci au niveau de mon cluster ! (et des workload clusters)
tanzu-admin@alg-linux-tanzu:~$ tanzu mc get
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
alg-tanzu-mgnt tkg-system running 1/1 1/1 v1.26.5+vmware.2 management dev
Details:
NAME READY SEVERITY REASON SINCE
/alg-tanzu-mgnt False Error VCenterUnreachable 14d
ClusterInfrastructure - VSphereCluster/alg-tanzu-mgnt-vtbkm False Error VCenterUnreachable 14d
ControlPlane - KubeadmControlPlane/alg-tanzu-mgnt-xjqfm True 114d
Machine/alg-tanzu-mgnt-xjqfm-zvjxt True 114d
Workers
MachineDeployment/alg-tanzu-mgnt-md-0-hv8q6 True 14d
Machine/alg-tanzu-mgnt-md-0-hv8q6-6d89d5c6d7xp8fdh-xdhw2 True 114d
Providers:
NAMESPACE NAME TYPE PROVIDERNAME VERSION
caip-in-cluster-system ipam-in-cluster IPAMProvider in-cluster v0.1.0
capi-kubeadm-bootstrap-system bootstrap-kubeadm BootstrapProvider kubeadm v1.4.5
capi-kubeadm-control-plane-system control-plane-kubeadm ControlPlaneProvider kubeadm v1.4.5
capi-system cluster-api CoreProvider cluster-api v1.4.5
capv-system infrastructure-vsphere InfrastructureProvider vsphere v1.7.1 on verifie les pods….
tanzu-admin@alg-linux-tanzu:~$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
caip-in-cluster-system caip-in-cluster-controller-manager-5b7554487f-qftbb 1/1 Running 0 23m
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-58f695646-bfrnq 1/1 Running 0 23m
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-5f6c55f8d9-5dmst 1/1 Running 0 23m
capi-system capi-controller-manager-ccc57b89f-xdcss 1/1 Running 0 23m
capv-system capv-controller-manager-68889d4cf-4v2z6 1/1 Running 0 23m
cert-manager cert-manager-74dddbfbf6-s9xn9 1/1 Running 5 (14d ago) 114d
cert-manager cert-manager-cainjector-66c67788bc-6pww9 1/1 Running 8 (14d ago) 114d
cert-manager cert-manager-webhook-7fcd79db9f-n8ngh 1/1 Running 0 114d
kube-system antrea-agent-rt74n 2/2 Running 0 114d
kube-system antrea-agent-v7kbg 2/2 Running 1 (81d ago) 114d
kube-system antrea-controller-854fdc96fd-dzwv6 1/1 Running 1 (81d ago) 114d
kube-system coredns-75f565d4dd-7ltx5 1/1 Running 0 114d
kube-system coredns-75f565d4dd-r8lf8 1/1 Running 0 114d
kube-system etcd-alg-tanzu-mgnt-xjqfm-zvjxt 1/1 Running 0 114d
kube-system kube-apiserver-alg-tanzu-mgnt-xjqfm-zvjxt 1/1 Running 3 (14d ago) 114d
kube-system kube-controller-manager-alg-tanzu-mgnt-xjqfm-zvjxt 1/1 Running 12 (14d ago) 114d
kube-system kube-proxy-5kbtd 1/1 Running 0 114d
kube-system kube-proxy-gnffv 1/1 Running 0 114d
kube-system kube-scheduler-alg-tanzu-mgnt-xjqfm-zvjxt 1/1 Running 9 (14d ago) 114d
kube-system kube-vip-alg-tanzu-mgnt-xjqfm-zvjxt 1/1 Running 6 (14d ago) 114d
kube-system metrics-server-774cf97dcc-44729 1/1 Running 0 114d
kube-system vsphere-cloud-controller-manager-24szw 1/1 Running 13 (14d ago) 114d
secretgen-controller secretgen-controller-666f4b9957-x5h2p 1/1 Running 0 114d
tanzu-auth tanzu-auth-controller-manager-68855cdd58-lk2xs 1/1 Running 0 22m
tkg-system-telemetry tkg-telemetry-28374840-nvc4z 0/1 Completed 0 15h
tkg-system-telemetry tkg-telemetry-28375200-bmlps 0/1 Completed 0 9h
tkg-system-telemetry tkg-telemetry-28375560-7ww7j 0/1 Completed 0 3h44m
tkg-system kapp-controller-8644d7bcb9-z8c6k 2/2 Running 0 114d
tkg-system object-propagation-controller-manager-69ff969bbb-gdcnf 1/1 Running 0 21m
tkg-system tanzu-addons-controller-manager-67fff9bf69-4rl5j 1/1 Running 0 22m
tkg-system tanzu-capabilities-controller-manager-8577c7bdd8-5tx8q 1/1 Running 0 22m
tkg-system tanzu-featuregates-controller-manager-6c6fc86cf9-8kk6k 1/1 Running 0 22m
tkg-system tkr-conversion-webhook-manager-7cc5bfbc9-tvh86 1/1 Running 0 21m
tkg-system tkr-resolver-cluster-webhook-manager-7db5455454-9m26x 1/1 Running 0 21m
tkg-system tkr-source-controller-manager-76794d6ddf-ph6t2 1/1 Running 0 21m
tkg-system tkr-status-controller-manager-894995449-lv8bk 1/1 Running 0 21m
tkg-system tkr-vsphere-resolver-webhook-manager-574cc6f8c-2v9vq 1/1 Running 0 20m
vmware-system-antrea register-placeholder-wb9pp 1/1 Running 0 4s
vmware-system-csi vsphere-csi-controller-69db6c6955-dhnfk 5/7 CrashLoopBackOff 9505 (27s ago) 114d
vmware-system-csi vsphere-csi-node-r6znb 3/3 Running 2 (114d ago) 114d
vmware-system-csi vsphere-csi-node-x6dft 3/3 Running 3 (81d ago) 114dAie le pod vsphere-csi-controller est en CrashLoopBackOff (plus de 9000 fois…)
On va vérifier les logs du pod
tanzu-admin@alg-linux-tanzu:~$ kubectl logs vsphere-csi-controller-69db6c6955-dhnfk -n vmware-system-csi
Defaulted container "csi-attacher" out of: csi-attacher, csi-resizer, vsphere-csi-controller, liveness-probe, vsphere-syncer, csi-provisioner, csi-snapshotter
W1207 16:12:10.486804 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:12:20.486505 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:12:30.487263 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:12:40.486882 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:12:50.486941 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:13:00.486938 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:13:10.486673 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:13:20.486476 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:13:30.487016 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:13:40.487527 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:13:50.487416 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:14:00.486439 1 connection.go:183] Still connecting to unix:///csi/csi.sock
W1207 16:14:10.486553 1 connection.go:183] Still connecting to unix:///csi/csi.sock
[...]On observe plusieurs pages de csi.sock… sans interet par contre, le pod contient plusieurs containers (7 !) on regarde les logs d’un autre container avec l’option -c
tanzu-admin@alg-linux-tanzu:~$ kubectl logs vsphere-csi-controller-69db6c6955-dhnfk -n vmware-system-csi -c vsphere-csi-controller
[...]
{"level":"error","time":"2023-12-14T09:42:09.035598113Z","caller":"cnsvolumeoperationrequest/cnsvolumeoperationrequest.go:370","msg":"failed to list VolumeSnapshotContents with error the server could not find the requested resource (get volumesnapshotcontents.snapshot.storage.k8s.io). Abandoning CnsVolumeOperationRequests clean up ...","TraceId":"da95014d-49f0-4c61-9585-f881be411040","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeoperationrequest.(*operationRequestStore).cleanupStaleInstances\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/internalapis/cnsvolumeoperationrequest/cnsvolumeoperationrequest.go:370"}
{"level":"error","time":"2023-12-14T09:42:09.10074258Z","caller":"vsphere/virtualcenter.go:171","msg":"failed to create new client with err: Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"bf34cecc-de64-40fb-95b9-1105952e7173","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).NewClient\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:171\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:284\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).Connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:259\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.GetVirtualCenterInstanceForVCenterConfig\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:645\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:234\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:188\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"error","time":"2023-12-14T09:42:09.100845971Z","caller":"vsphere/virtualcenter.go:285","msg":"failed to create govmomi client with err: Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"bf34cecc-de64-40fb-95b9-1105952e7173","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:285\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).Connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:259\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.GetVirtualCenterInstanceForVCenterConfig\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:645\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:234\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:188\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"error","time":"2023-12-14T09:42:09.100927697Z","caller":"vsphere/virtualcenter.go:287","msg":"failed to connect to vCenter using CA file: \"\"","TraceId":"bf34cecc-de64-40fb-95b9-1105952e7173","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:287\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).Connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:259\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.GetVirtualCenterInstanceForVCenterConfig\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:645\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:234\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:188\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"error","time":"2023-12-14T09:42:09.100996495Z","caller":"vsphere/virtualcenter.go:261","msg":"Cannot connect to vCenter with err: Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"bf34cecc-de64-40fb-95b9-1105952e7173","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.(*VirtualCenter).Connect\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:261\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.GetVirtualCenterInstanceForVCenterConfig\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:645\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:234\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:188\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"error","time":"2023-12-14T09:42:09.101022186Z","caller":"vsphere/virtualcenter.go:647","msg":"failed to connect to VirtualCenter host: \"vcenter.example.com\". Err: Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"bf34cecc-de64-40fb-95b9-1105952e7173","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere.GetVirtualCenterInstanceForVCenterConfig\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/common/cns-lib/vsphere/virtualcenter.go:647\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:234\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:188\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"error","time":"2023-12-14T09:42:09.101076335Z","caller":"vanilla/controller.go:236","msg":"failed to get vCenterInstance for vCenter \"vcenter.example.com\"err=Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"bf34cecc-de64-40fb-95b9-1105952e7173","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).Init\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/vanilla/controller.go:236\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:188\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"error","time":"2023-12-14T09:42:09.101094921Z","caller":"service/driver.go:189","msg":"failed to init controller. Error: failed to get vCenterInstance for vCenter \"vcenter.example.com\"err=Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"913e07f7-9498-437a-a39a-bc8957074d71","TraceId":"c1203a43-f7b7-4815-bbd9-76b725aa9ec7","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).BeforeServe\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:189\nsigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:202\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
{"level":"info","time":"2023-12-14T09:42:09.101147884Z","caller":"service/driver.go:109","msg":"Configured: \"csi.vsphere.vmware.com\" with clusterFlavor: \"VANILLA\" and mode: \"controller\"","TraceId":"913e07f7-9498-437a-a39a-bc8957074d71","TraceId":"c1203a43-f7b7-4815-bbd9-76b725aa9ec7"}
{"level":"error","time":"2023-12-14T09:42:09.101173921Z","caller":"service/driver.go:203","msg":"failed to run the driver. Err: +failed to get vCenterInstance for vCenter \"vcenter.example.com\"err=Post \"https://vcenter.example.com:443/sdk\": host \"vcenter.example.com:443\" thumbprint does not match \"EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2\"","TraceId":"913e07f7-9498-437a-a39a-bc8957074d71","stacktrace":"sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service.(*vsphereCSIDriver).Run\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/pkg/csi/service/driver.go:203\nmain.main\n\t/build/mts/release/bora-21991883/cayman_vsphere_csi_driver/vsphere_csi_driver/src/cmd/vsphere-csi/main.go:71\nruntime.main\n\t/build/mts/release/bora-21991883/compcache/cayman_go/ob-21619204/linux64/src/runtime/proc.go:250"}
Problème identifié ! un souci au niveau de l’empreinte (thumbprint) du certificat du vCenter… (thumbprint does not match EA:A2:E6:EF:0C:E3…. ).
Après une vérification, en effet, le certificat du vCenter a été renouvelé il y a une 15aine de jour suite à l’expiration approchant de ce dernier.
et si on regarde la configuration yaml du cluster, il y avais été configuré en mode « VSPHERE_INSECURE: FALSE » et l’empreinte
[...]
VSPHERE_INSECURE: "false"
VSPHERE_TLS_THUMBPRINT: EA:A2:E6:EF:0C:E3:46:0E:86:12:B1:60:F5:7C:6F:26:35:36:5F:F2
[...]La question est maintenant, comment mettre à jour cette empreinte ?
Mise à jour de l’empreinte (thumbprint) dans Tanzu
Comment mettre à jour facilement cette empreinte ?
Quand on cherche sur Google, on trouve des personnes qui ont créer des scripts pour cela ! mais depuis Tanzu 2, il est possible de faire cela directement avec la tanzu CLI.
Mais pour commencer, il faut récupérer la nouvelle empreinte ! on va utiliser le super outil GOVC dont j’ai déjà parlé dans un précèdent article. On tape la commande govc about.cert
tanzu-admin@alg-linux-tanzu:~$ govc about.cert
Certificate Status: ERROR tls: failed to verify certificate: x509: certificate signed by unknown authority
Issued To:
Common Name (CN): vcenter.example.com
Organization (O): VMware
Organizational Unit (OU): <Not Part Of Certificate>
Issued By:
Common Name (CN): vcenter.example.com
Organization (O): vcenter.example.com
Organizational Unit (OU): <Not Part Of Certificate>
Validity Period:
Issued On: 2023-11-28 08:14:04 +0000 UTC
Expires On: 2025-11-27 08:14:04 +0000 UTC
Thumbprints:
SHA-256 Thumbprint: E9:1B:E0:E7:95:49:84:94:85:37:58:84:06:B5:9B:09:4E:CE:B2:5E:87:B4:D5:2B:5D:93:55:7A:93:35:22:16
SHA-1 Thumbprint: F4:B4:45:53:C6:CC:C8:C9:EC:94:58:8A:24:5B:70:C7:0D:25:B5:7F
C’est le SHA-1 Thumbprint qui nous intéresse, on peux uniquement recuperer cela avec un govc about.cert -thumbprint
tanzu-admin@alg-linux-tanzu:~$ govc about.cert -thumbprint
vcenter.example.com F4:B4:45:53:C6:CC:C8:C9:EC:94:58:8A:24:5B:70:C7:0D:25:B5:7Fdonc pour mettre à jour, il suffit de taper la commande suivante depuis sa jumpbox : tanzu mc credentials update. Il faut ressaisir le compte de service et mot de passe utilisé pour la connexion avec le vCenter et on termine par le TLS thumbprint récupéré précédemment.
tanzu-admin@alg-linux-tanzu:~$tanzu mc credentials update
Downloading the TKG Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkg-bom:v2.3.1'
Downloading the TKr Bill of Materials (BOM) file from 'projects.registry.vmware.com/tkg/tkr-bom:v1.26.8_vmware.1-tkg.2'
the old providers folder /home/tanzu-admin/.config/tanzu/tkg/providers is backed up to /home/tanzu-admin/.config/tanzu/tkg/providers-20231214101823-pp0m0yeg
? Specify provider "vsphere" or "azure" (vsphere)
? Enter vSphere username [email protected]
? Enter vSphere password ***************
? Enter vSphere TLS thumbprint F4:B4:45:53:C6:CC:C8:C9:EC:94:58:8A:24:5B:70:C7:0D:25:B5:7F
Updating credentials for management cluster "alg-tanzu-mgnt"
Credentials for management cluster is being updatedOn vérifie au bout de quelques secondes avec un tanzu mc get
tanzu-admin@alg-linux-tanzu:~$ tanzu mc get
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN TKR
alg-tanzu-mgnt tkg-system running 1/1 1/1 v1.26.5+vmware.2 management dev v1.26.5---vmware.2-tkg.1
Details:
NAME READY SEVERITY REASON SINCE MESSAGE
/alg-tanzu-mgnt True 48s
ClusterInfrastructure - VSphereCluster/alg-tanzu-mgnt-vtbkm True 48s
ControlPlane - KubeadmControlPlane/alg-tanzu-mgnt-xjqfm True 114d
Machine/alg-tanzu-mgnt-xjqfm-zvjxt True 114d
Workers
MachineDeployment/alg-tanzu-mgnt-md-0-hv8q6 True 14d
Machine/alg-tanzu-mgnt-md-0-hv8q6-6d89d5c6d7xp8fdh-xdhw2 True 114d Tout est revenu à la normale ! on va pouvoir continuer à utiliser nos clusters Tanzu ! (et lancer la mise à jour…)
