[terminal]
May 06 21:41:42 node1.example.com systemd[26469]: I0506 21:41:42.386428 26469 nodeinfomanager.go:402] Failed to publish CSINode: the server could not find the requested resource
May 06 21:41:42 node1.example.com systemd[26469]: E0506 21:41:42.386469 26469 csi_plugin.go:273] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource
May 06 21:41:42 node1.example.com systemd[26469]: F0506 21:41:42.386474 26469 csi_plugin.go:287] Failed to initialize CSINode after retrying: timed out waiting for the condition
[/terminal]
or the CSI plugin pods are crashing with an error similar to below and they are in `CrashLoopBackState`:
[terminal]
I0519 03:09:55.089594 1 main.go:110] Version: v1.2.0-0-g6ef000ae
I0519 03:09:55.089644 1 connection.go:151] Connecting to unix:///csi/csi.sock
I0519 03:09:55.090799 1 node_register.go:58] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
I0519 03:09:55.091050 1 node_register.go:67] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock
I0519 03:09:55.937613 1 main.go:77] Received GetInfo call: &InfoRequest{}
I0519 03:09:56.937580 1 main.go:77] Received GetInfo call: &InfoRequest{}
I0519 03:09:58.343490 1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource,}
E0519 03:09:58.343558 1 main.go:89] Registration process failed with error: RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource, restarting registration container.
[/terminal]
Also:
[terminal]
csi-cephfsplugin-q6vk9 2/3 CrashLoopBackOff 26 8 15h
csi-cephfsplugin-yy9bv 2/3 CrashLoopBackOff 26 8 15h
csi-rbdplugin-88wn4 2/3 CrashLoopBackOff 199 15h
csi-rbdplugin-8j8q9 2/3 CrashLoopBackOff 264 15h
csi-rbdplugin-gz9t9 2/3 CrashLoopBackOff 26 8 15h
[/terminal]
Whats happening here? The CSI node-driver-registrar is a sidecar container that fetches driver information (using `NodeGetInfo`) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.From there on kubelet directly talk to the CSI driver through the registration socket for driver calls like NodeGetInfo, NodeStageVolume, NodePublishVolume ..etc. After registration, there is an object called CSINOde which is managed by kubelet: While the plugin comes up the registration failed for some reason.
[terminal]
apiVersion: storage.k8s.io/v1
kind: CSINode
metadata:
name: node1
spec:
drivers:
– name: mycsidriver.example.com
nodeID: storageNodeID1
topologyKeys: [‘mycsidriver.example.com/regions’, “mycsidriver.example.com/zones”]
[/terminal]
Few troubleshooting tips I can share here:
Whats your Kubernetes client and server versions?
Are they running on the same version ( say both are 1.17 or 1.18 ? ) ? if not, can you make sure both are the same or at least they are under-supported version skew? This is mostly popped up after an upgrade of one component to Kubernetes v1.17.
Copyright secured by Digiprove © 2020 Humble Chirammal