Failed to initialize CSINode: error updating CSINode annotation: timed out

May 06 21:41:42 node1.example.com  systemd[26469]: I0506 21:41:42.386428   26469 nodeinfomanager.go:402] Failed to publish CSINode: the server could not find the requested resource

May 06 21:41:42 node1.example.com  systemd[26469]: E0506 21:41:42.386469   26469 csi_plugin.go:273] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: the server could not find the requested resource

May 06 21:41:42 node1.example.com  systemd[26469]: F0506 21:41:42.386474   26469 csi_plugin.go:287] Failed to initialize CSINode after retrying: timed out waiting for the condition

or the CSI plugin pods are crashing with an error similar to below and they are in CrashLoopBackState:

I0519 03:09:55.089594       1 main.go:110] Version: v1.2.0-0-g6ef000ae

I0519 03:09:55.089644       1 connection.go:151] Connecting to unix:///csi/csi.sock

I0519 03:09:55.090799       1 node_register.go:58] Starting Registration Server at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock

I0519 03:09:55.091050       1 node_register.go:67] Registration Server started at: /registration/rook-ceph.cephfs.csi.ceph.com-reg.sock

I0519 03:09:55.937613       1 main.go:77] Received GetInfo call: &InfoRequest{}

I0519 03:09:56.937580       1 main.go:77] Received GetInfo call: &InfoRequest{}

I0519 03:09:58.343490       1 main.go:87] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource,}

E0519 03:09:58.343558       1 main.go:89] Registration process failed with error: RegisterPlugin error — plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: the server could not find the requested resource, restarting registration container.

Also:

csi-cephfsplugin-q6vk9                            2/3     CrashLoopBackOff   26 8        15h

csi-cephfsplugin-yy9bv                            2/3     CrashLoopBackOff   26 8        15h

csi-rbdplugin-88wn4                               2/3     CrashLoopBackOff   199        15h

csi-rbdplugin-8j8q9                               2/3     CrashLoopBackOff   264        15h

csi-rbdplugin-gz9t9                               2/3     CrashLoopBackOff   26 8        15h

Whats happening here? The CSI node-driver-registrar is a sidecar container that fetches driver information (using NodeGetInfo) from a CSI endpoint and registers it with the kubelet on that node using the kubelet plugin registration mechanism.From there on kubelet directly talk to the CSI driver through the registration socket for driver calls like NodeGetInfo, NodeStageVolume, NodePublishVolume ..etc. After registration, there is an object called CSINOde which is managed by kubelet: While the plugin comes up the registration failed for some reason.

apiVersion: storage.k8s.io/v1

kind: CSINode

metadata:

  name: node1

spec:

  drivers:

  – name: mycsidriver.example.com

    nodeID: storageNodeID1

    topologyKeys: [‘mycsidriver.example.com/regions’, “mycsidriver.example.com/zones”]

Few troubleshooting tips I can share here:

Whats your Kubernetes client and server versions?

Are they running on the same version ( say both are 1.17 or 1.18 ? ) ? if not, can you make sure both are the same or at least they are under-supported version skew? This is mostly popped up after an upgrade of one component to Kubernetes v1.17.

Share this post:

X (Twitter) Facebook LinkedIn Email