Topics covered in this article:
Problem statement:
All the pods are stuck in `pending` state.
NAME READY STATUS RESTARTS AGE
cspc-operator-656cd6cb79-94np2 0/1 Pending 0 12m
cvc-operator-59965778fd-xqtgm 0/1 Pending 0 12m
maya-apiserver-967595b5c-jtbjd 0/1 Pending 0 12m
openebs-admission-server-6c7fd4779-bvnp5 0/1 Pending 0 12m
openebs-localpv-provisioner-c6cdf9dbd-rsrrl 0/1 Pending 0 12m
openebs-ndm-operator-c66d6449d-b2kmm 0/1 Pending 0 12m
openebs-provisioner-7c7ccfdd69-tr4g7 0/1 Pending 0 12m
openebs-snapshot-operator-6c4559fc7c-jmtsz 0/2 Pending 0 12m
This was encountered when the nodes of a cluster got deleted and a new set of nodes were added to the cluster(either manually or by enabling autoscaling feature).
Why did this happen?
This problem occurred due to the absence of node management on clusters that have OpenEBS installed using Director.
On describing the ndm pod, a warning similar to the one given below will be mentioned.
Warning FailedScheduling 5m26s (x70 over 107m) default-scheduler 0/1 nodes are available: 1 node(s) didn't match node selector.
Troubleshooting:
The workaround for this can be adding labels to the nodes. This can be done using the below mentioned command:
kubectl label node --all mayadata.io/data-plane=true
kubectl label node --all mayadata.io/control-plane=true
In case only some specific nodes need to be labeled, replace "--all" with the desired nodeName.
kubectl label node <nodeName> mayadata.io/data-plane=true
kubectl label node <nodeName> mayadata.io/control-plane=true
All the pods should now be running. To verify, execute:
kubectl get pods -n openebs
Output:
NAME READY STATUS RESTARTS AGE
cspc-operator-656cd6cb79-94np2 1/1 Running 0 14m
cvc-operator-59965778fd-xqtgm 1/1 Running 0 14m
maya-apiserver-967595b5c-jtbjd 1/1 Running 0 14m
openebs-admission-server-6c7fd4779-bvnp5 1/1 Running 0 14m
openebs-localpv-provisioner-c6cdf9dbd-rsrrl 1/1 Running 0 14m
openebs-ndm-gpvmf 1/1 Running 0 12m
openebs-ndm-operator-c66d6449d-b2kmm 1/1 Running 0 14m
openebs-provisioner-7c7ccfdd69-tr4g7 1/1 Running 0 14m
openebs-snapshot-operator-6c4559fc7c-jmtsz 2/2 Running 0 14m