nickumia-reisys commented on issue #399: URL: https://github.com/apache/solr-operator/issues/399#issuecomment-1024745521
Hmmm.... so upon further research, it seems like there's a race condition happening in either the `solr` or `solr-operator` helm charts. Inspecting the pvc that `solr` tries to create,  Inspecting the pod that `solr-operator` tries to create, <details> <summary> Full Pod Inspection </summary> ``` nickumia@DL62-2-2MDD043:~/datagov-brokerpak$ kubectl describe pods solr-50d858e0985ecc7f-solrcloud-0 Name: solr-50d858e0985ecc7f-solrcloud-0 Namespace: default Priority: 2000001000 Priority Class Name: system-node-critical Node: <none> Labels: app.kubernetes.io/instance=solr-50d858e0985ecc7f app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=solr app.kubernetes.io/version=8.9.0 controller-revision-hash=solr-50d858e0985ecc7f-solrcloud-74f75459bd eks.amazonaws.com/fargate-profile=default-namespaces-k8s-5fcb154753056b9c helm.sh/chart=solr-0.5.0 solr-cloud=solr-50d858e0985ecc7f statefulset.kubernetes.io/pod-name=solr-50d858e0985ecc7f-solrcloud-0 technology=solr-cloud Annotations: kubernetes.io/psp: eks.privileged solr.apache.org/nextScheduledRestart: 2022-01-28T23:20:00Z solr.apache.org/solrXmlMd5: 843652bc6b529b66f46bcdae6764ab4e Status: Pending IP: IPs: <none> Controlled By: StatefulSet/solr-50d858e0985ecc7f-solrcloud Init Containers: cp-solr-xml: Image: library/busybox:1.28.0-glibc Port: <none> Host Port: <none> Command: sh -c cp /tmp/solr.xml /tmp-config/solr.xml Environment: <none> Mounts: /tmp from solr-xml (rw) /tmp-config from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-qzccw (ro) setup-zk: Image: docker.io/solr:8.11 Port: <none> Host Port: <none> Command: sh -c ZK_SECURITY_JSON=$(/opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost ${ZK_HOST} -cmd get /security.json); if [ ${#ZK_SECURITY_JSON} -lt 3 ]; then echo $SECURITY_JSON > /tmp/security.json; /opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost ${ZK_HOST} -cmd putfile /security.json /tmp/security.json; echo "put security.json in ZK"; fi Environment: ZK_HOST: solr-50d858e0985ecc7f-solrcloud-zookeeper-0.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-1.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-2.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181/ ZK_CHROOT: / ZK_SERVER: solr-50d858e0985ecc7f-solrcloud-zookeeper-0.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-1.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-2.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181 SECURITY_JSON: <set to the key 'security.json' in secret 'solr-50d858e0985ecc7f-solrcloud-security-bootstrap'> Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-qzccw (ro) Containers: solrcloud-node: Image: docker.io/solr:8.11 Port: 8983/TCP Host Port: 0/TCP Requests: cpu: 1 memory: 1G Liveness: http-get http://:8983/solr/admin/info/system delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8983/solr/admin/info/system delay=15s timeout=1s period=5s #success=1 #failure=3 Environment: SOLR_JAVA_MEM: -Xms300m -Xmx300m SOLR_HOME: /var/solr/data SOLR_PORT: 8983 SOLR_NODE_PORT: 80 POD_HOSTNAME: solr-50d858e0985ecc7f-solrcloud-0 (v1:metadata.name) SOLR_HOST: default-$(POD_HOSTNAME).sub-nickumia40.ssb-dev.data.gov SOLR_LOG_LEVEL: INFO GC_TUNE: SOLR_STOP_WAIT: 55 ZK_HOST: solr-50d858e0985ecc7f-solrcloud-zookeeper-0.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-1.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-2.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181/ ZK_CHROOT: / ZK_SERVER: solr-50d858e0985ecc7f-solrcloud-zookeeper-0.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-1.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-50d858e0985ecc7f-solrcloud-zookeeper-2.solr-50d858e0985ecc7f-solrcloud-zookeeper-headless.default.svc.cluster.local:2181 SOLR_OPTS: -DhostPort=$(SOLR_NODE_PORT) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-qzccw (ro) /var/solr/data from data (rw) Conditions: Type Status PodScheduled False Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-solr-50d858e0985ecc7f-solrcloud-0 ReadOnly: false solr-xml: Type: ConfigMap (a volume populated by a ConfigMap) Name: solr-50d858e0985ecc7f-solrcloud-configmap Optional: false default-token-qzccw: Type: Secret (a volume populated by a Secret) SecretName: default-token-qzccw Optional: false QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 42m fargate-scheduler Pod not supported on Fargate: volumes not supported: data not supported because: PVC data-solr-50d858e0985ecc7f-solrcloud-0 not bound ``` </details>  From my understanding the order of operations is, 1. Create PV 2. Create PVC 3. Create Pod It seems like the pod is waiting for the PVC to be bound but the PVC is not getting bound because it's waiting for the pod to start up. This may only be an issue because of Fargate issues. I was trying to inspect the [solr helm chart](https://github.com/apache/solr-operator/blob/main/helm/solr/templates/solrcloud.yaml) for an order of operations, but didn't see anything that would represent an order of operations. Can you help verify that there aren't any complex dependencies between the PVC creation and the Pod creation? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
