comet created SPARK-43329: ----------------------------- Summary: driver and executors shared same Kubernetes PVC in Spark 3.4+ Key: SPARK-43329 URL: https://issues.apache.org/jira/browse/SPARK-43329 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.4.0 Reporter: comet
I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below error. I would like all the executors and driver to mount the same PVC. Is this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because otherwise each of the executors will use an unique and separate PVC. "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors" below is how I enabled it pvc in spark 3.3 and it works, but does not work in Spark 3.4 *spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc \* *--conf spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir \* *--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc \* *--conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir* \ -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org