[ https://issues.apache.org/jira/browse/SPARK-43329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
comet updated SPARK-43329: -------------------------- Description: I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below error. I would like all the executors and driver to mount the same PVC. Is this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because otherwise each of the executors will use an unique and separate PVC. "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors" below is how I enabled it pvc in spark 3.3 and it works, but does not work in Spark 3.4 {code:sh} spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir {code} was: I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below error. I would like all the executors and driver to mount the same PVC. Is this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because otherwise each of the executors will use an unique and separate PVC. "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple executors" below is how I enabled it pvc in spark 3.3 and it works, but does not work in Spark 3.4 {code:shell} spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc --conf spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc --conf spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir {code} > driver and executors shared same Kubernetes PVC in Spark 3.4+ > ------------------------------------------------------------- > > Key: SPARK-43329 > URL: https://issues.apache.org/jira/browse/SPARK-43329 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.4.0 > Reporter: comet > Priority: Major > > I able to shared same PVC for spark 3.3. but on Spark 3.4 onward. i get below > error. I would like all the executors and driver to mount the same PVC. Is > this a bug ? I don't want to use SPARK_EXECUTOR_ID or OnDemand because > otherwise each of the executors will use an unique and separate PVC. > > "should contain OnDemand or SPARK_EXECUTOR_ID when requiring multiple > executors" > > below is how I enabled it pvc in spark 3.3 and it works, but does not work in > Spark 3.4 > {code:sh} > spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc > > --conf > spark.kubernetes.driver.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir > > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.options.claimName=rwxpvc > > --conf > spark.kubernetes.executor.volumes.persistentVolumeClaim.rwxpvc.mount.path=/opt/spark/work-dir > > > {code} > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org