[ 
https://issues.apache.org/jira/browse/SPARK-38223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17493259#comment-17493259
 ] 

Zimo Li commented on SPARK-38223:
---------------------------------

[~dongjoon] this may relate to your changes since you added 
ownPersistentVolumeClaim.

> PersistentVolumeClaim does not work in clusters with multiple nodes
> -------------------------------------------------------------------
>
>                 Key: SPARK-38223
>                 URL: https://issues.apache.org/jira/browse/SPARK-38223
>             Project: Spark
>          Issue Type: Bug
>          Components: Kubernetes
>    Affects Versions: 3.2.1
>         Environment: 
> [https://spark.apache.org/docs/latest/running-on-kubernetes.html#how-it-works]
> [https://spark.apache.org/docs/latest/running-on-kubernetes.html#using-kubernetes-volumes]
> [https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes]
>  
>            Reporter: Zimo Li
>            Priority: Minor
>
> We are using {{spark-submit}} to establish a ThriftServer warehouse on Google 
> Kubernetes Engine. The Spark documentation on running on Kubernetes suggests 
> that we can use 
> [persistentVolumeClaim|https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim]
>  for Spark applications.
> {code:bash}
> spark-submit \
>   --master k8s://$KUBERNETES_SERVICE_HOST \
>   --deploy-mode cluster \
>   --class $THRIFTSERVER \
>   --conf spark.sql.catalogImplementation=hive \
>   --conf spark.sql.hive.metastore.sharedPrefixes=org.postgresql \
>   --conf spark.hadoop.hive.metastore.schema.verification=false \
>   --conf spark.hadoop.datanucleus.schema.autoCreateTables=true \
>   --conf spark.hadoop.datanucleus.autoCreateSchema=false \
>   --conf spark.sql.parquet.int96RebaseModeInWrite=CORRECTED \
>   --conf 
> spark.hadoop.javax.jdo.option.ConnectionDriverName=org.postgresql.Driver \
>   --conf spark.hadoop.javax.jdo.option.ConnectionUserName=spark \
>   --conf spark.hadoop.javax.jdo.option.ConnectionPassword=Password1! \
>   --conf spark.sql.warehouse.dir=$MOUNT_PATH \
>   --conf spark.kubernetes.driver.pod.name=spark-hive-thriftserver-driver \
>   --conf spark.kubernetes.driver.label.app.kubernetes.io/name=thriftserver \
>   --conf 
> spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$CLAIM_NAME
>  \
>   --conf 
> spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH
>  \
>   --conf 
> spark.kubernetes.executor.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.readOnly=false
>  \
>   --conf 
> spark.kubernetes.driver.volumes.persistentVolumeClaim.$VOLUME_NAME.options.claimName=$CLAIM_NAME
>  \
>   --conf 
> spark.kubernetes.driver.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.path=$MOUNT_PATH
>  \
>   --conf 
> spark.kubernetes.driver.volumes.persistentVolumeClaim.$VOLUME_NAME.mount.readOnly=false
>  \
>   --conf spark.kubernetes.executor.deleteOnTermination=true \
>   --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-kube \
>   --conf spark.kubernetes.container.image=$IMAGE \
>   --conf spark.kubernetes.container.image.pullPolicy=Always \
>   --conf spark.executor.memory=2g \
>   --conf spark.driver.memory=2g \
>   local:///$JAR {code}
> When it ran, it created one driver and two executors. Each of these wanted to 
> use the same pvc. Unfortunately, at least one of these pods was scheduled on 
> a different node from the rest. As GKE mounts pvs to nodes in order to honor 
> pvcs for pods, that odd pod out was unable to attach the pv:
> {code:java}
> FailedMount
> Unable to attach or mount volumes: unmounted volumes=[spark-warehouse], 
> unattached volumes=[kube-api-access-grfld spark-conf-volume-exec 
> spark-warehouse spark-local-dir-1]: timed out waiting for the condition {code}
> This is because GKE like many cloud providers does not support 
> {{ReadWriteMany}} for pvcs/pvs.
> ----
> I suggest changing the documentation not to suggest using pvcs for 
> ThriftServers.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to