[
https://issues.apache.org/jira/browse/SPARK-26290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yinan Li resolved SPARK-26290.
------------------------------
Resolution: Not A Bug
> [K8s] Driver Pods no mounted volumes on submissions from older spark versions
> -----------------------------------------------------------------------------
>
> Key: SPARK-26290
> URL: https://issues.apache.org/jira/browse/SPARK-26290
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Environment: Kuberentes: 1.10.6
> Container: Spark 2.4.0
> Spark containers are built from the archive served by
> [www.apache.org/dist/spark/|http://www.apache.org/dist/spark/]
> Submission done by older spark versions integrated e.g. in livy
> Reporter: Martin Buchleitner
> Priority: Major
>
> I want to use the volume feature to mount an existing PVC as readonly volume
> into the driver and also executor.
> The executor gets the PVC mounted, but the driver is missing the mount
> {code:java}
> /opt/spark/bin/spark-submit \
> --deploy-mode cluster \
> --class org.apache.spark.examples.SparkPi \
> --conf spark.app.name=spark-pi \
> --conf spark.executor.instances=4 \
> --conf spark.kubernetes.namespace=spark-demo \
> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
> --conf spark.kubernetes.container.image.pullPolicy=Always \
> --conf spark.kubernetes.container.image=kube-spark:2.4.0 \
> --conf spark.master=k8s://https://<master-ip> \
> --conf
> spark.kubernetes.driver.volumes.persistentVolumeClaim.ddata.mount.path=/srv \
> --conf
> spark.kubernetes.driver.volumes.persistentVolumeClaim.ddata.mount.readOnly=true
> \
> --conf
> spark.kubernetes.driver.volumes.persistentVolumeClaim.ddata.options.claimName=nfs-pvc
> \
> --conf
> spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path=/srv \
> --conf
> spark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly=true
> \
> --conf
> spark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName=nfs-pvc
> \
> /srv/spark-examples_2.11-2.4.0.jar
> {code}
> When i use the jar included in the container
> {code:java}
> local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
> {code}
> the call works and i can review the pod descriptions to review the behavior
> *Driver description*
> {code:java}
> Name: spark-pi-1544018157391-driver
> [...]
> Containers:
> spark-kubernetes-driver:
> Container ID:
> docker://3a31d867c140183247cb296e13a8b35d03835f7657dd7e625c59083024e51e28
> Image: kube-spark:2.4.0
> Image ID: [...]
> Port: <none>
> Host Port: <none>
> State: Terminated
> Reason: Completed
> Exit Code: 0
> Started: Wed, 05 Dec 2018 14:55:59 +0100
> Finished: Wed, 05 Dec 2018 14:56:08 +0100
> Ready: False
> Restart Count: 0
> Limits:
> memory: 1408Mi
> Requests:
> cpu: 1
> memory: 1Gi
> Environment:
> SPARK_DRIVER_MEMORY: 1g
> SPARK_DRIVER_CLASS: org.apache.spark.examples.SparkPi
> SPARK_DRIVER_ARGS:
> SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
> SPARK_MOUNTED_CLASSPATH:
> /opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
> SPARK_JAVA_OPT_1:
> -Dspark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.path=/srv
> SPARK_JAVA_OPT_3: -Dspark.app.name=spark-pi
> SPARK_JAVA_OPT_4:
> -Dspark.kubernetes.driver.volumes.persistentVolumeClaim.ddata.mount.path=/srv
> SPARK_JAVA_OPT_5: -Dspark.submit.deployMode=cluster
> SPARK_JAVA_OPT_6: -Dspark.driver.blockManager.port=7079
> SPARK_JAVA_OPT_7:
> -Dspark.kubernetes.driver.volumes.persistentVolumeClaim.ddata.mount.readOnly=true
> SPARK_JAVA_OPT_8:
> -Dspark.kubernetes.authenticate.driver.serviceAccountName=spark
> SPARK_JAVA_OPT_9:
> -Dspark.driver.host=spark-pi-1544018157391-driver-svc.spark-demo.svc.cluster.local
> SPARK_JAVA_OPT_10:
> -Dspark.kubernetes.driver.pod.name=spark-pi-1544018157391-driver
> SPARK_JAVA_OPT_11:
> -Dspark.kubernetes.driver.volumes.persistentVolumeClaim.ddata.options.claimName=nfs-pvc
> SPARK_JAVA_OPT_12:
> -Dspark.kubernetes.executor.volumes.persistentVolumeClaim.data.mount.readOnly=true
> SPARK_JAVA_OPT_13: -Dspark.driver.port=7078
> SPARK_JAVA_OPT_14:
> -Dspark.jars=/opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar
> SPARK_JAVA_OPT_15:
> -Dspark.kubernetes.executor.podNamePrefix=spark-pi-1544018157391
> SPARK_JAVA_OPT_16: -Dspark.local.dir=/tmp/spark-local
> SPARK_JAVA_OPT_17: -Dspark.master=k8s://https://<master-ip>
> SPARK_JAVA_OPT_18:
> -Dspark.app.id=spark-89420bd5fa8948c3aa9d14a4eb6ecfca
> SPARK_JAVA_OPT_19: -Dspark.kubernetes.namespace=spark-demo
> SPARK_JAVA_OPT_21: -Dspark.executor.instances=4
> SPARK_JAVA_OPT_22:
> -Dspark.kubernetes.executor.volumes.persistentVolumeClaim.data.options.claimName=nfs-pvc
> SPARK_JAVA_OPT_23:
> -Dspark.kubernetes.container.image=kube-spark:2.4.0
> SPARK_JAVA_OPT_24:
> -Dspark.kubernetes.container.image.pullPolicy=Always
> Mounts:
> /tmp/spark-local from spark-local-dir-0-spark-local (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from spark-token-nhcdd
> (ro)
> Conditions:
> Type Status
> Initialized True
> Ready False
> PodScheduled True
> Volumes:
> spark-local-dir-0-spark-local:
> Type: EmptyDir (a temporary directory that shares a pod's lifetime)
> Medium:
> spark-token-nhcdd:
> Type: Secret (a volume populated by a Secret)
> SecretName: spark-token-nhcdd
> Optional: false
> QoS Class: Burstable
> Node-Selectors: <none>
> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
> node.kubernetes.io/unreachable:NoExecute for 300s
> Events: <none>
> {code}
> *Executor description:*
> {code:java}
> Name: spark-pi-1544018157391-exec-2
> [..]
> Controlled By: Pod/spark-pi-1544018157391-driver
> Containers:
> executor:
> Container ID:
> docker://053256f023805a0a2fa580815f78203d2a32b0bc4e8e17741f45d84dd20a5e44
> Image: kube-spark:2.4.0
> Image ID: [...]
> Port: 7079/TCP
> Host Port: 0/TCP
> Args:
> executor
> State: Running
> Started: Wed, 05 Dec 2018 14:56:04 +0100
> Ready: True
> Restart Count: 0
> Limits:
> memory: 1408Mi
> Requests:
> cpu: 1
> memory: 1408Mi
> Environment:
> SPARK_DRIVER_URL:
> spark://coarsegrainedschedu...@spark-pi-1544018157391-driver-svc.spark-demo.svc.cluster.local:7078
> SPARK_EXECUTOR_CORES: 1
> SPARK_EXECUTOR_MEMORY: 1g
> SPARK_APPLICATION_ID: spark-application-1544018162183
> SPARK_CONF_DIR: /opt/spark/conf
> SPARK_EXECUTOR_ID: 2
> SPARK_EXECUTOR_POD_IP: (v1:status.podIP)
> SPARK_LOCAL_DIRS: /tmp/spark-local
> Mounts:
> /srv from data (ro)
> /tmp/spark-local from spark-local-dir-1 (rw)
> /var/run/secrets/kubernetes.io/serviceaccount from default-token-5srsx
> (ro)
> Conditions:
> Type Status
> Initialized True
> Ready True
> PodScheduled True
> Volumes:
> spark-local-dir-1:
> Type: EmptyDir (a temporary directory that shares a pod's lifetime)
> Medium:
> data:
> Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim
> in the same namespace)
> ClaimName: nfs-pvc
> ReadOnly: true
> default-token-5srsx:
> Type: Secret (a volume populated by a Secret)
> SecretName: default-token-5srsx
> Optional: false
> {code}
> I also tried to use hostPath but it reflected the same behavior
> I also reviewed the code which is doing those jobs and tried to find all
> available parameters, but there are not any parameters available except the
> subPath options. The Code of executor and driver looks exactly the same from
> my point of view.
>
> *Update*
> This behavior is based on using a spark-submit version older than 2.4 and
> there is no output which is related that you cannot expect this behavior or
> get warned !
> i rebuilt our livy installation based on spark 2.4 containers and the
> submission from spark 2.4 works fine!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
