[ https://issues.apache.org/jira/browse/SPARK-35715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475135#comment-17475135 ]
jingxiong zhong commented on SPARK-35715: ----------------------------------------- t seems that spark 3 does not support the schema using local as the path. You can try file:///etc/xattr.conf > Option "--files" with local:// prefix is not honoured for Spark on kubernetes > ----------------------------------------------------------------------------- > > Key: SPARK-35715 > URL: https://issues.apache.org/jira/browse/SPARK-35715 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.0.2, 3.1.2 > Reporter: Pardhu Madipalli > Priority: Major > > When we provide a local file as a dependency using "--files" option, the file > is not getting copied to work directories of executors. > h5. Example 1: > > {code:java} > $SPARK_HOME/bin/spark-submit --master k8s://https://<ip-address-k8s> \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=1 \ > --conf spark.kubernetes.container.image=<spark-3.1.2-image> \ > --conf spark.kubernetes.driver.pod.name=sparkdriverpod \ > --files local:///etc/xattr.conf \ > local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar 1000000 > {code} > > h6. Content of Spark Executor work-dir: > > {code:java} > ~$ kubectl exec -n default spark-pi-22de6279f6bec01c-exec-1 ls > /opt/spark/work-dir/ > spark-examples_2.12-3.1.2.jar > {code} > > We can notice here that the file _/etc/xattr.conf_ is *NOT* copied to > _/opt/spark/work-dir/ ._ > > ---- > > {{Instead of using "–files", if we use "--jars" option the file is getting > copied as expected.}} > h5. Example 2: > {code:java} > $SPARK_HOME/bin/spark-submit --master k8s://https://<ip-address-k8s> \ > --deploy-mode cluster \ > --name spark-pi \ > --class org.apache.spark.examples.SparkPi \ > --conf spark.executor.instances=1 \ > --conf spark.kubernetes.container.image=<spark-3.1.2-image> \ > --conf spark.kubernetes.driver.pod.name=sparkdriverpod \ > --jars local:///etc/xattr.conf \ > local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar 1000000 > {code} > h6. Content of Spark Executor work-dir: > > {code:java} > ~$ kubectl exec -n default spark-pi-22de6279f6bec01c-exec-1 ls > /opt/spark/work-dir/ > spark-examples_2.12-3.1.2.jar > xattr.conf > {code} > We can notice here that the file _/etc/xattr.conf_ *IS COPIED* to > _/opt/spark/work-dir/ ._ > > I tested this with versions *3.1.2* and *3.0.2*. It is behaving the same way > in both cases. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org