[ https://issues.apache.org/jira/browse/SPARK-33782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17746127#comment-17746127 ]
Szymon Kuryło commented on SPARK-33782: --------------------------------------- [~pratik.malani] I have a similar problem, and thus a question: Why doesn't your destination path contain the randomly named directory? it should be created inside the `KubernetesUtils.uploadFileUri` method. > Place spark.files, spark.jars and spark.files under the current working > directory on the driver in K8S cluster mode > ------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-33782 > URL: https://issues.apache.org/jira/browse/SPARK-33782 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.2.0 > Reporter: Hyukjin Kwon > Assignee: Pralabh Kumar > Priority: Major > Fix For: 3.4.0 > > > In Yarn cluster modes, the passed files are able to be accessed in the > current working directory. Looks like this is not the case in Kubernates > cluset mode. > By doing this, users can, for example, leverage PEX to manage Python > dependences in Apache Spark: > {code} > pex pyspark==3.0.1 pyarrow==0.15.1 pandas==0.25.3 -o myarchive.pex > PYSPARK_PYTHON=./myarchive.pex spark-submit --files myarchive.pex > {code} > See also https://github.com/apache/spark/pull/30735/files#r540935585. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org