[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416846#comment-17416846 ]
Stavros Kontopoulos edited comment on SPARK-23153 at 9/17/21, 6:21 PM: ----------------------------------------------------------------------- [~xuzhoyin] sorry for the late reply, the local scheme in the past meant local in the container, had a different meaning (https://github.com/apache/spark/pull/21378). So this was intentional. Not sure the status now. was (Author: skonto): [~xuzhoyin] sorry for the late reply, the local scheme in the past meant local in the container, had a different meaning (https://github.com/apache/spark/pull/21378). Not sure the status now. > Support application dependencies in submission client's local file system > ------------------------------------------------------------------------- > > Key: SPARK-23153 > URL: https://issues.apache.org/jira/browse/SPARK-23153 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Spark Core > Affects Versions: 2.4.0 > Reporter: Yinan Li > Assignee: Stavros Kontopoulos > Priority: Major > Fix For: 3.0.0 > > > Currently local dependencies are not supported with Spark on K8S i.e. if the > user has code or dependencies only on the client where they run > {{spark-submit}} then the current implementation has no way to make those > visible to the Spark application running inside the K8S pods that get > launched. This limits users to only running applications where the code and > dependencies are either baked into the Docker images used or where those are > available via some external and globally accessible file system e.g. HDFS > which are not viable options for many users and environments -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org