[
https://issues.apache.org/jira/browse/SPARK-43540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Wang updated SPARK-43540:
-----------------------------
Description:
In Yarn cluster modes, the passed files/jars are able to be accessed in the
classloader. Looks like this is not the case in Kubernetes cluster mode.
After SPARK-33782, it places spark.files, spark.jars and spark.files under the
current working directory on the driver in K8S cluster mode. but the
spark.files and spark.jars seems are not accessible by the classloader.
we need to add the current working directory into classpath.
was:
In Yarn cluster modes, the passed files/jars are able to be accessed in the
classloader. Looks like this is not the case in Kubernetes cluster mode.
After SPARK-33782, it places spark.files, spark.jars and spark.files under the
current working directory on the driver in K8S cluster mode. but the
spark.files and spark.jars seems are not accessible by the classloader.
we need to add the current working directory to classpath.
> Add current working directory into classpath on the driver in K8S cluster mode
> ------------------------------------------------------------------------------
>
> Key: SPARK-43540
> URL: https://issues.apache.org/jira/browse/SPARK-43540
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Fei Wang
> Priority: Major
>
> In Yarn cluster modes, the passed files/jars are able to be accessed in the
> classloader. Looks like this is not the case in Kubernetes cluster mode.
> After SPARK-33782, it places spark.files, spark.jars and spark.files under
> the current working directory on the driver in K8S cluster mode. but the
> spark.files and spark.jars seems are not accessible by the classloader.
>
> we need to add the current working directory into classpath.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
