[
https://issues.apache.org/jira/browse/SPARK-32773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kousuke Saruta updated SPARK-32773:
-----------------------------------
Description:
Jars/files specified with --jars / --files options are listed by sc.listJars
and listFiles except when we run apps on YARN.
If we run apps not on YARN, those files are served by the embedded file server
in the driver and listJars/listFiles list the served files.
But with YARN, such files specified by the options are not served by the
embedded file server so listJars and listFiles don't list them.
was:
Jars/files specified with --jars / --files options are listed by sc.listJars
and listFiles except when we run apps on YARN.
If we run apps not on YARN, those files are served by the embedded file server
in the driver and listJars/listFiles list the served files.
But with YARN, such files specified by the options are not served by the
embedded file server so listJars and listFiles don't list them.
> The behavior of listJars and listFiles is not consistent between YARN and
> other cluster managers
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-32773
> URL: https://issues.apache.org/jira/browse/SPARK-32773
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 3.1.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Major
>
> Jars/files specified with --jars / --files options are listed by sc.listJars
> and listFiles except when we run apps on YARN.
> If we run apps not on YARN, those files are served by the embedded file
> server in the driver and listJars/listFiles list the served files.
> But with YARN, such files specified by the options are not served by the
> embedded file server so listJars and listFiles don't list them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
