[
https://issues.apache.org/jira/browse/SPARK-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-1755:
-----------------------------
Description:
In YARN client mode, --name is ignored because the deploy mode is client, and
the name is for some reason a [cluster
config|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170)].
In YARN cluster mode, --name is passed to the
org.apache.spark.deploy.yarn.Client as a command line argument. The Client
class, however, uses this name only as the [app name for the
RM|https://github.com/apache/spark/blob/master/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L80],
but not for Spark. In other words, when SparkConf attempts to load default
configs, application name is not set.
In both cases, passing --name to SparkSubmit does not actually cause Spark to
adopt it as its application name, despite what the usage promises.
was:
In YARN client mode, --name is ignored because the deploy mode is client, and
the name is for some reason a [cluster
config|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170)].
In YARN cluster mode, --name is passed to the
org.apache.spark.deploy.yarn.Client as a command line argument. The Client
class, however, uses this name only as the app name for the RM, but not for
Spark. In other words, when SparkConf attempts to load default configs,
application name is not set.
In both cases, passing --name to SparkSubmit does not actually cause Spark to
adopt it as its application name, despite what the usage promises.
> Spark-submit --name does not resolve to application name on YARN
> ----------------------------------------------------------------
>
> Key: SPARK-1755
> URL: https://issues.apache.org/jira/browse/SPARK-1755
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.0.0
> Reporter: Andrew Or
> Fix For: 1.0.1
>
>
> In YARN client mode, --name is ignored because the deploy mode is client, and
> the name is for some reason a [cluster
> config|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L170)].
> In YARN cluster mode, --name is passed to the
> org.apache.spark.deploy.yarn.Client as a command line argument. The Client
> class, however, uses this name only as the [app name for the
> RM|https://github.com/apache/spark/blob/master/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L80],
> but not for Spark. In other words, when SparkConf attempts to load default
> configs, application name is not set.
> In both cases, passing --name to SparkSubmit does not actually cause Spark to
> adopt it as its application name, despite what the usage promises.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
