[
https://issues.apache.org/jira/browse/SPARK-5222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Or updated SPARK-5222:
-----------------------------
Description:
The behavior is summarized in a table produced by [~WangTaoTheTonic] here:
https://github.com/apache/spark/pull/3557
SPARK_YARN_APP_NAME is respected only in client mode but not in cluster mode.
This results in the strange behavior where the app name changes if the user
runs the same application but uses a different deploy mode from before. We
should make sure the app name behavior is consistent across deploy modes
regardless of what variable or config is set.
Additionally, it should be noted that because "spark.app.name" is required of
all applications, the setting of "SPARK_YARN_APP_NAME" will not take effect
unless we handle it preemptively in Spark submit.
was:
The behavior is summarized in a table produced by [~WangTaoTheTonic] here:
https://github.com/apache/spark/pull/3557
SPARK_YARN_APP_NAME is respected only in client mode but not in cluster mode.
This results in the strange behavior where the app name changes if the user
runs the same application but uses a different deploy mode from before.
We should make sure the app name behavior is consistent across deploy modes
regardless of what variable or config is set.
> YARN client and cluster modes have different app name behaviors
> ---------------------------------------------------------------
>
> Key: SPARK-5222
> URL: https://issues.apache.org/jira/browse/SPARK-5222
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Andrew Or
> Assignee: WangTaoTheTonic
>
> The behavior is summarized in a table produced by [~WangTaoTheTonic] here:
> https://github.com/apache/spark/pull/3557
> SPARK_YARN_APP_NAME is respected only in client mode but not in cluster mode.
> This results in the strange behavior where the app name changes if the user
> runs the same application but uses a different deploy mode from before. We
> should make sure the app name behavior is consistent across deploy modes
> regardless of what variable or config is set.
> Additionally, it should be noted that because "spark.app.name" is required of
> all applications, the setting of "SPARK_YARN_APP_NAME" will not take effect
> unless we handle it preemptively in Spark submit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
