[jira] [Commented] (SPARK-1680) Clean up use of setExecutorEnvs in SparkConf
[ https://issues.apache.org/jira/browse/SPARK-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110979#comment-15110979 ] Apache Spark commented on SPARK-1680: - User 'weineran' has created a pull request for this issue: https://github.com/apache/spark/pull/10869 > Clean up use of setExecutorEnvs in SparkConf > - > > Key: SPARK-1680 > URL: https://issues.apache.org/jira/browse/SPARK-1680 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Patrick Wendell >Assignee: Thomas Graves >Priority: Blocker > Fix For: 1.1.0 > > > We should make this consistent between YARN and Standalone. Basically, YARN > mode should just use the executorEnvs from the Spark conf and not need > SPARK_YARN_USER_ENV. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-1680) Clean up use of setExecutorEnvs in SparkConf
[ https://issues.apache.org/jira/browse/SPARK-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068584#comment-14068584 ] Apache Spark commented on SPARK-1680: - User 'tgravescs' has created a pull request for this issue: https://github.com/apache/spark/pull/1512 > Clean up use of setExecutorEnvs in SparkConf > - > > Key: SPARK-1680 > URL: https://issues.apache.org/jira/browse/SPARK-1680 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Patrick Wendell >Assignee: Thomas Graves >Priority: Blocker > Fix For: 1.1.0 > > > We should make this consistent between YARN and Standalone. Basically, YARN > mode should just use the executorEnvs from the Spark conf and not need > SPARK_YARN_USER_ENV. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1680) Clean up use of setExecutorEnvs in SparkConf
[ https://issues.apache.org/jira/browse/SPARK-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14068581#comment-14068581 ] Thomas Graves commented on SPARK-1680: -- https://github.com/apache/spark/pull/1512 > Clean up use of setExecutorEnvs in SparkConf > - > > Key: SPARK-1680 > URL: https://issues.apache.org/jira/browse/SPARK-1680 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Patrick Wendell >Assignee: Thomas Graves >Priority: Blocker > Fix For: 1.1.0 > > > We should make this consistent between YARN and Standalone. Basically, YARN > mode should just use the executorEnvs from the Spark conf and not need > SPARK_YARN_USER_ENV. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1680) Clean up use of setExecutorEnvs in SparkConf
[ https://issues.apache.org/jira/browse/SPARK-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066872#comment-14066872 ] Thomas Graves commented on SPARK-1680: -- so what do we want to do here. Do we want to expose spark.executorEnv. to the user to allow them to set environment variables? I assume since it isn't documented it wasn't meant for end users but perhaps we just missed documenting it. I would like to add a config to replace the env variable SPARK_YARN_USER_ENV. Either way I will probably need another config to set env variables on the application master. I could make that one yarn specific like spark.yarn.appMasterEnv. > Clean up use of setExecutorEnvs in SparkConf > - > > Key: SPARK-1680 > URL: https://issues.apache.org/jira/browse/SPARK-1680 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Reporter: Patrick Wendell >Priority: Blocker > Fix For: 1.1.0 > > > We should make this consistent between YARN and Standalone. Basically, YARN > mode should just use the executorEnvs from the Spark conf and not need > SPARK_YARN_USER_ENV. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (SPARK-1680) Clean up use of setExecutorEnvs in SparkConf
[ https://issues.apache.org/jira/browse/SPARK-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13989548#comment-13989548 ] Thomas Graves commented on SPARK-1680: -- I think this is dup of SPARK-1665. > Clean up use of setExecutorEnvs in SparkConf > - > > Key: SPARK-1680 > URL: https://issues.apache.org/jira/browse/SPARK-1680 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 1.0.0 >Reporter: Patrick Wendell >Priority: Blocker > Fix For: 1.0.0 > > > This was added in 0.9.0 but the config change removed propagation of these > values to executors. We should make one of two decisions: > 1. Don't allow setting arbitrary executor envs in standalone mode. > 2. Document this option, respect the env variables when launching a job, and > consolidate this with SPARK_YARN_USER_ENV. -- This message was sent by Atlassian JIRA (v6.2#6252)
