Saisai Shao created SPARK-23635: ----------------------------------- Summary: Spark executor env variable is overwritten by same name AM env variable Key: SPARK-23635 URL: https://issues.apache.org/jira/browse/SPARK-23635 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 2.3.0 Reporter: Saisai Shao
In the current Spark on YARN code, AM always will copy and overwrite its env variables to executors, so we cannot set different values of executors. To reproduce issue, user could start spark-shell like: {code} ./bin/spark-shell --master yarn-client --conf spark.executorEnv.SPARK_ABC=executor_val --conf spark.yarn.appMasterEnv.SPARK_ABC=am_val {code} Then check executor env variables by {code} sc.parallelize(1 to 1).flatMap \{ i => sys.env.toSeq }.collect.foreach(println) {code} You will always get \{{am_val}} instead of {{executor_val}}. So we should not let AM to overwrite specifically set executor env variables. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org