[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 Ok, I will open another pr to remove it. Thanks a lot both of you. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17859 > Do we need remove the comments from template config? Ah, that would be a good idea. I also noticed it's still used in `YarnSparkHadoopUtil.scala`, so that could be removed too. I

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 @vanzin Thanks a lot for you review. Do we need remove the comments from template config? It doesn't work anymore in current version. --- If your project is set up for it, you can reply to this

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17859 This was intentionally removed. Env-based configs have been deprecated for a long time, and some were already removed. Use a proper configuration file (or command line option) for that. See

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 Hi @vanzin, can you help to take a look? The affect of `spark.executor.instances` changed after [SPARK-9092](https://github.com/apache/spark/pull/7657), but the comments in the config template

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 It only affect the YARN mode, just see the follow code: ``` OptionAssigner(args.numExecutors, YARN, ALL_DEPLOY_MODES, sysProp = "spark.executor.instances"), ``` But from now

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user srowen commented on the issue: https://github.com/apache/spark/pull/17859 This change causes it to affect all usages of spark-submit right? if you look at the comment about it in the config template, it suggests it is for YARN client mode only. I don't know the history of

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 Hi @srowen, the follow is the test result, the submit script: `./spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode client --verbose

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user srowen commented on the issue: https://github.com/apache/spark/pull/17859 No, I believe that only exists for YARN client mode. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17859 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #17859: [SPARK-20595][Deploy]Parse the 'SPARK_EXECUTOR_INSTANCES...

Github user ConeyLiu commented on the issue: https://github.com/apache/spark/pull/17859 @srowen @HyukjinKwon Can you take a look, thanks a lot. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not