[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/16819
So your current approach is to have a second connection to the RM, and ask
for the RM's available resources every time the scheduler tries to change the
number of resources.
Did you look at Tom's suggestion of using
{{AllocateResponse.getAvailableResources()}} instead? Seems like it would be
simpler, cheaper, and could all be handled internally in
{{YarnAllocator.scala}}.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/16819 @vanzin What do you think about current approach? I have tested on a same Spark hive-thriftserver, the `spark.dynamicAllocation.maxExecutors` wiil decrease if I kill 4 NodeManager: ``` 17/02/27 15:58:08 DEBUG ExecutorAllocationManager: Not adding executors because our current target total is already 94 (limit 94) 17/02/27 15:58:09 DEBUG ExecutorAllocationManager: Not adding executors because our current target total is already 94 (limit 94) 17/02/27 16:05:49 DEBUG ExecutorAllocationManager: Not adding executors because our current target total is already 85 (limit 85) 17/02/27 16:05:49 DEBUG ExecutorAllocationManager: Not adding executors because our current target total is already 85 (limit 85) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/16819 I agree with others, this is not the way to do this. There are different schedulers in yarn, each with different configs that could affect the actual resources you get. If you want to do something like this it should look at the available resources after calling the allocate call to yarn (allocateResponse.getAvailableResources). When yarn returns it tells you the available resources, which takes into account the various scheduler things. MapReduce refers to that as headroom and uses it to determine things like if it needs to kill a reducer to run a map. We could use this to help with dynamic allocation and do more intelligent things. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73515/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73515/testReport)** for PR 16819 at commit [`e4b3b0c`](https://github.com/apache/spark/commit/e4b3b0c9dfcb900b0c69329966cbc8030f68c1c3). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73515/testReport)** for PR 16819 at commit [`e4b3b0c`](https://github.com/apache/spark/commit/e4b3b0c9dfcb900b0c69329966cbc8030f68c1c3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16819 Getting the config only at the beginning, to me, is not an acceptable solution. Getting it every once in a while is better, but it's not the only possible approach. I even suggest something different in the bug I mention above. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/16819 @vanzin We must pull the configuration from ResourceManager, ResourceManager can't push. So setting the max before each stage? This feels too frequent. In fact, This is suitable for periodic tasks. e.g. ML, SQL, For streaming jobs, it is better set manually. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16819 I agree there's room for improvement in the current code; I even asked SPARK-18769 to be filed to track that work. But I don't think setting the max to a fixed value at startup is the right approach. Queue configs change, node managers go up and down, new ones are added, old ones are removed. If this value ends up being calculated at the wrong time, the application will suffer. If you want to investigate a more dynamic approach here I'm all for that, but I'm not a big fan of the current solution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73282/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73282/testReport)** for PR 16819 at commit [`cd306e2`](https://github.com/apache/spark/commit/cd306e2662b9ed60cf21fe8cb234b232ddc9aca2). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73282/testReport)** for PR 16819 at commit [`cd306e2`](https://github.com/apache/spark/commit/cd306e2662b9ed60cf21fe8cb234b232ddc9aca2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73277/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73277/testReport)** for PR 16819 at commit [`fabe2c5`](https://github.com/apache/spark/commit/fabe2c572db80bde639984f6e3834239b9590bfc). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73277/testReport)** for PR 16819 at commit [`fabe2c5`](https://github.com/apache/spark/commit/fabe2c572db80bde639984f6e3834239b9590bfc). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73151/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73151 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73151/testReport)** for PR 16819 at commit [`8e99701`](https://github.com/apache/spark/commit/8e9970107c8e74b57718398d4972af7d4709ec2d). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73151 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73151/testReport)** for PR 16819 at commit [`8e99701`](https://github.com/apache/spark/commit/8e9970107c8e74b57718398d4972af7d4709ec2d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73147/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73147/testReport)** for PR 16819 at commit [`4f81680`](https://github.com/apache/spark/commit/4f81680364c16e5e70b65e785a439c184b1313e3). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #73147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73147/testReport)** for PR 16819 at commit [`4f81680`](https://github.com/apache/spark/commit/4f81680364c16e5e70b65e785a439c184b1313e3). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/16819 @srowen . Dynamic set `spark.dynamicAllocation.maxExecutors` can avoid some strange problems: 1. [Spark application hang when dynamic allocation is enabled](https://issues.apache.org/jira/browse/SPARK-16441) 2. [Report failure reason from Reporter Thread](https://issues.apache.org/jira/browse/SPARK-19226) 3. CLI shows successful but web ui didn't, simally to [this](https://issues.apache.org/jira/secure/attachment/12846513/can-not-consume-taskEnd-events.jpg) I add a unit test just now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16819 What problem does this solve though? calling that function is not a problem. It seems like you get the right behavior in both cases. Are you saying there's some RPC problem? The target goes very high, but, as far as I can see it's correctly reflecting the fact that the app would use a lot of executors if it could -- that's fine. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/16819 It will reduce the function call on [CoarseGrainedSchedulerBackend.requestTotalExecutors()](https://github.com/apache/spark/blob/v2.1.0/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L491) after apply this PR: [before](https://github.com/apache/spark/files/756685/doNotApply-PR-16819.txt) [after-apply-this-PR](https://github.com/apache/spark/files/756684/apply-PR-16819.txt) Full log can be found [here](https://issues.apache.org/jira/secure/attachment/12851318/SPARK-16441-compare-apply-PR-16819.zip). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user andrewor14 commented on the issue: https://github.com/apache/spark/pull/16819 I agree. Resource managers generally expect applications to request more than what's available already so we don't have to do it again ourselves in Spark. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72434/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16819 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #72434 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72434/testReport)** for PR 16819 at commit [`97e5eee`](https://github.com/apache/spark/commit/97e5eee6aaf2335a2af62e816b767d408c37a59e). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16819 **[Test build #72434 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72434/testReport)** for PR 16819 at commit [`97e5eee`](https://github.com/apache/spark/commit/97e5eee6aaf2335a2af62e816b767d408c37a59e). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #16819: [SPARK-16441][YARN] Set maxNumExecutor depends on yarn c...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/16819 I don't think this is a necessary change. Already, you can't ask for more resources than the cluster has; the cluster won't grant them. Capping it here means the app can't use more resources if the cluster suddenly gets more. I see the problem you're trying to solve but the resource manager already ramps up requests slowly, so I don't think this is the issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
