GitHub user mariahualiu opened a pull request:
https://github.com/apache/spark/pull/17854
[SPARK-20564][Deploy] Reduce massive executor failures when executor count
is large (>2000)
## What changes were proposed in this pull request?
In applications that use over 2000 executors, we noticed a large number of
failed executors due to driver overloading with too many executor RPCs within a
short period of time (for example, retrieve spark properties, executor
registration). This patch adds an extra configuration
spark.yarn.launchContainer.count.simultaneously, which caps the maximal number
of containers that driver can ask for and launch in every
spark.yarn.scheduler.heartbeat.interval-ms. As a result, the number of
executors grows steadily. The number of executor failures is reduced and
applications can reach the desired number of executors faster.
## How was this patch tested?
1. Didn't break relevant unit tests
2. Tested with a spark application (2500 executors) on a Yarn cluster with
3000 machines.
A gentle ping to the contributors of YarnAllocator: @srowen @foxish
@jinxing64 @squito
A JIRA is opened: https://issues.apache.org/jira/browse/SPARK-20564
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mariahualiu/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17854.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17854
----
commit e1cc521817c49fdf3448fa9290f50129d837d8bc
Author: hualiu <hua...@microsoft.com>
Date: 2017-05-03T18:30:00Z
add spark.yarn.launchContainer.count.simultaneously to cap # of executors
to be launched simultaneously
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
