GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/4168
SPARK-4136. Under dynamic allocation, cancel outstanding executor requests when no longer needed [WIP] This takes advantage of the changes made in SPARK-4337 to cancel pending requests to YARN when they are no longer needed. Each time the timer in `ExecutorAllocationManager` strikes, we compute `maxNumNeededExecutors`, the maximum number of executors we could fill with the current load. This is calculated as the total number of running and pending tasks divided by the number of cores per executor. If `maxNumNeededExecutors` is below the total number of running and pending executors, we call `requestTotalExecutors(maxNumNeededExecutors)` to let the cluster manager know that it should cancel any pending requests above this amount. If not, `maxNumNeededExecutors` is just used as a bound in alongside the configured `maxExecutors` to limit the number of new requests. The patch modifies the API exposed by `ExecutorAllocationClient` for requesting additional executors by moving from `requestExecutors` to `requestTotalExecutors`. This makes the communication between the `ExecutorAllocationManager` and the `YarnAllocator` easier to reason about and removes some state that needed to be kept in the `CoarseGrainedSchedulerBackend`. I think an argument can be made that this makes for a less attractive user-facing API in `SparkContext`, but I'm having trouble envisioning situations where a user would want to use either of these APIs. This will likely break some tests, but I wanted to get feedback on the approach before adding tests and polishing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sryza/spark sandy-spark-4136 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4168.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4168 ---- commit 3ddb88526c06be83a1daf3eb73c7bfcc67c717de Author: Sandy Ryza <sa...@cloudera.com> Date: 2015-01-23T01:15:03Z SPARK-4136 commit 39c4de55de1fe8df6eb70db06bd9e2052324938d Author: Sandy Ryza <sa...@cloudera.com> Date: 2015-01-23T01:22:28Z Make ExecutorAllocationClient request a total commit 79763d98431e8764e29ef029992b1cd506eb8802 Author: Sandy Ryza <sa...@cloudera.com> Date: 2015-01-23T02:05:22Z Make things right ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org