GitHub user mridulm opened a pull request:
https://github.com/apache/spark/pull/22609
[SPARK-25594] [Core] Avoid maintaining task information when UI is disabled
## What changes were proposed in this pull request?
Avoid maintaining task information in live spark application when UI is
disabled.
For long running application, with large number of tasks, this ended up
causing OOM in our tests.
## How was this patch tested?
Long running test successfully ran for 34 hours with steady memory, when it
used to fail in 28 hours with OOM earlier.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mridulm/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22609.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22609
----
commit c2f6b4a50ef3693292f600a2d4d7743ea870b96e
Author: Mridul Muralidharan <mridul@...>
Date: 2018-10-02T08:03:41Z
SPARK-25594: Avoid maintaining task information when UI is disabled
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
