GitHub user jeanlyn opened a pull request: https://github.com/apache/spark/pull/11779
[SPARK-13845][CORE]Using onBlockUpdated to replace onTaskEnd avioding driver OOM ## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always driver OOM after few days, here is some driver heap dump before OOM ``` num #instances #bytes class name ---------------------------------------------- 1: 13845916 553836640 org.apache.spark.storage.BlockStatus 2: 14020324 336487776 org.apache.spark.storage.StreamBlockId 3: 13883881 333213144 scala.collection.mutable.DefaultEntry 4: 8907 89043952 [Lscala.collection.mutable.HashEntry; 5: 62360 65107352 [B 6: 163368 24453904 [Ljava.lang.Object; 7: 293651 20342664 [C ... ``` `BlockStatus` and `StreamBlockId` keep on growing, and the driver OOM in the end. After investigated, i found the `executorIdToStorageStatus` in `StorageStatusListener` seems never remove the blocks from `StorageStatus`. In order to fix the issue, i try to use `onBlockUpdated` replace `onTaskEnd ` , so we can update the block informations(add blocks, drop the block from memory to disk and delete the blocks) in time. ## How was this patch tested? Existing unit tests and manual tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/jeanlyn/spark fix_driver_oom Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11779.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11779 ---- commit f1f6df685b416b1f7fa47f9419dc5a0300e63070 Author: jeanlyn <jeanly...@gmail.com> Date: 2016-03-12T08:01:39Z using onBlockUpdated to replace onTaskEnd when the block updated commit 5255067c40b49efab597dfbb13ccfa9b6f7b0833 Author: jeanlyn <jeanly...@gmail.com> Date: 2016-03-12T12:52:04Z fix unit test commit ab4a863b71ee28791b43daea1af58453906af8c0 Author: jeanlyn <jeanly...@gmail.com> Date: 2016-03-15T01:01:53Z fix scala style commit 59150927dfe7c1eacd244a359fc5e8d040d4dc07 Author: jeanlyn <jeanly...@gmail.com> Date: 2016-03-17T06:24:01Z fix HistoryServerSuite commit 2e2c319cd0dafe3ee8ae8d4fb8fe958ae5ddcd4e Author: jeanlyn <jeanly...@gmail.com> Date: 2016-03-17T09:16:39Z fix typos ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org