GitHub user tashoyan opened a pull request: https://github.com/apache/spark/pull/19711
[SPARK-22471][SQL] SQLListener consumes much memory causing OutOfMemoryError ## What changes were proposed in this pull request? This PR addresses the issue [SPARK-22471](https://issues.apache.org/jira/browse/SPARK-22471). The modified version of `SQLListener` respects the setting `spark.ui.retainedStages` and keeps the number of the tracked stages within the specified limit. The hash map `_stageIdToStageMetrics` does not outgrow the limit, hence overall memory consumption does not grow with time anymore. A 2.2-compatible fix. Maybe incompatible with 2.3 due to #19681. ## How was this patch tested? A new unit test covers this fix - see `SQLListenerMemorySuite.scala`. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tashoyan/spark SPARK-22471-branch-2.2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/19711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19711 ---- commit 08b7c82be3effe094e40618fe992d3c50c3e2d98 Author: Arseniy Tashoyan <tasho...@gmail.com> Date: 2017-11-08T15:41:36Z Add reproducer for the issue SPARK-22471 commit 2502a7e9846e359d793c485db1d3abef8a2c1e12 Author: Arseniy Tashoyan <tasho...@gmail.com> Date: 2017-11-08T15:41:54Z Add fix for the issue SPARK-22471 commit 2a13530db9ec611b6ee55fc9d79bd8aac5c01862 Author: Arseniy Tashoyan <tasho...@gmail.com> Date: 2017-11-08T20:39:02Z Remove debug print and irrelevant checks. Add a reference to the issue. commit 98f7b23fb52ffd11ae92716c871e5aa06ea61428 Author: Arseniy Tashoyan <tasho...@gmail.com> Date: 2017-11-08T20:47:44Z Remove debug print and irrelevant checks. Add a reference to the issue. commit 80755ece91703b3b6436f88e14eb11251ae6678f Author: Arseniy Tashoyan <tasho...@gmail.com> Date: 2017-11-08T21:21:42Z Collect memory-related tests on SQLListener in the same suite ---- --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org