Re: OOM: Structured Streaming aggregation state not cleaned up properly

The issue looks like fixed in https://issues.apache.org/jira/browse/SPARK-23670, and likely 2.3.1 will include the fix. -Jungtaek Lim (HeartSaVioR) 2018년 5월 23일 (수) 오후 7:12, weand 님이 작성: > Thanks for clarification. So it really seem a Spark UI OOM Issue. > > After

Re: OOM: Structured Streaming aggregation state not cleaned up properly

Thanks for clarification. So it really seem a Spark UI OOM Issue. After setting: --conf spark.sql.ui.retainedExecutions=10 --conf spark.worker.ui.retainedExecutors=10 --conf spark.worker.ui.retainedDrivers=10 --conf spark.ui.retainedJobs=10 --conf spark.ui.retainedStages=10

Re: OOM: Structured Streaming aggregation state not cleaned up properly

Just to be clear, these screenshots are about the memory consumption of the driver. So this is nothing to do with streaming aggregation state which are kept in the memory of the executors, not the driver. On Tue, May 22, 2018 at 10:21 AM, Jungtaek Lim wrote: > 1. Could you

Re: OOM: Structured Streaming aggregation state not cleaned up properly

1. Could you share your Spark version? 2. Could you reduce "spark.sql.ui.retainedExecutions" and see whether it helps? This configuration is available in 2.3.0, and default value is 1000. Thanks, Jungtaek Lim (HeartSaVioR) 2018년 5월 22일 (화) 오후 4:29, weand 님이 작성: > You

Re: OOM: Structured Streaming aggregation state not cleaned up properly

You can see it even better on this screenshot: TOP Entries Collapsed #2 Sorry for the spam, attached a not so perfect screen in the mail before. -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

Re: OOM: Structured Streaming aggregation state not cleaned up properly

Instances of org.apache.spark.sql.execution.ui.SparkPlanGraphWrapper are not cleaned up, see TOP Entries Collapsed #2: TOP Entries All TOP Entries Collapsed #1

Re: OOM: Structured Streaming aggregation state not cleaned up properly

Hi, w.r.t. ElementTrackingStore, since it is backed by KVStore, there should be other classes which occupy significant memory. Can you pastebin the top 10 entries among the heap dump ? Thanks