I set those configurations by passing to spark-submit script: "bin/spark-submit --conf spark.ui.retainedJobs=20 ...". I have verified that these configurations are being passed correctly because they are listed in the environments tab and also by counting the number of job/stages that are listed. The "spark.sql.ui.retainedExecutions=0" only applies to the number of "completed" executions; there will always be a "running" execution. For some reason, I have one execution that consumes an excessive amount of memory.
Actually, I am not interested in the SQL UI, as I find the Job/Stages UI to have sufficient information. I am also using Spark Standalone cluster manager so have not had to use the history server. On Mon, Oct 12, 2015 at 8:17 PM, Shixiong Zhu <zsxw...@gmail.com> wrote: > Could you show how did you set the configurations? You need to set these > configurations before creating SparkContext and SQLContext. > > Moreover, the history sever doesn't support SQL UI. So > "spark.eventLog.enabled=true" doesn't work now. > > Best Regards, > Shixiong Zhu > > 2015-10-13 2:01 GMT+08:00 pnpritchard <nicholas.pritch...@falkonry.com>: > >> Hi, >> >> In my application, the Spark UI is consuming a lot of memory, especially >> the >> SQL tab. I have set the following configurations to reduce the memory >> consumption: >> - spark.ui.retainedJobs=20 >> - spark.ui.retainedStages=40 >> - spark.sql.ui.retainedExecutions=0 >> >> However, I still get OOM errors in the driver process with the default 1GB >> heap size. The following link is a screen shot of a heap dump report, >> showing the SQLListener instance having a retained size of 600MB. >> >> https://cloud.githubusercontent.com/assets/5124612/10404379/20fbdcfc-6e87-11e5-9415-27e25193a25c.png >> >> Rather than just increasing the allotted heap size, does anyone have any >> other ideas? Is it possible to disable the SQL tab specifically? I also >> thought about serving the UI from disk rather than memory with >> "spark.eventLog.enabled=true" and "spark.ui.enabled=false". Has anyone >> tried >> this before? >> >> Thanks, >> Nick >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-UI-consuming-lots-of-memory-tp25033.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
