[
https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019394#comment-14019394
]
Mridul Muralidharan commented on SPARK-2017:
--------------------------------------------
Currently, for our jobs, I run with spark.ui.retainedStages=3 (so that there is
some visibility into past stages) : this is to prevent OOM's in the master when
number of tasks per stage is not low (50k for example is not very high imo)
The stage details UI becomes very sluggish to pretty much unresponsive for our
tasks where tasks > 30k ... though that might also be a browser issue
(firefox/chrome) ?
> web ui stage page becomes unresponsive when the number of tasks is large
> ------------------------------------------------------------------------
>
> Key: SPARK-2017
> URL: https://issues.apache.org/jira/browse/SPARK-2017
> Project: Spark
> Issue Type: Sub-task
> Reporter: Reynold Xin
> Labels: starter
>
> {code}
> sc.parallelize(1 to 1000000, 1000000).count()
> {code}
> The above code creates one million tasks to be executed. The stage detail web
> ui page takes forever to load (if it ever completes).
> There are again a few different alternatives:
> 0. Limit the number of tasks we show.
> 1. Pagination
> 2. By default only show the aggregate metrics and failed tasks, and hide the
> successful ones.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
