[
https://issues.apache.org/jira/browse/SPARK-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054178#comment-14054178
]
Masayoshi TSUZUKI commented on SPARK-2017:
------------------------------------------
Pagination seems to be better because with aggregated metrics,
1. we can't identify the skew of tasks between the executors.
2. the same problem will appear again when many tasks fail in a certain stage.
In addition, when some errors or problems occur under the production
environment, we would like to see the status of tasks near the time even if
those tasks mostly succeeded. Although every status of tasks is written in the
log file, web ui is very useful in operation phase.
> web ui stage page becomes unresponsive when the number of tasks is large
> ------------------------------------------------------------------------
>
> Key: SPARK-2017
> URL: https://issues.apache.org/jira/browse/SPARK-2017
> Project: Spark
> Issue Type: Sub-task
> Reporter: Reynold Xin
> Labels: starter
>
> {code}
> sc.parallelize(1 to 1000000, 1000000).count()
> {code}
> The above code creates one million tasks to be executed. The stage detail web
> ui page takes forever to load (if it ever completes).
> There are again a few different alternatives:
> 0. Limit the number of tasks we show.
> 1. Pagination
> 2. By default only show the aggregate metrics and failed tasks, and hide the
> successful ones.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
