Patrick Wendell created SPARK-8667:
--------------------------------------
Summary: Improve Spark UI behavior at scale
Key: SPARK-8667
URL: https://issues.apache.org/jira/browse/SPARK-8667
Project: Spark
Issue Type: Improvement
Reporter: Patrick Wendell
Assignee: Shixiong Zhu
This is a parent ticket and we can create child tickets when solving specific
issues. The main problem I would like to solve is the fact that the Spark UI
has issues at very large scale.
The worst issue is when there is a stage page with more than a few thousand
tasks. In this case:
1. The page itself is very slow to load and becomes unresponsive with huge
number of tasks.
2. The Scala XML output can become so large that it crashes the driver program
due to OOM for a page with a huge number of tasks.
I am not sure if (1) is caused by javascript slowness, or maybe just the raw
amount of data sent over the wire. If it is the latter, it might be possible to
add compression to the HTTP payload to help improve load time.
It would be nice to reproduce+investigate these issues further and create
specific sub tasks to improve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
