[jira] [Commented] (SPARK-8667) Improve Spark UI behavior at scale
[ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14604000#comment-14604000 ] Patrick Wendell commented on SPARK-8667: Thanks Sean. I looked for a while for an older JIRA on this, but couldn't find it. This is definitely a dup of SPARK-2015. Improve Spark UI behavior at scale -- Key: SPARK-8667 URL: https://issues.apache.org/jira/browse/SPARK-8667 Project: Spark Issue Type: Improvement Components: Web UI Reporter: Patrick Wendell Assignee: Shixiong Zhu This is a parent ticket and we can create child tickets when solving specific issues. The main problem I would like to solve is the fact that the Spark UI has issues at very large scale. The worst issue is when there is a stage page with more than a few thousand tasks. In this case: 1. The page itself is very slow to load and becomes unresponsive with huge number of tasks. 2. The Scala XML output can become so large that it crashes the driver program due to OOM for a page with a huge number of tasks. I am not sure if (1) is caused by javascript slowness, or maybe just the raw amount of data sent over the wire. If it is the latter, it might be possible to add compression to the HTTP payload to help improve load time. It would be nice to reproduce+investigate these issues further and create specific sub tasks to improve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8667) Improve Spark UI behavior at scale
[ https://issues.apache.org/jira/browse/SPARK-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14603989#comment-14603989 ] Sean Owen commented on SPARK-8667: -- [~pwendell] there are a some old tickets and an old umbrella for this: https://issues.apache.org/jira/browse/SPARK-2015 -- would you be able to merge these or shall I? I'm concerned these get filed and forgotten Improve Spark UI behavior at scale -- Key: SPARK-8667 URL: https://issues.apache.org/jira/browse/SPARK-8667 Project: Spark Issue Type: Improvement Components: Web UI Reporter: Patrick Wendell Assignee: Shixiong Zhu This is a parent ticket and we can create child tickets when solving specific issues. The main problem I would like to solve is the fact that the Spark UI has issues at very large scale. The worst issue is when there is a stage page with more than a few thousand tasks. In this case: 1. The page itself is very slow to load and becomes unresponsive with huge number of tasks. 2. The Scala XML output can become so large that it crashes the driver program due to OOM for a page with a huge number of tasks. I am not sure if (1) is caused by javascript slowness, or maybe just the raw amount of data sent over the wire. If it is the latter, it might be possible to add compression to the HTTP payload to help improve load time. It would be nice to reproduce+investigate these issues further and create specific sub tasks to improve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
