[ https://issues.apache.org/jira/browse/SPARK-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084399#comment-14084399 ]
Patrick Wendell commented on SPARK-2016: ---------------------------------------- I think a major part of this was on the computation side as well - there were huge inefficiencies on this page. They were fixed in SPARK-2316. It would be interesting to see whether the test you give here is still unresponsive in that case (to dilineate whether this was a UI issue or not). > rdd in-memory storage UI becomes unresponsive when the number of RDD > partitions is large > ---------------------------------------------------------------------------------------- > > Key: SPARK-2016 > URL: https://issues.apache.org/jira/browse/SPARK-2016 > Project: Spark > Issue Type: Sub-task > Reporter: Reynold Xin > Labels: starter > > Try run > {code} > sc.parallelize(1 to 100, 1000000).cache().count() > {code} > And open the storage UI for this RDD. It takes forever to load the page. > When the number of partitions is very large, I think there are a few > alternatives: > 0. Only show the top 1000. > 1. Pagination > 2. Instead of grouping by RDD blocks, group by executors -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org