[
https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449250#comment-15449250
]
Thomas Graves commented on SPARK-17243:
---------------------------------------
I agree, there are a ton of ways to improve the history server. I think these
should be separate jiras though. Ideally it is much faster to load all the apps
and get the initial list very quickly. Only load the entire application as a
user requests or in the background to fill the cache. Like you mention could
have summary file written after loaded. They could be stored differently so
basic data is in dir or file path (like MapReduce history server), etc. I just
haven't had time to do this myself.
Right now this seems like a good workaround and as I mention in Pr
spark.history.retainedApplications used to do this limiting in of the display
but things have changed and I guess it broke/wasn't updated.
> Spark 2.0 history server summary page gets stuck at "loading history summary"
> with 10K+ application history
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17243
> URL: https://issues.apache.org/jira/browse/SPARK-17243
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0
> Environment: Linux
> Reporter: Gang Wu
>
> The summary page of Spark 2.0 history server web UI keep displaying "Loading
> history summary..." all the time and crashes the browser when there are more
> than 10K application history event logs on HDFS.
> I did some investigation, "historypage.js" file sends a REST request to
> /api/v1/applications endpoint of history server REST endpoint and gets back
> json response. When there are more than 10K applications inside the event log
> directory it takes forever to parse them and render the page. When there are
> only hundreds or thousands of application history it is running fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
