[
https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439352#comment-15439352
]
Steve Loughran commented on SPARK-17243:
----------------------------------------
The REST API actually lets you set a time range for querying entries coming
back, though not a limit.
This problem could presumably be addressed in a couple of ways
# add a {{limit}} argument to the REST API, declaring the max #of responses to
return
# leave the REST API alone but tweak the client code to work backwards from now
to try and get a range. That's more convoluted and is probably brittle to
clocks.
strategy #1 is simpler and would avoid the server being overloaded from large
requests made directly by arbitrary callers —that serialization is going to be
expensive too, and an easy to way to bring the history server down.
> Spark 2.0 history server summary page gets stuck at "loading history summary"
> with 10K+ application history
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17243
> URL: https://issues.apache.org/jira/browse/SPARK-17243
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0
> Environment: Linux
> Reporter: Gang Wu
>
> The summary page of Spark 2.0 history server web UI keep displaying "Loading
> history summary..." all the time and crashes the browser when there are more
> than 10K application history event logs on HDFS.
> I did some investigation, "historypage.js" file sends a REST request to
> /api/v1/applications endpoint of history server REST endpoint and gets back
> json response. When there are more than 10K applications inside the event log
> directory it takes forever to parse them and render the page. When there are
> only hundreds or thousands of application history it is running fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
