[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976364#comment-14976364 ]
Thomas Graves commented on SPARK-6951: -------------------------------------- Unfortunately for us I don't think the current timeline server will scale at this point. But I'll have to try it once things are ready. There are other ways to make this faster to startup. - we could simply do an ls and show applications without any details and then add details as they are loaded. Then if user clicks on one load that one first. - we could change file name or directory structures to have some basic information that could be displayed by a simple file listing (like the MapReduce history server does). > History server slow startup if the event log directory is large > --------------------------------------------------------------- > > Key: SPARK-6951 > URL: https://issues.apache.org/jira/browse/SPARK-6951 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 1.3.0 > Reporter: Matt Cheah > > I started my history server, then navigated to the web UI where I expected to > be able to view some completed applications, but the webpage was not > available. It turned out that the History Server was not finished parsing all > of the event logs in the event log directory that I had specified. I had > accumulated a lot of event logs from months of running Spark, so it would > have taken a very long time for the History Server to crunch through them > all. I purged the event log directory and started from scratch, and the UI > loaded immediately. > We should have a pagination strategy or parse the directory lazily to avoid > needing to wait after starting the history server. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org