[ 
https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apache Spark reassigned SPARK-17843:
------------------------------------

    Assignee:     (was: Apache Spark)

> History Server gives no feedback about the application list being incomplete
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-17843
>                 URL: https://issues.apache.org/jira/browse/SPARK-17843
>             Project: Spark
>          Issue Type: Improvement
>          Components: Web UI
>    Affects Versions: 2.0.0, 2.0.1
>            Reporter: Vinayak Joshi
>            Priority: Minor
>
> A user who lands on the history server's application listing page before the 
> history server's background processing has finished replaying the existing 
> event logs complains why a particular  application is not included in the 
> list. There is no indication on the UI about the possible reason, or that 
> refreshing the page after some time may show up the application the user 
> expects to see. This problem is very noticeable when there are large sized 
> event logs that take a long time to finish replaying.
> The usability problems with large (number or size) event logs and the history 
> server have been known. Particularly SPARK-5522, SPARK-13988 and other issues 
> referenced within them highlight the problems and the improvements done so 
> far. To improve the history server startup time and reduce the impact of 
> large event log files, the event logs are now processed (replayed) via a pool 
> of threads. This allows a user to browse to the application listing page 
> before the event logs have finished replaying. After history server startup, 
> a user expects to see any old completed applications to appear in the 
> application list page. But unless the corresponding event logs have finished 
> replaying, the application wont be in the list, and the user complains. There 
> is no feedback to the user about this on the UI, hence this JIRA to try and 
> address this problem. 
> Idea is to give some indication of the number of event logs that are pending 
> replay to the user. Note that the way the replay is currently designed, one 
> cycle of "check for logs that need to be replayed > replay the logs > update 
> application info" needs to complete before a new one begins. Therefore, it 
> should be possible for the FsApplicationHistoryProvider to send info about 
> the number of logs that are currently pending processing. This in turn would 
> address the user anxiety of not seeing the application they expect to see. 
> I will be attaching a pull request with my initial take on implementing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to