[jira] [Commented] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
[ https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15996013#comment-15996013 ] zhoukang commented on SPARK-20582: -- You are right [~vanzin],i searched into you implementation.Thanks for your time. > Speed up the restart of HistoryServer using ApplicationAttemptInfo > checkpointing > > > Key: SPARK-20582 > URL: https://issues.apache.org/jira/browse/SPARK-20582 > Project: Spark > Issue Type: Improvement > Components: Deploy >Affects Versions: 2.1.0 >Reporter: zhoukang >Priority: Critical > >In the current code of HistoryServer,jetty server will be started > after all the logs fetch from yarn has been replayed.However,when the number > of logs is becoming larger,the start time of jetty will be too long. >Here,we implement a solution which using ApplicationAttemptInfo > checkpointing to speed up the start of historyserver.When historyserver is > starting,it will load ApplicationAttemptInfo from checkpoint file first if > exists which is faster then replaying one by one. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
[ https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995958#comment-15995958 ] Marcelo Vanzin commented on SPARK-20582: You're talking about checkpointing the application info, which the code I have already does. With my changes the HTTP server comes up immediately. > Speed up the restart of HistoryServer using ApplicationAttemptInfo > checkpointing > > > Key: SPARK-20582 > URL: https://issues.apache.org/jira/browse/SPARK-20582 > Project: Spark > Issue Type: Improvement > Components: Deploy >Affects Versions: 2.1.0 >Reporter: zhoukang >Priority: Critical > >In the current code of HistoryServer,jetty server will be started > after all the logs fetch from yarn has been replayed.However,when the number > of logs is becoming larger,the start time of jetty will be too long. >Here,we implement a solution which using ApplicationAttemptInfo > checkpointing to speed up the start of historyserver.When historyserver is > starting,it will load ApplicationAttemptInfo from checkpoint file first if > exists which is faster then replaying one by one. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20582) Speed up the restart of HistoryServer using ApplicationAttemptInfo checkpointing
[ https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995951#comment-15995951 ] zhoukang commented on SPARK-20582: -- [~vanzin]I think you misunderstand this issue.This issue in order to let jetty server listen to given port faster.Since when there are too many logs need replaying,the time may be tens of minutes. > Speed up the restart of HistoryServer using ApplicationAttemptInfo > checkpointing > > > Key: SPARK-20582 > URL: https://issues.apache.org/jira/browse/SPARK-20582 > Project: Spark > Issue Type: Improvement > Components: Deploy >Affects Versions: 2.1.0 >Reporter: zhoukang >Priority: Critical > >In the current code of HistoryServer,jetty server will be started > after all the logs fetch from yarn has been replayed.However,when the number > of logs is becoming larger,the start time of jetty will be too long. >Here,we implement a solution which using ApplicationAttemptInfo > checkpointing to speed up the start of historyserver.When historyserver is > starting,it will load ApplicationAttemptInfo from checkpoint file first if > exists which is faster then replaying one by one. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
