[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data
[ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-953: - Attachment: YARN-953.5.patch I updated the a new patch, which coordinates the change of writer interface. The basic design is not changed: we still have a RMApplicationHistoryWriter, which handles writing requests from RM on different threads asynchronously. There're some other major changes: 1. Instead of handling all the writing events in one separate thread, I define a dispatcher vector to improve concurrency, and also ensure the events of one application is scheduled in the same thread. Therefore, the writing events of different applications will be processed concurrently, while events of the same application will be processed in the order where they are scheduled (It's important to ensure the events scheduled before applicationFinished to be processed first). 2. Make sure applicationFinished is called after all applicationAttemptsFinished, especially in the killing case, where RMApp moves to the final state before RMAppAttempt. 3. Improve the test cases. 4. Fix the break of other tests in RM project. There's something to be handled separately: 1. We need to make RMContainer have more information to fill ContainerHistoryData. It's going to be done in YARN-974 2. Like RMStateStore, RMApplicationHistoryWriter needs to flush all the pending events given RM stops. We can make use of the update in YARN-1121 later. [YARN-321] Change ResourceManager to use HistoryStorage to log history data --- Key: YARN-953 URL: https://issues.apache.org/jira/browse/YARN-953 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: YARN-953-5.patch, YARN-953.1.patch, YARN-953.2.patch, YARN-953.3.patch, YARN-953.4.patch, YARN-953.5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data
[ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated YARN-953: --- Attachment: YARN-953-5.patch Thanks [~zjshen] for the patch. I am updating it with latest YARN-321 branch and fixing some of the compilation failures. Thanks, Mayank [YARN-321] Change ResourceManager to use HistoryStorage to log history data --- Key: YARN-953 URL: https://issues.apache.org/jira/browse/YARN-953 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: YARN-953.1.patch, YARN-953.2.patch, YARN-953.3.patch, YARN-953.4.patch, YARN-953-5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data
[ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-953: - Attachment: YARN-953.4.patch Rebase the patch [YARN-321] Change ResourceManager to use HistoryStorage to log history data --- Key: YARN-953 URL: https://issues.apache.org/jira/browse/YARN-953 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: YARN-953.1.patch, YARN-953.2.patch, YARN-953.3.patch, YARN-953.4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data
[ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-953: - Attachment: YARN-953.3.patch Update the patch against the latest branch code, and add the code to init/start and stop the writer in case it is a service [YARN-321] Change ResourceManager to use HistoryStorage to log history data --- Key: YARN-953 URL: https://issues.apache.org/jira/browse/YARN-953 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: YARN-953.1.patch, YARN-953.2.patch, YARN-953.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data
[ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-953: - Attachment: YARN-953.1.patch Created the patch to make RM write the history data: 1. Move the removed code in YARN-934 here, to created a service with an independent dispatcher to write history data. The writer service provides non-blocking APIs to RM 2. The writer service is created when RM is inited and wrapped into RMContext. 3. RMApp, RMAppAttempt and RMContainer use their contained RMContext to write the history data when they reach the final state. 4. In the writer service, the implementation of ApplicationHistoryWriter is to be created. By default, the writing service is disabled in terms of using DummyApplicationHistoryWriter, which does nothing. 5. Not all the fields of ContainerHistoryData are filled, because this information is not available. YARN-974 will look after this issue. 6. Test case is added to test the writer service. [YARN-321] Change ResourceManager to use HistoryStorage to log history data --- Key: YARN-953 URL: https://issues.apache.org/jira/browse/YARN-953 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: YARN-953.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-953) [YARN-321] Change ResourceManager to use HistoryStorage to log history data
[ https://issues.apache.org/jira/browse/YARN-953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-953: - Summary: [YARN-321] Change ResourceManager to use HistoryStorage to log history data (was: [YARN-321] Change ResourceManager to use HistoryUsage to log history data) [YARN-321] Change ResourceManager to use HistoryStorage to log history data --- Key: YARN-953 URL: https://issues.apache.org/jira/browse/YARN-953 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
