[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3512: --- Resolution: Fixed Release Note: Batching JobHistory flushing to DFS so that we don't flush for every event slowing down AM. Status: Resolved (was: Patch Available) Just committed this to trunk and branch-0.23. Thanks Sid! > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3512: --- Attachment: MAPREDUCE-3512-20120111.txt Patch with trivial edits. > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3512: --- Fix Version/s: 0.23.1 Status: Open (was: Patch Available) > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3512: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Fix For: 0.23.1 > > Attachments: MAPREDUCE-3512-20120111.txt, MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3512: -- Attachment: MR3512.txt Updated with unit tests. > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Attachments: MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3512: -- Status: Patch Available (was: Open) > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Attachments: MR3512.txt, MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-3512: -- Attachment: MR3512.txt Initial patch - will upload another one later with some of the TODOs resolved and with unit tests. > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > Attachments: MR3512.txt > > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3512: - Priority: Critical (was: Major) > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Priority: Critical > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3512) Batch jobHistory disk flushes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3512: --- Issue Type: Sub-task (was: Improvement) Parent: MAPREDUCE-3561 > Batch jobHistory disk flushes > - > > Key: MAPREDUCE-3512 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3512 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth > > The mr-am flushes each individual job history event to disk for AM recovery. > The history even handler ends up with a significant backlog for tests like > MAPREDUCE-3402. > History events could be batched up based on num records / time / > TaskFinishedEvents to reduce the number of DFS writes - with the potential > drawback of having to rerun some tasks during AM recovery. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
