[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-1699: --- Fix Version/s: 0.23.0 0.22.0 JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.22.0, 0.23.0 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1699: - Release Note: (was: Fix committed 20.1xx) Fix Version/s: (was: 0.20.3) This was not fixed in the 0.20 branch. There are two options: 1) If the problem does not exist in trunk, please change resolution to won't fix 2) Reopen the bug, optionally attaching a patch for the Apache 0.20 branch JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699-8.patch Revised if JobHistory.log() fails for a job, the writer (PrintWriter) for that will be removed from FileManager to prevent repeated failed attempts JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699-7.patch Revised (from Arun's review) if logSubmitted writer fails we remove the entry from fileManager list (if not null) Patch for an earlier release. Not for commit in trunk JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1699: - Status: Open (was: Patch Available) This needs more work... # Remove JobHistory.disableHistory # JobHistory.logSubmitted shouldn't throw an IOException, failure to log job-history shouldn't fail a job. We probably need a map from JobID - disabled which should be checked everywhere in-lieu of JobHistory.disableHistory JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699-5.patch Arun I took out references to disableHistory I do not believe we need a separate structure to store disabled_state for each jobId. In jobHistory.logSubmitted, writer (entry point for history logging) is initialized for each jobID If fileManager.addWriter(jobId, writer) fails, we log an error in catch block (just like trunk) During subsequent writing of all log events (started/failed/killed/completed/inited/ogInfo ..) we check for fileManager.getWriter(jobId) != null (will be null if logSubmitted failed to initialize) This should be sufficient and is similar to what you proposed (jobId-disabled) I am modifying the patch accordingly Let me know! JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Status: Open (was: Patch Available) JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699-3.patch Modified per Chris's comments Removed changes related to CompletedJobStatusStore Fix logSubmitted - will throw an error if logging fails Fix for earlier release not for commit in the trunk JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Status: Patch Available (was: Open) Revised patch based on Chris review JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[
https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-1699:
-
Status: Open (was: Patch Available)
Assignee: Krishna Ramachandran
* It looks like MAPREDUCE-1778 is included in the latest patch
* {{JobHistory}} can still be disabled through {{JobInfo::logSubmitted}}
JobHistory shouldn't be disabled for any reason
---
Key: MAPREDUCE-1699
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.2
Reporter: Arun C Murthy
Assignee: Krishna Ramachandran
Fix For: 0.20.3
Attachments: mapred-1699-1.patch, mapred-1699.patch
Recently we have had issues with JobTracker silently disabling job-history
and starting to keep all completed jobs in memory. This leads to OOM on the
JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Status: Patch Available (was: Open) Yes fix for 1778 was included based on discussions with Arun oops I missed logSubmitted In my latest revision I catch IOException and rethrow JobHistory::JobInfo::logSubmitted JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Status: Open (was: Patch Available) JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699-2.patch JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Status: Patch Available (was: Open) I have revised the patch to address history being disabled from JobInfo.logSubmitted JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699-1.patch attached revision with test cases not for commit in the trunk fix for earlier release JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Status: Patch Available (was: Open) Patch for Y! distribution security branch With this change, if job history location is mis-configured JobTracker will throw an exception and fail. A similar fix in place for trunk (part of https://issues.apache.org/jira/browse/MAPREDUCE-157) This patch also addresses https://issues.apache.org/jira/browse/MAPREDUCE-1778 If CompletedJobStatusStore is active location is badly configured JobTracker will not start JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.20.3 Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krishna Ramachandran updated MAPREDUCE-1699: Attachment: mapred-1699.patch JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.20.3 Attachments: mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1699: - Fix Version/s: 0.20.3 (was: 0.22.0) It would be nice if this were fixed in the 0.20 branch JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.20.3 Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
