[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12968467#action_12968467 ] Todd Lipcon commented on MAPREDUCE-1699: Does this still need to be put in trunk for 22? JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699-3.patch, mapred-1699-5.patch, mapred-1699-7.patch, mapred-1699-8.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[
https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882320#action_12882320
]
Krishna Ramachandran commented on MAPREDUCE-1699:
-
yes - this is good. I have added a map and logSubmitted will initialize in case
of an error for that job and use that for subsequent logging (failed, killed
etc.,)
Will update patch once testing is completed
Am not able to entirely take out disableHistory because of this code (incorrect
javadoc - there is no param - jobID)
/**
* Logs history meta-info to the history file. This needs to be called once
* per history file.
* @param jobId job id, assigned by jobtracker.
*/
static void logMetaInfo(ArrayListPrintWriter writers){
if (!disableHistory){
if (null != writers){
JobHistory.log(writers, RecordTypes.Meta,
new Keys[] {Keys.VERSION},
new String[] {String.valueOf(VERSION)});
}
}
}
also there are couple of (public) methods that use this flag. though I do not
see where they are called
/**
* Returns history disable status. by default history is enabled so this
* method returns false.
* @return true if history logging is disabled, false otherwise.
*/
public static boolean isDisableHistory() {
return disableHistory;
}
/**
* Enable/disable history logging. Default value is false, so history
* is enabled by default.
* @param disableHistory true if history should be disabled, false otherwise.
*/
public static void setDisableHistory(boolean disableHistory) {
JobHistory.disableHistory = disableHistory;
}
JobHistory shouldn't be disabled for any reason
---
Key: MAPREDUCE-1699
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.2
Reporter: Arun C Murthy
Assignee: Krishna Ramachandran
Fix For: 0.20.3
Attachments: mapred-1699-1.patch, mapred-1699-2.patch,
mapred-1699-3.patch, mapred-1699.patch
Recently we have had issues with JobTracker silently disabling job-history
and starting to keep all completed jobs in memory. This leads to OOM on the
JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[
https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12882402#action_12882402
]
Krishna Ramachandran commented on MAPREDUCE-1699:
-
In trunk I believe, all job state loggings are handled in
jobHistory.logEvent(jobId) {
/**
* Method to log the specified event
* @param event The event to log
* @param id The Job ID of the event
*/
public void logEvent(HistoryEvent event, JobID id) {
try {
final MetaInfo mi = fileMap.get(id);
if (mi != null) {
mi.writeEvent(event);
}
} catch (IOException e) {
LOG.error(Error Logging event, + e.getMessage());
}
}
}
if logging fails just log a message. This is a lot cleaner - disableHistory is
never used
JobHistory shouldn't be disabled for any reason
---
Key: MAPREDUCE-1699
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.20.2
Reporter: Arun C Murthy
Assignee: Krishna Ramachandran
Fix For: 0.20.3
Attachments: mapred-1699-1.patch, mapred-1699-2.patch,
mapred-1699-3.patch, mapred-1699.patch
Recently we have had issues with JobTracker silently disabling job-history
and starting to keep all completed jobs in memory. This leads to OOM on the
JobTracker. We should never do this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876058#action_12876058 ] Hadoop QA commented on MAPREDUCE-1699: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446441/mapred-1699-2.patch against trunk revision 951832. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/225/console This message is automatically generated. JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876059#action_12876059 ] Hadoop QA commented on MAPREDUCE-1699: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446441/mapred-1699-2.patch against trunk revision 951832. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/226/console This message is automatically generated. JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Assignee: Krishna Ramachandran Fix For: 0.20.3 Attachments: mapred-1699-1.patch, mapred-1699-2.patch, mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12873195#action_12873195 ] Krishna Ramachandran commented on MAPREDUCE-1699: - the patch is for 20.1xxS not for commit here JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.20.3 Attachments: mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872019#action_12872019 ] Hadoop QA commented on MAPREDUCE-1699: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445602/mapred-1699.patch against trunk revision 947758. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/207/console This message is automatically generated. JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.20.3 Attachments: mapred-1699.patch Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1699) JobHistory shouldn't be disabled for any reason
[ https://issues.apache.org/jira/browse/MAPREDUCE-1699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12856739#action_12856739 ] Sharad Agarwal commented on MAPREDUCE-1699: --- This has been fixed by MAPREDUCE-157 in 0.21 JobHistory shouldn't be disabled for any reason --- Key: MAPREDUCE-1699 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1699 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.2 Reporter: Arun C Murthy Fix For: 0.22.0 Recently we have had issues with JobTracker silently disabling job-history and starting to keep all completed jobs in memory. This leads to OOM on the JobTracker. We should never do this. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
