[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070916#comment-13070916 ] Sherry Chen commented on MAPREDUCE-2589: Mahadev, Per Bharath (Mundlapudi), his change in MAPREDUCE-2415 distributes TaskTracker userlogs onto multiple disks, it went into 0.20.204. Changes in MAPREDUCE-2415 strengthened TaskTracker reliability with regard to disk failures, also have addOldUserLogsForDeletion() to add the job log directories for deletion with default retain hours. I tested with MAPREDUCE-2415 change in, all old job log directories will be deleted after default retain hours. I think my change here is not necessary, I would like to withdraw it. Is it OK with you? TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Fix For: 0.20.205.0 Attachments: MAPREDUCE-2589.patch, MAPREDUCE-2589_1.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070918#comment-13070918 ] Mahadev konar commented on MAPREDUCE-2589: -- Sherry, That sounds good to me! Please resolve this as WONT FIX. TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Fix For: 0.20.205.0 Attachments: MAPREDUCE-2589.patch, MAPREDUCE-2589_1.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070295#comment-13070295 ] Mahadev konar commented on MAPREDUCE-2589: -- Sherry, Sorry I looked at it again and I think its good if we make the DEFAULT_USER_LOG_RETAIN_MAX_HOURS configurable. Also, -1 should disable the feature. I think its important to be able to switch off misbehaving configuration. Also, why do we need a call to get jobs that are still running? I thought the call was made only on restart/reinit? We should be able to clean old user logs without calling jc.jobstocomplete? I think we should avoid adding a dependency on calling jobtracker client methods in the tasktracker itself. what do you think? TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Fix For: 0.20.205.0 Attachments: MAPREDUCE-2589.patch, MAPREDUCE-2589_1.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069288#comment-13069288 ] Hadoop QA commented on MAPREDUCE-2589: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487372/MAPREDUCE-2589_1.patch against trunk revision 1149323. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/493//console This message is automatically generated. TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Fix For: 0.20.205.0 Attachments: MAPREDUCE-2589.patch, MAPREDUCE-2589_1.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067516#comment-13067516 ] Devaraj K commented on MAPREDUCE-2589: -- One improvement can be done in the patch, now for every file in the user log directory it is getting the jobs which are to be completed every time and checking. Instead of this it can get the jobs list once and can check for all the files in the user log directory whether it belongs to running job or not. TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Fix For: 0.20.205.0 Attachments: MAPREDUCE-2589.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13064759#comment-13064759 ] Hadoop QA commented on MAPREDUCE-2589: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482575/MAPREDUCE-2589.patch against trunk revision 1145889. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/464//console This message is automatically generated. TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Attachments: MAPREDUCE-2589.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13062159#comment-13062159 ] Sherry Chen commented on MAPREDUCE-2589: Aaron, I believe MAPREDUCE-1100 and related JIRAs change how task-logs work. We may not need to have this patch for trunk at this point. TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Attachments: MAPREDUCE-2589.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2589) TaskTracker not purging userlog directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13049310#comment-13049310 ] Aaron T. Myers commented on MAPREDUCE-2589: --- Hi Sherry, does this issue not also affect trunk? If so, would you mind preparing a trunk patch as well? TaskTracker not purging userlog directories --- Key: MAPREDUCE-2589 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2589 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0 Environment: 0.20.205 Reporter: Sherry Chen Assignee: Sherry Chen Priority: Minor Attachments: MAPREDUCE-2589.patch, cleanup_userlogs.py UserLogCleaner is not robust. Leftover userlogs after a restart sometimes have to be manually cleaned. Things can accumulate over a period of time. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
