[jira] [Updated] (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1100: - Target Version/s: 0.22.0 > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.21.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.21.1, 0.22.0 > > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, > patch-1100-fix-ydist.2.txt, reducetask-log-level.patch > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated MAPREDUCE-1100: --- Priority: Major (was: Blocker) I agree. Unblocking this. Target for 0.22.1 > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.21.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.21.1, 0.22.0 > > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, > patch-1100-fix-ydist.2.txt, reducetask-log-level.patch > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-1100: --- Attachment: reducetask-log-level.patch The YDH patch for this issue also includes these changes to the logging level in ReduceTask > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.21.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.21.1, 0.22.0 > > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, > patch-1100-fix-ydist.2.txt, reducetask-log-level.patch > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-1100: --- Priority: Blocker (was: Major) This is a pretty serious bug, let's make it a blocker for 22. > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.21.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.21.1, 0.22.0 > > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, > patch-1100-fix-ydist.2.txt > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated MAPREDUCE-1100: --- Fix Version/s: 0.21.1 0.22.0 Affects Version/s: 0.20.2 0.20.1 > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.20.1, 0.20.2, 0.21.0 >Reporter: Vinod K V >Assignee: Vinod K V > Fix For: 0.21.1, 0.22.0 > > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, > patch-1100-fix-ydist.2.txt > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1100: - Attachment: patch-1100-fix-ydist.2.txt The patch MAPREDUCE-1100-20091216.2.txt above that tried to solve log truncation for yahoo distribution has a bug w.r.t updating index files correctly. Attached patch fixes that bug. The patch is not for commit to any of apache branches. > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt, > patch-1100-fix-ydist.2.txt > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-1100: - Attachment: MAPREDUCE-1100-20091216.2.txt Attaching Vinod's yhadoop20 patch on his behalf. > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1100-20091102.txt, > MAPREDUCE-1100-20091106.txt, MAPREDUCE-1100-20091216.2.txt > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated MAPREDUCE-1100: - Attachment: MAPREDUCE-1100-20091106.txt Attaching a patch that handles cases with JVM-reuse. > User's task-logs filling up local disks on the TaskTrackers > --- > > Key: MAPREDUCE-1100 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-1100-20091102.txt, MAPREDUCE-1100-20091106.txt > > > Some user's jobs are filling up TT disks by outrageous logging. > mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are > getting filled up before task-log cleanup via > mapred.task.userlog.retain.hours can kick in. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1100) User's task-logs filling up local disks on the TaskTrackers
[
https://issues.apache.org/jira/browse/MAPREDUCE-1100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated MAPREDUCE-1100:
-
Attachment: MAPREDUCE-1100-20091102.txt
Attaching a first patch.
Introducing the following configuration items:
- Job Configuration:
-- {{JobContext.MAP_USERLOG_LIMIT}} : Per task limit on how much each log
file can grow to. Used by {{killRunningTasksOverLimit()}} for killing tasks
that write excessive logging.
-- {{JobContext.REDUCE_USERLOG_LIMIT}} : Same as above for reduces.
-- {{JobContext.MAP_USERLOG_RETAIN_SIZE}} : Per task configuration of how
much tail of the each log file has to be retained. Each task-log file is
truncated to this amount after the task finishes. Used by
{{truncateLogsOfFinishedTasks()}}
-- {{JobContext.REDUCE_USERLOG_RETAIN_SIZE}} : Same as above for reduces.
- TT configuration
-- {{TTConfig.TT_USERLOG_RETAIN_HOURS}} : TT configuraton of how long logs
of each finished task has to be retained on this TT. Used by
{{retireOldLogs()}} to cleanup very old logs.
-- {{TTConfig.TT_USERLOG_CUMULATIVE_LIMIT}} : TT configuration limiting the
total usage of log files across all tasks. If the total usage grows beyond this
limit, {{removeOldFilesToControlCumulativeUsage()}} removes old log files
irrespective of their age w.r.t {{TTConfig.TT_USERLOG_RETAIN_HOURS}}.
Moved clean-up of task-logs from child into TaskLogsMonitor which does the
following:
{code}
while(true) {
retireOldLogs(); // remove very old logs
truncateLogsOfFinishedTasks(); // truncate finished tasks' logs. Also set
no-writable permissions.
killRunningTasksOverLimit(); // kill tasks going over per-task per-file limit
removeOldFilesToControlCumulativeUsage(); // remove very old logs if total
usage is alarming irrespective of retain.hours
}
{code}
> User's task-logs filling up local disks on the TaskTrackers
> ---
>
> Key: MAPREDUCE-1100
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1100
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.21.0
>Reporter: Vinod K V
>Assignee: Vinod K V
> Attachments: MAPREDUCE-1100-20091102.txt
>
>
> Some user's jobs are filling up TT disks by outrageous logging.
> mapreduce.task.userlog.limit.kb is not enabled on the cluster. Disks are
> getting filled up before task-log cleanup via
> mapred.task.userlog.retain.hours can kick in.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
