[jira] Commented: (MAPREDUCE-1943) Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920079#action_12920079 ] Liyin Liang commented on MAPREDUCE-1943: your latest patch is based on your previous patch, why? > Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes > > > Key: MAPREDUCE-1943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1943-0.20-yahoo.patch, > MAPREDUCE-1943-0.20-yahoo.patch, MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S-fix.patch, > MAPREDUCE-1943-yahoo-hadoop-0.20S.patch > > > We have come across issues in production clusters wherein users abuse > counters, statusreport messages and split sizes. One such case was when one > of the users had 100 million counters. This leads to jobtracker going out of > memory and being unresponsive. In this jira I am proposing to put sane limits > on the status report length, the number of counters and the size of block > locations returned by the input split. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1943) Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888693#action_12888693 ] Amareshwari Sriramadasu commented on MAPREDUCE-1943: Limiting task diagnostic info and status are done in MAPREDUCE-1482. > Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes > > > Key: MAPREDUCE-1943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Mahadev konar >Assignee: Mahadev konar > Attachments: MAPREDUCE-1943-0.20-yahoo.patch > > > We have come across issues in production clusters wherein users abuse > counters, statusreport messages and split sizes. One such case was when one > of the users had 100 million counters. This leads to jobtracker going out of > memory and being unresponsive. In this jira I am proposing to put sane limits > on the status report length, the number of counters and the size of block > locations returned by the input split. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1943) Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes
[ https://issues.apache.org/jira/browse/MAPREDUCE-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888557#action_12888557 ] Scott Chen commented on MAPREDUCE-1943: --- +1 to the idea. We have seen the huge split-size kills JT. This will help. > Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes > > > Key: MAPREDUCE-1943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1943 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Mahadev konar >Assignee: Mahadev konar > > We have come across issues in production clusters wherein users abuse > counters, statusreport messages and split sizes. One such case was when one > of the users had 100 million counters. This leads to jobtracker going out of > memory and being unresponsive. In this jira I am proposing to put sane limits > on the status report length, the number of counters and the size of block > locations returned by the input split. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
