[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857640#action_12857640 ] Hong Tang commented on MAPREDUCE-1317: -- Patch "mapreduce-1317-20091223.patch" applies cleanly to yahoop-hadoop-0.20.1xx branch. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798371#action_12798371 ] Hudson commented on MAPREDUCE-1317: --- Integrated in Hadoop-Mapreduce-trunk #200 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/200/]) > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797381#action_12797381 ] Tamas Sarlos commented on MAPREDUCE-1317: - Patch looks good to me. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796401#action_12796401 ] Hadoop QA commented on MAPREDUCE-1317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428801/mapreduce-1317-20091223.patch against trunk revision 894964. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/console This message is automatically generated. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796288#action_12796288 ] Hong Tang commented on MAPREDUCE-1317: -- The failed tests are not related to the patch. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794499#action_12794499 ] Hadoop QA commented on MAPREDUCE-1317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428801/mapreduce-1317-20091223.patch against trunk revision 893800. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/342/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/342/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/342/console This message is automatically generated. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794472#action_12794472 ] Hong Tang commented on MAPREDUCE-1317: -- The failed tests are not related to the patch. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12794295#action_12794295 ] Hadoop QA commented on MAPREDUCE-1317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428801/mapreduce-1317-20091223.patch against trunk revision 893469. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/339/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/339/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/339/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/339/console This message is automatically generated. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793994#action_12793994 ] Hong Tang commented on MAPREDUCE-1317: -- The 2 failed unit tests in rumen were caused by my false assumption that LoggedXXX objects are immutable - while in fact the HadoopLogAnalyzer actually mutates the List object returned from the getter method. I restore the original semantics by creating an empty list instead of using Collections.emptyList(). I filed MAPREDUCE-1330 to propose to make LoggedXXX APIs more consistent in this regard. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793943#action_12793943 ] Hadoop QA commented on MAPREDUCE-1317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428791/mapreduce-1317-20091222-2.patch against trunk revision 893361. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/239/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/239/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/239/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/239/console This message is automatically generated. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793821#action_12793821 ] Hadoop QA commented on MAPREDUCE-1317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428720/mapreduce-1317-20091222.patch against trunk revision 893055. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/334/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/334/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/334/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/334/console This message is automatically generated. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792441#action_12792441 ] Hong Tang commented on MAPREDUCE-1317: -- Through YourKit profiling, we found two places where we could save memory: - LoggedLocation - we should share references to the same LoggedLocation for the same preferred location for different map tasks. - LoggedTaskAttempt.hostName - we should keep a cache of all host names for the cluster and share the references. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Hong Tang > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
