[ https://issues.apache.org/jira/browse/MAPREDUCE-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13557835#comment-13557835 ]
Hadoop QA commented on MAPREDUCE-4946: -------------------------------------- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565590/MAPREDUCE-4946.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 5 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs: org.apache.hadoop.mapreduce.v2.app.TestRecovery {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3253//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3253//console This message is automatically generated. > Type conversion of map completion events leads to performance problems with > large jobs > -------------------------------------------------------------------------------------- > > Key: MAPREDUCE-4946 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4946 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am > Affects Versions: 2.0.2-alpha, 0.23.5 > Reporter: Jason Lowe > Assignee: Jason Lowe > Priority: Critical > Attachments: MAPREDUCE-4946.patch > > > We've seen issues with large jobs (e.g.: 13,000 maps and 3,500 reduces) where > reducers fail to connect back to the AM after being launched due to > connection timeout. Looking at stack traces of the AM during this time we > see a lot of IPC servers stuck waiting for a lock to get the application ID > while type converting the map completion events. What's odd is that normally > getting the application ID should be very cheap, but in this case we're > type-converting thousands of map completion events for *each* reducer > connecting. That means we end up type-converting the map completion events > over 45 million times during the lifetime of the example job (13,000 * 3,500). > We either need to make the type conversion much cheaper (i.e.: lockless or at > least read-write locked) or, even better, store the completion events in a > form that does not require type conversion when serving them up to reducers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira