[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790112#action_12790112 ] Ravi Gummadi commented on MAPREDUCE-1252: - In HADOOP-6105, old key takes precedence irrespective of the order in which resources are loaded. I propose that we resolve based on the order of loading of resources irrespective of old/new key is seen. Thoughts ? > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784278#action_12784278 ] Owen O'Malley commented on MAPREDUCE-1252: -- To be clearer: 1. The previous patch fixes the warnings, but not the problem. Thus, it is necessary but not sufficient. 2. On a one node cluster with 0.21.0-dev from yesterday: a. Have mapred.map.tasks set to 12 in hadoop-site.xml. b. Submit a word count example on a two file input directory. 3. The 2 maps run fine. 4. The 2 reduces both lock up at 5.5% with a status of 2 of 12 maps fetched. Looking at the system directory the job.xml has mapreduce.job.maps = 2, but the task and the web ui show it as 12. If I change the config to use mapreduce.job.maps to set the default, everything works correctly. > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784114#action_12784114 ] Owen O'Malley commented on MAPREDUCE-1252: -- This patch is necessary to suppress the warnings, but doesn't fix the problem. The problem is that I was using a configuration with the old attribute name. The attribute name isn't being translated during the xml loading. I'm not sure how it is happening, but in some context, we get 2 and others 12. When I change the config to use the new attribute name, it all goes away. I suspect this is caused by a bug in HADOOP-6105. > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1252) Shuffle deadlocks on wrong number of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-1252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12784011#action_12784011 ] Hadoop QA commented on MAPREDUCE-1252: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426479/mr-1252.patch against trunk revision 885530. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/154/console This message is automatically generated. > Shuffle deadlocks on wrong number of maps > - > > Key: MAPREDUCE-1252 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1252 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task >Affects Versions: 0.21.0, 0.22.0 >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.21.0, 0.22.0 > > Attachments: mr-1252.patch > > > The new shuffle assumes that the number of maps is correct. The new > JobSubmitter sets the old value. Something misfires in the middle causing: > 09/12/01 00:00:15 WARN conf.Configuration: mapred.job.split.file is > deprecated. Instead, use mapreduce.job.splitfile > 09/12/01 00:00:15 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > But my reduces got stuck at 2 maps / 12 when there were only 2 maps in the > job. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
