[jira] [Commented] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057279#comment-13057279 ] Hudson commented on MAPREDUCE-2185: --- Integrated in Hadoop-Mapreduce-trunk #722 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/]) > Infinite loop at creating splits using CombineFileInputFormat > - > > Key: MAPREDUCE-2185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Reporter: Hairong Kuang >Assignee: Ramkumar Vadali > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2185.patch > > > This is caused by a missing block in HDFS. So the block's locations are > empty. The following code adds the block to blockToNodes map but not to > rackToBlocks map. Later on when generating splits, only blocks in > rackToBlocks are removed from blockToNodes map. So blockToNodes map can never > become empty therefore causing infinite loop > {code} > // add this block to the block --> node locations map > blockToNodes.put(oneblock, oneblock.hosts); > // add this block to the rack --> block map > for (int j = 0; j < oneblock.racks.length; j++) { > .. > } > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13045186#comment-13045186 ] Hudson commented on MAPREDUCE-2185: --- Integrated in Hadoop-Mapreduce-trunk-Commit #717 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/717/]) MAPREDUCE-2185. Fix infinite loop at creating splits using CombineFileInputFormat. (Ramkumar Vadali via schen) schen : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1132807 Files : * /hadoop/mapreduce/trunk/CHANGES.txt * /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java * /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java > Infinite loop at creating splits using CombineFileInputFormat > - > > Key: MAPREDUCE-2185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Reporter: Hairong Kuang >Assignee: Ramkumar Vadali > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2185.patch > > > This is caused by a missing block in HDFS. So the block's locations are > empty. The following code adds the block to blockToNodes map but not to > rackToBlocks map. Later on when generating splits, only blocks in > rackToBlocks are removed from blockToNodes map. So blockToNodes map can never > become empty therefore causing infinite loop > {code} > // add this block to the block --> node locations map > blockToNodes.put(oneblock, oneblock.hosts); > // add this block to the rack --> block map > for (int j = 0; j < oneblock.racks.length; j++) { > .. > } > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039357#comment-13039357 ] dhruba borthakur commented on MAPREDUCE-2185: - +1 code looks good to me. > Infinite loop at creating splits using CombineFileInputFormat > - > > Key: MAPREDUCE-2185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Reporter: Hairong Kuang >Assignee: Ramkumar Vadali > Attachments: MAPREDUCE-2185.patch > > > This is caused by a missing block in HDFS. So the block's locations are > empty. The following code adds the block to blockToNodes map but not to > rackToBlocks map. Later on when generating splits, only blocks in > rackToBlocks are removed from blockToNodes map. So blockToNodes map can never > become empty therefore causing infinite loop > {code} > // add this block to the block --> node locations map > blockToNodes.put(oneblock, oneblock.hosts); > // add this block to the rack --> block map > for (int j = 0; j < oneblock.racks.length; j++) { > .. > } > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2185) Infinite loop at creating splits using CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038301#comment-13038301 ] Hadoop QA commented on MAPREDUCE-2185: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480153/MAPREDUCE-2185.patch against trunk revision 1126591. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/297//console This message is automatically generated. > Infinite loop at creating splits using CombineFileInputFormat > - > > Key: MAPREDUCE-2185 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2185 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Reporter: Hairong Kuang >Assignee: Ramkumar Vadali > Attachments: MAPREDUCE-2185.patch > > > This is caused by a missing block in HDFS. So the block's locations are > empty. The following code adds the block to blockToNodes map but not to > rackToBlocks map. Later on when generating splits, only blocks in > rackToBlocks are removed from blockToNodes map. So blockToNodes map can never > become empty therefore causing infinite loop > {code} > // add this block to the block --> node locations map > blockToNodes.put(oneblock, oneblock.hosts); > // add this block to the rack --> block map > for (int j = 0; j < oneblock.racks.length; j++) { > .. > } > {code} -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira