[ https://issues.apache.org/jira/browse/MAPREDUCE-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sadayuki Furuhashi updated MAPREDUCE-2862: ------------------------------------------ Attachment: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch Breaks infinite loop in CombineFileInputFormat.getMoreSplits by ignoring corrupted blocks. > Infinite loop in CombineFileInputFormat#getMoreSplits(), with missing blocks > ---------------------------------------------------------------------------- > > Key: MAPREDUCE-2862 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2862 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Kazuki Ohta > Attachments: MAPREDUCE-2862-warn-and-ignore-corrupted-blocks.patch > > > Hi, we met the infinite loop on CombineFileInputFormat#getMoreSplits(). > At first, we lost some blocks by mis-operation :-(. Then, one job tried to > use these missing blocks. At that time getMoreSplits() goes into the infinite > loop. > From our investigation, this List could be an empty array. > > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L363 > Then 'for' loop just after that line does nothing, and entry is not removed > from 'blockToNodes'. > Finally this line goes into the infinite loop. > > https://github.com/apache/hadoop-mapreduce/blob/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java#L348 > We're now creating the patch against this problem... -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira