william zhu created HADOOP-12684:
------------------------------------
Summary: CombineFileInputFormat may cause splits uneveness
Key: HADOOP-12684
URL: https://issues.apache.org/jira/browse/HADOOP-12684
Project: Hadoop Common
Issue Type: Wish
Environment: Hadoop 2.6
Reporter: william zhu
CombieFileInputFormat add blocks into one split use below codes :
```
if (maxSize != 0 && curSplitSize >= maxSize) {
// create an input split and add it to the splits array
addCreatedSplit(splits, nodes, validBlocks);
curSplitSize = 0;
validBlocks.clear();
}
```
but , in some cases , the file is not splitable. Then files may contains both
large and small . In this case, one split will be added a large file and then
reach the threshold .
I think some algorithm such as sort first will make the splits more even .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
