[
https://issues.apache.org/jira/browse/HADOOP-4565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HADOOP-4565:
-------------------------------------
Attachment: CombineMultiFile4.patch
1. Inherit from MultiFileInputFormat.
2. Do not combine spilled blocks with blocks from the next rack. Instead after
all racks are processed, combine all spilled blocks into a single split
3.Renamed nodesToBlocks to be racksToBlocks.
4. CombineFileInputFormat is an abstract class. CombineFileRecordReader can be
used as a RecordReader.
5. The sorting is required because we want the algorithm to be capable of
creating large splits. If block B1 is in racks R1, R2 and R3 and block B2 is in
racks R1 and R4, then we want to process rack R1 before rack R4. If we process
R1 first then block B1 and B2 can be combined into a single split. (Instead if
we process rack R4, then one split will contain B2 while another split will
have block B1.
> MultiFileInputSplit can use data locality information to create splits
> ----------------------------------------------------------------------
>
> Key: HADOOP-4565
> URL: https://issues.apache.org/jira/browse/HADOOP-4565
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: CombineMultiFile.patch, CombineMultiFile2.patch,
> CombineMultiFile3.patch, CombineMultiFile4.patch
>
>
> The MultiFileInputFormat takes a set of paths and creates splits based on
> file sizes. Each splits contains a few files an each split are roughly equal
> in size. It would be efficient if we can extend this InputFormat to create
> splits such each all the blocks in one split and either node-local or
> rack-local.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
