[ https://issues.apache.org/jira/browse/MAPREDUCE-1649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated MAPREDUCE-1649: ---------------------------------- Attachment: MAPREDUCE-1649.1.branch-0.20.patch A simple fix (MAPREDUCE-1649.1.branch-0.20.patch) is to ignore the splits that start with non-0 offset in {{TextInputFormat}} when the file is non-splittable. > Compressed files with TextInputFormat does not work with > CombineFileInputFormat > ------------------------------------------------------------------------------- > > Key: MAPREDUCE-1649 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1649 > Project: Hadoop Map/Reduce > Issue Type: Bug > Affects Versions: 0.20.2 > Reporter: Zheng Shao > Assignee: Zheng Shao > Attachments: MAPREDUCE-1649.1.branch-0.20.patch > > > {{CombineFileInputFormat}} creates splits based on blocks, regardless whether > the underlying {{FileInputFormat}} is splittable or not.. > This means that we can have 2 or more splits for a compressed text file with > {{TextInputFormat}}. For each of these splits, > {{TextInputFormat.getRecordReader}} will return a {{RecordReader}} for the > whole compressed file, thus causing duplicate input data. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.