[ https://issues.apache.org/jira/browse/MAPREDUCE-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536698#comment-14536698 ]
Hudson commented on MAPREDUCE-2094: ----------------------------------- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2138 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2138/]) MAPREDUCE-2094. LineRecordReader should not seek into non-splittable, compressed streams. (cdouglas: rev 2edcf931d7843cddcf3da5666a73d6ee9a10d00d) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/LineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/pom.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/TestSafeguardSplittingUnsplittableFiles.txt.gz * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/LineRecordReader.java > LineRecordReader should not seek into non-splittable, compressed streams. > ------------------------------------------------------------------------- > > Key: MAPREDUCE-2094 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2094 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task > Reporter: Niels Basjes > Assignee: Niels Basjes > Fix For: 2.8.0 > > Attachments: M2094-1.patch, M2094.patch, > MAPREDUCE-2094-2011-05-19.patch, > MAPREDUCE-2094-20140727-svn-fixed-spaces.patch, > MAPREDUCE-2094-20140727-svn.patch, MAPREDUCE-2094-20140727.patch, > MAPREDUCE-2094-2015-05-05-2328.patch, > MAPREDUCE-2094-FileInputFormat-docs-v2.patch > > > When implementing a custom derivative of FileInputFormat we ran into the > effect that a large Gzipped input file would be processed several times. > A near 1GiB file would be processed around 36 times in its entirety. Thus > producing garbage results and taking up a lot more CPU time than needed. > It took a while to figure out and what we found is that the default > implementation of the isSplittable method in > [org.apache.hadoop.mapreduce.lib.input.FileInputFormat | > http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java?view=markup > ] is simply "return true;". > This is a very unsafe default and is in contradiction with the JavaDoc of the > method which states: "Is the given filename splitable? Usually, true, but if > the file is stream compressed, it will not be. " . The actual implementation > effectively does "Is the given filename splitable? Always true, even if the > file is stream compressed using an unsplittable compression codec. " > For our situation (where we always have Gzipped input) we took the easy way > out and simply implemented an isSplittable in our class that does "return > false; " > Now there are essentially 3 ways I can think of for fixing this (in order of > what I would find preferable): > # Implement something that looks at the used compression of the file (i.e. do > migrate the implementation from TextInputFormat to FileInputFormat). This > would make the method do what the JavaDoc describes. > # "Force" developers to think about it and make this method abstract. > # Use a "safe" default (i.e. return false) -- This message was sent by Atlassian JIRA (v6.3.4#6332)