If you're case is like mine, where you have lots of .gz files and you
don't want splits in the middle of those files, you can use the code I
just sent in the thread about traversing subdirectories. In brief, your
RecordReader could do something like:
public static class MyRecordReader
implements RecordReader<DocLocation, Text> {
private CompressionCodecFactory compressionCodecs = null;
private long start;
private long end;
private long pos;
private Path file;
private LineRecordReader.LineReader in;
public MyRecordReader(JobConf job, FileSplit split)
throws IOException {
file = split.getPath();
start = 0;
end = split.getLength();
compressionCodecs = new CompressionCodecFactory(job);
CompressionCodec codec = compressionCodecs.getCodec(file);
FileSystem fs = file.getFileSystem(job);
FSDataInputStream fileIn = fs.open(file);
if (codec != null) {
in = new LineRecordReader.LineReader(codec.createInputStream(fil
eIn), job);
} else {
in = new LineRecordReader.LineReader(fileIn, job);
}
pos = 0;
}
Alex Loddengaard <a...@cloudera.com> writes:
> Hi Adam,
>
> Gzipped files don't play that nicely with Hadoop, because they aren't
> splittable. Can you use bzip2 instead? bzip2 files play more nicely with
> Hadoop, because they're splittable. If you're stuck with gzip, then take a
> look here: <http://issues.apache.org/jira/browse/HADOOP-437>. I don't know
> if you'll have to set the same JobConf parameter in newer versions of
> Hadoop, but it's worth trying out.
>
> Hope this helps.
>
> Alex
>
> On Wed, Jun 3, 2009 at 11:50 AM, Adam Silberstein
> <silbe...@yahoo-inc.com>wrote:
>
>> Hi,
>>
>> I have some hadoop code that works properly when the input files are not
>> compressed, but it is not working for the gzipped versions of those
>> files. My files are named with *.gz, but the format is not being
>> recognized. I'm under the impression I don't need to set any JobConf
>> parameters to indicate compressed input.
>>
>>
>>
>> I'm actually taking a directory name as input, and modeled that aspect
>> of my application after the MultiFileWordCount.java example in
>> org.apache.hadoop.examples. Not sure if this is part of the problem.
>>
>>
>>
>> Thanks,
>>
>> Adam
>>
>>
