Hi Guys, It's not a hadoop solution but it works : cdh-hadoop@sayonara:~/mscdata/edgecast/201010 (85) 21:41:47 201010:> for i in `hadoop fs -ls /user/cdh-hadoop/mscdata/edgecast/201010/ | awk '{print $8}'`; do echo $i; hadoop fs -cat $i | gzip -t; done
/user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0000.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0001.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0002.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0003.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0004.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0005.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0006.log.gz gzip: stdin: invalid compressed data--crc error gzip: stdin: invalid compressed data--length error /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0007.log.gz /user/cdh-hadoop/mscdata/edgecast/201010/wpc_NORM_201010-0001-0008.log.gz The file right before the error message will be corrupted. On Thu, Feb 10, 2011 at 6:57 PM, Charles Gonçalves <charles...@gmail.com>wrote: > Hy Guys, > > I have 2 TB of data to process on my MSC work, but I share resources with > others students and don't have all that space. > So I gzipped my files in splits with sizes similars with the block (to > benefit from multiple maps). > > The problem is that i'm getting a lot of those errors: > > java.io.IOException: incorrect data check > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.inflateBytesDirect(Native > Method) > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.decompress(ZlibDecompressor.java:221) > at > org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:80) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74) > at java.io.InputStream.read(InputStream.java:85) > at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134) > at > org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:97) > at msc.pig.EdgeLoader.getValidFields(Unknown Source) > at msc.pig.EdgeLoader.getNext(Unknown Source) > at > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:187) > at > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:423) > at > org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > Does anyone has a suggestion on how can I check which file(s) has this > problem? > > > Thanks > > -- > *Charles Ferreira Gonçalves * > http://homepages.dcc.ufmg.br/~charles/ > UFMG - ICEx - Dcc > Cel.: 55 31 87741485 > Tel.: 55 31 34741485 > Lab.: 55 31 34095840 > -- *Charles Ferreira Gonçalves * http://homepages.dcc.ufmg.br/~charles/ UFMG - ICEx - Dcc Cel.: 55 31 87741485 Tel.: 55 31 34741485 Lab.: 55 31 34095840