Hey there,
I've been running a cluster for over a year and was getting a lzo
decompressing exception less than once a month. Suddenly it happens almost
once per day. Any ideas what could be causing it? I'm with hadoop 0.20.2
I've thought in moving to snappy but would like to know why this happens
more often now
The exception happens always when the reducer gets data from the map and
looks like:
Error: java.lang.InternalError: lzo1x_decompress returned: -8
at
com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native
Method)
at
com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305)
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76)
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285)
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216)
Thanks in advance.
--
View this message in context:
http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3783652.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
