Which version of the Hadoop LZO library are you using? It looks like something I'm pretty sure was fixed in a newer version.
-Joey On Feb 28, 2012, at 4:58, Marc Sturlese <marc.sturl...@gmail.com> wrote: > Hey there, > I've been running a cluster for over a year and was getting a lzo > decompressing exception less than once a month. Suddenly it happens almost > once per day. Any ideas what could be causing it? I'm with hadoop 0.20.2 > I've thought in moving to snappy but would like to know why this happens > more often now > > The exception happens always when the reducer gets data from the map and > looks like: > > Error: java.lang.InternalError: lzo1x_decompress returned: -8 > at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native > Method) > at > com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305) > at > org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285) > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216) > > Thanks in advance. > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3783652.html > Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
