I am using Cloudera distribution cdh3u1. When trying to check native codecs for better decompression performance such as Snappy or LZO, I ran into issues with random access using MapFile.Reader.get(key, value) method. First call of MapFile.Reader.get() works but a second call fails.
Also I am getting different exceptions depending on number of entries in a map file. With LzoCodec and 10 record file, jvm gets aborted. At the same time the DefaultCodec works fine for all cases, as well as record compression for the native codecs. I created a simple test program (attached) that creates map files locally with sizes of 10 and 100 records for three codecs: Default, Snappy, and LZO. (The test requires corresponding native library available) The summary of problems are given below: Map Size: 100 Compression: RECORD ================== DefaultCodec: OK SnappyCodec: OK LzoCodec: OK Map Size: 10 Compression: RECORD ================== DefaultCodec: OK SnappyCodec: OK LzoCodec: OK Map Size: 100 Compression: BLOCK ================ DefaultCodec: OK SnappyCodec: java.io.EOFException at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:114) LzoCodec: java.io.EOFException at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:114) Map Size: 10 Compression: BLOCK ================== DefaultCodec: OK SnappyCodec: java.lang.NoClassDefFoundError: Ljava/lang/InternalError at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompressBytesDirect(Native Method) LzoCodec: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00002b068ffcbc00, pid=6385, tid=47304763508496 # # JRE version: 6.0_21-b07 # Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode linux-amd64 ) # Problematic frame: # C [liblzo2.so.2+0x13c00] lzo1x_decompress+0x1a0 # # An error report file with more information is saved as: # /hadoop/user/yurgis/testapp/hs_err_pid6385.log # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. #
