If you really confirmed that libsnappy.so.1 is in the correct location, and being loaded into java library path, working in your test program, but still didn't work in MR, there is one another possibility which was puzzling me before. How do you get the libhadoop.so in your hadoop environment? Did you compile it by yourself, or is it from some vendors? You want to make sure the Java native methods of invoking snappy is also being compiled and available in libhadoop.so. For example, the following command will prove it: $ nm ./libhadoop.so | grep snappy00000000000035c0 T Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_compressBytesDirect0000000000003960 T Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs0000000000003bb0 T Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_decompressBytesDirect0000000000003f60 T Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs0000000000206cf0 b dlsym_snappy_compress0000000000206d20 b dlsym_snappy_uncompress Without these Java native methods being compiled and available in the libhadoop.so, MR runtime will also complain that "native snappy library not available". Yong Date: Thu, 2 Jan 2014 13:37:46 +0200 Subject: Re: Setting up Snappy compression in Hadoop From: am...@infolinks.com To: user@hadoop.apache.org
I did everything mentioned in the link Ted mentioned, and the test actually works, but using Snappy for MapReduce map output compression still fails with "native snappy library not available". On Wed, Jan 1, 2014 at 6:37 PM, bharath vissapragada <bharathvissapragada1...@gmail.com> wrote: Did you build it for your platform? You can do an "ldd" on the .so file to check if the dependent libs are present. Also make sure you placed it in the right directory for your platform (Linux-amd64-64 or Linux-i386-32) On Wed, Jan 1, 2014 at 10:02 PM, Ted Yu <yuzhih...@gmail.com> wrote: Please take a look at http://hbase.apache.org/book.html#snappy.compression Cheers On Wed, Jan 1, 2014 at 8:05 AM, Amit Sela <am...@infolinks.com> wrote: Hi all, I'm running on Hadoop 1.0.4 and I'd like to use Snappy for map output compression.I'm adding the configurations: configuration.setBoolean("mapred.compress.map.output", true); configuration.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec"); And I've added libsnappy.so.1 to $HADOOP_HOME/lib/native/Linux-amd64-64/ Still, all map tasks fail with "native snappy library not available". Could anyone elaborate on how to install Snappy for Hadoop ? Thanks, Amit.
