Hope this may help you: http://blogs.impetus.com/big_data/big_data_technologies/SnappyCompressionInHBase.do
On Thu, Dec 11, 2014 at 7:25 AM, Fabio <[email protected]> wrote: > Plain Apache Hadoop 2.5.0. > Too bad it didn't work, hope someone can help. > > > On 12/10/2014 06:22 PM, peterm_second wrote: > >> Hi Fabio , >> Thanks for the reply, but unfortunately it didn't work. I am using >> vanilla hadoop 2.4 with vanilla hive 0.14 and so on, I am using the vanilla >> distros. >> I did set the HADOOP_COMMON_LIB_NATIVE_DIR but that didn't make any >> change. What version were you using ? >> >> Peter >> >> >> On 10.12.2014 16:23, Fabio wrote: >> >>> Not sure it will help, but if the problem is native library loading, I >>> spent a loooong time trying anything to make it work. >>> I may suggest to try also: >>> export JAVA_LIBRARY_PATH=/opt/yarn/hadoop-2.5.0/lib/native >>> export HADOOP_COMMON_LIB_NATIVE_DIR=/opt/yarn/hadoop-2.5.0/lib >>> I have this both in the bash "init" script (/etc/profile.p/...) and in >>> /opt/yarn/hadoop-2.5.0/etc/hadoop/hadoop-env.sh; quite sure it's >>> redundant, but as long as it works I don't change it. >>> I see here I commented out my attempts to set HADOOP_OPTS, so maybe it's >>> not necessary. >>> I don't see anything in my .xml config files. >>> Also, someone says to compile the libraries under your 64 bit system, >>> since the ones in Hadoop are for a 32bit architecture. >>> >>> Good luck >>> >>> Fabio >>> >>> On 12/10/2014 02:57 PM, peterm_second wrote: >>> >>>> Hi guys, >>>> I have a hadoop + hbase + hive application, >>>> For some reason my cluster is unable to find the snappy native library >>>> Here is the exception : >>>> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z >>>> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native >>>> Method) >>>> at >>>> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) >>>> >>>> at >>>> org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) >>>> >>>> at org.apache.hadoop.io.compress.CodecPool.getCompressor( >>>> CodecPool.java:148) >>>> at org.apache.hadoop.io.compress.CodecPool.getCompressor( >>>> CodecPool.java:163) >>>> at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115) >>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer. >>>> sortAndSpill(MapTask.java:1583) >>>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush( >>>> MapTask.java:1462) >>>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) >>>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) >>>> at java.security.AccessController.doPrivileged(Native Method) >>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>> at org.apache.hadoop.security.UserGroupInformation.doAs( >>>> UserGroupInformation.java:1548) >>>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) >>>> >>>> >>>> I am working with a 64bit ubuntu 14.04LTS. I've installed snappy on my >>>> os and added the coppied the libs to hadoop_home/lib/native >>>> I've also added the libs to the JRE, but it still fails as if nothing >>>> is present. >>>> I've added HADOOP_OPTS="-Djava.net.preferIPv4Stack=true >>>> $GC_DEBUG_OPTS -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native >>>> $HADOOP_OPTS" >>>> in my yarn xml I have >>>> <property> >>>> <name>yarn.app.mapreduce.am.env</name> >>>> <value>LD_LIBRARY_PATH=$HADOOP_HOME/lib/native</value> >>>> </property> >>>> >>>> in my mapred-site.xml i have >>>> <property> >>>> <name>mapred.child.java.opts</name> >>>> <value> -Djava.library.path=/usr/local/hadoop-2.4.0/lib/native< >>>> /value> >>>> </property> >>>> <property> >>>> <name>mapreduce.reduce.java.opts</name> >>>> <value>-Djava.library.path=/usr/local/hadoop-2.4.0/lib/native</value> >>>> </property> >>>> >>>> The last two were a desperation move. >>>> The result is always the same. Any ideas would be welcomed. >>>> >>>> Thanks, >>>> Peter >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >> . >> >> > -- *Thanks & Regards* *Hanish Bansal*
