Re: buildSupportsSnappy exception when reading the snappy file in Spark

2015-09-08 Thread Akhil Das 
Looks like you are having different versions of snappy library. Here's a
similar discussion if you haven't seen it already
http://stackoverflow.com/questions/22150417/hadoop-mapreduce-java-lang-unsatisfiedlinkerror-org-apache-hadoop-util-nativec

Thanks
Best Regards

On Mon, Sep 7, 2015 at 7:41 AM, dong.yajun  wrote:

> hi all,
>
> I met problem that can't read the file with snappy encoding from HDFS in
> Spark1.4.1,
>
> I have configured the SPARK_LIBRARY_PATH property in conf/spark-env.sh to
> the native path of Hadoop and restarted the spark cluster
>
> SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/app/install/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native
>
>
> the partial exception:
>
> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
> reading HFile Trailer from file
> hdfs://nameservice1/hbase/data/default/IM_ItemBase/02296539242087aea77877dced9ba3d5/BaseInfo/9fe36f74334c4d30ba1bfc17bbd717f5
>
>  at
> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:478)
>
>  at
> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:521)
>
>  at
> com.newegg.ec.bigdata.dump.CombineHFileRecordReader.(CombineHFileRecordReader.java:33)
>
>  ... 19 more
>
> Caused by: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>
>  at
> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
>
>  at
> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
>
>  at
> org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:192)
>
>  at
> org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176)
>
>  at
> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:328)
>
>  at
> org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:423)
>
>  at
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90)
>
> --
> *Ric Dong*
>
>

Re: buildSupportsSnappy exception when reading the snappy file in Spark

2015-09-08 Thread dong.yajun 
hi Akhil,

I just use property key LD_LIBRARY_PATH in conf/spark-env.xml instead of
SPARK_LIBRARY_PATH which points to the path of native, it works.

thanks.

On Tue, Sep 8, 2015 at 6:14 PM, Akhil Das 
wrote:

> Looks like you are having different versions of snappy library. Here's a
> similar discussion if you haven't seen it already
> http://stackoverflow.com/questions/22150417/hadoop-mapreduce-java-lang-unsatisfiedlinkerror-org-apache-hadoop-util-nativec
>
> Thanks
> Best Regards
>
> On Mon, Sep 7, 2015 at 7:41 AM, dong.yajun  wrote:
>
>> hi all,
>>
>> I met problem that can't read the file with snappy encoding from HDFS in
>> Spark1.4.1,
>>
>> I have configured the SPARK_LIBRARY_PATH property in conf/spark-env.sh to
>> the native path of Hadoop and restarted the spark cluster
>>
>> SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/app/install/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native
>>
>>
>> the partial exception:
>>
>> Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem 
>> reading HFile Trailer from file
>> hdfs://nameservice1/hbase/data/default/IM_ItemBase/02296539242087aea77877dced9ba3d5/BaseInfo/9fe36f74334c4d30ba1bfc17bbd717f5
>>
>>  at
>> org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:478)
>>
>>  at
>> org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:521)
>>
>>  at
>> com.newegg.ec.bigdata.dump.CombineHFileRecordReader.(CombineHFileRecordReader.java:33)
>>
>>  ... 19 more
>>
>> Caused by: java.lang.UnsatisfiedLinkError:
>> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
>>
>>  at
>> org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
>>
>>  at
>> org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
>>
>>  at
>> org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:192)
>>
>>  at
>> org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176)
>>
>>  at
>> org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:328)
>>
>>  at
>> org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:423)
>>
>>  at
>> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90)
>>
>> --
>> *Ric Dong*
>>
>>
>


-- 
*Ric Dong*