hi all,
I met problem that can't read the file with snappy encoding from HDFS in
Spark1.4.1,
I have configured the SPARK_LIBRARY_PATH property in conf/spark-env.sh to
the native path of Hadoop and restarted the spark cluster
SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/app/install/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/lib/native
the partial exception:
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
Problem reading HFile Trailer from file
hdfs://nameservice1/hbase/data/default/IM_ItemBase/02296539242087aea77877dced9ba3d5/BaseInfo/9fe36f74334c4d30ba1bfc17bbd717f5
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:478)
at
org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:521)
at
com.newegg.ec.bigdata.dump.CombineHFileRecordReader.<init>(CombineHFileRecordReader.java:33)
... 19 more
Caused by: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
at
org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63)
at
org.apache.hadoop.io.compress.SnappyCodec.getDecompressorType(SnappyCodec.java:192)
at
org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:176)
at
org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getDecompressor(Compression.java:328)
at
org.apache.hadoop.hbase.io.compress.Compression.decompress(Compression.java:423)
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:90)
--
*Ric Dong*
