L. C. Hsieh created SPARK-36681:
-----------------------------------
Summary: Fail to load Snappy codec
Key: SPARK-36681
URL: https://issues.apache.org/jira/browse/SPARK-36681
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
snappy-java as a native library should not be relocated in Hadoop shaded client
libraries. Currently we use Hadoop shaded client libraries in Spark. If trying
to use SnappyCodec to write sequence file, we will encounter the following
error:
{code}
[info] Cause: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
[info] at
org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawCompress(Native
Method)
[info] at
org.apache.hadoop.shaded.org.xerial.snappy.Snappy.compress(Snappy.java:151)
[info] at
org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressDirectBuf(SnappyCompressor.java:282)
[info] at
org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:210)
[info] at
org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:149)
[info] at
org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:142)
[info] at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.writeBuffer(SequenceFile.java:1589)
[info] at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.sync(SequenceFile.java:1605)
[info] at
org.apache.hadoop.io.SequenceFile$BlockCompressWriter.close(SequenceFile.java:1629)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
