Yuqi Gu created HBASE-19387: ------------------------------- Summary: HBase-spark snappy.SnappyError on Arm64 Key: HBASE-19387 URL: https://issues.apache.org/jira/browse/HBASE-19387 Project: HBase Issue Type: Bug Components: spark, test Affects Versions: 3.0.0 Reporter: Yuqi Gu
When running the hbase-spark Unit tests on Arm64, the failures are shown as follows: {code:java} scalatest-maven-plugin:1.0:test (test) @ hbase-spark --- Discovery starting. Discovery completed in 2 seconds, 837 milliseconds. Run starting. Expected test count is: 79 HBaseDStreamFunctionsSuite: Formatting using clusterid: testClusterID - bulkput to test HBase client *** FAILED *** java.lang.reflect.InvocationTargetException: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63) ... Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=aarch64 at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) ... Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux and os.arch=aarch64 at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152) at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46) at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) ... Formatting using clusterid: testClusterID PartitionFilterSuite: *** RUN ABORTED *** java.lang.reflect.InvocationTargetException: at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63) ... Cause: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) ... Cause: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy at org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) {code} Root Cause: The spark version in hbase is 1.6.0. And the snappy version based on hbase-1.6.0 do not support the Arm64. Fix it by upgrading snappy-java to 1.1.4 in hbase-spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)