[ https://issues.apache.org/jira/browse/HBASE-19387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272275#comment-16272275 ]
Yuqi Gu commented on HBASE-19387: --------------------------------- Everything is ok for running hbase-spark unit tests both on x86 and Arm64. > HBase-spark snappy.SnappyError on Arm64 > --------------------------------------- > > Key: HBASE-19387 > URL: https://issues.apache.org/jira/browse/HBASE-19387 > Project: HBase > Issue Type: Bug > Components: spark, test > Affects Versions: 3.0.0 > Reporter: Yuqi Gu > Priority: Minor > Attachments: HBASE-19387.patch > > > When running the hbase-spark Unit tests on Arm64, the failures are shown as > follows: > > {code:java} > scalatest-maven-plugin:1.0:test (test) @ hbase-spark --- > Discovery starting. > Discovery completed in 2 seconds, 837 milliseconds. > Run starting. Expected test count is: 79 > HBaseDStreamFunctionsSuite: > Formatting using clusterid: testClusterID > - bulkput to test HBase client *** FAILED *** > java.lang.reflect.InvocationTargetException: > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) > at > org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) > at > org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) > at > org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63) > ... > Cause: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: > [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux > and os.arch=aarch64 > at > org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) > at > org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) > at > org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) > ... > Cause: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=aarch64 > at org.xerial.snappy.SnappyLoader.findNativeLibrary(SnappyLoader.java:331) > at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:171) > at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:152) > at org.xerial.snappy.Snappy.<clinit>(Snappy.java:46) > at > org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) > ... > Formatting using clusterid: testClusterID > PartitionFilterSuite: > *** RUN ABORTED *** > java.lang.reflect.InvocationTargetException: > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) > at > org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) > at > org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) > at > org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:63) > ... > Cause: java.lang.IllegalArgumentException: java.lang.NoClassDefFoundError: > Could not initialize class org.xerial.snappy.Snappy > at > org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:156) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) > at > org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) > at > org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) > ... > Cause: java.lang.NoClassDefFoundError: Could not initialize class > org.xerial.snappy.Snappy > at > org.apache.spark.io.SnappyCompressionCodec.<init>(CompressionCodec.scala:154) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:72) > at > org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:65) > at > org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) > at > org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:80) > at > org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34) > {code} > Root Cause: > The spark version in hbase is 1.6.0. > And the snappy version based on hbase-1.6.0 do not support the Arm64. > Fix it by upgrading snappy-java to 1.1.4 in hbase-spark. -- This message was sent by Atlassian JIRA (v6.4.14#64029)