[ https://issues.apache.org/jira/browse/SPARK-13690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181362#comment-15181362 ]
Santiago M. Mola commented on SPARK-13690: ------------------------------------------ snappy-java does not have any fallback, but snappy seems to work on arm64 correctly. I submitted a PR for snappy-java, so a future version should have support. This issue will have to wait until such version is out. I don't expect active support for arm64, but given the latest developments on arm64 servers, I'm interested in experimenting with it. It seems I'm not the first one to think about it: http://www.sparkonarm.com/ ;-) > UnsafeShuffleWriterSuite fails on arm64 (SnappyError, no native library is > found) > --------------------------------------------------------------------------------- > > Key: SPARK-13690 > URL: https://issues.apache.org/jira/browse/SPARK-13690 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.0.0 > Environment: $ java -version > java version "1.8.0_73" > Java(TM) SE Runtime Environment (build 1.8.0_73-b02) > Java HotSpot(TM) 64-Bit Server VM (build 25.73-b02, mixed mode) > $ uname -a > Linux spark-on-arm 4.2.0-55598-g45f70e3 #5 SMP Tue Feb 2 10:14:08 CET 2016 > aarch64 aarch64 aarch64 GNU/Linux > Reporter: Santiago M. Mola > Priority: Minor > Labels: arm64, porting > > UnsafeShuffleWriterSuite fails because of missing Snappy native library on > arm64. > {code} > Tests run: 19, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 6.437 sec > <<< FAILURE! - in org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite > mergeSpillsWithFileStreamAndSnappy(org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite) > Time elapsed: 0.072 sec <<< ERROR! > java.lang.reflect.InvocationTargetException > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.testMergingSpills(UnsafeShuffleWriterSuite.java:337) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.mergeSpillsWithFileStreamAndSnappy(UnsafeShuffleWriterSuite.java:389) > Caused by: java.lang.IllegalArgumentException: org.xerial.snappy.SnappyError: > [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=Linux > and os.arch=aarch64 > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.testMergingSpills(UnsafeShuffleWriterSuite.java:337) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.mergeSpillsWithFileStreamAndSnappy(UnsafeShuffleWriterSuite.java:389) > Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no > native library is found for os.name=Linux and os.arch=aarch64 > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.testMergingSpills(UnsafeShuffleWriterSuite.java:337) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.mergeSpillsWithFileStreamAndSnappy(UnsafeShuffleWriterSuite.java:389) > mergeSpillsWithTransferToAndSnappy(org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite) > Time elapsed: 0.041 sec <<< ERROR! > java.lang.reflect.InvocationTargetException > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.testMergingSpills(UnsafeShuffleWriterSuite.java:337) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.mergeSpillsWithTransferToAndSnappy(UnsafeShuffleWriterSuite.java:384) > Caused by: java.lang.IllegalArgumentException: > java.lang.NoClassDefFoundError: Could not initialize class > org.xerial.snappy.Snappy > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.testMergingSpills(UnsafeShuffleWriterSuite.java:337) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.mergeSpillsWithTransferToAndSnappy(UnsafeShuffleWriterSuite.java:384) > Caused by: java.lang.NoClassDefFoundError: Could not initialize class > org.xerial.snappy.Snappy > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.testMergingSpills(UnsafeShuffleWriterSuite.java:337) > at > org.apache.spark.shuffle.sort.UnsafeShuffleWriterSuite.mergeSpillsWithTransferToAndSnappy(UnsafeShuffleWriterSuite.java:384) > Running org.apache.spark.JavaAPISuite > Tests run: 90, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 52.526 sec - > in org.apache.spark.JavaAPISuite > Running org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.761 sec - > in org.apache.spark.unsafe.map.BytesToBytesMapOnHeapSuite > Running org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite > Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.967 sec - > in org.apache.spark.unsafe.map.BytesToBytesMapOffHeapSuite > Running org.apache.spark.api.java.OptionalSuite > Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - > in org.apache.spark.api.java.OptionalSuite > Results : > Tests in error: > > UnsafeShuffleWriterSuite.mergeSpillsWithFileStreamAndSnappy:389->testMergingSpills:337 > » InvocationTarget > > UnsafeShuffleWriterSuite.mergeSpillsWithTransferToAndSnappy:384->testMergingSpills:337 > » InvocationTarget > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org