[
https://issues.apache.org/jira/browse/ARROW-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651642#comment-16651642
]
Wes McKinney commented on ARROW-3476:
-------------------------------------
[~kiszk] there is nothing structurally preventing the Arrow columnar format
from being used on big-endian platforms, it is an implementation and testing
question. If IBM could make some hardware available for testing and
development, it would be a benefit to the project.
The columnar memory format itself uses little endian integer representation, so
byte swapping would be necessary in many places. I'm interested to hear how big
endian support could be introduced in Java and C++ without introducing
performance regressions on little endian systems (particular in Java).
> [Java] mvn test in memory fails on a big-endian platform
> --------------------------------------------------------
>
> Key: ARROW-3476
> URL: https://issues.apache.org/jira/browse/ARROW-3476
> Project: Apache Arrow
> Issue Type: Bug
> Components: Java
> Reporter: Kazuaki Ishizaki
> Priority: Major
>
> Apache Arrow is becoming commonplace to exchange data among important
> emerging analytics frameworks such as Pandas, Numpy, and Spark.
> [IBM Z|https://en.wikipedia.org/wiki/IBM_Z] is one of platforms to process
> critical transactions such as bank or credit card. Users of IBM Z want to
> extract insights from these transactions using the emerging analytics systems
> on IBM Z Linux. These analytics pipelines can be also fast and effective on
> IBM Z Linux by using Apache Arrow on memory.
> From the technical perspective, since IBM Z Linux uses big-endian data
> format, it is not possible to use Apache Arrow in this pipeline. If Apache
> Arrow could support big-endian, the use case would be expanded.
> When I ran test case of Apache arrow on a big-endian platform (ppc64be),
> {{mvn test}} in memory causes a failure due to an assertion.
> In {{TestEndianess.testLittleEndian}} test suite, the assertion occurs during
> an allocation of a {{RootAllocator}} class.
> {code}
> $ uname -a
> Linux ppc64be.novalocal 4.5.7-300.fc24.ppc64 #1 SMP Fri Jun 10 20:29:32 UTC
> 2016 ppc64 ppc64 ppc64 GNU/Linux
> $ arch
> ppc64
> $ cd java/memory
> $ mvn test
> [INFO] Scanning for projects...
> [INFO]
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Building Arrow Memory 0.12.0-SNAPSHOT
> [INFO]
> ------------------------------------------------------------------------
> [INFO]
> ...
> [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082
> s - in org.apache.arrow.memory.TestAccountant
> [INFO] Running org.apache.arrow.memory.TestLowCostIdentityHashMap
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012
> s - in org.apache.arrow.memory.TestLowCostIdentityHashMap
> [INFO] Running org.apache.arrow.memory.TestBaseAllocator
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.746
> s <<< FAILURE! - in org.apache.arrow.memory.TestEndianess
> [ERROR] testLittleEndian(org.apache.arrow.memory.TestEndianess) Time
> elapsed: 0.313 s <<< ERROR!
> java.lang.ExceptionInInitializerError
> at
> org.apache.arrow.memory.TestEndianess.testLittleEndian(TestEndianess.java:31)
> Caused by: java.lang.IllegalStateException: Arrow only runs on LittleEndian
> systems.
> at
> org.apache.arrow.memory.TestEndianess.testLittleEndian(TestEndianess.java:31)
> [ERROR] Tests run: 22, Failures: 0, Errors: 21, Skipped: 1, Time elapsed:
> 0.055 s <<< FAILURE! - in org.apache.arrow.memory.TestBaseAllocator
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
