Hey Micah, thanks for starting the discussion. I just skimmed that thread and it isn't entirely clear that there was a conclusion that the overhead was worth it. I think everybody agrees that it would be nice to have the code work on both platforms. On the flipside, the code noise for a rare case makes the cost-benefit questionable.
In the Java code, we wrote the code to explicitly disallow big endian platforms and put preconditions checks in. I definitely think if we want to support this, it should be done holistically across the code with appropriate test plan (both functional and perf). To me, the question is really about how many use cases are blocked by this. I'm not sure I've heard anyone say that the limiting factor to leveraging Java Arrow was the block on endianess. Keep in mind that until very recently, using any Arrow Java code would throw a preconditions check before you could even get started on big-endian and I don't think we've seen a bunch of messages on that exception. Adding if conditions throughout the codebase like this patch: [1] isn't exactly awesome and it can also risk performance impacts depending on how carefully it is done. If there isn't a preponderance of evidence of many users being blocked by this capability, I don't think we should accept the code. We already have a backlog of items that we need to address just ensure existing use cases work well. Expanding to new use cases that there is no clear demand for will likely just increase code development cost at little benefit. What do others think? [1] https://github.com/apache/arrow/pull/7923#issuecomment-674311119 On Fri, Aug 14, 2020 at 4:36 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > Kazuaki Ishizak has started working on Big Endian support in Java > (including setting up CI for it). Thank you! > > We previously discussed support for Big Endian architectures in C++ [1] and > generally agreed that it was a reasonable thing to do. > > Similar to C++ I think as long as we have a working CI setup it is > reasonable for Java to support Big Endian machines. > > But I think there might be differing opinions so it is worth a discussion > to see if there are technical blockers or other reasons for not supporting > Big Endian architectures in the existing java implementation. > > Thanks, > Micah > > > [1] > > https://lists.apache.org/thread.html/rcae745f1d848981bb5e8dddacfc4554641aba62e3c949b96bfd8b019%40%3Cdev.arrow.apache.org%3E >
