[jira] [Updated] (ARROW-101) Fix java warnings emitted by java compiler

2016-08-01 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney updated ARROW-101: --- Assignee: Laurent Goujon > Fix java warnings emitted by java compiler > ---

[jira] [Resolved] (ARROW-101) Fix java warnings emitted by java compiler

2016-08-01 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-101. Resolution: Fixed Issue resolved by pull request 60 [https://github.com/apache/arrow/pull/60] > Fix

[jira] [Commented] (ARROW-246) [Java] UnionVector doesn't call allocateNew() when creating it's vectorType

2016-08-01 Thread Deneche A. Hakim (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402958#comment-15402958 ] Deneche A. Hakim commented on ARROW-246: created pull request [#110|https://github.

Re: Core dump while building with CPP

2016-08-01 Thread Wes McKinney
You can either: - Set the environment variable CXXFLAGS or - Pass CMAKE_CXX_FLAGS to the cmake command: cmake -DCMAKE_CXX_FLAGS=" ... " Setting CMAKE_CXX_FLAGS as an environment variable does not do anything AFAIK (double check the actual command lines with `make VERBOSE=1`) On Mon, Aug 1, 201

[jira] [Updated] (ARROW-246) [Java] UnionVector doesn't call allocateNew() when creating it's vectorType

2016-08-01 Thread Deneche A. Hakim (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim updated ARROW-246: --- Description: UnionVector uses a UINT1Vector to store the types of every value in the vector., b

RE: Core dump while building with CPP

2016-08-01 Thread Sanjay Rao
Didn't help much, I actually rebuilt whole arrow cpp with export CMAKE_CXX_FLAGS="-DFLATBUFFERS_LITTLEENDIAN=0" and then tried to build simple debug, still it's failing in same step. Thanks,Sanjay > From: wesmck...@gmail.com > Date: Mon, 1 Aug 2016 13:07:11 -0700 > Subject: Re: Core dump while b

Re: Core dump while building with CPP

2016-08-01 Thread Micah Kornfield
This looks like somehow flatbuffers (a third party library) isn't getting setup correctly for big-endian systems. According to the documentation [1] it should be able to work. It would take some investigations to figure why it isn't. [1] https://google.github.io/flatbuffers/md__internals.html O

Re: Core dump while building with CPP

2016-08-01 Thread Wes McKinney
See from flatbuffers.h // The wire format uses a little endian encoding (since that's efficient for // the common platforms). #if !defined(FLATBUFFERS_LITTLEENDIAN) #if defined(__GNUC__) || defined(__clang__) #ifdef __BIG_ENDIAN__ #define FLATBUFFERS_LITTLEENDIAN 0 #else #def

Core dump while building with CPP

2016-08-01 Thread Sanjay Rao
cmake -DARROW_BUILD_BENCHMARKS=ON .make[ 0%] Running flatc compiler on /u01/hadoop/src/arrow/arrow/cpp/../format/Message.fbsAssertion failed: *reinterpret_cast(&endiantest) == FLATBUFFERS_LITTLEENDIAN, file /u01/hadoop/src/arrow/arrow/cpp/thirdparty/flatbuffers-1.3.0/include/flatbuffers/fl

[jira] [Assigned] (ARROW-246) [Java] UnionVector doesn't call allocateNew() when creating it's vectorType

2016-08-01 Thread Deneche A. Hakim (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deneche A. Hakim reassigned ARROW-246: -- Assignee: Deneche A. Hakim > [Java] UnionVector doesn't call allocateNew() when creating

[jira] [Created] (ARROW-246) [Java] UnionVector doesn't call allocateNew() when creating it's vectorType

2016-08-01 Thread Deneche A. Hakim (JIRA)
Deneche A. Hakim created ARROW-246: -- Summary: [Java] UnionVector doesn't call allocateNew() when creating it's vectorType Key: ARROW-246 URL: https://issues.apache.org/jira/browse/ARROW-246 Project:

[jira] [Commented] (ARROW-243) Use generic HDFS component instead of libhdfs

2016-08-01 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402593#comment-15402593 ] Wes McKinney commented on ARROW-243: Yes -- part of the reason for using dlopen for lib

[jira] [Resolved] (ARROW-244) [C++] Some global APIs of IPC module should be visible to the outside

2016-08-01 Thread Wes McKinney (JIRA)
[ https://issues.apache.org/jira/browse/ARROW-244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wes McKinney resolved ARROW-244. Resolution: Fixed Issue resolved by pull request 109 [https://github.com/apache/arrow/pull/109] > [C

RE: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Sanjay Rao
Hi Wes, Hi Micah, I understood what you meant, so point 2. Arrow working with Big Endian machine to Big Endian shouldn't be an issue right ? Please confirm. Thanks,Sanjay > From: wesmck...@gmail.com > Date: Mon, 1 Aug 2016 11:07:07 -0700 > Subject: Re: Is there plan to support BigEndian Systems li

Re: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Wes McKinney
hey Micah, On Mon, Aug 1, 2016 at 11:02 AM, Micah Kornfield wrote: > Hi Wes, > The point I was trying to argue from an earlier thread is that the most > common cases for relocation are: > 1. Little endian machine to little endian machine (most likely same > machine) > 2. big endian machine to b

Re: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Micah Kornfield
Hi Wes, The point I was trying to argue from an earlier thread is that the most common cases for relocation are: 1. Little endian machine to little endian machine (most likely same machine) 2. big endian machine to big endian machine (most likely same machine) 3. big endian machine to little end

Re: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Wes McKinney
We do not have any tests that consider endianness. This would be a welcome contribution, particularly for the IPC / memory sharing side of things. On Mon, Aug 1, 2016 at 10:56 AM, Sanjay Rao wrote: > Hi Wes, > Thanks for your reply, so can I say that C++ version of Arrow works on Big > Endian sy

RE: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Sanjay Rao
Hi Wes, Thanks for your reply, so can I say that C++ version of Arrow works on Big Endian system(although not tested officially) ? Also would need your help in getting me details on how to run and one sample program using APIs. Thanks a lot,Sanjay > From: wesmck...@gmail.com > Date: Mon, 1 Aug 2

Re: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Wes McKinney
I replied on the other thread (missed this one). I believe that since the C++ codebase uses native endianness, we would need to add byte-swapping logic in the IPC code path to support big endian systems. On Mon, Aug 1, 2016 at 9:31 AM, Sanjay Rao wrote: > Hi Micah, > Thanks, I am trying to write

Re: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Wes McKinney
On Mon, Aug 1, 2016 at 10:26 AM, Wes McKinney wrote: > hi Sanjay, > > You may have seen: > > https://github.com/apache/arrow/blob/master/format/Layout.md#byte-order-endianness > > For big-endian systems, they are free to work with integers with the > native byte order, but for the purposes of any

[jira] [Created] (ARROW-245) [Format] Clarify Arrow's relationship with big endian platforms

2016-08-01 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-245: -- Summary: [Format] Clarify Arrow's relationship with big endian platforms Key: ARROW-245 URL: https://issues.apache.org/jira/browse/ARROW-245 Project: Apache Arrow

Re: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Wes McKinney
hi Sanjay, You may have seen: https://github.com/apache/arrow/blob/master/format/Layout.md#byte-order-endianness For big-endian systems, they are free to work with integers with the native byte order, but for the purposes of any memory sharing / IPC / RPC of Arrow memory layout, my understanding

Re: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Parth Chandra
Sorry, please disregard my reply. I misunderstood this to be a question on the Drill mailing list. On Mon, Aug 1, 2016 at 9:50 AM, Parth Chandra wrote: > Short answer is no. > Drill's in memory format assumes little endian and it would be very > disruptive to have to change that. > There is a J

Re: Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Parth Chandra
Short answer is no. Drill's in memory format assumes little endian and it would be very disruptive to have to change that. There is a JIRA to fix the Drill client to allow little endian and we should fix that, but it is currently low in priority. On Mon, Aug 1, 2016 at 9:27 AM, Sanjay Rao wrote:

Re: Can someone help me how should I start using Arrow Java Jars ?

2016-08-01 Thread Ted Dunning
Cloning the git repository gives you the full source code. On Mon, Aug 1, 2016 at 8:11 AM, Sanjay Rao wrote: > Hi Kiril, > Thanks a lot for your reply, Can I have the full source code ? It would > help me, also could you help me with Java doc link if any as such. > Thanks again,Sanjay > > > Fr

RE: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Sanjay Rao
Hi Micah, Thanks, I am trying to write Java examples with Java libraries of Apache Arrow on Big Endian system, I already got JARs built, this is part of a research I am doing. I ran "mvn test" on Big Endian system and it failed in TestEndianess, kindly let me know if at least Java version of Apa

Re: Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Micah Kornfield
Hi Sanjay, There was some discussion on the mailing list a little while ago about this [1]. We discussed making endianness part of the IPC metadata. I don't think we've gotten far enough in the implementation to take action on the discussion though. Right now I think the C++ code at least is end

Is there plan to support BigEndian Systems like SUN SPARC Hardware ?

2016-08-01 Thread Sanjay Rao
As Apache SPARK supports Big Endian systems. Thanks,Sanjay

Is LittleEndian system mandatory for using Apache Arrow ?

2016-08-01 Thread Sanjay Rao
Hi, I read that Integer representation is assumed to LittleEndian in Arrow, does this mean we cannot use Arrow in Big Endian Systems like SPARC ? Thanks,Sanjay

RE: Can someone help me how should I start using Arrow Java Jars ?

2016-08-01 Thread Sanjay Rao
Hi Kiril, Thanks a lot for your reply, Can I have the full source code ? It would help me, also could you help me with Java doc link if any as such. Thanks again,Sanjay > From: kmenshi...@gmail.com > Date: Mon, 1 Aug 2016 16:37:52 +0300 > Subject: Re: Can someone help me how should I start using

Re: Can someone help me how should I start using Arrow Java Jars ?

2016-08-01 Thread Kiril Menshikov
Hi Sanjay, You need to check out latest arrow: git clone https://github.com/apache/arrow Go to the java folder and run maven: mvn install After you will find two jars: memory/target/arrow-memory-0.1-SNAPSHOT.jar vector/target/vector-0.1-SNAPSHOT.jar Arrow example: BufferAllocator allocator =