Re: arrow read/write examples in Java

2018-01-11 Thread Wes McKinney
I'm working on getting all the docs updated for 0.8.0 -- there are some issues blocking a more automated update, so I may update them piecemeal until this is resolved: https://github.com/apache/arrow/pull/1472 On Tue, Jan 2, 2018 at 10:41 AM, Wes McKinney wrote: > I'll take a look at updating th

Re: arrow read/write examples in Java

2018-01-02 Thread Wes McKinney
I'll take a look at updating the site docs today. Thanks for pointing this out! On Wed, Dec 27, 2017 at 4:57 AM, Animesh Trivedi wrote: > Hello everyone, > > I solved the issue with my writer. Now everything is working fine, > including HDFS file reads and writes. I also wrote a parquet to arrow

Re: arrow read/write examples in Java

2017-12-27 Thread Animesh Trivedi
Hello everyone, I solved the issue with my writer. Now everything is working fine, including HDFS file reads and writes. I also wrote a parquet to arrow converter (on HDFS) that works fine. I noticed that Arrow javadocs are still at the 0.7 release. Can someone please update them? FWIW: I wrote

Re: arrow read/write examples in Java

2017-12-20 Thread Animesh Trivedi
I think the null pointer exception happens due to some issue in my new writer (which used my implementation of the ByteBuffer writable interface)...let me narrow it down first. The basic code, that does not use my writer's implementation, seems to work. This is the code which is at github. I did n

Re: arrow read/write examples in Java

2017-12-20 Thread Animesh Trivedi
Wes, Emilio, Siddharth - many thanks for helpful replies and comments ! I managed to upgrade the code to 0.8 API. I have to say that 0.8 API is much more intuitive ;) I will summarize my code example with some documentation in a blog post soon (and post it here too). - Is there 1st class support

Re: arrow read/write examples in Java

2017-12-19 Thread Siddharth Teotia
>From Arrow 0.8, the second step "Grab the corresponding mutator and accessor objects by calls to getMutator(), getAccessor()" is not needed. In fact, it is not even there. On Tue, Dec 19, 2017 at 10:01 AM, Siddharth Teotia wrote: > Hi Animesh, > > Firstly I would like to suggest switching over

Re: arrow read/write examples in Java

2017-12-19 Thread Siddharth Teotia
Hi Animesh, Firstly I would like to suggest switching over to Arrow 0.8 release asap since you are writing JAVA programs and the API usage has changed drastically. The new APIs are much simpler with good javadocs and detailed internal comments. If you are writing stop-gap implementation then it i

Re: arrow read/write examples in Java

2017-12-19 Thread Emilio Lahr-Vivaz
This has probably changed with the Java code refactor, but I've posted some answers inline, to the best of my understanding. Thanks, Emilio On 12/16/2017 12:17 PM, Animesh Trivedi wrote: Thanks Wes for you help. Based upon some code reading, I managed to code-up a basic working example. The

Re: arrow read/write examples in Java

2017-12-19 Thread Wes McKinney
hi, These are great questions. Note that the Java API has changed in regards to some of your questions in 0.8.0. Could someone more experienced in the Java library create some JIRAs about creating user documentation to help get started using the Java API (if even some simple example programs to p

Re: arrow read/write examples in Java

2017-12-16 Thread Animesh Trivedi
Thanks Wes for you help. Based upon some code reading, I managed to code-up a basic working example. The code is here: https://github.com/animeshtrivedi/ArrowExample/tree/master/src/main/java/com/github/animeshtrivedi/arrowexample . However, I do have some questions about the concepts in Arrow 1

Re: arrow read/write examples in Java

2017-12-14 Thread Wes McKinney
hi Animesh, I suggest you try the ArrowStreamReader/Writer or ArrowFileReader/Writer classes. See https://github.com/apache/arrow/blob/master/java/tools/src/main/java/org/apache/arrow/tools/Integration.java for example working code for this - Wes On Thu, Dec 14, 2017 at 8:30 AM, Animesh Trivedi

arrow read/write examples in Java

2017-12-14 Thread Animesh Trivedi
Hi all, It might be a trivial question, so please let me know if I am missing something. I am trying to write and read files in the Arrow format in Java. My data is simple flat schema with primitive types. I already have the data in Java. So my questions are: 1. Is this possible or am I fundament