[JAVA] Figuring out whats shifted from Drill/Java

Hi Everyone, I'm looking to help get started with Arrow & Spark and to that end I'd like to start with getting the Java implementation closer to the spec / C implementation. I'm wondering what places people know the differences are between the two? Cheers, Holden :)

Re: [JAVA] Figuring out whats shifted from Drill/Java

I am also interested in this. On Tue, Jun 7, 2016 at 17:37 Holden Karau wrote: > Hi Everyone, > > I'm looking to help get started with Arrow & Spark and to that end I'd like > to start with getting the Java implementation closer to the spec / C > implementation. I'm wondering what places people k

Re: [JAVA] Figuring out whats shifted from Drill/Java

How can I unsubscribe from this chain? Too much streaming! No options to unsubscribe. Arrow team please help! Thanks, Shravan V. -- Email: shra...@hotmail.com -- > On Jun 7, 2016, at 5:54 PM, Leif Walsh wrote: > > I am also interested in this. >> On Tue, Jun 7, 2016 at 17:37 Holden Karau wro

Re: [JAVA] Figuring out whats shifted from Drill/Java

Please send an e-mail to dev-unsubscr...@arrow.apache.org. Dear readers: if you would like to participate in higher level discussions but not see JIRA activity I strongly suggest you set up appropriate e-mail filters to hide or filter the JIRA activity. Thanks On Tue, Jun 7, 2016 at 8:14 PM, Shr

Re: [JAVA] Figuring out whats shifted from Drill/Java

If you want to unsubscribe from the list I believe emailing dev-unsubscr...@arrow.apache.org will do the trick. On Tuesday, June 7, 2016, Shravan V wrote: > How can I unsubscribe from this chain? Too much streaming! No options to > unsubscribe. Arrow team please help! > > Thanks, > > Shravan V.

Re: [JAVA] Figuring out whats shifted from Drill/Java

I am currently working on a patch that addresses this, as well as removing some of the residual code from Drill that isn't really needed in Arrow, (such as the Drill types, MaterializedField, etc.) I will be posting this within a few days. On Tue, Jun 7, 2016 at 5:54 PM, Leif Walsh wrote: > I a

Re: [JAVA] Figuring out whats shifted from Drill/Java

Hi Steven, Is the patch focused on the alignment/padding. Or are there other issues as well? Thanks, Micah On Tue, Jun 7, 2016 at 11:22 PM, Steven Phillips wrote: > I am currently working on a patch that addresses this, as well as removing > some of the residual code from Drill that isn't reall

Re: [JAVA] Figuring out whats shifted from Drill/Java

On Wed, Jun 8, 2016 at 9:26 PM, Micah Kornfield wrote: > Hi Steven, > Is the patch focused on the alignment/padding. Or are there other > issues as well? > I'm interested on this as well > > Thanks, > Micah > > On Tue, Jun 7, 2016 at 11:22 PM, Steven Phillips > wrote: > > I am currently w

Re: [JAVA] Figuring out whats shifted from Drill/Java

I am also interested on this . Do we need to know drill before start implementing not a for arrow . On Jun 10, 2016 9:45 PM, "Wail Alkowaileet" wrote: > On Wed, Jun 8, 2016 at 9:26 PM, Micah Kornfield > wrote: > > > Hi Steven, > > Is the patch focused on the alignment/padding. Or are there othe

Re: [JAVA] Figuring out whats shifted from Drill/Java

Steven is on vacation for a couple of days. His focus as I understand it is rationalizing the code so it is cleaner, correct for arrow versus drill representation differences (such as decimal, nulls, etc) and has more unit tests. Once he gets back in the next day or two, hopefully he can post a wip

Re: [JAVA] Figuring out whats shifted from Drill/Java

Also, one does not need to understand or work with Drill to leverage arrow. Especially once Steven cleans up some of the Drill leakage. On Jun 11, 2016 12:45 PM, "Jacques Nadeau" wrote: > Steven is on vacation for a couple of days. His focus as I understand it > is rationalizing the code so it is

Re: [JAVA] Figuring out whats shifted from Drill/Java

Hi, Does this mean that offset must be adjusted depending on the UDLE memory. So new memory address will be align to 64 bits? > > The first thing we should do for the alignment in Java is adjust the > allocator so that it always allocates on a 64 bit offset. Does someone want > to look at that?

Re: [JAVA] Figuring out whats shifted from Drill/Java

Yes, I think there are two main components. Also, I accidentally said 64 bits when I should have said 64 bytes. 1. New allocations should be on 64 byte offsets 2. Serializing existing vectors must be done such that they are always in an increment of 64 bytes. This is necessary to avoid copying whe

Re: [JAVA] Figuring out whats shifted from Drill/Java

Netty buffer always allocate memory aligned to 64-bytes. So each new ArrowBuf will be aligned to 64-bytes as well, with offset = 0. ​I don't fully understand why new allocations should be on 64-bytes offset?​ ​-Kiril​ On Jun 14, 2016, at 00:22, Jacques Nadeau wrote: Yes, I think there are tw

Re: [JAVA] Figuring out whats shifted from Drill/Java

> Netty buffer always allocate memory aligned to 64-bytes. So each new > ArrowBuf will be aligned to 64-bytes as well, with offset = 0. > You confirmed that both the Netty chunk as well as buffer allocations (ArrowBufs returned from here [1]) are on 64-byte offsets? Can you maybe write some tests/

Re: [JAVA] Figuring out whats shifted from Drill/Java

Yes, I’d rather write the test. > On Jun 16, 2016, at 18:59, Jacques Nadeau wrote: > >> Netty buffer always allocate memory aligned to 64-bytes. So each new >> ArrowBuf will be aligned to 64-bytes as well, with offset = 0. >> > > You confirmed that both the Netty chunk as well as buffer alloca

Re: [JAVA] Figuring out whats shifted from Drill/Java

Hi, I’m back from vacation and finalize offset test. I found that memory is not necessarily 64-bytes align. So now the offset is calculated before ArrowBuf initialization. https://github.com/apache/arrow/pull/98 -Kiril > On Jun 16, 2016, at 21:32, Ki