Re: ARROW-3191: Making ArrowBuf work with arbitrary memory and setting io.netty.tryReflectionSetAccessible to true for java builds

Hi Bryan, AFAIK, there is not other impact. So we should be good. The last few integration issues that I had been chasing are now fixed (got a clean build with my previous commit pushed over the weekend). I just pushed a new commit with some cleanup and the changes are now ready. We should plan t

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory and setting io.netty.tryReflectionSetAccessible to true for java builds

Hi Sidd, Does setting the system property io.netty.tryReflectionSetAccessible to true have any other adverse effect other than those warnings during build? Bryan On Thu, May 2, 2019 at 8:43 PM Jacques Nadeau wrote: > I'm onboard with this change. > > On Fri, Apr 26, 2019 at 2:14 AM Siddharth T

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory and setting io.netty.tryReflectionSetAccessible to true for java builds

I'm onboard with this change. On Fri, Apr 26, 2019 at 2:14 AM Siddharth Teotia wrote: > As part of working on this patch < > https://github.com/apache/arrow/pull/4151>, > I ran into a problem with jdk 9 and 11 builds. Since memory underlying > ArrowBuf may not necessarily be a ByteBuf (or any o

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory and setting io.netty.tryReflectionSetAccessible to true for java builds

As part of working on this patch , I ran into a problem with jdk 9 and 11 builds. Since memory underlying ArrowBuf may not necessarily be a ByteBuf (or any of its extensions), methods like nioBuffer() can no longer be delegated as UnsafeDirectLittleEndian

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory

I have made all the necessary changes in java code to work with new ArrowBuf, ReferenceManager interfaces. More importantly, there is a wrapper buffer NettyArrowBuf interface to comply with usage in RPC and Netty related code. It will be good to get feedback on this one (and of course all other ch

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory

Are there any other general comments here? If not, let's get this done and merged. On Mon, Apr 15, 2019, 4:19 PM Siddharth Teotia wrote: > I believe reader/writer indexes are typically used when we send buffers > over the wire -- so may not be necessary for all users of ArrowBuf. I am > okay wi

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory

I believe reader/writer indexes are typically used when we send buffers over the wire -- so may not be necessary for all users of ArrowBuf. I am okay with the idea of providing a simple wrapper to ArrowBuf to manage the reader/writer indexes with a couple of APIs. Note that some APIs like writeInt

Re: ARROW-3191: Making ArrowBuf work with arbitrary memory

Hey Sidd, Thanks for pulling this together. This looks very promising. One quick thought: do we think the concept of the reader and writer index need to be on ArrowBuf? It seems like something that could be added as an additional decoration/wrapper when needed instead of being part of the core str