Hi Jacques and Micah, Thanks for the fruitful discussion.
It seems netty based allocator and unsafe based allocator have their specific advantages. Maybe we can implement both as independent allocators, to support different scenarios. This should not be difficult, as [1] has laid a solid ground for this. Best, Liya Fan [1] https://issues.apache.org/jira/browse/ARROW-7329 On Mon, Jan 20, 2020 at 11:38 AM Micah Kornfield <emkornfi...@gmail.com> wrote: > Hmm, somehow I missed those two alternatives, thanks for pointing them out. > > I agree that these are probably better than taking a new dependency. Of > the two of them, it seems like using Unsafe directly might be better since > it would also solve a the issue of setting special environment variables > for Netty [1], but it might be two big of a change to couple the two > together. > > The other point brought on the JIRA about honoring -XX:MaxDirectMemorySize > is a good one. The one downside to this is it potentially comes with a > performance penalty [2] (this is quite dated though). But I think we can > always explore other options after doing the simplest thing first. > > -Micah > > [1] https://issues.apache.org/jira/browse/ARROW-7223 > [2] > > http://mail.openjdk.java.net/pipermail/hotspot-dev/2015-February/017089.html > > On Sun, Jan 19, 2020 at 3:03 PM Jacques Nadeau <jacq...@apache.org> wrote: > > > It seems like jna is overkill & unnecessary for simply allocating/freeing > > memory. > > > > A simple way to do this is either to use unsafe directly or call the > > existing netty unsafe facade directly. > > > > PlatformDependent.allocateMemory(long) > > PlatformDependent.freeMemory(long) > > > > Should be relatively straightforward to add to the existing Netty-based > > allocator. > > > > On Sat, Jan 18, 2020 at 8:14 PM Fan Liya <liya.fa...@gmail.com> wrote: > > > >> Hi Micah, > >> > >> Thanks for the good suggestion. JNA seems like a good and reasonable > tool > >> for allocating large memory chunks. > >> > >> How about we directly use Java UNSAFE API? It seems the allocateMemory > API > >> is also based on the malloc method of the native implementation [1]. > >> > >> Best, > >> Liya Fan > >> > >> [1] > >> > >> > http://hg.openjdk.java.net/jdk7/jdk7/hotspot/file/4fc084dac61e/src/share/vm/prims/unsafe.cpp > >> > >> On Sat, Jan 18, 2020 at 12:58 PM Micah Kornfield <emkornfi...@gmail.com > > > >> wrote: > >> > >> > With the recently merged changes to the underlying ArrowBuf APIs to > >> allow > >> > 64-bit memory address spaces there is some follow-up work to actually > >> > confirm it works. I opened a JIRA [1] to track this work. > >> > > >> > The main question is how to provide an allocator that supports larger > >> > memory chunks. It appears the Netty API only takes an 32-bit integer > >> for > >> > array sizes. Doing a little bit of investigation it seems like JNA > [2] > >> > exposes a direct call to malloc of 64-bit integers [3]. > >> > > >> > The other options would seem to be rolling our own allocator via JNI. > >> > > >> > Is there anybody worked with JNA and can share experiences? > >> > Is anyone familiar with other options? > >> > > >> > Thanks, > >> > Micah > >> > > >> > [1] https://issues.apache.org/jira/browse/ARROW-7606 > >> > [2] https://github.com/java-native-access/jna > >> > [3] > >> > > >> > > >> > https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/Native.java#L2265 > >> > > >> > > >
