+ dev@ There's several examples of sending record batches from Java to Python (and vice versa) over a network socket -- e.g. Jacques and I are working on a prototype of a general purpose Arrow-native RPC framework in Java and C++ respectively. Where there's some R&D needed is in Java interactions with shared memory. So if you want to do a zero copy read from a memory mapped file, then some development in the Arrow Java libraries is required.
I'm not an expert but it seems like Netty has a mechanism to interact with ByteBuffer, which should include MappedByteBuffer https://github.com/netty/netty/blob/4.1/buffer/src/main/java/io/netty/buffer/ReadOnlyUnsafeDirectByteBuf.java#L25 Correspondingly, an interface could be developed to enable the Java IPC code path to write to a shared memory region. To the Java developers, could we create some JIRA issues (if there are not already) around Java shared memory IPC? - Wes On Sat, Sep 1, 2018 at 2:32 PM ALBERTO Bocchinfuso <alberto_boc...@hotmail.it> wrote: > > I want to reinforce this request. I am interested in the same topic. > > I’d like an example specially focused on the creation of a RecordBatch passed > from a Java program to a python one and vice-versa. > > > > Thanks, > > Alberto > > > > ________________________________ > Da: Clive Cox <c...@seldon.io> > Inviato: Saturday, September 1, 2018 6:12:29 PM > A: u...@arrow.apache.org > Oggetto: IPC Example > > Hi, > > Is there any example of how to do say Java - Python IPC? I'm not sure how to > get started. > > I'm thinking of using Arrow IPC to replace REST/gRPC APIs for communication > when everything can be run on a single computer node and low-latency is the > goal - hoping to remove the cost of serialization/deserialization and network > costs. Would this make sense. > > Thanks, > > Clive >
