A question has been raised on this PR as to whether we should publish a separate crate for the format/ipc generated code. I think there might be some merit in this and wanted to raise it here to see what everyone thinks.
This would mean having two directories under the rust directory ... one for arrow-format (or maybe the name should be arrow-ipc?) and one for the main arrow crate. It is possible we might want other crates in the future e.g. arrow-parquet or whatever. Andy. On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <andygrov...@gmail.com> wrote: > I have created a PR this morning that adds the generated Flatbuffers code > to the Rust implementation (https://github.com/apache/arrow/pull/2986). > > Just throwing these files into the repo doesn't add a ton of value until > we use them for something, but it took some effort to generate them (and > then fix a couple things by hand) so it seems useful to contribute this. > The PR includes a brief README with the instructions for generating the > files. > > I have taken this week off for Thanksgiving so I actually have some > dedicated time to contribute to Arrow. I suppose my next step on this is to > write a test that can serialize some data and then de-serialize it again. > > Andy. > > > > > > > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <wesmck...@gmail.com> wrote: > >> hi Andy, >> >> AFAIK in the other implementations, the Flatbuffers serialization is >> an implementation detail of offering support for the columnar IPC >> protocol (which in turn depends on the columnar data structures, >> memory management, etc.). I'm not sure what a crate offering the >> Flatbuffers bindings alone would offer, outside of helping people >> create new Rust Arrow implementations, I guess. In any case, if you >> wanted to do that, we would need to make a release first. >> >> - Wes >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <andygrov...@gmail.com> wrote: >> > >> > Brief update on this. I have now been able to use flatbuffers to >> generate >> > Rust code from the Arrow schema, and it compiles. In theory, this can be >> > published as a standalone "arrow-format" crate but I'm wondering what >> the >> > plan should be here. >> > >> > I will start reviewing how this is handled in the other implementations >> but >> > if anyone has suggestions, please let me know. >> > >> > Thanks, >> > >> > Andy. >> > >> > >> > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <andygrov...@gmail.com> >> wrote: >> > >> > > Flatbuffers now has support for Rust - the PR was just merged in the >> last >> > > couple days: >> > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers >> > > >> > > I hope to find some time in the next week or two to start >> experimenting >> > > with this for Arrow IPC. >> > > >> > > Andy. >> > > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <andygrov...@gmail.com> >> wrote: >> > > >> > >> The author of the Rust version of FlatBuffers is now working on a >> public >> > >> fork and seems to be making good progress. I just wanted to send a >> quick >> > >> update on this since I am now back onto working on Arrow IPC in Rust >> (I'm >> > >> sure this is going to take quite a while though so don't want to get >> > >> anyones hopes up too much). For anyone that is interested, here is >> the >> > >> FlatBuffers branch: >> https://github.com/rw/flatbuffers/tree/2018-02--rust >> > >> >> > >> Andy. >> > >> >> > >> >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <wesmck...@gmail.com> >> wrote: >> > >> >> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and >> > >>> Rust. One possibility is to build an interface to the Flatbuffers >> data >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if >> these >> > >>> bindings are header-only like C++ and if that makes things any >> easier >> > >>> for you. >> > >>> >> > >>> If you defined an API in Rust for reading and writing the Arrow >> > >>> metadata that does not expose any Flatbuffers-specific details, then >> > >>> once there is a production-grade Rust implementation of Flatbuffers, >> > >>> the implementation details could be swapped out without disruption. >> > >>> >> > >>> > my recent attempts at contacting the author have been unsuccessful >> > >>> >> > >>> Have you e-mailed the author directly or only pings on GitHub? Pings >> > >>> may not be making it to their inbox. >> > >>> >> > >>> Appreciate your efforts on this; I think we'll see a lot more work >> in >> > >>> data processing in Rust in the coming years. >> > >>> >> > >>> - Wes >> > >>> >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <andygrov...@gmail.com> >> > >>> wrote: >> > >>> > Hi, >> > >>> > >> > >>> > Now that the refactor I've been working on has been merged, the >> next >> > >>> > priority for me personally with the Rust implementation is >> getting IPC >> > >>> and >> > >>> > integration testing working. >> > >>> > >> > >>> > Unfortunately the official Flatbuffers Rust version is not >> available >> > >>> yet >> > >>> > and my recent attempts at contacting the author have been >> unsuccessful >> > >>> so I >> > >>> > have started working with this fork of Flatbuffers which has Rust >> > >>> support: >> > >>> > https://github.com/josephDunne/flatbuffers. >> > >>> > >> > >>> > I was able to generate code from Schema.fbs but it doesn't >> compile and >> > >>> I've >> > >>> > started filing issues and debugging this. >> > >>> > >> > >>> > Working with this fork isn't ideal but I don't see what other >> choice I >> > >>> > have, other than just waiting for the official project to support >> Rust. >> > >>> > >> > >>> > I'm interested to hear if anyone has any alternate suggestions. I >> know >> > >>> it >> > >>> > would be possible to wrap C code but I'd like to keep the Rust >> > >>> > implementation as a pure Rust project if possible. >> > >>> > >> > >>> > Thanks, >> > >>> > >> > >>> > Andy. >> > >>> >> > >> >> >
