I have created a PR this morning that adds the generated Flatbuffers code to the Rust implementation (https://github.com/apache/arrow/pull/2986).
Just throwing these files into the repo doesn't add a ton of value until we use them for something, but it took some effort to generate them (and then fix a couple things by hand) so it seems useful to contribute this. The PR includes a brief README with the instructions for generating the files. I have taken this week off for Thanksgiving so I actually have some dedicated time to contribute to Arrow. I suppose my next step on this is to write a test that can serialize some data and then de-serialize it again. Andy. On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <wesmck...@gmail.com> wrote: > hi Andy, > > AFAIK in the other implementations, the Flatbuffers serialization is > an implementation detail of offering support for the columnar IPC > protocol (which in turn depends on the columnar data structures, > memory management, etc.). I'm not sure what a crate offering the > Flatbuffers bindings alone would offer, outside of helping people > create new Rust Arrow implementations, I guess. In any case, if you > wanted to do that, we would need to make a release first. > > - Wes > On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <andygrov...@gmail.com> wrote: > > > > Brief update on this. I have now been able to use flatbuffers to generate > > Rust code from the Arrow schema, and it compiles. In theory, this can be > > published as a standalone "arrow-format" crate but I'm wondering what the > > plan should be here. > > > > I will start reviewing how this is handled in the other implementations > but > > if anyone has suggestions, please let me know. > > > > Thanks, > > > > Andy. > > > > > > > > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <andygrov...@gmail.com> wrote: > > > > > Flatbuffers now has support for Rust - the PR was just merged in the > last > > > couple days: > > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers > > > > > > I hope to find some time in the next week or two to start experimenting > > > with this for Arrow IPC. > > > > > > Andy. > > > > > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <andygrov...@gmail.com> > wrote: > > > > > >> The author of the Rust version of FlatBuffers is now working on a > public > > >> fork and seems to be making good progress. I just wanted to send a > quick > > >> update on this since I am now back onto working on Arrow IPC in Rust > (I'm > > >> sure this is going to take quite a while though so don't want to get > > >> anyones hopes up too much). For anyone that is interested, here is the > > >> FlatBuffers branch: > https://github.com/rw/flatbuffers/tree/2018-02--rust > > >> > > >> Andy. > > >> > > >> > > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <wesmck...@gmail.com> > wrote: > > >> > > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and > > >>> Rust. One possibility is to build an interface to the Flatbuffers > data > > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if > these > > >>> bindings are header-only like C++ and if that makes things any easier > > >>> for you. > > >>> > > >>> If you defined an API in Rust for reading and writing the Arrow > > >>> metadata that does not expose any Flatbuffers-specific details, then > > >>> once there is a production-grade Rust implementation of Flatbuffers, > > >>> the implementation details could be swapped out without disruption. > > >>> > > >>> > my recent attempts at contacting the author have been unsuccessful > > >>> > > >>> Have you e-mailed the author directly or only pings on GitHub? Pings > > >>> may not be making it to their inbox. > > >>> > > >>> Appreciate your efforts on this; I think we'll see a lot more work in > > >>> data processing in Rust in the coming years. > > >>> > > >>> - Wes > > >>> > > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <andygrov...@gmail.com> > > >>> wrote: > > >>> > Hi, > > >>> > > > >>> > Now that the refactor I've been working on has been merged, the > next > > >>> > priority for me personally with the Rust implementation is getting > IPC > > >>> and > > >>> > integration testing working. > > >>> > > > >>> > Unfortunately the official Flatbuffers Rust version is not > available > > >>> yet > > >>> > and my recent attempts at contacting the author have been > unsuccessful > > >>> so I > > >>> > have started working with this fork of Flatbuffers which has Rust > > >>> support: > > >>> > https://github.com/josephDunne/flatbuffers. > > >>> > > > >>> > I was able to generate code from Schema.fbs but it doesn't compile > and > > >>> I've > > >>> > started filing issues and debugging this. > > >>> > > > >>> > Working with this fork isn't ideal but I don't see what other > choice I > > >>> > have, other than just waiting for the official project to support > Rust. > > >>> > > > >>> > I'm interested to hear if anyone has any alternate suggestions. I > know > > >>> it > > >>> > would be possible to wrap C code but I'd like to keep the Rust > > >>> > implementation as a pure Rust project if possible. > > >>> > > > >>> > Thanks, > > >>> > > > >>> > Andy. > > >>> > > >> >
