I think I suggested this. The idea is to isolate the auto-generated code from the other source files, and also to isolate future changes due to updates on either format or flatbuffers versions. IMO this makes the repo cleaner and more organized. Of course, ideally we should auto-generate these files when compiling.
On Mon, Nov 19, 2018 at 11:55 AM Wes McKinney <wesmck...@gmail.com> wrote: > Could you explain why having a separate crate would be a good idea? > On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <andygrov...@gmail.com> wrote: > > > > A question has been raised on this PR as to whether we should publish a > > separate crate for the format/ipc generated code. I think there might be > > some merit in this and wanted to raise it here to see what everyone > thinks. > > > > This would mean having two directories under the rust directory ... one > for > > arrow-format (or maybe the name should be arrow-ipc?) and one for the > main > > arrow crate. > > > > It is possible we might want other crates in the future e.g. > arrow-parquet > > or whatever. > > > > Andy. > > > > > > On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <andygrov...@gmail.com> > wrote: > > > > > I have created a PR this morning that adds the generated Flatbuffers > code > > > to the Rust implementation (https://github.com/apache/arrow/pull/2986 > ). > > > > > > Just throwing these files into the repo doesn't add a ton of value > until > > > we use them for something, but it took some effort to generate them > (and > > > then fix a couple things by hand) so it seems useful to contribute > this. > > > The PR includes a brief README with the instructions for generating the > > > files. > > > > > > I have taken this week off for Thanksgiving so I actually have some > > > dedicated time to contribute to Arrow. I suppose my next step on this > is to > > > write a test that can serialize some data and then de-serialize it > again. > > > > > > Andy. > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > >> hi Andy, > > >> > > >> AFAIK in the other implementations, the Flatbuffers serialization is > > >> an implementation detail of offering support for the columnar IPC > > >> protocol (which in turn depends on the columnar data structures, > > >> memory management, etc.). I'm not sure what a crate offering the > > >> Flatbuffers bindings alone would offer, outside of helping people > > >> create new Rust Arrow implementations, I guess. In any case, if you > > >> wanted to do that, we would need to make a release first. > > >> > > >> - Wes > > >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <andygrov...@gmail.com> > wrote: > > >> > > > >> > Brief update on this. I have now been able to use flatbuffers to > > >> generate > > >> > Rust code from the Arrow schema, and it compiles. In theory, this > can be > > >> > published as a standalone "arrow-format" crate but I'm wondering > what > > >> the > > >> > plan should be here. > > >> > > > >> > I will start reviewing how this is handled in the other > implementations > > >> but > > >> > if anyone has suggestions, please let me know. > > >> > > > >> > Thanks, > > >> > > > >> > Andy. > > >> > > > >> > > > >> > > > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <andygrov...@gmail.com> > > >> wrote: > > >> > > > >> > > Flatbuffers now has support for Rust - the PR was just merged in > the > > >> last > > >> > > couple days: > > >> > > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers > > >> > > > > >> > > I hope to find some time in the next week or two to start > > >> experimenting > > >> > > with this for Arrow IPC. > > >> > > > > >> > > Andy. > > >> > > > > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <andygrov...@gmail.com > > > > >> wrote: > > >> > > > > >> > >> The author of the Rust version of FlatBuffers is now working on a > > >> public > > >> > >> fork and seems to be making good progress. I just wanted to send > a > > >> quick > > >> > >> update on this since I am now back onto working on Arrow IPC in > Rust > > >> (I'm > > >> > >> sure this is going to take quite a while though so don't want to > get > > >> > >> anyones hopes up too much). For anyone that is interested, here > is > > >> the > > >> > >> FlatBuffers branch: > > >> https://github.com/rw/flatbuffers/tree/2018-02--rust > > >> > >> > > >> > >> Andy. > > >> > >> > > >> > >> > > >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney < > wesmck...@gmail.com> > > >> wrote: > > >> > >> > > >> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers > and > > >> > >>> Rust. One possibility is to build an interface to the > Flatbuffers > > >> data > > >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if > > >> these > > >> > >>> bindings are header-only like C++ and if that makes things any > > >> easier > > >> > >>> for you. > > >> > >>> > > >> > >>> If you defined an API in Rust for reading and writing the Arrow > > >> > >>> metadata that does not expose any Flatbuffers-specific details, > then > > >> > >>> once there is a production-grade Rust implementation of > Flatbuffers, > > >> > >>> the implementation details could be swapped out without > disruption. > > >> > >>> > > >> > >>> > my recent attempts at contacting the author have been > unsuccessful > > >> > >>> > > >> > >>> Have you e-mailed the author directly or only pings on GitHub? > Pings > > >> > >>> may not be making it to their inbox. > > >> > >>> > > >> > >>> Appreciate your efforts on this; I think we'll see a lot more > work > > >> in > > >> > >>> data processing in Rust in the coming years. > > >> > >>> > > >> > >>> - Wes > > >> > >>> > > >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove < > andygrov...@gmail.com> > > >> > >>> wrote: > > >> > >>> > Hi, > > >> > >>> > > > >> > >>> > Now that the refactor I've been working on has been merged, > the > > >> next > > >> > >>> > priority for me personally with the Rust implementation is > > >> getting IPC > > >> > >>> and > > >> > >>> > integration testing working. > > >> > >>> > > > >> > >>> > Unfortunately the official Flatbuffers Rust version is not > > >> available > > >> > >>> yet > > >> > >>> > and my recent attempts at contacting the author have been > > >> unsuccessful > > >> > >>> so I > > >> > >>> > have started working with this fork of Flatbuffers which has > Rust > > >> > >>> support: > > >> > >>> > https://github.com/josephDunne/flatbuffers. > > >> > >>> > > > >> > >>> > I was able to generate code from Schema.fbs but it doesn't > > >> compile and > > >> > >>> I've > > >> > >>> > started filing issues and debugging this. > > >> > >>> > > > >> > >>> > Working with this fork isn't ideal but I don't see what other > > >> choice I > > >> > >>> > have, other than just waiting for the official project to > support > > >> Rust. > > >> > >>> > > > >> > >>> > I'm interested to hear if anyone has any alternate > suggestions. I > > >> know > > >> > >>> it > > >> > >>> > would be possible to wrap C code but I'd like to keep the Rust > > >> > >>> > implementation as a pure Rust project if possible. > > >> > >>> > > > >> > >>> > Thanks, > > >> > >>> > > > >> > >>> > Andy. > > >> > >>> > > >> > >> > > >> > > > >
