Auto generation sounds like a good idea, similarly like rust-bindgen does it: https://rust-lang-nursery.github.io/rust-bindgen/tutorial-3.html Of course it depends on the complexity of manual edits.
We can still separate the crates later if that's desired. On Tue, Nov 20, 2018 at 6:39 PM Chao Sun <sunc...@uber.com.invalid> wrote: > I think I suggested this. The idea is to isolate the auto-generated code > from the other source files, and also to isolate future changes due to > updates on either format or flatbuffers versions. IMO this makes the repo > cleaner and more organized. Of course, ideally we should auto-generate > these files when compiling. > > On Mon, Nov 19, 2018 at 11:55 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > Could you explain why having a separate crate would be a good idea? > > On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <andygrov...@gmail.com> > wrote: > > > > > > A question has been raised on this PR as to whether we should publish a > > > separate crate for the format/ipc generated code. I think there might > be > > > some merit in this and wanted to raise it here to see what everyone > > thinks. > > > > > > This would mean having two directories under the rust directory ... one > > for > > > arrow-format (or maybe the name should be arrow-ipc?) and one for the > > main > > > arrow crate. > > > > > > It is possible we might want other crates in the future e.g. > > arrow-parquet > > > or whatever. > > > > > > Andy. > > > > > > > > > On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <andygrov...@gmail.com> > > wrote: > > > > > > > I have created a PR this morning that adds the generated Flatbuffers > > code > > > > to the Rust implementation ( > https://github.com/apache/arrow/pull/2986 > > ). > > > > > > > > Just throwing these files into the repo doesn't add a ton of value > > until > > > > we use them for something, but it took some effort to generate them > > (and > > > > then fix a couple things by hand) so it seems useful to contribute > > this. > > > > The PR includes a brief README with the instructions for generating > the > > > > files. > > > > > > > > I have taken this week off for Thanksgiving so I actually have some > > > > dedicated time to contribute to Arrow. I suppose my next step on this > > is to > > > > write a test that can serialize some data and then de-serialize it > > again. > > > > > > > > Andy. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > >> hi Andy, > > > >> > > > >> AFAIK in the other implementations, the Flatbuffers serialization is > > > >> an implementation detail of offering support for the columnar IPC > > > >> protocol (which in turn depends on the columnar data structures, > > > >> memory management, etc.). I'm not sure what a crate offering the > > > >> Flatbuffers bindings alone would offer, outside of helping people > > > >> create new Rust Arrow implementations, I guess. In any case, if you > > > >> wanted to do that, we would need to make a release first. > > > >> > > > >> - Wes > > > >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <andygrov...@gmail.com> > > wrote: > > > >> > > > > >> > Brief update on this. I have now been able to use flatbuffers to > > > >> generate > > > >> > Rust code from the Arrow schema, and it compiles. In theory, this > > can be > > > >> > published as a standalone "arrow-format" crate but I'm wondering > > what > > > >> the > > > >> > plan should be here. > > > >> > > > > >> > I will start reviewing how this is handled in the other > > implementations > > > >> but > > > >> > if anyone has suggestions, please let me know. > > > >> > > > > >> > Thanks, > > > >> > > > > >> > Andy. > > > >> > > > > >> > > > > >> > > > > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <andygrov...@gmail.com> > > > >> wrote: > > > >> > > > > >> > > Flatbuffers now has support for Rust - the PR was just merged in > > the > > > >> last > > > >> > > couple days: > > > >> > > > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers > > > >> > > > > > >> > > I hope to find some time in the next week or two to start > > > >> experimenting > > > >> > > with this for Arrow IPC. > > > >> > > > > > >> > > Andy. > > > >> > > > > > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove < > andygrov...@gmail.com > > > > > > >> wrote: > > > >> > > > > > >> > >> The author of the Rust version of FlatBuffers is now working > on a > > > >> public > > > >> > >> fork and seems to be making good progress. I just wanted to > send > > a > > > >> quick > > > >> > >> update on this since I am now back onto working on Arrow IPC in > > Rust > > > >> (I'm > > > >> > >> sure this is going to take quite a while though so don't want > to > > get > > > >> > >> anyones hopes up too much). For anyone that is interested, here > > is > > > >> the > > > >> > >> FlatBuffers branch: > > > >> https://github.com/rw/flatbuffers/tree/2018-02--rust > > > >> > >> > > > >> > >> Andy. > > > >> > >> > > > >> > >> > > > >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney < > > wesmck...@gmail.com> > > > >> wrote: > > > >> > >> > > > >> > >>> Sorry to hear, it's a bit of a rough situation with > Flatbuffers > > and > > > >> > >>> Rust. One possibility is to build an interface to the > > Flatbuffers > > > >> data > > > >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder > if > > > >> these > > > >> > >>> bindings are header-only like C++ and if that makes things any > > > >> easier > > > >> > >>> for you. > > > >> > >>> > > > >> > >>> If you defined an API in Rust for reading and writing the > Arrow > > > >> > >>> metadata that does not expose any Flatbuffers-specific > details, > > then > > > >> > >>> once there is a production-grade Rust implementation of > > Flatbuffers, > > > >> > >>> the implementation details could be swapped out without > > disruption. > > > >> > >>> > > > >> > >>> > my recent attempts at contacting the author have been > > unsuccessful > > > >> > >>> > > > >> > >>> Have you e-mailed the author directly or only pings on GitHub? > > Pings > > > >> > >>> may not be making it to their inbox. > > > >> > >>> > > > >> > >>> Appreciate your efforts on this; I think we'll see a lot more > > work > > > >> in > > > >> > >>> data processing in Rust in the coming years. > > > >> > >>> > > > >> > >>> - Wes > > > >> > >>> > > > >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove < > > andygrov...@gmail.com> > > > >> > >>> wrote: > > > >> > >>> > Hi, > > > >> > >>> > > > > >> > >>> > Now that the refactor I've been working on has been merged, > > the > > > >> next > > > >> > >>> > priority for me personally with the Rust implementation is > > > >> getting IPC > > > >> > >>> and > > > >> > >>> > integration testing working. > > > >> > >>> > > > > >> > >>> > Unfortunately the official Flatbuffers Rust version is not > > > >> available > > > >> > >>> yet > > > >> > >>> > and my recent attempts at contacting the author have been > > > >> unsuccessful > > > >> > >>> so I > > > >> > >>> > have started working with this fork of Flatbuffers which has > > Rust > > > >> > >>> support: > > > >> > >>> > https://github.com/josephDunne/flatbuffers. > > > >> > >>> > > > > >> > >>> > I was able to generate code from Schema.fbs but it doesn't > > > >> compile and > > > >> > >>> I've > > > >> > >>> > started filing issues and debugging this. > > > >> > >>> > > > > >> > >>> > Working with this fork isn't ideal but I don't see what > other > > > >> choice I > > > >> > >>> > have, other than just waiting for the official project to > > support > > > >> Rust. > > > >> > >>> > > > > >> > >>> > I'm interested to hear if anyone has any alternate > > suggestions. I > > > >> know > > > >> > >>> it > > > >> > >>> > would be possible to wrap C code but I'd like to keep the > Rust > > > >> > >>> > implementation as a pure Rust project if possible. > > > >> > >>> > > > > >> > >>> > Thanks, > > > >> > >>> > > > > >> > >>> > Andy. > > > >> > >>> > > > >> > >> > > > >> > > > > > > >
