Yes it is possible - we have done similar things in parquet-rs to generate Thrift files, using build.rs (see here <https://github.com/sunchao/parquet-rs/blob/f2f0993098bdeaa62a0790be18faac2f5bf2fe3b/build.rs>). Once the manual edits are not required we can explore that path.
On Tue, Nov 20, 2018 at 1:20 PM Krisztián Szűcs <szucs.kriszt...@gmail.com> wrote: > Auto generation sounds like a good idea, similarly like rust-bindgen > does it: https://rust-lang-nursery.github.io/rust-bindgen/tutorial-3.html > Of course it depends on the complexity of manual edits. > > We can still separate the crates later if that's desired. > > On Tue, Nov 20, 2018 at 6:39 PM Chao Sun <sunc...@uber.com.invalid> wrote: > > > I think I suggested this. The idea is to isolate the auto-generated code > > from the other source files, and also to isolate future changes due to > > updates on either format or flatbuffers versions. IMO this makes the repo > > cleaner and more organized. Of course, ideally we should auto-generate > > these files when compiling. > > > > On Mon, Nov 19, 2018 at 11:55 AM Wes McKinney <wesmck...@gmail.com> > wrote: > > > > > Could you explain why having a separate crate would be a good idea? > > > On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <andygrov...@gmail.com> > > wrote: > > > > > > > > A question has been raised on this PR as to whether we should > publish a > > > > separate crate for the format/ipc generated code. I think there might > > be > > > > some merit in this and wanted to raise it here to see what everyone > > > thinks. > > > > > > > > This would mean having two directories under the rust directory ... > one > > > for > > > > arrow-format (or maybe the name should be arrow-ipc?) and one for the > > > main > > > > arrow crate. > > > > > > > > It is possible we might want other crates in the future e.g. > > > arrow-parquet > > > > or whatever. > > > > > > > > Andy. > > > > > > > > > > > > On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <andygrov...@gmail.com> > > > wrote: > > > > > > > > > I have created a PR this morning that adds the generated > Flatbuffers > > > code > > > > > to the Rust implementation ( > > https://github.com/apache/arrow/pull/2986 > > > ). > > > > > > > > > > Just throwing these files into the repo doesn't add a ton of value > > > until > > > > > we use them for something, but it took some effort to generate them > > > (and > > > > > then fix a couple things by hand) so it seems useful to contribute > > > this. > > > > > The PR includes a brief README with the instructions for generating > > the > > > > > files. > > > > > > > > > > I have taken this week off for Thanksgiving so I actually have some > > > > > dedicated time to contribute to Arrow. I suppose my next step on > this > > > is to > > > > > write a test that can serialize some data and then de-serialize it > > > again. > > > > > > > > > > Andy. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <wesmck...@gmail.com> > > > wrote: > > > > > > > > > >> hi Andy, > > > > >> > > > > >> AFAIK in the other implementations, the Flatbuffers serialization > is > > > > >> an implementation detail of offering support for the columnar IPC > > > > >> protocol (which in turn depends on the columnar data structures, > > > > >> memory management, etc.). I'm not sure what a crate offering the > > > > >> Flatbuffers bindings alone would offer, outside of helping people > > > > >> create new Rust Arrow implementations, I guess. In any case, if > you > > > > >> wanted to do that, we would need to make a release first. > > > > >> > > > > >> - Wes > > > > >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <andygrov...@gmail.com> > > > wrote: > > > > >> > > > > > >> > Brief update on this. I have now been able to use flatbuffers to > > > > >> generate > > > > >> > Rust code from the Arrow schema, and it compiles. In theory, > this > > > can be > > > > >> > published as a standalone "arrow-format" crate but I'm wondering > > > what > > > > >> the > > > > >> > plan should be here. > > > > >> > > > > > >> > I will start reviewing how this is handled in the other > > > implementations > > > > >> but > > > > >> > if anyone has suggestions, please let me know. > > > > >> > > > > > >> > Thanks, > > > > >> > > > > > >> > Andy. > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove < > andygrov...@gmail.com> > > > > >> wrote: > > > > >> > > > > > >> > > Flatbuffers now has support for Rust - the PR was just merged > in > > > the > > > > >> last > > > > >> > > couple days: > > > > >> > > > > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers > > > > >> > > > > > > >> > > I hope to find some time in the next week or two to start > > > > >> experimenting > > > > >> > > with this for Arrow IPC. > > > > >> > > > > > > >> > > Andy. > > > > >> > > > > > > >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove < > > andygrov...@gmail.com > > > > > > > > >> wrote: > > > > >> > > > > > > >> > >> The author of the Rust version of FlatBuffers is now working > > on a > > > > >> public > > > > >> > >> fork and seems to be making good progress. I just wanted to > > send > > > a > > > > >> quick > > > > >> > >> update on this since I am now back onto working on Arrow IPC > in > > > Rust > > > > >> (I'm > > > > >> > >> sure this is going to take quite a while though so don't want > > to > > > get > > > > >> > >> anyones hopes up too much). For anyone that is interested, > here > > > is > > > > >> the > > > > >> > >> FlatBuffers branch: > > > > >> https://github.com/rw/flatbuffers/tree/2018-02--rust > > > > >> > >> > > > > >> > >> Andy. > > > > >> > >> > > > > >> > >> > > > > >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney < > > > wesmck...@gmail.com> > > > > >> wrote: > > > > >> > >> > > > > >> > >>> Sorry to hear, it's a bit of a rough situation with > > Flatbuffers > > > and > > > > >> > >>> Rust. One possibility is to build an interface to the > > > Flatbuffers > > > > >> data > > > > >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I > wonder > > if > > > > >> these > > > > >> > >>> bindings are header-only like C++ and if that makes things > any > > > > >> easier > > > > >> > >>> for you. > > > > >> > >>> > > > > >> > >>> If you defined an API in Rust for reading and writing the > > Arrow > > > > >> > >>> metadata that does not expose any Flatbuffers-specific > > details, > > > then > > > > >> > >>> once there is a production-grade Rust implementation of > > > Flatbuffers, > > > > >> > >>> the implementation details could be swapped out without > > > disruption. > > > > >> > >>> > > > > >> > >>> > my recent attempts at contacting the author have been > > > unsuccessful > > > > >> > >>> > > > > >> > >>> Have you e-mailed the author directly or only pings on > GitHub? > > > Pings > > > > >> > >>> may not be making it to their inbox. > > > > >> > >>> > > > > >> > >>> Appreciate your efforts on this; I think we'll see a lot > more > > > work > > > > >> in > > > > >> > >>> data processing in Rust in the coming years. > > > > >> > >>> > > > > >> > >>> - Wes > > > > >> > >>> > > > > >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove < > > > andygrov...@gmail.com> > > > > >> > >>> wrote: > > > > >> > >>> > Hi, > > > > >> > >>> > > > > > >> > >>> > Now that the refactor I've been working on has been > merged, > > > the > > > > >> next > > > > >> > >>> > priority for me personally with the Rust implementation is > > > > >> getting IPC > > > > >> > >>> and > > > > >> > >>> > integration testing working. > > > > >> > >>> > > > > > >> > >>> > Unfortunately the official Flatbuffers Rust version is not > > > > >> available > > > > >> > >>> yet > > > > >> > >>> > and my recent attempts at contacting the author have been > > > > >> unsuccessful > > > > >> > >>> so I > > > > >> > >>> > have started working with this fork of Flatbuffers which > has > > > Rust > > > > >> > >>> support: > > > > >> > >>> > https://github.com/josephDunne/flatbuffers. > > > > >> > >>> > > > > > >> > >>> > I was able to generate code from Schema.fbs but it doesn't > > > > >> compile and > > > > >> > >>> I've > > > > >> > >>> > started filing issues and debugging this. > > > > >> > >>> > > > > > >> > >>> > Working with this fork isn't ideal but I don't see what > > other > > > > >> choice I > > > > >> > >>> > have, other than just waiting for the official project to > > > support > > > > >> Rust. > > > > >> > >>> > > > > > >> > >>> > I'm interested to hear if anyone has any alternate > > > suggestions. I > > > > >> know > > > > >> > >>> it > > > > >> > >>> > would be possible to wrap C code but I'd like to keep the > > Rust > > > > >> > >>> > implementation as a pure Rust project if possible. > > > > >> > >>> > > > > > >> > >>> > Thanks, > > > > >> > >>> > > > > > >> > >>> > Andy. > > > > >> > >>> > > > > >> > >> > > > > >> > > > > > > > > > > >
