Could you explain why having a separate crate would be a good idea?
On Mon, Nov 19, 2018 at 2:40 PM Andy Grove <andygrov...@gmail.com> wrote:
>
> A question has been raised on this PR as to whether we should publish a
> separate crate for the format/ipc generated code. I think there might be
> some merit in this and wanted to raise it here to see what everyone thinks.
>
> This would mean having two directories under the rust directory ... one for
> arrow-format (or maybe the name should be arrow-ipc?) and one for the main
> arrow crate.
>
> It is possible we might want other crates in the future e.g. arrow-parquet
> or whatever.
>
> Andy.
>
>
> On Sat, Nov 17, 2018 at 10:04 AM Andy Grove <andygrov...@gmail.com> wrote:
>
> > I have created a PR this morning that adds the generated Flatbuffers code
> > to the Rust implementation (https://github.com/apache/arrow/pull/2986).
> >
> > Just throwing these files into the repo doesn't add a ton of value until
> > we use them for something, but it took some effort to generate them (and
> > then fix a couple things by hand) so it seems useful to contribute this.
> > The PR includes a brief README with the instructions for generating the
> > files.
> >
> > I have taken this week off for Thanksgiving so I actually have some
> > dedicated time to contribute to Arrow. I suppose my next step on this is to
> > write a test that can serialize some data and then de-serialize it again.
> >
> > Andy.
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Nov 4, 2018 at 11:11 AM Wes McKinney <wesmck...@gmail.com> wrote:
> >
> >> hi Andy,
> >>
> >> AFAIK in the other implementations, the Flatbuffers serialization is
> >> an implementation detail of offering support for the columnar IPC
> >> protocol (which in turn depends on the columnar data structures,
> >> memory management, etc.). I'm not sure what a crate offering the
> >> Flatbuffers bindings alone would offer, outside of helping people
> >> create new Rust Arrow implementations, I guess. In any case, if you
> >> wanted to do that, we would need to make a release first.
> >>
> >> - Wes
> >> On Sat, Nov 3, 2018 at 1:01 PM Andy Grove <andygrov...@gmail.com> wrote:
> >> >
> >> > Brief update on this. I have now been able to use flatbuffers to
> >> generate
> >> > Rust code from the Arrow schema, and it compiles. In theory, this can be
> >> > published as a standalone "arrow-format" crate but I'm wondering what
> >> the
> >> > plan should be here.
> >> >
> >> > I will start reviewing how this is handled in the other implementations
> >> but
> >> > if anyone has suggestions, please let me know.
> >> >
> >> > Thanks,
> >> >
> >> > Andy.
> >> >
> >> >
> >> >
> >> > On Mon, Sep 3, 2018 at 9:08 PM Andy Grove <andygrov...@gmail.com>
> >> wrote:
> >> >
> >> > > Flatbuffers now has support for Rust - the PR was just merged in the
> >> last
> >> > > couple days:
> >> > > https://github.com/google/flatbuffers/tree/master/rust/flatbuffers
> >> > >
> >> > > I hope to find some time in the next week or two to start
> >> experimenting
> >> > > with this for Arrow IPC.
> >> > >
> >> > > Andy.
> >> > >
> >> > > On Tue, Jun 19, 2018 at 6:09 PM Andy Grove <andygrov...@gmail.com>
> >> wrote:
> >> > >
> >> > >> The author of the Rust version of FlatBuffers is now working on a
> >> public
> >> > >> fork and seems to be making good progress. I just wanted to send a
> >> quick
> >> > >> update on this since I am now back onto working on Arrow IPC in Rust
> >> (I'm
> >> > >> sure this is going to take quite a while though so don't want to get
> >> > >> anyones hopes up too much). For anyone that is interested, here is
> >> the
> >> > >> FlatBuffers branch:
> >> https://github.com/rw/flatbuffers/tree/2018-02--rust
> >> > >>
> >> > >> Andy.
> >> > >>
> >> > >>
> >> > >> On Sat, May 19, 2018 at 4:05 PM Wes McKinney <wesmck...@gmail.com>
> >> wrote:
> >> > >>
> >> > >>> Sorry to hear, it's a bit of a rough situation with Flatbuffers and
> >> > >>> Rust. One possibility is to build an interface to the Flatbuffers
> >> data
> >> > >>> via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if
> >> these
> >> > >>> bindings are header-only like C++ and if that makes things any
> >> easier
> >> > >>> for you.
> >> > >>>
> >> > >>> If you defined an API in Rust for reading and writing the Arrow
> >> > >>> metadata that does not expose any Flatbuffers-specific details, then
> >> > >>> once there is a production-grade Rust implementation of Flatbuffers,
> >> > >>> the implementation details could be swapped out without disruption.
> >> > >>>
> >> > >>> > my recent attempts at contacting the author have been unsuccessful
> >> > >>>
> >> > >>> Have you e-mailed the author directly or only pings on GitHub? Pings
> >> > >>> may not be making it to their inbox.
> >> > >>>
> >> > >>> Appreciate your efforts on this; I think we'll see a lot more work
> >> in
> >> > >>> data processing in Rust in the coming years.
> >> > >>>
> >> > >>> - Wes
> >> > >>>
> >> > >>> On Fri, May 18, 2018 at 9:21 AM, Andy Grove <andygrov...@gmail.com>
> >> > >>> wrote:
> >> > >>> > Hi,
> >> > >>> >
> >> > >>> > Now that the refactor I've been working on has been merged, the
> >> next
> >> > >>> > priority for me personally with the Rust implementation is
> >> getting IPC
> >> > >>> and
> >> > >>> > integration testing working.
> >> > >>> >
> >> > >>> > Unfortunately the official Flatbuffers Rust version is not
> >> available
> >> > >>> yet
> >> > >>> > and my recent attempts at contacting the author have been
> >> unsuccessful
> >> > >>> so I
> >> > >>> > have started working with this fork of Flatbuffers which has Rust
> >> > >>> support:
> >> > >>> > https://github.com/josephDunne/flatbuffers.
> >> > >>> >
> >> > >>> > I was able to generate code from Schema.fbs but it doesn't
> >> compile and
> >> > >>> I've
> >> > >>> > started filing issues and debugging this.
> >> > >>> >
> >> > >>> > Working with this fork isn't ideal but I don't see what other
> >> choice I
> >> > >>> > have, other than just waiting for the official project to support
> >> Rust.
> >> > >>> >
> >> > >>> > I'm interested to hear if anyone has any alternate suggestions. I
> >> know
> >> > >>> it
> >> > >>> > would be possible to wrap C code but I'd like to keep the Rust
> >> > >>> > implementation as a pure Rust project if possible.
> >> > >>> >
> >> > >>> > Thanks,
> >> > >>> >
> >> > >>> > Andy.
> >> > >>>
> >> > >>
> >>
> >
