Re: Rust IPC and Integration Testing

Yes it is possible - we have done similar things in parquet-rs to generate Thrift files, using build.rs (see here ). Once the manual edits are not required we can explore that path. On Tue, Nov 20, 2018

Re: Rust IPC and Integration Testing

Auto generation sounds like a good idea, similarly like rust-bindgen does it: https://rust-lang-nursery.github.io/rust-bindgen/tutorial-3.html Of course it depends on the complexity of manual edits. We can still separate the crates later if that's desired. On Tue, Nov 20, 2018 at 6:39 PM Chao

Re: Rust IPC and Integration Testing

I think I suggested this. The idea is to isolate the auto-generated code from the other source files, and also to isolate future changes due to updates on either format or flatbuffers versions. IMO this makes the repo cleaner and more organized. Of course, ideally we should auto-generate these

Re: Rust IPC and Integration Testing

Could you explain why having a separate crate would be a good idea? On Mon, Nov 19, 2018 at 2:40 PM Andy Grove wrote: > > A question has been raised on this PR as to whether we should publish a > separate crate for the format/ipc generated code. I think there might be > some merit in this and

Re: Rust IPC and Integration Testing

A question has been raised on this PR as to whether we should publish a separate crate for the format/ipc generated code. I think there might be some merit in this and wanted to raise it here to see what everyone thinks. This would mean having two directories under the rust directory ... one for

Re: Rust IPC and Integration Testing

I have created a PR this morning that adds the generated Flatbuffers code to the Rust implementation (https://github.com/apache/arrow/pull/2986). Just throwing these files into the repo doesn't add a ton of value until we use them for something, but it took some effort to generate them (and then

Re: Rust IPC and Integration Testing

hi Andy, AFAIK in the other implementations, the Flatbuffers serialization is an implementation detail of offering support for the columnar IPC protocol (which in turn depends on the columnar data structures, memory management, etc.). I'm not sure what a crate offering the Flatbuffers bindings

Re: Rust IPC and Integration Testing

Brief update on this. I have now been able to use flatbuffers to generate Rust code from the Arrow schema, and it compiles. In theory, this can be published as a standalone "arrow-format" crate but I'm wondering what the plan should be here. I will start reviewing how this is handled in the other

Re: Rust IPC and Integration Testing

Flatbuffers now has support for Rust - the PR was just merged in the last couple days: https://github.com/google/flatbuffers/tree/master/rust/flatbuffers I hope to find some time in the next week or two to start experimenting with this for Arrow IPC. Andy. On Tue, Jun 19, 2018 at 6:09 PM Andy

Re: Rust IPC and Integration Testing

The author of the Rust version of FlatBuffers is now working on a public fork and seems to be making good progress. I just wanted to send a quick update on this since I am now back onto working on Arrow IPC in Rust (I'm sure this is going to take quite a while though so don't want to get anyones

Re: Rust IPC and Integration Testing

Sorry to hear, it's a bit of a rough situation with Flatbuffers and Rust. One possibility is to build an interface to the Flatbuffers data via flatcc (https://github.com/dvidelabs/flatcc) -- I wonder if these bindings are header-only like C++ and if that makes things any easier for you. If you

Rust IPC and Integration Testing

Hi, Now that the refactor I've been working on has been merged, the next priority for me personally with the Rust implementation is getting IPC and integration testing working. Unfortunately the official Flatbuffers Rust version is not available yet and my recent attempts at contacting the