Re: [Rust] Is Rust Arrow's deserialization code zero-copy?

2021-04-07 Thread Andrew Lamb
> Is my understanding correct? I believe your understanding is correct. > Are there additional considerations in rust which means it's not possible for objects accessed by application code (e.g. record-batches, arrays) to keep their buffers in IPC messages? I don't think there is anything specif

Re: [Rust] Is Rust Arrow's deserialization code zero-copy?

2021-04-06 Thread Vertexclique
Hi Al; Yeah, that's true. The ownership rules of Rust prevents resource ownership outside the container that the resource lifetime taken. That's why there are buffer copies to take ownership and operate on it. Imo, it should be also removed from the docs if it's written like that (for Rust). B

Re: [Rust] Is Rust Arrow's deserialization code zero-copy?

2021-04-06 Thread Dominik Moritz
I ran into this issue as well. See https://issues.apache.org/jira/projects/ARROW/issues/ARROW-11696 . As I understand it, https://github.com/jorgecarleitao/arrow2 will make reading more efficient. On Apr 6, 2021

[Rust] Is Rust Arrow's deserialization code zero-copy?

2021-04-06 Thread Al Taylor
Hi, I was reading around the rust-arrow codebase, evaluating it for potential future use. I'm particularly interested in zero-copy processing. I could very well be wrong here, as I don't have a lot of rust experience, but it looks like the code for reading buffers out of IPC messages is copyin