Re: [DISCUSS] Memory alignment in rust - what to do?

To bring closure to this thread: #8401 implements the necessary functionality to import and export from and to the C data interface, includes an integration test to run these against the API provided by pyarrow, and the CI is green. Special thanks to An

Re: [DISCUSS] Memory alignment in rust - what to do?

Le 24/09/2020 à 17:18, Antoine Pitrou a écrit : >> 1. in pyarrow, I was only able to find Array.from_buffers and from_pandas. >> Is the ABI implemented but not documented? > > It is implemented in C++ and also exposed (but undocumented) in Python. > > The Python methods are called `Array._impor

Re: [DISCUSS] Memory alignment in rust - what to do?

Hi Jorge, Le 24/09/2020 à 17:04, Jorge Cardoso Leitão a écrit : > Thank you both for the prompt response. > > Just to check I understand, Antoine, your recommendation is: > > 1. Rust implementation should expose the ABI > 2. Rust implementation should be able to consume (and use) the ABI (with

Re: [DISCUSS] Memory alignment in rust - what to do?

Thank you both for the prompt response. Just to check I understand, Antoine, your recommendation is: 1. Rust implementation should expose the ABI 2. Rust implementation should be able to consume (and use) the ABI (without owning it, but still call the `release`) And, likewise, 1. C/Pyarrow shou

Re: [DISCUSS] Memory alignment in rust - what to do?

Not a Rust expert yet, but here are my 2 (or more) cents: The alignment was chosen (as far as I know) so that separately allocated buffers would not share the same cache line, which could cause performance issues when the same cache line is accessed by multiple threads. So aligning our buffers tha

Re: [DISCUSS] Memory alignment in rust - what to do?

Le 22/09/2020 à 19:16, Jorge Cardoso Leitão a écrit : > Hi, > > I had some time to look at https://issues.apache.org/jira/browse/ARROW-10039, > wrt to the alignment requirements that rust implementation currently > imposes. > > The gist is that it is not that easy, and I would like to request s

[DISCUSS] Memory alignment in rust - what to do?

Hi, I had some time to look at https://issues.apache.org/jira/browse/ARROW-10039, wrt to the alignment requirements that rust implementation currently imposes. The gist is that it is not that easy, and I would like to request some guidance. Some facts: 1. Our current implementation does not acce