By the way, half-relatedly, you may also be interested in the C data interface at some point: https://arrow.apache.org/docs/format/CDataInterface.html
We're also planning an experimental C stream interface above the C data interface: https://github.com/apache/arrow/pull/8052 https://github.com/apache/arrow/blob/b71d9652c8f41764ea5c0036c720cea786ee564a/docs/source/format/CStreamInterface.rst Regards Antoine. Le 18/09/2020 à 18:03, Jorge Cardoso Leitão a écrit : > Hi, > > I am trying to convert pyarrow buffers into Rust buffers and vice-versa, to > perform zero-copy from and to pyarrow, to and from Rust's library. > > I was able to perform the operation rust -> pyarrow, using something along > the lines of > > // 64 bits system > let pointer = buffer.raw_data() as i64; > pyarrow.call1("foreign_buffer > <https://arrow.apache.org/docs/python/generated/pyarrow.foreign_buffer.html>", > (pointer, buffer.len())) > > However, I am not being able to perform the opposite transformation, due to > a memory misalignment check that is failing. I.e. > > let address = buffer.getattr("address")?.extract::<i64>()?; > let size = buffer.getattr("size")?.extract::<i64>()? as usize; > > // panics with "memory not aligned" > Buffer::from_raw_parts(address as *const u8, size, size) > > I get an address such as `4624199872` (i64), but, when converted to `*const > u8`, our rust implementation does not consider it to be memory aligned (at > least in x86_64) and panics. Has anyone worked in this problem? Any hints > on what I am doing wrong? > > AFAIK, our rust implementation requires different alignments, depending on > the architecture (see memory.rs > <https://github.com/apache/arrow/blob/master/rust/arrow/src/memory.rs>). > Wasn't this supposed to be a single alignment, consistent throughout the > arrow? I am confused. > > Any help would be greatly appreciated! > > Best, > Jorges >
