viirya opened a new issue, #4431:
URL: https://github.com/apache/arrow-rs/issues/4431
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
Recently some memory alignment error was found while passing arrays from
Java arrow to Rust arrow. The error comes from `From<Buffer>` for
`ScalarBuffer<T>`. I inspected the error further and am wondering if the
alignment requirement doesn't make sense (at least for working with external
allocated memory).
The idea of alignment check in `From<Buffer>` is making sure that the buffer
pointer is aligned with scalar type `T`. The existing test shows that:
```rust
fn test_unaligned() {
let expected = [0_i32, 1, 2];
let buffer = Buffer::from_iter(expected.iter().cloned());
let buffer = buffer.slice(1);
ScalarBuffer::<i32>::new(buffer, 0, 2);
}
```
So the buffer slice is not aligned anymore with i32, so converting it to i32
ScalarBuffer panics with alignment error. But the assumption here is the memory
region of the buffer is allocated with type. In Java Arrow, it allocates buffer
simply with required number of bytes without type information, it's basically a
contiguous memory region with `n` bytes. So obviously it doesn't necessarily
have alignment with some type `T` there. When we try to pass array backed by
the buffer to Rust arrow, the alignment check in `From<Buffer>` will fail
occasionally (sometimes it passes the check if the allocated buffer happens to
align with `T`).
**To Reproduce**
<!--
Steps to reproduce the behavior:
-->
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
**Additional context**
<!--
Add any other context about the problem here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
