mbutrovich opened a new pull request, #10030: URL: https://github.com/apache/arrow-rs/pull/10030
# Which issue does this PR close? - Closes #10028. # Rationale for this change `from_ffi` / `from_ffi_and_data_type` (and therefore `ArrowArrayStreamReader`) panic inside `ScalarBuffer::<i128>::from` when an FFI producer hands over a `Decimal128` buffer that is 8-byte aligned but not 16-byte aligned. The producer is spec-conformant — the C Data Interface only recommends 8-byte alignment — but `align_of::<i128>() == 16` since Rust 1.77 on x86 (always on ARM), so arrow-rs's typed arrays require 16. JVM producers like arrow-java's `NettyAllocationManager` hit this regularly. The IPC reader already handles this by calling `ArrayData::align_buffers()` on import (default of `IpcReadOptions::require_alignment`, see #5554), and `arrow-pyarrow` was patched the same way for #6471 / apache/arrow#43552. The C Data Interface entry points were the missing piece. # What changes are included in this PR? - `arrow::ffi::from_ffi` and `from_ffi_and_data_type`: call `data.align_buffers()` after `consume()`. No-op when buffers are already aligned; depends on #6462 making `align_buffers` recursive over child data. - `arrow-pyarrow`: drop the now-redundant `array_data.align_buffers()` call; it's covered by `from_ffi`. # Are these changes tested? Yes. New regression test `test_decimal128_under_aligned_round_trip` in `arrow-array/src/ffi.rs` constructs an 8-aligned-not-16-aligned `Decimal128` buffer via `Buffer::from_vec(...).slice(8)`, imports through `from_ffi`, and asserts the resulting `Decimal128Array` values are correct. The test panics without the fix with the exact error from #10028. # Are there any user-facing changes? No API changes. Behavior change: `from_ffi` / `from_ffi_and_data_type` (and `ArrowArrayStreamReader::next`) now silently realign under-aligned buffers instead of panicking. Already-aligned producers are unaffected; misaligned producers that previously panicked now succeed with a one-time copy of the offending buffer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
