ninglinLiu opened a new pull request, #11905: URL: https://github.com/apache/arrow-rs/pull/11905
# Which issue does this PR close? - Part of #11855 # Rationale for this change `ArrayData::validate_nulls` for a `FixedSizeList` with a non-nullable child always expanded the parent null buffer to `len * element_len` bits whenever the parent had nulls, even when the child had no nulls at all. In that case the expanded mask is never consulted (a child without nulls trivially satisfies the non-nullable check), but with a large `element_len` and a buffer-less child (e.g. a `Null` array) a tiny input can force a very large allocation, which can abort on OOM when validating untrusted input such as IPC. # What changes are included in this PR? - Only slice/expand when the visible range of the child actually contains nulls (`null_count() > 0`); otherwise skip the check, which is exactly what `validate_non_nullable` would have concluded anyway. Behavior for children that do have nulls is unchanged. - This addresses the "expansion is not needed" half of the issue; making the expansion itself fallible is left for a follow-up, hence "Part of". # Are these changes tested? Yes. Added `test_fixed_size_list_validate_nulls_skips_expansion_without_child_nulls` (element size `1 << 30`, `Null` child, parent with nulls). Before this change the test allocates ~512 MiB and takes ~1s; with it, validation does no allocation and finishes instantly. The existing FixedSizeList nullability tests still pass (`cargo test -p arrow-data`, `cargo clippy -p arrow-data --all-targets -- -D warnings`). # Are there any user-facing changes? No API changes. This PR was prepared with AI assistance and reviewed/tested by the author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
