ninglinLiu opened a new pull request, #11903: URL: https://github.com/apache/arrow-rs/pull/11903
# Which issue does this PR close? - Part of #11902 # Rationale for this change `Null` arrays never have a validity bitmap (`into_builder` already drops it), but `MutableArrayData::try_with_capacities` still allocates one for a `Null` array whenever `use_nulls` is set, either by the caller or inherited from a nullable parent. For deeply nested types such as `FixedSizeList<FixedSizeList<Null>>` the `Null` leaf has the largest capacity, so this is the biggest wasted allocation (and the memory exhaustion case described in the issue). This PR handles the `Null` part of the issue. It does not change when nested non-`Null` children get a bitmap from a nullable parent: today `FixedSizeList`/`Struct` `extend_nulls` forward nulls to their children, so that part needs a separate design decision. # What changes are included in this PR? - Never allocate a null buffer for `DataType::Null` in `try_with_capacities`. - `build_extend_null_bits` is a no-op for `Null` source arrays. - `try_extend_nulls` on a `Null` array only bumps `len`/`null_count`, since every slot is already null. This keeps `extend_nulls` on a nullable parent (e.g. `FixedSizeList<Null>`) working without the child bitmap. # Are these changes tested? Yes. Two unit tests in `arrow-data/src/transform/mod.rs`: one for a plain `Null` array with `use_nulls = true`, and one adapted from the reproducer in the issue (nested `FixedSizeList` with a `Null` leaf and a null parent), which also extends and freezes the result and validates it. Both fail on `main`. # Are there any significant user-facing changes? No API changes. One behavior difference: `try_extend_nulls` on a `MutableArrayData` of type `Null` now succeeds even when it was created with `use_nulls = false`, since a `Null` array can always hold nulls. --- AI disclosure: this change and its tests were prepared with AI assistance; I reviewed the code and ran the `arrow-data`, `arrow-select` and `arrow` transform tests, `cargo fmt` and `cargo clippy -p arrow-data` locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
