Jefffrey opened a new issue, #11902:
URL: https://github.com/apache/arrow-rs/issues/11902
### Describe the bug
For nested types, `MutableArrayData::try_with_capacities` may allocate a
null bitmap for child types even if the child type doesn't need one: for
example, it is `Null` datatype or simply the parent having a null bitmap
doesn't force the child to either (if the input child didn't have null bitmap
either)
### To Reproduce
Test generated by codex:
```rust
// arrow-data/src/transform/mod.rs
#[test]
fn test_nested_fixed_size_list_does_not_allocate_child_null_bitmaps() {
let values = ArrayData::new_null(&DataType::Null, 8);
let inner_type =
DataType::FixedSizeList(Arc::new(Field::new("item",
DataType::Null, true)), 4);
let inner = ArrayData::builder(inner_type.clone())
.len(2)
.add_child_data(values)
.build()
.unwrap();
let outer = ArrayData::builder(DataType::FixedSizeList(
Arc::new(Field::new("item", inner_type, true)),
2,
))
.len(1)
.nulls(Some(NullBuffer::new_null(1)))
.add_child_data(inner)
.build()
.unwrap();
let mutable =
MutableArrayData::try_with_capacities(vec![&outer], false,
Capacities::Array(1))
.unwrap();
// Copying a null parent does not require null bitmaps for its
children.
let inner = &mutable.data.child_data[0];
let values = &inner.data.child_data[0];
assert!(inner.data.null_buffer.is_none());
// Null arrays never need a validity bitmap.
assert!(values.data.null_buffer.is_none());
}
```
### Expected behavior
We shouldn't allocate memory for a null bitmap here for the inner child,
since it is of `Null` type so this allocation is wasteful.
In the worst case, a crafted input via IPC for example can cause memory
exhaustion; e.g. have a `FixedSizeList<FixedSizeList<Null>>` input array where
the list size is 2^30 at each level. This is valid, since `Null` itself would
hold no memory, but going through `MutableArrayData` currently allocates a null
bitmap in case the parent has nulls, so this can cause a massive allocation for
the `Null` child which can be avoided
### Additional context
Reported by Adam Korczynski from ADA Logics, identified by Anthropic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
