mbutrovich opened a new issue, #11583:
URL: https://github.com/apache/arrow-rs/issues/11583
### Describe the bug
`byte_view_equal` checks `lhs.is_null(idx)` with `idx` counted from the
start of the compared range, not from the start of the array. When a list, list
view, fixed-size list or struct compares a `Utf8View` or `BinaryView` child
range that starts past 0, it reads the null bit of the wrong slot. Null slots
are then compared by their undefined content, and valid slots can be skipped.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::{make_array, Array, ArrayRef, ListArray, StringViewArray};
use arrow_buffer::{NullBuffer, OffsetBuffer};
use arrow_schema::{DataType, Field};
/// [["x"], [null]], where the null item hides `hidden`
fn list(hidden: &str) -> ArrayRef {
let child = StringViewArray::from(vec!["x", hidden]).into_data();
let child = child.into_builder().nulls(Some(NullBuffer::from(vec![true,
false]))).build().unwrap();
let field = Arc::new(Field::new("item", DataType::Utf8View, true));
Arc::new(ListArray::new(field, OffsetBuffer::from_lengths([1, 1]),
make_array(child), None))
}
fn main() {
let a = list("a long string hidden under a null");
let b = list("another long string under the null");
// Both are [[null]], differing only in data under the null
println!("{}", a.slice(1, 1).to_data() == b.slice(1, 1).to_data()); //
false, expected true
}
```
### Expected behavior
The two arrays hold the same values, `[[null]]`, so they compare equal. Data
under a null slot is unspecified
([Columnar.rst](https://github.com/apache/arrow/blob/fcf8f3014227cafe24deeb044c2d51f5fc9902e5/docs/source/format/Columnar.rst?plain=1#L442-L445))
and must not affect equality.
### Additional context
The check is at
[byte_view.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-data/src/equal/byte_view.rs#L33-L37).
Using `lhs.is_null(lhs_start + idx)` would fix it. Comparing the children
directly does not show the bug, because `StringViewArray::slice` re-slices the
views buffer so the compared range starts at 0. A `ListView` whose rows point
past child index 0 shows it without slicing.
The code has been unchanged since view types were added in #5481. #9323 was
a similar offset bug in the equality of `Utf8` arrays.
Found with property-based tests that build arrays in every physical layout
the Arrow format allows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
