mbutrovich opened a new issue, #11590:
URL: https://github.com/apache/arrow-rs/issues/11590
### Describe the bug
`substring` on a `StringArray` or `LargeStringArray` checks char boundaries
in every slot, including null slots. If a null slot covers a multi-byte
character and the cut falls inside it, the whole call returns an error,
although the output for that slot is null.
### To Reproduce
```rust
use arrow_array::{BooleanArray, StringArray};
use arrow_select::nullif::nullif;
use arrow_string::substring::substring;
fn main() {
let array = nullif(&StringArray::from(vec!["\u{e9}", "ab"]),
&BooleanArray::from(vec![true, false])).unwrap();
println!("{:?}", substring(&array, 1, None));
// Err(ComputeError("The offset 1 is at an invalid utf-8 boundary."))
}
```
### Expected behavior
`Ok` with `[null, "b"]`. Only non-null values should be able to cause the
documented char boundary error.
### Additional context
The [Columnar
format](https://github.com/apache/arrow/blob/fcf8f3014227cafe24deeb044c2d51f5fc9902e5/docs/source/format/Columnar.rst?plain=1#L442-L445)
lets a null slot cover a non-empty range with undefined contents.
`byte_substring` computes the new start and end of every slot through
`check_char_boundary`, with no null check
([substring.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-string/src/substring.rs#L379-L400)).
`string_view_substring` skips null slots before the same check
([substring.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-string/src/substring.rs#L287-L300)),
so a `StringViewArray` with a null view over the same bytes returns `[null,
"b"]`.
`nullif` keeps the bytes of the slots it sets to null, so its output has
this shape. Arrays from other producers through FFI or IPC can too.
Found with property-based tests that build arrays in every physical layout
the Arrow format allows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
