edubraqd opened a new issue, #11094:
URL: https://github.com/apache/arrow-rs/issues/11094
**Describe the bug**
The comparison kernels in `arrow_ord::cmp` (`eq`, `lt`, `distinct`, ...)
panic with `internal error: entered unreachable code` when given a `Dictionary`
array whose values are themselves a `Dictionary` (or a `RunEndEncoded`). They
return `Err(InvalidArgumentError)` for other unsupported inputs, such as nested
types or mismatched types, so a panic here is inconsistent with the rest of the
kernel and takes down the caller's thread.
`compare_op` unwraps one run-end-encoded layer and one dictionary layer,
then checks `is_nested()` and type equality on what is left.
`DataType::is_nested` returns `false` for `Dictionary(_, Utf8)`, and both sides
have the same type, so the checks pass and `downcast_primitive_array!` falls
into its `_ => unreachable!()` arm (`arrow-ord/src/cmp.rs:290` in 59.3.0, line
314 on `main`).
`main` already has the right predicate, `supports_distinct` (added in
#9642), but only `partition.rs` consults it; `compare_op` itself still panics.
**To Reproduce**
```rust
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, DictionaryArray, Int32Array, StringArray,
UInt32Array};
use arrow_ord::cmp::eq;
fn main() {
// Dictionary<Int32, Dictionary<UInt32, Utf8>>
let inner: ArrayRef = Arc::new(DictionaryArray::new(
UInt32Array::from(vec![0, 1]),
Arc::new(StringArray::from(vec!["a", "b"])),
));
let outer = DictionaryArray::new(Int32Array::from(vec![0, 1]), inner);
println!("type: {}", outer.data_type());
match eq(&outer, &outer) {
Ok(a) => println!("ok: {a:?}"),
Err(e) => println!("err: {e}"),
}
}
```
Output with `arrow-array`/`arrow-ord`/`arrow-schema` `=59.3.0`:
```
type: Dictionary(Int32, Dictionary(UInt32, Utf8))
thread 'main' panicked at .../arrow-ord-59.3.0/src/cmp.rs:290:18:
internal error: entered unreachable code
```
Found through DataFusion, where `ScalarValue::partial_cmp` on a
`List<Dictionary<Dictionary<Utf8>>>` reaches this kernel
(apache/datafusion#24916).
**Expected behavior**
`Err(ArrowError::InvalidArgumentError(..))`, in line with the existing
`Nested comparison: ... (hint: use make_comparator instead)` error, or support
for the nested encoding.
**Additional context**
I am happy to send a PR that returns an error from `compare_op` when the
unwrapped type is still a `Dictionary` or `RunEndEncoded`, reusing
`supports_distinct`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
