andygrove opened a new issue, #5101: URL: https://github.com/apache/datafusion-comet/issues/5101
### What is the problem the feature request solves? Two array functions hand-roll nested element comparison that arrow's `make_comparator` provides: - `native/spark-expr/src/array_funcs/arrays_overlap.rs` (`structural_eq` / `list_structural_eq` / `struct_structural_eq`): recursive element-by-element structural equality for nested List/LargeList/FixedSizeList/Struct elements with null==null treated as equal, comparing leaves via a per-element one-row `eq` kernel call (a kernel dispatch per element pair). - `native/spark-expr/src/array_funcs/array_position.rs` (`position_fallback`): materializes a `ScalarValue` per candidate element inside the inner loop (an allocation per element) and compares with `ScalarValue::eq`. Arrow's `make_comparator` (arrow-ord) supports fully nested types, treats null==null as Equal, and is built once per array pair, replacing both the recursion and the per-element dispatch/allocation. ### Describe the potential solution Build one comparator per (probe, search) array pair with `SortOptions::default()`, then compare indices with `cmp(i, j) == Ordering::Equal` per element. ### Additional context Float caveat: the current `arrays_overlap` leaf comparison uses the `eq` kernel (IEEE: NaN != NaN, -0.0 == 0.0), while `make_comparator` uses total order (NaN == NaN, -0.0 < 0.0). Spark's `ordering.equiv` (Java `Double.compare`) matches total order, so the comparator is arguably more Spark-correct, but it is a behavior change for NaN/-0.0 inside nested elements and needs test coverage before switching. For `array_position`, `ScalarValue` equality already matches total-order semantics, so that switch is behavior-preserving. The typed fast paths in both files are not affected; there is no "search within list segments" kernel. Found during an audit of native code that replicates existing arrow-rs kernels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
