andygrove opened a new pull request, #331:
URL: https://github.com/apache/datafusion-spark/pull/331

   Closes #140.
   
   ## Summary
   
   Adds Spark-compatible `array_max` and `array_min`, ported from Comet's
   `native/spark-expr/src/array_funcs/array_extrema.rs`.
   
   - NULL elements are skipped. A NULL array, an empty array, or an array of 
only NULL
     elements returns NULL.
   - Elements are compared with Spark's ordering:
     - NaN is greater than every other value, including infinity, and all NaNs 
are equal.
     - `-0.0` equals `0.0`. On a tie the first element wins and is returned 
unchanged, so
       `array_max(array(-0.0, 0.0))` is `-0.0` and `array_max(array(0.0, 
-0.0))` is `0.0`.
       The NaN payload bits are kept as well.
     - These float rules also apply inside nested arrays and structs.
     - Strings and binary compare bytewise.
     - Arrays compare element by element and then by length. Structs compare 
field by field.
       A NULL nested inside an array or struct sorts first.
   - Supported element types: integers, floats, decimals, booleans, strings, 
binary, dates,
     timestamps, and nested arrays and structs of these. Any other type that 
arrow's
     `make_comparator` can order is accepted too; interval elements are 
accepted but not tested
     here. `List` and `LargeList` are
     supported, and `FixedSizeList` is coerced to `List`.
   - An untyped `NULL` argument and `array()` are accepted and return a NULL of 
the NULL type,
     as in Spark (`void`).
   - Planning errors use Spark's error classes:
     - `INVALID_ORDERING_TYPE` for map elements at any depth.
     - `UNEXPECTED_INPUT_TYPE` for an argument that is not an array.
   - How it works: the element values are canonicalized with the existing 
`normalize_array`
     helper (NaN to the canonical NaN, `-0.0` to `0.0`, at any depth) and 
compared with arrow's
     `make_comparator`, with nulls first. The winner is then taken from the 
original,
     un-normalized values. `invoke_with_args` does the work itself and does not 
depend on
     `simplify`.
   - `contains_map` in `array_position.rs` is now `pub(super)` so that both 
functions can use it.
   - Both functions are documented with `#[user_doc]`.
   
   ## Differences from Comet's version
   
   - Comet wraps DataFusion's built-in `array_max`/`array_min`: it reuses their 
signature and
     return type, and hands non-float, non-nested element types to the 
built-in. This version
     has its own signature and `coerce_types`, and one code path for all 
element types.
   - Comet has a dedicated float scan and a hand-written `take` for nested 
values, which keeps
     child buffer capacity bounded when a large list loses. This version uses 
`normalize_array`
     plus `make_comparator` and arrow's `take`. It is simpler, but it may be 
slower on long float
     lists and can over-reserve child capacity for nested results. Both produce 
the same values.
   - Comet supports only 32-bit `List`. This version also supports `LargeList`, 
and coerces
     `FixedSizeList` to `List`.
   - Comet does not check types. This version rejects map elements and 
non-array arguments with
     Spark's error classes.
   
   ## Testing
   
   - New unit tests in `array_extrema.rs`, adapted from Comet's tests:
     - first-winner float semantics for f32 and f64: NaN payloads, signed 
zeros, infinities,
       long lists
     - scalar and sliced input
     - strings and binary
     - `LargeList`
     - nested lists and structs
     - NULL-typed input
     - type rejection
   - New `array/array_extrema.slt` (34 records). All 34 match Spark when run 
with the PySpark
     SLT runner on `pyspark==4.2.0`. I also checked the following directly in 
PySpark 4.2.0:
     - error messages for maps, non-arrays and the wrong number of arguments
     - the `void` result type for `NULL` and `array()`
     - that the zero sign is kept
   - The sign of `-0.0` is checked with `CAST(... AS STRING)`, because the test 
runner prints both
     zeros as `0`. `atan2` cannot be used for this: Spark adds `0.0` to its 
arguments, which turns
     `-0.0` into `0.0`.
   - Not covered in the `.slt` file, because of literal typing (#97):
     - `array_max(array(1.0, 2))`: Spark types the array as `DECIMAL(11,1)` and 
returns `2.0`,
       while DataFusion returns a `DOUBLE`.
   - Not tested:
     - Interval elements. Spark orders `DAY TO SECOND` intervals, and PySpark 
cannot collect
       `YEAR TO MONTH` intervals.
   - The `ArrayMax`/`ArrayMin` definitions in the Spark 3.5.8 and 4.1.1 sources 
have the same
     logic. I did not do a full multi-version audit.
   - Commands, all passing:
     - `cargo fmt --all`
     - `cargo clippy --all-targets --all-features -- -D warnings`
     - `cargo test -p datafusion-spark`: 345 passed
     - `cargo test -p datafusion-spark-sqllogictest`: 250 of 250 files passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to