andygrove opened a new issue, #221:
URL: https://github.com/apache/datafusion-spark/issues/221

   Part of #178 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `array_distinct(array)` returns the array with duplicate values removed, 
keeping the first occurrence of each value in input order. At most one NULL 
element is kept, at the position of the first NULL.
   
   The argument must be an `ARRAY` (`ExpectsInputTypes`, `inputTypes = 
Seq(ArrayType)`). The element type must be orderable 
(`TypeUtils.checkForOrderingExpr`), so `MAP`, `VARIANT` and interval-calendar 
elements are rejected with `DATATYPE_MISMATCH.INVALID_ORDERING_TYPE`. The 
return type is the input type unchanged, including `containsNull`. The function 
is null-intolerant: a NULL array gives NULL. Available since 2.4.0, no aliases.
   
   ### Examples
   
   ```sql
   SELECT array_distinct(array(1, 2, 3, null, 3));
   -- [1,2,3,null]
   SELECT array_distinct(array(2, 1, 2, 3, 4, 4, 5));
   -- [2,1,3,4,5]
   SELECT array_distinct(array('b', null, 'a', null, 'a', null));
   -- ["b",null,"a"]
   SELECT array_distinct(array(null, null, null));
   -- [null]
   SELECT array_distinct(CAST(array() AS ARRAY<INT>));
   -- []
   SELECT array_distinct(array(array(1, 2), array(3, 4), array(1, 2)));
   -- [[1,2],[3,4]]
   -- Note: 0.0 and -0.0 here are DECIMAL literals, so -0.0 is already 0 before 
the cast to DOUBLE.
   SELECT array_distinct(array(0.0, -0.0, -0.0, DOUBLE("NaN"), DOUBLE("NaN")));
   -- [0.0,NaN]
   -- Spark 4.2.0 (SPARK-54918):
   SELECT array_distinct(array(-0.0D, 0.0D));
   -- [0.0]
   -- Spark 3.5.x to 4.1.3: the same query returns [-0.0,0.0]
   ```
   
   ### Edge cases to test
   
   - NULL array (column and `CAST(NULL AS ARRAY<INT>)`) returns NULL. Empty 
array returns empty.
   - Several NULL elements: only the first is kept, in place (`["b",null,"a"]`).
   - `DOUBLE`/`FLOAT` NaN: all NaNs are one value on every version, including 
NaNs with different bit patterns such as `double('NaN')` and `-double('NaN')`. 
The first NaN is kept.
   - Signed zero in a flat `DOUBLE`/`FLOAT` array: Spark 4.2.0 treats `-0.0` 
and `0.0` as one value and outputs `+0.0`, even for a single `-0.0` 
(`array_distinct(array(-0.0D))` returns `[0.0]`). Earlier versions keep both 
and leave the sign as is (see below).
   - Floats nested in `STRUCT` or `ARRAY` elements: Spark compares these with 
`ordering.equiv` (`SQLOrderingUtil`), so `-0.0 = 0.0` and NaN = NaN on every 
version. Before 4.2 the first one seen is kept with its original sign. 4.2.0 
also normalizes nested floats to `+0.0`/canonical NaN.
   - `BINARY` elements are compared by content (`[[5,6],[1,2],[1,2],[5,6]]` 
gives `[[5,6],[1,2]]`), and so are `ARRAY` and `STRUCT` elements.
   - Strings with a non-`UTF8_BINARY` collation (4.0+) are compared under that 
collation. For example, `'aaa'` and `'AAA'` are one value under `UTF8_LCASE`.
   - `BOOLEAN`, `TINYINT`, `SMALLINT`, `DECIMAL`, `DATE`, `TIMESTAMP` elements.
   - Scalar (literal) argument as well as column argument. Sliced list arrays.
   - Result with more than `MAX_ROUNDED_ARRAY_LENGTH` elements raises 
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` (4.0+). This cannot normally be 
reached because the input is already bounded.
   
   ### Differences across Spark versions
   
   - Signed zero in flat float arrays:
     - 3.4: `OpenHashSet` compares with `==` but hashes by bits, so whether 
`-0.0` and `0.0` are merged depends on hash collisions. In practice they 
usually stay distinct.
     - 3.5.x to 4.1.3: `OpenHashSet` uses `equals` (SPARK-45599), so they 
always stay distinct.
     - 4.2.0: `NormalizeFloatingNumbers` wraps the argument in a normalizer 
(SPARK-54918), so `-0.0` becomes `0.0` and every NaN becomes the canonical NaN, 
at any depth.
     - Comet's docs say SPARK-59602 moves that normalization into evaluation 
for 4.0.5, 4.1.4 and 4.2.1. Those releases are not in the source trees checked 
here.
   - 4.0: `NullIntolerant` became the `nullIntolerant` flag (no behavior 
change). The size-limit error changed from `_LEGACY_ERROR_TEMP_2162` to 
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`. Collation-aware string comparison 
was added.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`,
 `ArrayDistinct` (with `trait ArraySetLike`). Atomic types with proper `equals` 
use `SQLOpenHashSet` 
(`sql/catalyst/src/main/scala/org/apache/spark/sql/util/SQLOpenHashSet.scala`), 
which tracks NULL and NaN separately. Other types (`BINARY`, `STRUCT`, `ARRAY`, 
collated strings) use a nested loop with `ordering.equiv`. For the 4.2 
normalization, see 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NormalizeFloatingNumbers.scala`.
   - DataFusion core has `array_distinct` 
(`datafusion/functions-nested/src/set_ops.rs`, `general_array_distinct`). It 
already matches Spark on first-occurrence order, a single NULL kept in place, 
NULL array giving NULL, and the return type equal to the input type. It also 
turns `-0.0` into `+0.0` at any depth (`normalize_float_zero`), which matches 
Spark 4.2.0. It differs in two ways:
     - Values are compared by their arrow-row bytes, and `normalize_float_zero` 
keeps NaN payloads. So NaNs with different bit patterns (for example `NaN` and 
`-NaN`) stay distinct, while Spark merges them on every version.
     - It cannot reproduce the pre-4.2 behavior, where flat arrays keep `-0.0` 
and `0.0` distinct and the sign survives.
     A Spark version could wrap core's function after canonicalizing NaN (and 
`-0.0` for 4.2 semantics) in the input.
   - datafusion-spark's `array_contains` 
(`crates/spark/src/function/array/array_contains.rs`) has a private 
`normalize_floats`/`normalize_array` that rewrites `-0.0` to `0.0` and every 
NaN to the canonical NaN through lists and structs (see 
apache/datafusion-spark#34). Move it into a shared helper (for example next to 
`functions_nested_utils.rs`) and reuse it here and in `array_union`, 
`array_intersect`, `array_except` and `arrays_overlap`.
   - Put the function in `crates/spark/src/function/array/`. Add 
`crates/sqllogictest/test_files/array/array_distinct.slt` following existing 
files such as `array_contains.slt`, and document it with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to