andygrove opened a new issue, #221:
URL: https://github.com/apache/datafusion-spark/issues/221
Part of #178 and apache/datafusion-comet#7300.
### What the function does
`array_distinct(array)` returns the array with duplicate values removed,
keeping the first occurrence of each value in input order. At most one NULL
element is kept, at the position of the first NULL.
The argument must be an `ARRAY` (`ExpectsInputTypes`, `inputTypes =
Seq(ArrayType)`). The element type must be orderable
(`TypeUtils.checkForOrderingExpr`), so `MAP`, `VARIANT` and interval-calendar
elements are rejected with `DATATYPE_MISMATCH.INVALID_ORDERING_TYPE`. The
return type is the input type unchanged, including `containsNull`. The function
is null-intolerant: a NULL array gives NULL. Available since 2.4.0, no aliases.
### Examples
```sql
SELECT array_distinct(array(1, 2, 3, null, 3));
-- [1,2,3,null]
SELECT array_distinct(array(2, 1, 2, 3, 4, 4, 5));
-- [2,1,3,4,5]
SELECT array_distinct(array('b', null, 'a', null, 'a', null));
-- ["b",null,"a"]
SELECT array_distinct(array(null, null, null));
-- [null]
SELECT array_distinct(CAST(array() AS ARRAY<INT>));
-- []
SELECT array_distinct(array(array(1, 2), array(3, 4), array(1, 2)));
-- [[1,2],[3,4]]
-- Note: 0.0 and -0.0 here are DECIMAL literals, so -0.0 is already 0 before
the cast to DOUBLE.
SELECT array_distinct(array(0.0, -0.0, -0.0, DOUBLE("NaN"), DOUBLE("NaN")));
-- [0.0,NaN]
-- Spark 4.2.0 (SPARK-54918):
SELECT array_distinct(array(-0.0D, 0.0D));
-- [0.0]
-- Spark 3.5.x to 4.1.3: the same query returns [-0.0,0.0]
```
### Edge cases to test
- NULL array (column and `CAST(NULL AS ARRAY<INT>)`) returns NULL. Empty
array returns empty.
- Several NULL elements: only the first is kept, in place (`["b",null,"a"]`).
- `DOUBLE`/`FLOAT` NaN: all NaNs are one value on every version, including
NaNs with different bit patterns such as `double('NaN')` and `-double('NaN')`.
The first NaN is kept.
- Signed zero in a flat `DOUBLE`/`FLOAT` array: Spark 4.2.0 treats `-0.0`
and `0.0` as one value and outputs `+0.0`, even for a single `-0.0`
(`array_distinct(array(-0.0D))` returns `[0.0]`). Earlier versions keep both
and leave the sign as is (see below).
- Floats nested in `STRUCT` or `ARRAY` elements: Spark compares these with
`ordering.equiv` (`SQLOrderingUtil`), so `-0.0 = 0.0` and NaN = NaN on every
version. Before 4.2 the first one seen is kept with its original sign. 4.2.0
also normalizes nested floats to `+0.0`/canonical NaN.
- `BINARY` elements are compared by content (`[[5,6],[1,2],[1,2],[5,6]]`
gives `[[5,6],[1,2]]`), and so are `ARRAY` and `STRUCT` elements.
- Strings with a non-`UTF8_BINARY` collation (4.0+) are compared under that
collation. For example, `'aaa'` and `'AAA'` are one value under `UTF8_LCASE`.
- `BOOLEAN`, `TINYINT`, `SMALLINT`, `DECIMAL`, `DATE`, `TIMESTAMP` elements.
- Scalar (literal) argument as well as column argument. Sliced list arrays.
- Result with more than `MAX_ROUNDED_ARRAY_LENGTH` elements raises
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION` (4.0+). This cannot normally be
reached because the input is already bounded.
### Differences across Spark versions
- Signed zero in flat float arrays:
- 3.4: `OpenHashSet` compares with `==` but hashes by bits, so whether
`-0.0` and `0.0` are merged depends on hash collisions. In practice they
usually stay distinct.
- 3.5.x to 4.1.3: `OpenHashSet` uses `equals` (SPARK-45599), so they
always stay distinct.
- 4.2.0: `NormalizeFloatingNumbers` wraps the argument in a normalizer
(SPARK-54918), so `-0.0` becomes `0.0` and every NaN becomes the canonical NaN,
at any depth.
- Comet's docs say SPARK-59602 moves that normalization into evaluation
for 4.0.5, 4.1.4 and 4.2.1. Those releases are not in the source trees checked
here.
- 4.0: `NullIntolerant` became the `nullIntolerant` flag (no behavior
change). The size-limit error changed from `_LEGACY_ERROR_TEMP_2162` to
`COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION`. Collation-aware string comparison
was added.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`,
`ArrayDistinct` (with `trait ArraySetLike`). Atomic types with proper `equals`
use `SQLOpenHashSet`
(`sql/catalyst/src/main/scala/org/apache/spark/sql/util/SQLOpenHashSet.scala`),
which tracks NULL and NaN separately. Other types (`BINARY`, `STRUCT`, `ARRAY`,
collated strings) use a nested loop with `ordering.equiv`. For the 4.2
normalization, see
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NormalizeFloatingNumbers.scala`.
- DataFusion core has `array_distinct`
(`datafusion/functions-nested/src/set_ops.rs`, `general_array_distinct`). It
already matches Spark on first-occurrence order, a single NULL kept in place,
NULL array giving NULL, and the return type equal to the input type. It also
turns `-0.0` into `+0.0` at any depth (`normalize_float_zero`), which matches
Spark 4.2.0. It differs in two ways:
- Values are compared by their arrow-row bytes, and `normalize_float_zero`
keeps NaN payloads. So NaNs with different bit patterns (for example `NaN` and
`-NaN`) stay distinct, while Spark merges them on every version.
- It cannot reproduce the pre-4.2 behavior, where flat arrays keep `-0.0`
and `0.0` distinct and the sign survives.
A Spark version could wrap core's function after canonicalizing NaN (and
`-0.0` for 4.2 semantics) in the input.
- datafusion-spark's `array_contains`
(`crates/spark/src/function/array/array_contains.rs`) has a private
`normalize_floats`/`normalize_array` that rewrites `-0.0` to `0.0` and every
NaN to the canonical NaN through lists and structs (see
apache/datafusion-spark#34). Move it into a shared helper (for example next to
`functions_nested_utils.rs`) and reuse it here and in `array_union`,
`array_intersect`, `array_except` and `arrays_overlap`.
- Put the function in `crates/spark/src/function/array/`. Add
`crates/sqllogictest/test_files/array/array_distinct.slt` following existing
files such as `array_contains.slt`, and document it with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]