andygrove opened a new issue, #216:
URL: https://github.com/apache/datafusion-spark/issues/216

   Part of #178 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `array_join(array, delimiter[, nullReplacement])` joins the elements of a 
string array with `delimiter`. With
   no `nullReplacement`, NULL elements are skipped and no delimiter is written 
for them. With one, every NULL
   element is written as `nullReplacement`. The result is a STRING (in 4.0+, 
the collation of the array's element
   type).
   
   The result is NULL if `array` or `delimiter` is NULL. It is also NULL if 
`nullReplacement` is given and NULL,
   even when the array has no NULL elements. Arguments are evaluated in order 
and evaluation stops at the first
   NULL: array, then delimiter, then replacement. The result is nullable if any 
argument is nullable.
   
   `array` must be `ARRAY<STRING>`. A dedicated coercion rule in 
`TypeCoercionHelper` casts any other `ARRAY<T>` to
   `ARRAY<STRING>` (keeping `containsNull`) when `T` can be implicitly cast to 
STRING. `delimiter` and
   `nullReplacement` use `ExpectsInputTypes` and are not cast, so 
`array_join(x, 1)` fails with
   `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` (requiredType `"STRING"`).
   
   The function exists since Spark 2.4.0 and has no aliases.
   
   ### Examples
   
   ```sql
   SELECT array_join(array('hello', 'world'), ' ');
   -- hello world
   SELECT array_join(array('hello', null ,'world'), ' ');
   -- hello world
   SELECT array_join(array('hello', null ,'world'), ' ', ',');
   -- hello , world
   SELECT array_join(array(2.0D, 1.0D), ', ');
   -- 2.0, 1.0
   SELECT array_join(array(true, false), ', ');
   -- true, false
   SELECT array_join(array(timestamp '2016-11-15 20:54:00.000', timestamp 
'2016-11-12 20:54:00.000'), ', ');
   -- 2016-11-15 20:54:00, 2016-11-12 20:54:00
   SELECT array_join(array(9223372036854775809, 9223372036854775808), ', ');
   -- 9223372036854775809, 9223372036854775808
   ```
   
   From `CollectionExpressionsSuite`, with delimiter `','`, first without 
`nullReplacement` and then with
   `nullReplacement` `'NULL'`:
   
   | input             | no replacement | replacement `'NULL'` |
   |-------------------|----------------|----------------------|
   | `['a','b']`       | `a,b`          | `a,b`                |
   | `['a',null,'b']`  | `a,b`          | `a,NULL,b`           |
   | `[null]`          | `''`           | `NULL`               |
   | `['a','b',null]`  | `a,b`          | `a,b,NULL`           |
   | `[null,'a','b']`  | `a,b`          | `NULL,a,b`           |
   | `[]`              | `''`           | `''`                 |
   
   The following inputs return NULL:
   - `array_join(NULL, ',')`
   - `array_join(array(NULL), NULL)`
   - `array_join(array(NULL), ',', NULL)`
   
   ### Edge cases to test
   
   - A NULL array, a NULL delimiter, and a NULL `nullReplacement` each give 
NULL. A NULL `nullReplacement` gives
     NULL even for `array('a')`, which has no NULL elements.
   - An empty array gives `''`, not NULL. An array of only NULLs gives `''` 
without a replacement.
   - Leading, trailing and consecutive NULLs, with and without a replacement. 
No delimiter may be emitted for a
     skipped NULL.
   - An empty delimiter, a multi-character delimiter, an empty 
`nullReplacement`, and multi-byte UTF-8 delimiters.
   - Non-string element types go through Spark's CAST to STRING, so the output 
must match Spark's formatting:
     - DOUBLE/FLOAT print as `2.0` and `1.0E10`, NaN as `NaN`, and Infinity as 
`Infinity`.
     - DECIMAL keeps its scale.
     - DATE and TIMESTAMP output depends on `spark.sql.session.timeZone`.
     - BINARY elements are cast to STRING.
   - Nested arrays (`ARRAY<ARRAY<INT>>`) fail analysis with 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`. Neither
     `TypeCoercion` nor `AnsiTypeCoercion` implicitly casts an ARRAY element to 
STRING, so the coercion rule leaves
     the argument unchanged. DataFusion's `array_to_string` instead flattens 
nested lists recursively, and the
     kernel must not inherit that.
   - `delimiter` and `nullReplacement` may each be a literal or a column. A 
per-row NULL in a column gives NULL
     only for that row.
   - Collations (4.0+): the result type carries the element type's collation, 
and trim collations are accepted.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5: the input types are `ArrayType(StringType)` and `StringType`, 
and the result is a plain `StringType`.
   - 4.0: the input types become 
`AbstractArrayType(StringTypeWithCollation(supportsTrimCollation = true))`, and
     `dataType` is the array's element string type, so the collation is 
propagated.
   - 4.1: adds `contextIndependentFoldable` (planner-only).
   - 4.2: fixes a codegen bug where a non-nullable array and delimiter with a 
nullable `nullReplacement` returned
     NULL. The interpreted path was already correct. 4.2 also marks the 
expression `expensive`.
   - The result semantics are otherwise unchanged.
   
   ### Implementation notes
   
   - Spark implements this in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`
     (`ArrayJoin`). Coercion lives in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala`.
   - DataFusion core's `array_to_string` (aliases `array_join`, `list_join`, 
`list_to_string`) is in
     `datafusion/functions-nested/src/string.rs`. It differs from Spark as 
follows:
     - A NULL `null_string` means "skip NULL elements" instead of returning 
NULL (apache/datafusion-comet#3178).
     - It recurses into nested lists and flattens them.
     - It formats floats with Rust `{}`, so `2.0` prints as `2`. Other 
non-primitive types go through the Arrow
       cast to `Utf8`, whose timestamp and decimal formatting is not Spark's.
     - It always returns `Utf8`.
   - The empty-array, NULL-array, NULL-delimiter and NULL-element-skipping 
behaviour already matches Spark.
   - A Spark version can do the element-to-string conversion in `coerce_types` 
with Spark's cast (see
     `crates/spark/src/function/conversion/cast.rs`), and then either:
     - call `array_to_string` after a NULL check on the replacement, or
     - use a small dedicated kernel over `ARRAY<STRING>`, which avoids the 
recursive flattening and makes nested
       arrays a planning error.
   - The function belongs in `crates/spark/src/function/array/`. Add
     `crates/sqllogictest/test_files/array/array_join.slt` and document it with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to