andygrove opened a new issue, #213:
URL: https://github.com/apache/datafusion-spark/issues/213

   Part of #178 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `map_concat(map, ...)` returns the union of all its map arguments. It takes 
zero or more arguments, and every
   one must be a `MAP`. If the arguments have different map types, the analyzer 
casts them all to the wider common
   map type (`findWiderCommonType`). If no common type exists, analysis fails 
with
   `DATATYPE_MISMATCH.DATA_DIFF_TYPES`. A non-map argument fails with 
`DATATYPE_MISMATCH.MAP_CONCAT_DIFF_TYPES`.
   The result type is the merged map type, and `valueContainsNull` is true if 
it is true for any input.
   With no arguments the result is an empty `MAP<STRING, STRING>`.
   
   The result is NULL if any argument is NULL. It is nullable only when some 
child is nullable. Entries are added
   in argument order, then in each map's entry order, through 
`ArrayBasedMapBuilder`. A key keeps the position
   where it first appeared. Duplicate keys are governed by 
`spark.sql.mapKeyDedupPolicy` (default `EXCEPTION`).
   Under `LAST_WIN`, a later value replaces the earlier one but stays at the 
earlier key's position.
   
   The function exists since Spark 2.4.0 and has no aliases. DataFusion core 
has no `map_concat`.
   
   ### Examples
   
   ```sql
   SELECT map_concat(map(1, 'a', 2, 'b'), map(3, 'c'));
   -- {1:"a",2:"b",3:"c"}
   SELECT map_concat(map('a', '1', 'b', '2'), map('d', '4', 'e', '5'), map('f', 
'1', 'g', '2'));
   -- {"a":"1","b":"2","d":"4","e":"5","f":"1","g":"2"}
   SELECT map_concat(map('d', '4', 'e', '5'), map('a', NULL, 'c', '3'));
   -- {"d":"4","e":"5","a":null,"c":"3"}
   SELECT map_concat(map('a', '1', 'b', '2'), CAST(NULL AS MAP<STRING, 
STRING>));
   -- NULL
   SELECT map_concat();
   -- {}
   SELECT map_concat(map('a', '1', 'b', '2'), map('c', '3', 'a', '4'));
   -- error [DUPLICATED_MAP_KEY] Duplicate map key a was found, please check 
the input data.
   --   If you want to remove the duplicated keys, you can set 
"spark.sql.mapKeyDedupPolicy" to "LAST_WIN"
   --   so that the key inserted at last takes precedence.
   SET spark.sql.mapKeyDedupPolicy=LAST_WIN;
   SELECT map_concat(map('a', '1', 'b', '2'), map('c', '3', 'a', '4'));
   -- {"a":"4","b":"2","c":"3"}
   ```
   
   (Spark's `.sql.out` golden files sort map entries when they print them, so 
do not take entry order from them.
   The order shown above comes from `CollectionExpressionsSuite` and the 
`ArrayBasedMapBuilder` code.)
   
   ### Edge cases to test
   
   - A NULL map in any position (first, middle, last, or the only argument) 
gives NULL.
   - Zero arguments give an empty `MAP<STRING, STRING>`, and a single argument 
returns a copy of that map.
   - Empty maps mixed with non-empty ones.
   - NULL values are kept, and `valueContainsNull` of the result is the OR over 
all inputs.
   - Duplicate keys:
     - Under `EXCEPTION`, Spark raises `DUPLICATED_MAP_KEY` (SQLSTATE `23505`). 
This applies to a key repeated
       across arguments. A key repeated inside one input map can't occur, 
because the map builders that create
       maps already dedupe.
     - Under `LAST_WIN`, the key keeps its first position and takes the last 
value.
     - Include a key that appears in three maps.
   - Floating-point keys. From Spark 4.0, `ArrayBasedMapBuilder` normalizes 
`FLOAT`/`DOUBLE` keys before lookup
     (`-0.0` becomes `0.0`, and every NaN becomes the canonical NaN). The 
internal config
     `spark.sql.legacy.disableMapKeyNormalization` (default `false`) turns this 
off.
     - `map_concat(map(0.0D, 1), map(-0.0D, 2))` is therefore a duplicate key: 
an error under `EXCEPTION`, or
       `{0.0:2}` under `LAST_WIN`. A lone `-0.0` key is stored as `0.0`.
     - In 3.4/3.5 there is no normalization and keys are compared as boxed 
`java.lang.Double`. Two NaNs collide,
       but `-0.0` and `0.0` are distinct keys.
   - Struct and array keys. Spark finds duplicates with a `TreeMap` in 
interpreted ordering, so `-0.0` equals
     `0.0` inside nested keys, and NaNs are equal.
   - Collated string keys (Spark 4.0+): a non-binary collation such as 
`UTF8_LCASE` compares keys through a
     `TreeMap`, so `'a'` and `'A'` are duplicates. Supporting only 
`UTF8_BINARY` is fine, but document it.
   - Binary keys are compared by content.
   - Type coercion (`typeCoercion/native/mapconcat.sql`):
     - `map<tinyint,tinyint>` with `map<smallint,smallint>` widens.
     - `map<string,string>` with `map<date,date>` fails with `DATA_DIFF_TYPES`.
     - A non-map argument (`map_concat(m, 12)`) fails with 
`MAP_CONCAT_DIFF_TYPES`.
   - The total entry count above `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH` 
raises
     `COLLECTION_SIZE_LIMIT_EXCEEDED`.
   - Inputs can be literals, columns, or a mix, with per-row NULLs in a column 
batch.
   - Sliced `MapArray` inputs, where offsets do not start at 0.
   
   ### Differences across Spark versions
   
   - 3.4 and 3.5: no float-key normalization, so `-0.0` and `0.0` are distinct 
keys.
   - 4.0: float-key normalization arrives, with the legacy config 
`spark.sql.legacy.disableMapKeyNormalization`.
     Collated string keys become possible, and `stateful = true` is added 
(planner-only).
   - 4.1 and 4.2: no change to `MapConcat` itself.
   
   ### Implementation notes
   
   - Spark implements this in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala`
     (`MapConcat`) and 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ArrayBasedMapBuilder.scala`.
     Coercion lives in `TypeCoercionHelper.scala`.
   - DataFusion core has no equivalent, so this needs a new kernel.
   - datafusion-spark already has the dedup logic in 
`crates/spark/src/function/map/utils.rs`:
     `map_from_keys_values_offsets_nulls` / `map_deduplicate_keys`. It keeps 
the first position and overwrites
     the value under `LAST_WIN`, and it is wired to 
`datafusion.spark.map_key_dedup_policy`. Concatenating each
     row's key and value slices and passing them through that helper should 
give Spark's ordering and errors.
     Before hashing, normalize float keys (including floats nested inside 
struct and array keys) to match 4.0+,
     because `ScalarValue` equality is bitwise.
   - The function belongs in `crates/spark/src/function/map/`, next to 
`map_from_arrays` and `map_from_entries`.
   - Add `crates/sqllogictest/test_files/map/map_concat.slt`, including `set 
datafusion.spark.map_key_dedup_policy`
     cases as in `map_from_entries.slt`, and document the function with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to