andygrove opened a new issue, #252:
URL: https://github.com/apache/datafusion-spark/issues/252
Part of #176 and apache/datafusion-comet#7300.
### What the function does
`map_zip_with(map1, map2, func)` merges two maps into one by calling
`func(k, v1, v2)` for every key in the union of
their keys. For a key present in only one map, NULL is passed for the
missing side. If an input map contains
duplicated keys, only the first entry of each key is used. The result type is
`MAP<keyType, func.dataType>` with `valueContainsNull = func.nullable`. The
result is NULL if either map is NULL.
Spark evaluates `map1` first and skips `map2` if `map1` is NULL. The lambda
parameters are `k` (non-null) and `v1`,
`v2` (always nullable). Non-lambda `func` arguments are wrapped in a hidden
lambda. Available since Spark 3.0.0. No
aliases.
Key types. If the two key types differ, the `MapZipWithCoercion` rule casts
both maps to the wider key type
(`findWiderTypeForTwo`), unless the cast would force nullability. If no
common type exists, analysis fails with
`DATATYPE_MISMATCH.MAP_ZIP_WITH_DIFF_TYPES`, for example `INT` against
`MAP<INT, INT>`. The key type must be
orderable.
Output order. Keys from `map1` come first in their original order, followed
by keys that appear only in `map2`, in
`map2` order. Spark builds a `LinkedHashMap` for key types with proper
equality, and does a linear search with
interpreted ordering otherwise (binary, struct and array keys). No dedup
policy is applied to the result.
### Examples
```sql
SELECT map_zip_with(map(1, 'a', 2, 'b'), map(1, 'x', 2, 'y'), (k, v1, v2) ->
concat(v1, v2));
-- {1:"ax",2:"by"}
SELECT map_zip_with(map('a', 1, 'b', 2), map('b', 3, 'c', 4), (k, v1, v2) ->
coalesce(v1, 0) + coalesce(v2, 0));
-- {"a":1,"b":5,"c":4}
SELECT map_zip_with(map('a', 1), map('a', 10), 100);
-- {"a":100}
SELECT map_zip_with(map('a', 1), map('a', 10), struct('left', 1, 'right',
10));
-- {"a":{"col1":"left","col2":1,"col3":"right","col4":10}}
```
These come from the `ExpressionDescription` and
`higher-order-functions.sql.out`. `HigherOrderFunctionsSuite`, with
`(k, v1, v2) -> k * v1 * v2`, adds:
| map1 | map2 | result |
|---|---|---|
| `{1:10, 2:20, 3:30}` | `{1:-1, 2:-2, 4:-4}` | `{1:-10, 2:-80, 3:null,
4:null}` |
| `{1:10, 2:20, 3:30}` | `{1:null, 2:-2, 3:null}` | `{1:null, 2:-80,
3:null}` |
| `{1:10, 2:20, 3:30}` | `{}` | `{1:null, 2:null, 3:null}` |
| `{1:10, 2:20, 3:30}` | NULL | NULL |
With `concat(k, v1, v2)` on strings, `{"d":"b","b":"d"}` and
`{"c":null,"b":"t","a":null}` give
`{"d":null,"b":"bdt","c":null,"a":null}`. The same test also covers binary
keys, which use the linear-search path.
### Edge cases to test
- Either map NULL gives NULL. Both maps empty gives `{}`. One map empty
gives every key of the other, with NULL for
the missing side.
- Key-only-in-one-side handling, NULL values that are actually present
(indistinguishable from missing values inside
the lambda), and the output order described above.
- Duplicate keys inside an input map. Only the first occurrence is used, and
later occurrences are ignored rather
than raising `DUPLICATED_MAP_KEY`. Normal map construction forbids
duplicates, but Arrow input can contain them.
- Floating-point keys. With `spark.sql.mapZipWithUsesJavaCollections=true`
(internal, default true, 4.1+), keys are
normalized with `NormalizeFloatingNumbers`, so `NaN` matches `NaN` and
`-0.0` matches `0.0`. 3.4 through 4.0 use a
Scala `LinkedHashMap` with no normalization, and the config's doc says the
Java path exists to avoid "NaN equality
issues". Test `NaN` and `-0.0` keys and expect the older versions to
differ, at least for `NaN`.
- Key type coercion: `map(1, ...)` (INT) against `map(1L, ...)` (BIGINT)
gives BIGINT keys. Also test incompatible key
types (`MAP_ZIP_WITH_DIFF_TYPES`) and non-orderable keys.
- Different value types on the two sides, for example `MAP<STRING, INT>` and
`MAP<STRING, STRING>`.
- Lambda arity: exactly 3 parameters. `(x, y) -> ...` fails with
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`.
- Captured outer columns, nondeterministic bodies (once per output key), and
ANSI errors inside the body.
- Very large maps: `mapSizeExceedArraySizeWhenZipMapError` on the
brute-force path.
### Differences across Spark versions
- 4.1.0 adds `spark.sql.mapZipWithUsesJavaCollections` (internal, default
`true`). The fast path then uses a Java
`LinkedHashMap` with float/double key normalization, which fixes `NaN`
equality. 3.4, 3.5 and 4.0 use a Scala
`LinkedHashMap` without normalization.
- 4.0 changed the key-type comparison to `DataTypeUtils.sameType`. That is
an internal refactor.
### Implementation notes
- Spark: `MapZipWith` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`
(`getKeysWithIndexesFastAsJava`, `getKeysWithIndexesFastUsingScala`,
`getKeysWithIndexesBruteForce`). Coercion
is `MapZipWithTypeCoercion` in
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala`.
- Infrastructure. DataFusion's higher-order function framework
(`HigherOrderUDFImpl` in
`datafusion/expr/src/higher_order_function.rs`, and
`LambdaArgument::evaluate` with `spread_captures` for captured
columns) is on `branch-56`, which this crate uses. `lambda_parameters`
should offer
`[key (non-null), v1 (nullable), v2 (nullable)]`. Key coercion has to
happen in `coerce_value_types`.
- DataFusion core has no equivalent. Implementation outline:
1. For each row, compute the key union and two index vectors into the
value arrays, with NULL for a missing key.
Use `arrow_row` or a hash on normalized keys.
2. `take` the keys and both value arrays to get three aligned flattened
columns.
3. Evaluate the lambda once over those columns.
4. Rebuild the `MapArray`.
- This crate's `crates/spark/src/function/lambda/mod.rs` is an empty
`ScalarUDF` placeholder. It needs a
`HigherOrderUDF` list and registration through
`register_higher_order_function` /
`SessionStateBuilder::higher_order_functions`.
`crates/spark/src/function/map/utils.rs` has `MapArray` helpers.
- Add `crates/sqllogictest/test_files/lambda/map_zip_with.slt` and document
the function with `#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]