andygrove opened a new issue, #254:
URL: https://github.com/apache/datafusion-spark/issues/254

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `transform_values(expr, func)` returns a map with the same keys, in the same 
order, whose values are `func(k, v)`
   for each entry of the map `expr`. The result type is `MAP<keyType, 
func.dataType>` with
   `valueContainsNull = func.nullable`. A NULL map gives NULL, and the result 
is nullable if the input is. The lambda
   takes exactly 2 parameters: `k` (non-null) and `v` (nullable when the 
input's `valueContainsNull` is true). `expr`
   must be a `MAP` (`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` otherwise). A 
non-lambda `func` is wrapped in a hidden
   lambda. Available since Spark 3.0.0. No aliases.
   
   Keys are not modified, so there is no dedup policy and no `NULL_MAP_KEY` 
check. Spark reuses the input key array
   directly (`new ArrayBasedMapData(map.keyArray(), resultValues)`).
   
   ### Examples
   
   ```sql
   SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, 
v) -> v + 1);
   -- {1:2,2:3,3:4}
   SELECT transform_values(map_from_arrays(array(1, 2, 3), array(1, 2, 3)), (k, 
v) -> k + v);
   -- {1:2,2:4,3:6}
   -- nested view: (1, map(1,1,2,2,3,3)), (2, map(4,4,5,5,6,6)) as t(x, ys)
   SELECT transform_values(ys, (k, v) -> k + v) FROM nested;
   -- {1:2,2:4,3:6} / {4:8,5:10,6:12}
   SELECT transform_values(map('a', 1, 'b', 2), 999);
   -- {"a":999,"b":999}
   SELECT transform_values(map('a', 1, 'b', 2), struct('val', 999));
   -- {"a":{"col1":"val","col2":999},"b":{"col1":"val","col2":999}}
   ```
   
   These come from the `ExpressionDescription` and 
`higher-order-functions.sql.out`. `HigherOrderFunctionsSuite` adds:
   
   | map | lambda | result |
   |---|---|---|
   | `{1:1, 2:null, 3:3}` | `v + 1` | `{1:2, 2:null, 3:4}` |
   | `{1:1, 2:null, 3:3}` | `k * k` | `{1:1, 2:4, 3:9}` |
   | `{"a":"xy", "bb":"yz", "ccc":"zx"}` | `length(v) + 1` | `{"a":3, "bb":3, 
"ccc":3}` |
   | `{"a":"xy", "bb":null, "ccc":"zx"}` | `concat(k, v)` | `{"a":"axy", 
"bb":null, "ccc":"ccczx"}` |
   | `{}` | any | `{}` |
   | NULL | any | NULL |
   
   ### Edge cases to test
   
   - A NULL map gives NULL. An empty map gives `{}`, also when the value type 
changes (`MAP<STRING, INT>` from
     `length(v) + 1`).
   - NULL values passed to the lambda: `v + 1` gives NULL there, while `k * k` 
ignores `v` and gives a value. The result
     `valueContainsNull` follows the lambda's nullability.
   - Value-type changes (string to int, int to struct, int to map). Chained 
calls:
     `transform_values(transform_values(m, (k, v) -> concat(k, v)), (k, v) -> 
concat(k, v))` on
     `{"a":"xy","bb":null,"ccc":"zx"}` gives 
`{"a":"aaxy","bb":null,"ccc":"cccccczx"}`.
   - Key order and key values are preserved exactly, including float keys such 
as `-0.0` and `NaN`, which are not
     re-normalized. Use the input key array as it is.
   - Lambda arity: `v -> v + 1` and `(k, v, x) -> ...` must fail with 
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`.
     DataFusion's planner allows a lambda to declare fewer parameters than 
offered, so this check has to be explicit.
   - Captured outer columns, nondeterministic value expressions (once per 
entry), and ANSI errors inside the lambda
     (`v / 0`, overflow).
   - Sliced `MapArray` input and values hidden under NULL map rows. The lambda 
must not be evaluated on them, because
     they could raise errors Spark never raises.
   
   ### Differences across Spark versions
   
   - None found in `TransformValues` between 3.4 and 4.2. The literal-argument 
cases in the golden file were added in
     4.1, which only adds tests. ANSI mode is on by default from 4.0.
   
   ### Implementation notes
   
   - Spark: `TransformValues` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
   - Infrastructure. DataFusion's higher-order function framework 
(`HigherOrderUDFImpl` in
     `datafusion/expr/src/higher_order_function.rs`, and 
`LambdaArgument::evaluate` with `spread_captures` for captured
     columns) is on `branch-56`, which this crate uses. `lambda_parameters` 
should offer `[key (non-null), value]`.
   - DataFusion core has no higher-order map functions. This is the simplest of 
the map functions to build:
     1. Evaluate the lambda once over the flattened entry keys and values, 
spreading captures with the per-entry row
        index, as `array_transform` does in 
`datafusion/functions-nested/src/array_transform.rs`.
     2. Rebuild the `MapArray` with the original offsets, nulls and keys, and 
the new values.
   - This crate's `crates/spark/src/function/lambda/mod.rs` is an empty 
`ScalarUDF` placeholder. It needs a
     `HigherOrderUDF` list and registration through 
`register_higher_order_function` /
     `SessionStateBuilder::higher_order_functions`. 
`crates/spark/src/function/map/utils.rs` has `MapArray` helpers.
   - Add `crates/sqllogictest/test_files/lambda/transform_values.slt` and 
document the function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to