andygrove opened a new issue, #248:
URL: https://github.com/apache/datafusion-spark/issues/248

   Part of #176 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `exists(expr, pred)` returns whether `pred` holds for at least one element 
of the array `expr`. `pred` must be a
   one-argument lambda returning `BOOLEAN`. A non-boolean body fails with 
`DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE`
   (`requiredType "BOOLEAN"`), and so does a non-lambda literal such as 
`exists(array(1, 2, 3), 1)`. There is no index
   argument, so `(x, i) -> ...` fails with 
`INVALID_LAMBDA_FUNCTION_CALL.NUM_ARGS_MISMATCH`. The result type is
   `BOOLEAN`. Available since Spark 2.4.0. No aliases.
   
   Semantics (`ArrayExists`, with `followThreeValuedLogic` captured from
   `spark.sql.legacy.followThreeValuedLogicInArrayExists`, default `true`):
   
   - A NULL array gives NULL.
   - Elements are visited in order, and evaluation **stops at the first 
`true`**.
   - If any element gives `true`, the result is `true`.
   - Otherwise, if any element gave NULL and three-valued logic is on, the 
result is NULL.
   - Otherwise the result is `false`. An empty array gives `false`.
   
   With the legacy config set to `false`, NULL predicate results count as 
`false`, and the result is NULL only for a
   NULL array. Nullability is `argument.nullable || function.nullable` with 
three-valued logic, and only
   `argument.nullable` without it.
   
   ### Examples
   
   ```sql
   SELECT exists(array(1, 2, 3), x -> x % 2 == 0);
   -- true
   SELECT exists(array(1, 2, 3), x -> x % 2 == 10);
   -- false
   SELECT exists(array(1, null, 3), x -> x % 2 == 0);
   -- NULL
   SELECT exists(array(0, null, 2, 3, null), x -> x IS NULL);
   -- true
   SELECT exists(array(1, 2, 3), x -> x IS NULL);
   -- false
   SELECT exists(cast(null as array<int>), y -> y > 30);
   -- NULL
   ```
   
   These come from the `ExpressionDescription` and 
`higher-order-functions.sql.out`.
   
   `HigherOrderFunctionsSuite` covers both settings of the legacy config. In 
the table, "3VL" means
   `followThreeValuedLogic`.
   
   | input | predicate | 3VL = true | 3VL = false |
   |---|---|---|---|
   | `[1,2,3]` | `x % 2 == 0` | true | true |
   | `[1,2,3]` | `NULL` | NULL | false |
   | `[1,null,3]` | `x % 2 == 0` | NULL | false |
   | `[1,null,3]` | `x IS NULL OR x % 2 == 1` | true | true |
   | `[1,null,3]` | `false` | false | false |
   | `NULL` | any | NULL | NULL |
   
   ### Edge cases to test
   
   - A NULL array, an empty array, NULL elements, and a predicate that returns 
NULL for every element.
   - Both values of `spark.sql.legacy.followThreeValuedLogicInArrayExists`. The 
value is captured when the expression
     is constructed. Also check the declared nullability under each setting.
   - Short-circuiting. After the first `true`, Spark never evaluates the 
predicate on later elements. By code reading,
     `exists(array(1, 0), x -> 1 / x > 0)` returns `true` even with ANSI mode 
on, because `1 / 0` is never evaluated.
     With ANSI on and the order reversed (`array(0, 1)`), it raises 
`DIVIDE_BY_ZERO`.
   - Nested use: `transform(aai, a -> exists(a, x -> x IS NULL OR x % 2 == 1))` 
on `[[1,2,3],null,[4,5]]` gives
     `[true,null,true]`.
   - Captured outer columns in the predicate (`exists(ys, y -> y > x)`), and 
nondeterministic predicates
     (`x -> rand() > 0.5`), which must be evaluated per element and in element 
order.
   - Strings and non-primitive elements, for example `exists(array('a0','b1'), 
x -> startswith(x, 'a'))`, and nested
     arrays.
   - SQL parsing. sqlparser 0.63 treats `exists(` as a function call only for 
`DatabricksDialect`. Under
     `SparkSqlDialect`, which this crate's sqllogictest runner uses, `exists(` 
is parsed as an `EXISTS (subquery)`
     expression, so the `.slt` tests need a parser fix or a different dialect.
   
   ### Differences across Spark versions
   
   - None found between 3.4 and 4.2. The legacy config has existed since 3.0 
with default `true`. ANSI mode is on by
     default from 4.0, which matters for errors inside the predicate.
   
   ### Implementation notes
   
   - Spark: `ArrayExists` in 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala`.
     The config is `LEGACY_ARRAY_EXISTS_FOLLOWS_THREE_VALUED_LOGIC` in 
`SQLConf.scala`.
   - Infrastructure. DataFusion's higher-order function framework
     (`HigherOrderUDFImpl` in `datafusion/expr/src/higher_order_function.rs`, 
and `LambdaArgument::evaluate` with
     `spread_captures` for captured columns) is on `branch-56`, which this 
crate uses. This crate's
     `crates/spark/src/function/lambda/mod.rs` is an empty `ScalarUDF` 
placeholder. It needs a `HigherOrderUDF` list and
     registration through `register_higher_order_function` / 
`SessionStateBuilder::higher_order_functions`.
   - DataFusion core has `array_any_match` (aliases `any_match`, 
`list_any_match`) in
     `datafusion/functions-nested/src/array_any_match.rs`. It already follows 
Spark's default three-valued logic
     (true / NULL / false, and NULL for a NULL array). The differences are:
     - It evaluates the predicate once over all flattened elements of the 
batch, with no short-circuit. Errors in later
       elements (ANSI arithmetic, casts) are raised where Spark returns `true`. 
Nondeterministic predicates are also
       evaluated on elements Spark would skip.
     - It does not support the legacy `false` mode.
   
     A Spark version can reuse its result reduction, but it needs 
short-circuit-aware error handling. One option is to
     evaluate vectorized and fall back to element-by-element evaluation when 
the body is fallible or volatile.
   - Add `crates/sqllogictest/test_files/lambda/exists.slt` and document the 
function with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to