Anton Kovalevsky created SPARK-50533:
----------------------------------------
Summary: MERGE fails with INVALID_NON_DETERMINISTIC_EXPRESSIONS if
MATCHED case is presented
Key: SPARK-50533
URL: https://issues.apache.org/jira/browse/SPARK-50533
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.5.3
Environment: Spark 3.5.0/3.5.3
Reporter: Anton Kovalevsky
Such query runs well:
{code:sql}
MERGE INTO "2" AS target
USING (
select name from "1"
) AS source
ON true
WHEN MATCHED THEN UPDATE SET name = source."name"
WHEN NOT MATCHED THEN INSERT (name) VALUES (source.name)
{code}
But if some non-detemenistic filter applied, e.x.
{code:sql}
MERGE INTO "2" AS target
USING (
select name
from "1"
qualify
row_number() over (
partition by name
order by rand()
) = 1
) AS source
ON true
WHEN MATCHED THEN UPDATE SET name = source."name"
WHEN NOT MATCHED THEN INSERT (name) VALUES (source.name)
{code}
then INVALID_NON_DETERMINISTIC_EXPRESSIONS error appears:
{code}
[INVALID_NON_DETERMINISTIC_EXPRESSIONS] The operator expects a deterministic
expression, but the actual expression is "true", "exists()".
{code}
1. After removing WHEN MATCHED case query compiles again.
2. Merge condition doesn't matter.
I didn't find any explicit requirements regarding MERGE's input, so it feels
like a bug.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
