joelrobin18 opened a new pull request, #58244:
URL: https://github.com/apache/spark/pull/58244
### What changes were proposed in this pull request?
SQL variables declared with `DECLARE` cannot be referenced in the conditions
of an
`UPDATE` or `MERGE INTO` statement. This passes `includeLastResort = true`
at the seven
condition-resolution sites that make up those clauses, matching what
SPARK-57260 did for
`OverwriteByExpression.deleteExpr`.
Variable resolution only runs from `resolveColsLastResort`, which is reached
when
`resolveExpressionByPlanOutput` / `resolveExpressionByPlanChildren` are
called with
`includeLastResort = true`. Both default the flag to `false`. Plans with no
dedicated
resolution rule fall through to the generic operator case in
`ResolveReferences`, which
does pass the flag -- which is why `DELETE ... WHERE` already works.
`UPDATE` and
`MERGE INTO` each have a dedicated rule that omitted it:
- `UPDATE` condition (`ResolveReferencesInUpdate`)
- `MERGE` `ON` condition
- `MERGE` `WHEN MATCHED` `DELETE` / `UPDATE` conditions
- `MERGE` `UPDATE *` condition
- `MERGE` `WHEN NOT MATCHED BY SOURCE` `DELETE` / `UPDATE` conditions
Assignment values (`SET col = var`, `INSERT VALUES (var)`) are affected by
the same root
cause, but they resolve through `resolveExprInAssignment`, which sets
`includeLastResort = false` explicitly rather than by default. That is left
unchanged
here pending a decision on whether the explicit `false` was deliberate, so
this PR is
scoped to conditions only.
### Why are the changes needed?
Variables resolve in `SELECT`, `INSERT` (including `REPLACE WHERE`) and
`DELETE`, but in
no condition of `UPDATE` or `MERGE INTO`, which fails analysis with:
```
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function
parameter with
name `target_dep` cannot be resolved. SQLSTATE: 42703
```
The message itself offers "A column, variable, or function parameter", so
the analyzer
reports that variable resolution was attempted. Nothing in
`error-conditions.json`, the
tests, or the docs records a restriction here. Where Spark does
intentionally block
variables (SPARK-57360, generated columns) it uses an explicit validation
and a dedicated
error class, so the inconsistency looks like an oversight rather than a
deliberate
limitation.
This affects both session variables and SQL scripting local variables.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix.
Before, against a table using the built-in DSv2 row-level operation
framework:
```sql
DECLARE OR REPLACE VARIABLE target_dep STRING DEFAULT 'hr';
UPDATE cat.ns1.test_table SET salary = 0 WHERE dep = target_dep;
-- [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... `target_dep` cannot be resolved.
SQLSTATE: 42703
```
After, the statement analyzes and executes, resolving `target_dep` to the
declared
variable. Statements that previously succeeded are unaffected: the flag only
enables a
last-resort resolution step that runs after normal column resolution fails,
so column
references continue to take precedence over same-named variables.
### How was this patch tested?
Four new tests, plus a shared `withSessionVariable` helper in
`RowLevelOperationSuiteBase`:
- `UpdateTableSuiteBase` -- a session variable and a SQL scripting local
variable in an
`UPDATE` condition.
- `MergeIntoTableSuiteBase` -- a variable in the `MERGE` `ON` condition, and
in the
`WHEN MATCHED` / `WHEN NOT MATCHED BY SOURCE` conditions.
All four were verified to fail on master without the fix (each raising
`UNRESOLVED_COLUMN`) and to pass with it.
Suites run: `GroupBasedUpdateTableSuite` + `GroupBasedMergeIntoTableSuite`
(137 tests)
and `DeltaBasedUpdateTableSuite` + `DeltaBasedMergeIntoTableSuite` (148
tests), all
passing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
