AMC-hawk opened a new pull request, #58760:
URL: https://github.com/apache/spark/pull/58760
### What changes were proposed in this pull request?
The SQL pipe `SET` operator is built in `AstBuilder.visitOperatorPipeSet` as
a star
expansion that excludes the assigned column and appends a replacement of the
same name:
```scala
UnresolvedStarExceptOrReplace(
target = None, excepts = Seq(Seq(ident)), replacements =
Some(Seq(replacement)))
```
The excluded source attribute is dropped from the project list entirely, so
nothing is
left for a later qualified reference such as `t.a` to resolve to.
This PR retains the excluded attribute as the hidden output of the `Project`
that `SET`
builds, using the existing `Project.hiddenOutputTag` mechanism that USING
joins already
use to hide their duplicated join keys. Concretely:
* `UnresolvedStarExceptOrReplace` gains `retainExceptedColumnsAsHidden`
(default `false`),
so that ordinary `SELECT * EXCEPT` / `* REPLACE` is unaffected. At the
point of star
expansion the two are otherwise indistinguishable.
* `visitOperatorPipeSet` sets it to `true`.
* `ResolveReferences`, in the same place it expands the star and builds the
new `Project`,
records the excluded attributes on `Project.hiddenOutputTag`, mirroring
`commonNaturalJoinProcessing` and `ResolveAsOfJoin`.
The retained attribute is spliced back into the project list by the existing
`AddMetadataColumns` rule, and only when it is actually referenced:
```
-- |> SELECT t.a
Project [a#x]
+- Project [(a#x + 1) AS a#x, b#x, a#x] <- original retained on demand
+- SubqueryAlias t
+- LocalRelation [a#x, b#x]
-- |> SELECT *
Project [a#x, b#x]
+- Project [(a#x + 1) AS a#x, b#x] <- unchanged, no extra column
+- SubqueryAlias t
+- LocalRelation [a#x, b#x]
```
The docs already describe this behavior, so no wording change was needed.
The `SET`
section did contain the same example twice by accident; the duplicate is
replaced with one
that demonstrates the alias retention. Happy to drop that hunk if it is
considered out of
scope.
### Why are the changes needed?
`docs/sql-pipe-syntax.md` documents that after a `SET` assignment "top-level
column names
are updated but table aliases still refer to the original row values". That
holds for
columns `SET` did not touch, but not for the assigned column itself:
```sql
VALUES (1, 10) AS t(a, b)
|> SET a = a + 1
|> SELECT t.a;
```
```
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function
parameter with name
`t`.`a` cannot be resolved. Did you mean one of the following?
[`a`, `t`.`b`]. SQLSTATE: 42703
```
The suggestion list shows the asymmetry: `a` resolves and `t`.`b` resolves,
but `t`.`a`
does not. SPARK-50772 (#49420) introduced the alias retention for
`SET`/`EXTEND`/`DROP`,
but its regression scenario reads only untouched qualified source columns;
it does not
read back a column that `SET` assigned.
This blocks the documented before/after use of pipe `SET` -- auditing,
computing a delta
against the source value, and join disambiguation.
### Does this PR introduce _any_ user-facing change?
Yes. A qualified reference to a column affected by pipe `SET` now resolves
to the original
input value instead of failing analysis. This is a change relative to master
and to
released versions; it makes the behavior match the documentation.
Before:
```sql
VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT t.a;
-- [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... `t`.`a` cannot be resolved
```
After:
```sql
VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT t.a;
-- 1
VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT a, t.a, t.b;
-- 2 1 10
```
The output schema of `SET` itself is unchanged; the retained column is
hidden and is only
materialized when referenced by name.
### How was this patch tested?
Golden file tests added to `pipe-operators.sql`, covering the reported
repro, reading the
new and original values together, that `SELECT *` still yields the original
schema,
qualified access through an alias introduced by `|> AS`, and a sequence of
two `SET`s on
the same column.
The full `SQLQueryTestSuite` passes, which covers the existing `SELECT *
EXCEPT` /
`* REPLACE` golden files. The regenerated golden diff contains only
additions; no existing
expected output changed.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
