cloud-fan commented on code in PR #58760:
URL: https://github.com/apache/spark/pull/58760#discussion_r4047867586
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -2085,6 +2086,29 @@ class Analyzer(
}.map(_.asInstanceOf[NamedExpression])
}
+ /**
+ * The SQL pipe SET operator is implemented as a star expansion that
excludes the assigned
+ * column and appends a replacement of the same name. That drops the
original attribute from
+ * the project list, which would also make it unreachable through its
table alias, contradicting
+ * the documented behavior that table aliases keep referring to the
original row values after an
+ * assignment. Retain the excluded attributes as hidden output instead,
the same way USING joins
+ * hide their duplicated join keys (SPARK-59146).
+ */
+ private def retainExceptedColumnsAsHiddenOutput(original: Project,
expanded: Project): Unit = {
+ val retain = original.projectList.exists {
+ case s: UnresolvedStarExceptOrReplace =>
s.retainExceptedColumnsAsHidden
+ case _ => false
+ }
+ if (retain) {
+ val excepted =
expanded.child.output.filterNot(expanded.outputSet.contains)
+ if (excepted.nonEmpty) {
+ expanded.setTagValue(
Review Comment:
**Non-blocking (P2):** `transformUpWithPruning` normally forwards the
original `Project` tags after this rewrite, but `copyTagsFrom` only does so
when the destination tag map is empty. Setting `hiddenOutputTag` here first
makes the map nonempty, so a Spark Connect relation rooted at pipe `SET` loses
`PLAN_ID_TAG`; DataFrame columns bound to that relation can then fail with
`CANNOT_RESOLVE_DATAFRAME_COLUMN`. Please preserve or merge the original
project's tags when adding this tag.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -2085,6 +2086,29 @@ class Analyzer(
}.map(_.asInstanceOf[NamedExpression])
}
+ /**
+ * The SQL pipe SET operator is implemented as a star expansion that
excludes the assigned
+ * column and appends a replacement of the same name. That drops the
original attribute from
+ * the project list, which would also make it unreachable through its
table alias, contradicting
+ * the documented behavior that table aliases keep referring to the
original row values after an
+ * assignment. Retain the excluded attributes as hidden output instead,
the same way USING joins
+ * hide their duplicated join keys (SPARK-59146).
+ */
+ private def retainExceptedColumnsAsHiddenOutput(original: Project,
expanded: Project): Unit = {
+ val retain = original.projectList.exists {
+ case s: UnresolvedStarExceptOrReplace =>
s.retainExceptedColumnsAsHidden
+ case _ => false
+ }
+ if (retain) {
+ val excepted =
expanded.child.output.filterNot(expanded.outputSet.contains)
+ if (excepted.nonEmpty) {
+ expanded.setTagValue(
+ Project.hiddenOutputTag,
+ excepted.map(_.markAsQualifiedAccessOnly()) ++
expanded.child.metadataOutput)
Review Comment:
**Blocking (P1):** `UnresolvedStar` expands qualified-only `metadataOutput`
before visible output, so prepending the retained attributes here does not
preserve the aliased row's column order. For example, `VALUES (1, 2, 3) AS t(a,
b, c) |> SET b = 20 |> SELECT t.*` expands as `(b, a, c)` and returns `(2, 1,
3)` instead of `(1, 2, 3)`. Please construct the retained output so `t.*`
follows the original source order, including after repeated `SET`s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
