ykisana commented on code in PR #59199:
URL: https://github.com/apache/spark/pull/59199#discussion_r4209095489
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanPartitioningAndOrdering.scala:
##########
@@ -104,8 +104,12 @@ object V2ScanPartitioningAndOrdering extends
Rule[LogicalPlan] with Logging {
// scan output: truncating it here would also drop the sort orders on
a partition key past a
// pruned column, which still hold.
`DataSourceV2ScanRelation.doCanonicalize` and
// `DataSourceV2ScanExecBase.outputOrdering` restrict it to the scan
output instead.
+ // A nested sort key such as `s.x` resolves to a new
`Alias(GetStructField(...))` on every
+ // run of this rule. Strip the alias so repeated runs give equal plans
and the ordering
+ // can satisfy a required ordering on the same field.
val ordering =
V2ExpressionUtils.toCatalystOrdering(reportedOrdering, relation,
relation.funCatalog)
+ .map(o => o.copy(child = trimAliases(o.child)))
Review Comment:
Added the MergeSubplansSuite test in 79e88aa. Confirmed it fails without the
fix ('had length 2 instead of expected length 1').
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
