ulysses-you commented on PR #58775: URL: https://github.com/apache/spark/pull/58775#issuecomment-5673650485
Thanks @peter-toth. All three addressed in `74f2edb0e91`: 1. The partial node now goes only when its own local sort goes with it (`RemoveRedundantWindowGroupLimits.scala:66`); the source-ordered shape keeps both nodes, with the reducer between the scan and the sort above the grouping. Regression of this PR: before it the rule could not see through the grouping, so that shape was untouched. 2. Folded into `remove redundant WindowGroupLimits` beside `checkWindowGroupLimits(query1, 1)`, asserting the sort stayed where the partial node was. 3. Dropped; the placement is now asserted on the final node's child in `KeyGroupedPartitioningSuite`. Keeping the reducer is the cheap answer in this shape: with the partial node removed the sort above the grouping reads the whole scan, and reclaiming `tryEnableSortedMerge` instead is no better (606-608 ms against master's 372-415 ms). The description now says what the surviving sort sees. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
