xumingming commented on code in PR #58144:
URL: https://github.com/apache/spark/pull/58144#discussion_r3880443379
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:
##########
@@ -1834,12 +1834,339 @@ object CollapseWindow extends Rule[LogicalPlan] {
}
/**
- * Transpose Adjacent Window Expressions.
- * - If the partition spec of the parent Window expression is compatible with
the partition spec
- * of the child window expression, transpose them.
+ * Reorder a stack of `Window` operators so that the fewest possible exchanges
are inserted
+ * for it. Window operators only append their output columns and each of them
+ * (re-)partitions/(re-)sorts its input independently of its position in the
stack, so two
+ * windows can be swapped freely as long as neither references the other's
output. Physically,
+ * a window rides the exchange created for the nearest window below it iff
that exchange's key
+ * set is a subset of the window's partition spec (`HashPartitioning`
satisfying
Review Comment:
Thanks @uros-b, both points are valid and are fixed in 87c2c81ac58.
For duplicate partition keys: `HashPartitioning(a, a)` is indistinguishable
from `HashPartitioning(a)` under the relaxed subset semantics, so
`optimalOrder` now reasons over normalized specs (semantic duplicates removed).
`[a,a]` therefore normalizes to `[a]`, sorts before `[a,b]` despite the equal
raw lengths, and its exchange serves the `[a,b]` window — one exchange for
both. Under `requireAllClusterKeysForDistribution` the spec is deliberately
kept as is, since there an exact ordered key match is required and duplicates
do matter.
For the pinned rank window: its spec is now passed to `optimalOrder` as a
tail constraint — the group whose keys the pinned window can ride is scheduled
last, so the pinned window rides that exchange instead of paying its own (`[b]
→ [a] → pinned[a]` needs two exchanges, as you described). Each group is also
now led by a window whose spec keys are exactly the minimal spec's, which is
what actually keys the group exchange on the minimal spec and makes the
one-exchange-per-group invariant hold with duplicate keys present.
Covered by new tests in `TransposeWindowSuite` (duplicate-key ordering +
strict-mode guard, pinned-window order) and `TransposeWindowQuerySuite`
(exchange counts asserted on both sides of the config: 1 vs 2, and 2 vs 3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
