ulysses-you commented on PR #58884:
URL: https://github.com/apache/spark/pull/58884#issuecomment-5770540139
Thanks @dongjoon-hyun. All four are in 4174278b339.
1. `detachAggregate` now takes the aggregate the pair folds into and asks
`isCompatibleAggregates` at the leaf, before the chain is rebuilt, so the
grouping is re-decided over the new child only for a pair that folds. That
check lives there and nowhere else: `isPartialAgg` is now the mode half alone
and `combinedAggregate` no longer repeats the compatibility part.
2. `withKeyPositionsFor` turns away a child that no longer reports the
partitioning this node was decided for, and reads the expressions from the
stored `childPartitioning`. The positions name keys in that space, so moving
them against another one re-targets the grouping. Same judgement as
`outputPartitioning` and `checkChildStillMatches`.
3. The sort below the partial aggregate goes only when it satisfies that
aggregate's `requiredChildOrdering`. The check is vacuous for the hash and
object-hash arms, whose required ordering is empty; there the crossing rests on
the pair being compatible, which pins the two groupings.
4. Both shapes are pinned.
- The composing branch needs a node that already projects, and no planner
shape reaches the fold with one: a partial aggregate's reported keys are its
grouping attributes, which are the cluster keys it is asked to satisfy, so
every position survives and `EnsureRequirements` hands out positions only for a
coalescing node. Measured over the suite's seven tests: all 11 crossings arrive
with `joinKeyPositions = None`. It is covered directly in
`GroupPartitionsExecSuite` now, on a node projecting position 1 of a child
reporting the same keys in the other order.
- The sort that stays is covered in `KeyGroupedPartitioningSuite`: a hash
pair with a local sort under the partial aggregate, which the fold leaves under
the regrouping.
Both new tests discriminate by mutation: dropping the lower sort
unconditionally fails the sort test, and dropping the child-partitioning
turn-away fails the position test, re-parenting with `JoinKeyPositions: [0]`
over a child whose key at that position is another one.
Ran `KeyGroupedPartitioningSuite` and `GroupPartitionsExecSuite` (224) and
the nine related suites (392), 616 in total, no failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
