walterddr opened a new pull request, #12079:
URL: https://github.com/apache/pinot/pull/12079
step 2 in #12015
this PR:
1. loosen the criteria for "co-assigning workers to the same partition to
avoid shuffling" to non-leaf children
2. reshape the rule of partition assignment & mailbox rule.
- Exchange governs whether an exchange is "pre-partitioned" --> e.g. the
input partition is the same desired by the exchange (exchange might still be
needed if the keys are the same but functions are different)
- For worker assignment
- Exchange "pre-partitioned" is required for assigning the same set
of workers from the 1st child
- For leaf-stage, worker assignment is only controlled by
`tableOptions`
- For mailbox assignment
- Exchange "pre-partitioned" is necessary and sufficient to assign
direct exchange for 1st child; it is NOT sufficient for the other children
(check logic needs to be applied)
- `partition_parallelism` is only used between leaf-children and
intermediate parent
- if they are both leaf (semi-join, partition_parallelism
shouldn't apply; num_worker == num_partition of table)
- if they are both intermediate stage, partition_parallelism
shouldn't apply; num_worker == num_partition * partition_parallelism from
leaf-table already)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
