gortiz commented on PR #19396:
URL: https://github.com/apache/pinot/pull/19396#issuecomment-5478898366
I've been working on my own fix for `SortedMailboxReceiverOperator`, which
goes back to what used to be:
```
┌───────────────────────────────────┬─────────────────────────┬──────────────────────┐
│ │ #19396 │ My
branch │
├───────────────────────────────────┼─────────────────────────┼──────────────────────┤
│ │ sortOnSender = │
sortOnReceiver = │
│ PinotWindowExchangeNodeInsertRule │ **true**, keeps │ **false** +
explicit │
│ │ sortOnReceiver = true │ Sort
│
├───────────────────────────────────┼─────────────────────────┼──────────────────────┤
│ │ +336 lines — k-way │
│
│ SortedMailboxReceiveOperator │ merge over sorted │ @Deprecated
│
│ │ senders │
│
├───────────────────────────────────┼─────────────────────────┼──────────────────────┤
│ MailboxSendOperator │ +143 — sorts sender │ untouched
│
│ │ output, 10k blocks │
│
├───────────────────────────────────┼─────────────────────────┼──────────────────────┤
│ │ new sortedOnSender │
│
│ Mailbox protocol │ confirmation over gRPC │ untouched
│
│ │ + in-memory │
│
├───────────────────────────────────┼─────────────────────────┼──────────────────────┤
│ PinotSortExchangeNodeInsertRule │ comment only — │
sortOnReceiver = │
│ │ deliberately deferred │ false
│
└───────────────────────────────────┴─────────────────────────┴──────────────────────┘
```
The idea I have is to stop generating SortedMailboxReceiveOperator and
substitute that with a sort on the receiver side (with optional sort on the
sender side when the limit is small). Once we have that, we can start thinking
about recovering SortedMailboxReceiveOperator as a k-way merge when the senders
guarantee data is send in order, which is what you and #19121 are doing
(although #19121 keeps both the current and the k-way merge).
Also, I don't think `MailboxSendOperator` should sort on send. Instead, we
should add a sort operator on the sender opchain and keep MailboxSendOperator
agnostic about ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
