gortiz opened a new pull request, #15024: URL: https://github.com/apache/pinot/pull/15024
This PR fixes two errors detected when spooling intermediate stages: 1. Queries failed when spooling an intermediate stage on PlanFragment when trying to assign a fragment to the children of the pruned stage. This is solved by removing these orphan children, as it was expected. 2. On multi-senders, stats were sent randomly to different children, as we used to do in regular broadcasts. When stats are merged in `MultiStageQueryStats.mergeUpstream`, we cannot merge stats sent from an early stage. This requirement was added when spools didn't exist; therefore, we could guarantee an order in stage ids. With spools, that order is broken in the general case (a parent stage may receive data from children with smaller IDs). Instead of fixing that requirement, this PR ensures that our mailbox send operators only send stats to the smaller receiver stage. By doing that, we can still guarantee the previous constraint when merging stats because a mailbox send must always send data to its original parent. This PR also includes a test that verifies that intermediate stages can be spooled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
