ankitsultana opened a new pull request, #10289: URL: https://github.com/apache/pinot/pull/10289
Fixes several issues with the current scheduler: 1. _seenMail can track entries for Mailbox which were never registered and over time become really large. 2. computeReady call is done too frequently and can be costly. 3. available queue can have starvation if QPS is high enough. This issue was added by my PR (which fixed the issue with too many computeReady calls): https://github.com/apache/pinot/pull/10141 4. callback can take a while to get scheduled if _ready queue continues getting queries. Similar to 3 and again added by my PR. This PR should fix all issues and all the operations should be super-fast (basic concurrent set/map ops done under a lock). Key design change is that OpChainSchedulerService doesn't take any locks and the OpChainScheduler implementation needs to be thread-safe. That gives the scheduler much more flexibility in controlling consistency on its end. For `RoundRobinScheduler`, the key change is that it only tracks OpChains after they are registered, and once they are de-registered it removes all references to it from its state. **Test-Plan:** TBD. We are running this in our internal clusters atm. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
