tjiuming opened a new issue, #18004: URL: https://github.com/apache/pulsar/issues/18004
### Motivation Pulsar doesn't have offload throttling for now. The offload task can have a chance to use up all the broker resources(CPU, network). When offload tasks use too much broker resources, the latency of messaging may increase and it makes Pulsar broker unstable. For the purpose of saving broker resources and improving broker stability, we have to introduce `Offload throttling` to Pulsar. It is a mechanism to limit the speed of Offload. With the mechanism, we can easily limit broker/namespace/topic level Offload speed, and schedule offload resources flexibly. ### Goal #### 1. Provide broker/namespace/topic level offload limiting configurations. #### 2. Schedule offload resources flexibly, to prevent some topics from using up all the offload resources(Thread). ### API Changes _No response_ ### Implementation The current implementation of ManagedLedgerOffloader cannot meet the requirements for Ledger offload throttling. The task granularity based on Ledger is relatively large, all the Ledgers’ offload task sharing a same OrderScheduler, before submit an offload task, fetch an Executor by calling `scheduler.chooseThread(handleId)`. If we block the thread when the offload task is restricted, all the offload tasks which share the Executor will be blocked too. This makes it impossible to make full use of offload resources, and some tasks with higher priority cannot run normally. Based on the above reasons, we need to split the current large task into small tasks that cannot be split further. After a small task is finished, it should determine if there are more entries needed to offload, if there are, submit the next task to the OrderedScheduler, if there aren’t, complete the ledger offload. ### Alternatives _No response_ ### Anything else? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
