lhotari commented on issue #24490: URL: https://github.com/apache/pulsar/issues/24490#issuecomment-3210153267
> Hi [@lhotari](https://github.com/lhotari), > > I tested by applying the parameters `dispatcherRetryBackoffInitialTimeInMs=0` and `dispatcherRetryBackoffMaxTimeInMs=0`. This brings down the latency, and when compared with 3.0.x, it’s now ~0%. The results are also consistent across re-runs. Cc: [@dlg99](https://github.com/dlg99) [@sandeep-mst](https://github.com/sandeep-mst) @Ganesh-Chandra Thanks for confirming and performing the testing, this is very helpful. We don't currently have a good place to document performance tuning of Pulsar. It would be great to have this documented and also document the tradeoff explained below: The main reason to add the setting is to be reduce unnecessary CPU spent in a spin loop when no messages were dispatched. In the case of Key_Shared subscriptions, it's to reduce the unnecessary CPU spent in the loop where the messages in replay hit the limits configured with `keySharedLookAheadMsgInReplayThresholdPerConsumer` and `keySharedLookAheadMsgInReplayThresholdPerSubscription` settings. When a Pulsar broker has a large amount of topics, it is necessary to avoid CPU spin loops. One potential optimization could be to support more fine grain resolution than having 1 ms as the initial backoff. However this is hard to achieve since schedulers don't really support shorter delays than 1 ms in scheduling a task. One possible way around this would be to have a separate rate limit that will start the backoff only if the rate goes over the limit. For example, in that way, it would be possible to allow 100 calls per second and then backoff for the initial 1ms delay instead of immediately backing off when there's a single call. The way dispatchers work currently are not very optimal. One of the problems I recently noticed again is https://github.com/apache/pulsar/issues/23865 . It's one area that I'd like to improve in the future. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
