lhotari commented on PR #22541: URL: https://github.com/apache/pulsar/pull/22541#issuecomment-2071568113
There's a problem with backpressure handling with async requests in the Pulsar code base. Since this PR limits the Pulsar Admin client to 16 connections per host, it now shows up problems. The namespace unloading is a good example: https://github.com/apache/pulsar/blob/d7d54522933b63f6a74ec7139c6dedebe8ad9149/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java#L829-L841 All bundles in the namespace are unloaded at once without limiting concurrency. There was a dev mailing list discussion about backpressure and Pulsar Admin API implementation in https://lists.apache.org/thread/03w6x9zsgx11mqcp5m4k4n27cyqmp271 . However we didn't come across resolving the problem. The solution for the namespace unloading issue is to have a way to limit the outstanding CompletableFutures that are in progress and use that as a way to "backpressure" the sending of new requests. The current solution of sending out all requests and then waiting for the results is a problematic solution since it doesn't use any sort of feedback from the system to adjust the speed. In other words, there's currently no proper backpressure solution for async Pulsar Admin calls within Pulsar broker. I'll experiment with some ways to add backpressure to cases where a large amount of async calls are triggered and then results are waited. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
