tuteng commented on issue #5417: Broker hangs and crashes when listing non-persistent topics URL: https://github.com/apache/pulsar/issues/5417#issuecomment-562382897 I tested this problem. under a cluster (including three brokers and three bookies), I used the `ab` tool to test. numHttpServerThreads defaults to 8. when the number of query requests initiated is significantly higher than the number that the thread can handle, it will indeed block the broker. I think this is because `non-persisten topic` is stored in memory. when there are multiple brokers, your request is sent to one of them, and the broker completes the remaining work. after the broker receives the request, it will forward the request to all brokers under the cluster. Look up all bundles of all broker, then traverse all topics under these bundles, and finally return topics. A large part of this operation is network delay. Therefore, when concurrent requests are large, the above problems will occur. I have not found a suitable way to solve this problem through fix code, but I have two alternatives to query `non-persistent topic`. 1. Reasonably evaluate your query request and configure the appropriate number of numHttpServerThreads, but this method cannot completely solve the above problems. 2. The proposal is divided into the following three steps a. Get all broker addresses ``` curl -v http://any-broker-ip:8080/admin/v2/brokers/cluster-name ``` b. Loop the following rest API to obtain bundle ``` curl -v http://broker-1:8080/admin/v2/non-persistent/test/test-namespace curl -v http://broker-2:8080/admin/v2/non-persistent/test/test-namespace ``` The following results will be returned ``` "bundles" : { "boundaries" : [ "0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ], "numBundles" : 4 }, ``` c. Get the topic under the bundle through the bundle. ``` ... http://broker-ip:8080/admin/v2/non-persistent/test/test-namespace/0x00000000_0x40000000 http://broker-ip:8080/admin/v2/non-persistent/test/test-namespace/0x40000000_0x80000000 ... ``` I think the second method will solve the problem of blocking the broker due to the high number of concurrent requests.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
