rdhabalia opened a new pull request #5298: [pulsar-broker] Fix: invalidate cache on zk-cache timeout URL: https://github.com/apache/pulsar/pull/5298 ### Motivation Right now, while creating producer/consumer, broker tries to authorize them by fetching namespace policies using zkCache. But sometimes, zkCache times-out while getting zk-node into cache and zk-cache keeps pending key for 5 mins. Because of that client can't create producer/consumer on that topic for 5 mins. Fix: broker should invalidate key if it receives timeout while accessing zk-cache so, subsequent requests can successfully fetch it ``` 01:07:14.847 [pulsar-ordered-OrderedExecutor-2-0] WARN org.apache.pulsar.broker.service.BrokerService - Got exception when reading persistence policy for persistent://prop/cluster/ns/topic: null java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?] at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:95) ~[pulsar-zookeeper-utils-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.pulsar.broker.service.BrokerService.lambda$getManagedLedgerConfig$119(BrokerService.java:723) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.util.SafeRun$2.safeRun(SafeRun.java:49) [managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.9.0.jar:4.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:834) [?:?] 01:07 : : 01:07:14.848 [bookkeeper-ml-workers-OrderedExecutor-3-0] WARN org.apache.pulsar.broker.service.persistent.DispatchRateLimiter - Failed to get message-rate for persistent://prop/cluster/ns/topic java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886) ~[?:?] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021) ~[?:?] at org.apache.pulsar.broker.service.persistent.DispatchRateLimiter.getPolicies(DispatchRateLimiter.java:254) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.pulsar.broker.service.persistent.DispatchRateLimiter.isDispatchRateNeeded(DispatchRateLimiter.java:155) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.pulsar.broker.service.persistent.PersistentTopic.initializeDispatchRateLimiterIfNeeded(PersistentTopic.java:248) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.pulsar.broker.service.persistent.PersistentTopic.<init>(PersistentTopic.java:199) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.pulsar.broker.service.BrokerService$3.openLedgerComplete(BrokerService.java:656) ~[pulsar-broker-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$98(ManagedLedgerFactoryImpl.java:328) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo ] at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?] at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeComplete(ManagedLedgerFactoryImpl.java:316) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3$1.operationComplete(ManagedLedgerImpl.java:467) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$1.operationComplete(ManagedCursorImpl.java:273) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl$1.operationComplete(ManagedCursorImpl.java:246) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.impl.MetaStoreImplZookeeper.lambda$null$120(MetaStoreImplZookeeper.java:241) ~[managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [managed-ledger-original-2.4.3-yahoo.jar:2.4.3-yahoo] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.9.0.jar:4.9.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:834) [?:?] ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services