JingTyagi-ICON opened a new issue, #24901: URL: https://github.com/apache/pulsar/issues/24901
### Search before reporting - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### User environment Broker version (check with bin/pulsar version) pulsar-toolset-0:/pulsar$ bin/pulsar version Current version of pulsar is: 4.0.6 Git Revision 4538ef7645c45a3c8686092128fde6c5d61c762b Git Branch branch-4.0 Built by Lari Hotari <[email protected]> on Laris-MBP.lan at 2025-07-30T13:37:25+0300 Broker Operating system and hardware type (check with uname -a on Unixes / systeminfo on Windows) Linux pulsar-toolset-0 5.10.244-240.970.amzn2.x86_64 #1 SMP Tue Oct 7 09:40:03 UTC 2025 x86_64 GNU/Linux Broker Java version (check with java -version) pulsar-toolset-0:/pulsar$ java -version openjdk version "21.0.8" 2025-07-15 LTS OpenJDK Runtime Environment Corretto-21.0.8.9.1 (build 21.0.8+9-LTS) OpenJDK 64-Bit Server VM Corretto-21.0.8.9.1 (build 21.0.8+9-LTS, mixed mode) ### Issue Description We observed that from client side with the Error message: System.TimeoutException: Could not send message to broker within given timeout. Then we find out some issues in the pulsar broker. Please see the ERROR messages below. ### Error messages ```text 2025-10-27T19:02:50,853+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Fencing exception on write: L165820 E41 on pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181 2025-10-27T19:02:50,853+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 165820 due to LedgerFencedException: Ledger has been fenced off. Some other client must have opened it to read 2025-10-27T19:02:50,853+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L165820 E41 2025-10-27T19:02:50,854+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Creating a new ledger after closed 165820 2025-10-27T19:02:50,855+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.PendingAddOp - Fencing exception on write: L165820 E41 on pulsar-bookie-3.pulsar-bookie.pulsar.svc.cluster.local:3181 2025-10-27T19:02:50,855+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181] is not adhering to Placement Policy. quarantinedBookies: [] 2025-10-27T19:02:50,855+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 165820 due to LedgerFencedException: Ledger has been fenced off. Some other client must have opened it to read 2025-10-27T19:02:50,858+0000 [ZKC-connect-executor-0-EventThread] WARN org.apache.bookkeeper.meta.AbstractZkLedgerManager - Ledger node does not exist in ZooKeeper: ledgerId=165820 2025-10-27T19:02:50,858+0000 [ZKC-connect-executor-0-EventThread] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=165820,loopId=6e454574) Error writing metadata to store org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server at org.apache.bookkeeper.meta.AbstractZkLedgerManager$5.processResult(AbstractZkLedgerManager.java:541) [org.apache.bookkeeper-bookkeeper-server-4.17.2.jar:4.17.2] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$22$1.processResult(ZooKeeperClient.java:1094) [org.apache.bookkeeper-bookkeeper-server-4.17.2.jar:4.17.2] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:617) [org.apache.zookeeper-zookeeper-3.9.3.jar:3.9.3] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:542) [org.apache.zookeeper-zookeeper-3.9.3.jar:3.9.3] 2025-10-27T19:02:50,858+0000 [ZKC-connect-executor-0-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2025-10-27T19:02:50,859+0000 [ZKC-connect-executor-0-EventThread] WARN org.apache.bookkeeper.client.LedgerHandle - Close failed: NoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server 2025-10-27T19:02:50,863+0000 [ZKC-connect-executor-0-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [pulsar-bookie-0.pulsar-bookie.pulsar.svc.cluster.local:3181, pulsar-bookie-2.pulsar-bookie.pulsar.svc.cluster.local:3181] for ledger: 165841 2025-10-27T19:02:50,863+0000 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Created new ledger 165841 2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Error updating meta data with the new list of ledgers: org.apache.pulsar.metadata.api.MetadataStoreException$BadVersionException: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /managed-ledgers/Research/TimepointConfiguration/persistent/DataEvents 2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Research/TimepointConfiguration/persistent/DataEvents Moving to Fenced state 2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [Research/TimepointConfiguration/persistent/DataEvents] Failed to update ledger list. z-node version mismatch. Closing managed ledger 2025-10-27T19:02:50,870+0000 [bookkeeper-ml-scheduler-OrderedScheduler-1-0] INFO org.apache.pulsar.broker.service.Producer - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://Research/TimepointConfiguration/DataEvents}, client=[id: 0xb4740df9, L:/100.64.10.130:6650 - R:/100.64.6.123:41984] [SR:100.64.6.123, state:Connected], producerName=pulsar-876-15695, producerId=29}, assignedBrokerLookupData: Optional.empty ``` ### Reproducing the issue This happened in our production environment, we haven't done any thing regarding to reproduce the issue. ### Additional information _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
