thetumbled opened a new issue, #25116: URL: https://github.com/apache/pulsar/issues/25116
### Search before reporting - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### User environment 3.0.x. ### Issue Description I stopped a bookie running on a faulty disk, and another machine subsequently shut down due to a mechanical fault. This machine hosts one broker and more than 10 bookies. Then, the throughput in and out decline dramatically. The topic policy module can't initialize successfully, resulting into most of the topics can't be initialized. <img width="830" height="414" alt="Image" src="https://github.com/user-attachments/assets/ebbf40f9-6480-4c60-a6e8-b9a69a90529a"; /> <img width="830" height="334" alt="Image" src="https://github.com/user-attachments/assets/fb32eb66-72b4-4d19-8ffd-f7890b28e926"; /> Though most of the throughput has decreased, there is still a small portion of inbound and outbound throughput. We **turn off the topic policy module and restart the broker cluster to avoid the issue temporarily.** It is noticed that there are many read request for bookie and the read latency soars to minutes level, which is suspected to be the read request for change_event topic. <img width="832" height="636" alt="Image" src="https://github.com/user-attachments/assets/a23e9607-a123-4c1c-a08a-c8673b9804ba"; /> ### Error messages ```text ERROR org.apache.pulsar.broker.service.SystemTopicBasedTopicPoliciesService - [XX/XX] Failed to create reader on __change_events topic java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$AlreadyClosedException: Consumer already closed ``` ### Reproducing the issue we can't reproduce the issue for now. ### Additional information _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
