lhotari commented on issue #23920: URL: https://github.com/apache/pulsar/issues/23920#issuecomment-2630561117
> Another observation when we encounter this issue is that no messages are delivered to subscriptions. We tried consuming data from topic using Pulsar's CLI client from earliest position but no records were delivered. This detail hints into the direction that there could be a deadlock on the broker side when the problem occurs. In the Pulsar docker containers `jstack -l 1` is how you can capture a thread dump with lock information. If you'd like to capture multiple thread dumps and a heapdump, there's a script that can automate this when there's `kubectl` access to the cluster: https://github.com/lhotari/pulsar-contributor-toolbox/blob/master/scripts/collect_jvm_diagnostics_from_pod.sh Notice: the heapdump could contain sensitive information, so it should never be shared publicly. Usually the only possibly sensisitive information in threaddumps is the Zookeeper client thread names that contain a IP address. Would you be able to capture a thread dump with `jstack -l 1` to ensure that there isn't a deadlock? In Pulsar 4.0.1 there's a known deadlock bug that has been fixed in 4.0.2 with #23854. That applies to Key_Shared subscriptions. In this case, please share (an possibly sanitize before sharing) the topic stats Commands for creating stats and stats-internal for a partitioned topic: ```shell pulsar-admin topics partitioned-stats --per-partition persistent://public/default/picks-teams-stream > topic_stats_picks-teams-stream_$(date +%s).json pulsar-admin topics partitioned-stats-internal persistent://public/default/picks-teams-stream > topic_stats_picks-teams-stream_$(date +%s).json ``` You might want to sanitize IP addresses in stats etc. before sharing. then use GitHub cli tool gh to create a secret gist at [gist.github.com](https://gist.github.com/): ```shell gh gist create topic_stats_picks-teams-stream_*.json topic_stats_picks-teams-stream_*.json ``` After creating a gist, you can check the content and then share the link in this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
