+1 Thanks for your great work! Wenbing Shen <oliver.shen...@gmail.com> 于2022年12月26日周一 17:29写道：
> Hi BookKeepers, I've changed the limitStatsLogging default value to true > from false: > BP-60 <https://github.com/apache/bookkeeper/issues/3718> > > Motivation > > We have an efficient online bookie cluster with hundreds of bookie nodes > deployed on SSD disks. > > We separate the AutoRecovery cluster and the Bookie cluster for independent > deployment. > > I observed that our AutoRecovery cluster GC is very frequent. After > investigation, I found that the limitStatsLogging of the bookkeeper client > PCBC is disabled by default, and a large number of channel monitoring > indicators are generated. Due to the large number of bookie cluster nodes, > this metric data occupies a large amount of heap memory. > > A single StringWriter object occupies 16MB of memory, of which nearly 70 > StringWriter objects are waiting for the next GC to be destroyed, occupying > 1GB+ heap memory. > Proposal > > In my use, I haven't found any usefulness of these PCBC monitoring metrics > data, at least so far, I haven't used it effectively. > > If our AutoRecovery and Bookie cluster are mixed in one process, these > large objects will affect the performance and stability of Bookie cluster. > > Since I can't find the meaning of these metrics by default, I suggest to > adjust the default value of limitStatsLogging to true. > > Everyone can choose to turn it on or off, but by default, it is difficult > for users to find out what effect this parameter will have, so that when > their cluster grows to hundreds or thousands, when they realize the problem > sometimes, it is necessary to restart hundreds to thousands of bookies in a > rolling manner. > > At the same time, I observed that in pulsar, various monitoring of the > bookkeeper client is turned off by default, because they really affect the > performance of the pulsar service, which is enough to show that we should > try to change it, especially some very redundant metrics created based on > channels. > Compatibility, Deprecation, and Migration PlanClients that rely on PCBC > metrics monitoring need to pay attention to this upgrade, but this will not > affect the actual functions of the client, only the metrics data, and users > can choose to open it again. > > > What do you think about it? > > Best. > Wenbing >
