wenbingshen commented on code in PR #4258: URL: https://github.com/apache/bookkeeper/pull/4258#discussion_r1582074412
########## site3/website/src/pages/bps/BP-66-support-throttling-for-zookeeper-read-of-rereplication.md: ########## @@ -0,0 +1,26 @@ +# BP-66: support throttling for zookeeper read of rereplication + +### Motivation + +Each time the cluster trigger the re-replication, all replicators will read data from zookeeper. This can cause a great pressure on Zookeeper. We need to support throttling for zookeeper read of re-replication. + +For example, in a Pulsar cluster, we enable auto-recovery for every bookie. There are 400 bookies in a cluster, which means there are 400 replicators in the cluster. +And there are about 3000 ledgers in each bookie, 1/3 of them are small ledgers, whose size is less than 0.1MB, that is 1000 small ledgers in each bookie. +If we decommission one bookie, the read latency of zookeeper will increase to minutes. + + +### Configuration +add the following configuration: Review Comment: Thank you very much for your work. Currently I maintain a bookie cluster with 200 nodes. I applied the following speed limit PR. The bookie process disabled autorecovery and deployed about 10 AutoRecovery processes independently. https://github.com/apache/bookkeeper/pull/2778 So far, the cluster service operation and maintenance work are relatively good. I think you can separate the AutoRecovery service and set the corresponding replication limit, which may help you. For the work of this proposal, by individually limiting the frequency of reading zk, the zk service can be reasonably limited and protected, but it is not convenient to limit the byte rate of reading and copying entries because the size of the entry changes; On the contrary, I feel that PR #2778 can protect zk's read speed through speed limiting. Let's hear what others have to say. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
