Yes, I am thinking that handling metrics reporting in a separate thread, so it doesn't impact the "main" thread.
Not sure about the idea of merging at the end of a reporting period. Can you elaborate a bit on it? Thanks, Li On Mon, Apr 26, 2021 at 9:11 PM Ted Dunning <ted.dunn...@gmail.com> wrote: > Would it help to keep per thread metrics that are either reported > independently or are merged at the end of a reporting period? > > > > On Mon, Apr 26, 2021 at 8:51 PM Li Wang <li4w...@gmail.com> wrote: > > > Hi Community, > > > > I've done further investigation on the issue and found the following > > > > 1. The perf of the read operation was decreased due to the lock > contention > > in the Prometheus TimeWindowQuantiles APIs. 3 out of 4 CommitProcWorker > > threads were blocked on the TimeWindowQuantiles.insert() API when the > test > > was. > > > > 2. The perf of the write operation was decreased because of the high CPU > > usage from Prometheus Summary type of metrics. The CPU usage of > > CommitProcessor increased about 50% when Prometheus was disabled compared > > to enabled (46% vs 80% with 4 CPU, 63% vs 99% with 12 CPU). > > > > > > Prometheus integration is a great feature, however the negative > performance > > impact is very significant. I wonder if anyone has any thoughts on how > to > > reduce the perf impact. > > > > > > > > Thanks, > > > > > > Li > > > > > > On Tue, Apr 6, 2021 at 12:33 PM Li Wang <li4w...@gmail.com> wrote: > > > > > Hi, > > > > > > I would like to reach out to the community to see if anyone has some > > > insights or experience with the performance impact of enabling > prometheus > > > metrics. > > > > > > I have done load comparison tests for Prometheus enabled vs disabled > and > > > found the performance is reduced about 40%-60% for both read and write > > > oeprations (i.e. getData, getChildren and createNode). > > > > > > The load test was done with Zookeeper 3.7, cluster size of 5 > participants > > > and 5 observers, each ZK server has 10G heap size and 4 cpu, 500 > > concurrent > > > users sending requests. > > > > > > The performance impact is quite significant. I wonder if this is > > expected > > > and what are things we can do to have ZK performing the same while > > > leveraging the new feature of Prometheus metric. > > > > > > Best, > > > > > > Li > > > > > > > > > > > > > > >
