Hi, I would like to reach out to the community to see if anyone has some insights or experience with the performance impact of enabling prometheus metrics.
I have done load comparison tests for Prometheus enabled vs disabled and found the performance is reduced about 40%-60% for both read and write oeprations (i.e. getData, getChildren and createNode). The load test was done with Zookeeper 3.7, cluster size of 5 participants and 5 observers, each ZK server has 10G heap size and 4 cpu, 500 concurrent users sending requests. The performance impact is quite significant. I wonder if this is expected and what are things we can do to have ZK performing the same while leveraging the new feature of Prometheus metric. Best, Li
