Hello, This can be very handy when dealing with large numbers of partitions on a broker.
I was recently experimenting with a third-party monitoring framework which provides a JMX collector [1] with the same mechanism to filter out the JMX beans retrieved from Kafka. When running a couple of tests with all filters removed, the time to fetch all beans could become quickly prohibitive as the number of partitions on the tested broker increased. After some investigation, the main source of "friction" was found in the (too) many RMI RPCs required to fetch the names and attributes of the JMX beans. Configuring the same JMX collector to run as a JVM agent, and taking care of unplugging the JMX-RMI connector, yielded significant gains (*). Note that this was obtained by fetching the beans via HTTP, with all values sent in a batch. I find one of the potential follow-up mentioned (exposing the beans via an alternative API) also very interesting from a performance perspective. [1] https://github.com/prometheus/jmx_exporter (*) On a 4-cores Xeon 8175M broker, hosting 1,000 replicas, the time to fetch all beans dropped from 13 seconds to ~400 ms. Le ven. 8 nov. 2019 à 17:29, Guozhang Wang <wangg...@gmail.com> a écrit : > Sounds good, thanks. > > Guozhang > > On Fri, Nov 8, 2019 at 9:26 AM Xavier Léauté <xav...@confluent.io> wrote: > > > > > > > 1. I do feel there're similar needs for clients make JMX configurable. > > Some > > > context: in modules like Connect and Streams we have added / > refactored a > > > large number of metrics so far [0, 1], and although we've added a > > reporting > > > level config [2] to clients, this is statically defined at code and > > cannot > > > be dynamically changed either. > > > > > > > Thanks for providing some context there, I have updated the KIP to add > > equivalent configs for clients, streams, and connect > > > > > > > 2. This may be out of the scope of this KIP, but have you thought about > > how > > > to make the metrics collection to be configurable (i.e. basically for > > those > > > metrics which we know would not be exposed, we do not collect them > > either) > > > dynamically? > > > > > > Yes, given what you described above, it would make sense to look into > this. > > One difficulty though, is that we'd probably want to define this at the > > sensor level, > > which does not always map to the metric names users understand. > > > > There are also cases where someone may want to expose different sets of > > metrics > > using different reporters, so I think a reporting level config is still > > useful. > > For this KIP, I am proposing we stick to making reporting configurable, > > independent of the underlying collection mechanism. > > > > > -- > -- Guozhang >
