Yana, The full integration test passed without issues. Can you create a PR of your findings and the change so we can document it and give credit?
-Doug On Friday, December 13, 2024 at 1:44:47 AM UTC-5 Doug Hoard wrote: > Using String intern() can help in some scenarios and cause performance > issues in other scenarios. > > I made your change on my development branch and ran a quick test > (./quick-test.sh) and didn't see any unit test or integration test issues. > > I just started a full regression test and will have some results in the > morning (US time.) (It takes a little over 3 hours.) > > -Doug > > On Thursday, December 12, 2024 at 11:02:02 AM UTC-5 Яна Илиева wrote: > >> Hello, >> >> While using the Prometheus JMX Exporter as a Java agent in our >> application, we observed frequent GC clean ups and it turns out it is >> because the available heap space got exhausted by the caching of the >> Prometheus JMX Exporter. >> >> For context - we have an exporter yaml file with about 40 rules defined >> and there are many JMX MBeans which we process in order to get the required >> metrics. Without caching enabled for the rules, the request can take a >> minimum of 1 minute to complete. In an attempt to reduce this time, we >> enabled caching and observed that the MatchedRulesCache object can take >> around 600 MB in a particular case. There is a realistic potential in out >> case that this object grows above 1 GB, which is a huge amount of space for >> this kind of process. >> >> We identified that the major reason for the large size of >> MatchedRulesCache is that it contains duplicating String objects for the >> same MBean names. The object keeps all MBean names the rules were matched >> against previously, in order to avoid expensive pattern matching later. >> Although each rule caches the same set of MBean names, those are in fact >> separate objects in the heap. >> >> The origin of the matter was found in >> https://github.com/prometheus/jmx_exporter/blob/a3dac9acee1464531cd87502579178a1fec1cc76/collector/src/main/java/io/prometheus/jmx/JmxCollector.java#L584 >> >> where for each rule with caching enabled, there is a new String object >> created, although such could already exist in memory from a previous >> iteration. >> >> The duplication is by a factor of the number of rules with caching >> enabled. >> >> I experimented with interning the String, >> >> *String matchName = (beanName + attributeName + ": " + >> matchBeanValue).intern();* >> >> so that the JVM's string pool is utilized and the String objects reused. >> The result in speed and heap space used by cache is described below: >> >> [image: Screenshot from 2024-12-12 14-53-26.png] >> >> What do you think about this and do you find this suggestion could have >> negative consequences in certain cases? >> >> Kind regards, >> Yana Ilieva >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-developers/da7d418b-5fd4-4946-84dd-4d2897053190n%40googlegroups.com.
