[
https://issues.apache.org/jira/browse/KAFKA-7660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702387#comment-16702387
]
John Roesler commented on KAFKA-7660:
-------------------------------------
[~pkleindl],
Ah, yes, now I'm remembering why I probably didn't wind up making this change
before. I restructured the sensor tracking so that we don't use parentSensors
for any of the Streams metrics.
The parentSensors map should be non-empty only if your code calls one of:
*
org.apache.kafka.streams.processor.internals.metrics.StreamsMetrics#addThroughputSensor
*
org.apache.kafka.streams.processor.internals.metrics.StreamsMetrics#addLatencyAndThroughputSensor
Is this the case? If so, you would need to call
org.apache.kafka.streams.StreamsMetrics#removeSensor in the relevant close()
method of the object that created the metrics.
If this is not the case, then I'm curious what the references are. I don't
suppose you can cause the apparent memory leak and then grab a heap dump? I'm
not sure if you'd want to put the whole heap dump on this forum, but maybe you
could take a look in it and see what objects are being referenced from the
parentSensors map.
Of course, it's possible (likely?) that the apparent memory leak has a
different cause, but it would be nice to rule this out, since it's on our radar.
I'll still keep my PR open, since, even if it's not affecting you, it would
still be a memory leak for anyone creating sensors via those methods.
> Stream Metrics - Memory Analysis
> --------------------------------
>
> Key: KAFKA-7660
> URL: https://issues.apache.org/jira/browse/KAFKA-7660
> Project: Kafka
> Issue Type: Bug
> Components: metrics, streams
> Affects Versions: 2.0.0
> Reporter: Patrik Kleindl
> Priority: Minor
> Attachments: Mem_Collections.jpeg, Mem_DuplicateStrings.jpeg,
> Mem_DuplicateStrings2.jpeg, Mem_Hotspots.jpeg, Mem_KeepAliveSet.jpeg,
> Mem_References.jpeg
>
>
> During the analysis of JVM memory two possible issues were shown which I
> would like to bring to your attention:
> 1) Duplicate strings
> Top findings:
> string_content="stream-processor-node-metrics" count="534,277"
> string_content="processor-node-id" count="148,437"
> string_content="stream-rocksdb-state-metrics" count="41,832"
> string_content="punctuate-latency-avg" count="29,681"
>
> "stream-processor-node-metrics" seems to be used in Sensors.java as a
> literal and not interned.
>
> 2) The HashMap parentSensors from
> org.apache.kafka.streams.processor.internals.StreamThread$StreamsMetricsThreadImpl
> was reported multiple times as suspicious for potentially keeping alive a
> lot of objects. In our case the reported size was 40-50MB each.
> I haven't looked too deep in the code but noticed that the class Sensor.java
> which is used as a key in the HashMap does not implement equals or hashCode
> method. Not sure this is a problem though.
>
> The analysis was done with Dynatrace 7.0
> We are running Confluent 5.0/Kafka2.0-cp1 (Brokers as well as Clients)
>
> Screenshots are attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
