[
https://issues.apache.org/jira/browse/KAFKA-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13692071#comment-13692071
]
Senthil Chittibabu commented on KAFKA-936:
------------------------------------------
Finally the memory leak issue resolved after providing "consumer.id" property
during consumer creation. We found that kafka is not cleaning up all the
metrics object during the shutdown process. We read messages based on the
interval, so the application open/close consumer frequently.
By providing the static "consumer.id" looks like kafka is reusing the metrics
object, hence it stopped creating lots of metrics object. However now we see
duplicate message consumption by the consumer within the same group as side
effect. Basically we have 10 partition and 10 consumer thread, all reading the
same message even though they all have same "group.id".
I believe cleaning up all the metrics object during shutdown process is the
right fix for this memory leak. If you setup small testcase which open/close
consumer frequently, you can see the memory leak immediately. All Metrics
object like MetricName, EWMA, Histogram, etc will never gets garbage collected.
I am not sure why kafka needs unique consumer id to read the message from the
stream. Please advice.
> Kafka Metrics Memory Leak
> --------------------------
>
> Key: KAFKA-936
> URL: https://issues.apache.org/jira/browse/KAFKA-936
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8
> Environment: centos linux, jdk 1.6, jboss
> Reporter: Senthil Chittibabu
> Assignee: Neha Narkhede
> Priority: Critical
>
> I am using kafka_2.8.0-0.8.0-SNAPSHOT version. I am running into
> OutOfMemoryError in PermGen Space. I have set the -XX:MaxPermSize=512m, but I
> still get the same error. I used profiler to trace the memory leak, and found
> the following kafka classes to be the cause for the memory leak. Please let
> me know if you need any additional information to debug this issue.
> kafka.server.FetcherLagMetrics
> kafka.consumer.FetchRequestAndResponseMetrics
> kafka.consumer.FetchRequestAndResponseStats
> kafka.metrics.KafkaTimer
> kafka.utils.Pool
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
