[
https://issues.apache.org/jira/browse/KAFKA-7304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Yang updated KAFKA-7304:
---------------------------
Description:
We are testing secured writing to kafka through ssl. Testing at small scale,
ssl writing to kafka was fine. However, when we enabled ssl writing at a larger
scale (>40k clients writes concurrently), the kafka brokers soon hit
OutOfMemory issue with 4G memory setting. We have tried with increasing the
heap size to 10Gb, and hit the same issue.
We took a few heap dump , and found that most of the heap memory is referenced
through org.apache.kafka.common.network.Selector object. There are two Channel
maps field in Selector. It seems that somehow the objects is not deleted from
the map in a timely manner.
{code}
private final Map<String, KafkaChannel> channels;
private final Map<String, KafkaChannel> closingChannels;
{code}
Please see the attached images and the following link for sample gc analysis.
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTgvMDgvMTcvLS1nYy5sb2cuMC5jdXJyZW50Lmd6LS0yLTM5LTM0
was:
We are testing secured writing to kafka through ssl. Testing at small scale,
ssl writing to kafka was fine. However, when we enable ssl writing at scale
(>40k clients writes concurrently), the kafka brokers soon hit OutOfMemory
issue with 4G memory setting. We have tried with increasing the heap size to
10Gb, and hit the same issue.
We took a few heap dump , and found that most of the heap memory is referenced
through org.apache.kafka.common.network.Selector object. There are two Channel
maps field in Selector. It seems that somehow the objects is not deleted from
the map in a timely manner.
{code}
private final Map<String, KafkaChannel> channels;
private final Map<String, KafkaChannel> closingChannels;
{code}
Please see the attached images and the following link for sample gc analysis.
http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTgvMDgvMTcvLS1nYy5sb2cuMC5jdXJyZW50Lmd6LS0yLTM5LTM0
> memory leakage in org.apache.kafka.common.network.Selector
> ----------------------------------------------------------
>
> Key: KAFKA-7304
> URL: https://issues.apache.org/jira/browse/KAFKA-7304
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Yu Yang
> Priority: Major
> Attachments: Screen Shot 2018-08-16 at 11.04.16 PM.png, Screen Shot
> 2018-08-16 at 11.06.38 PM.png, Screen Shot 2018-08-16 at 12.41.26 PM.png,
> Screen Shot 2018-08-16 at 4.26.19 PM.png
>
>
> We are testing secured writing to kafka through ssl. Testing at small scale,
> ssl writing to kafka was fine. However, when we enabled ssl writing at a
> larger scale (>40k clients writes concurrently), the kafka brokers soon hit
> OutOfMemory issue with 4G memory setting. We have tried with increasing the
> heap size to 10Gb, and hit the same issue.
> We took a few heap dump , and found that most of the heap memory is
> referenced through org.apache.kafka.common.network.Selector object. There
> are two Channel maps field in Selector. It seems that somehow the objects is
> not deleted from the map in a timely manner.
> {code}
> private final Map<String, KafkaChannel> channels;
> private final Map<String, KafkaChannel> closingChannels;
> {code}
> Please see the attached images and the following link for sample gc
> analysis.
> http://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMTgvMDgvMTcvLS1nYy5sb2cuMC5jdXJyZW50Lmd6LS0yLTM5LTM0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
