Re: Kafka producer huge memory usage (leak?)

2018-09-21 Thread Manikumar
Hi, In that case, it is not related to KAFKA-4493. I see StackOverFlowException in the uploaded heapdump. I am pretty sure this is related to StackOverFlowException. Some objects are dangling after this exception. You need to check/find out when this exception is occuring. check your application s

Re: Kafka producer huge memory usage (leak?)

2018-09-21 Thread Shantanu Deshmukh
Hi Manikumar, I am using correct protocol. SSL and truststore is also correctly configured and there is a single port which is for SSL. Otherwise it simply wouldn't work. Data is getting produced all fine, it's just that producer object is consuming massive amount of memory. On Fri, Sep 21, 2018 a

Re: Kafka producer huge memory usage (leak?)

2018-09-21 Thread Manikumar
Hi, Instead trying the PR, make sure you are setting valid security protocol and connecting to valid broker port. also looks for any errors in producer logs. Thanks, On Fri, Sep 21, 2018 at 12:35 PM Shantanu Deshmukh wrote: > Hi Manikumar, > > I checked this issue. There is this patch avai

Re: Kafka producer huge memory usage (leak?)

2018-09-21 Thread Shantanu Deshmukh
Hi Manikumar, I checked this issue. There is this patch available. https://github.com/apache/kafka/pull/2408.patch I pulled Kafka 0.10.1.0 from github. Then tried applying this patch. But several places I am getting error that patch doesn't apply. I am new to git and patching process. Can you gui

Re: Kafka producer huge memory usage (leak?)

2018-09-19 Thread Manikumar
Similar issue reported here:KAFKA-7304, but on broker side. maybe you can create a JIRA and upload the heap dump for analysis. On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh wrote: > Any thoughts on this matter? Someone, please help. > > On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh >

Re: Kafka producer huge memory usage (leak?)

2018-09-18 Thread Subash Konar
Hi Shantanu, Please refer this <https://stackoverflow.com/questions/52386300/kafka-producer-huge-memory-usage-leak/52386713#52386713>. Hope it helps. Thanks, Subash On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh wrote: > Any thoughts on this matter? Someone, please help. > &

Re: Kafka producer huge memory usage (leak?)

2018-09-18 Thread Shantanu Deshmukh
Any thoughts on this matter? Someone, please help. On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh wrote: > Additionally, here's the producer config > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092 > kafka.acks=0 > kafka.key.serializer=org.apache.kafka.common.serialization.St

Re: Kafka producer huge memory usage (leak?)

2018-09-18 Thread Shantanu Deshmukh
Additionally, here's the producer config kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092 kafka.acks=0 kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer kafka.max.block.ms=1000 kaf

Kafka producer huge memory usage (leak?)

2018-09-18 Thread Shantanu Deshmukh
Hello, We have a 3 broker Kafka 0.10.1.0 deployment in production. There are some applications which have Kafka Producers embedded in them which send application logs to a topic. This topic has 10 partitions with replication factor of 3. We are observing that memory usage on some of these applica