Kafka producer huge memory usage (leak?)

Hello, We have a 3 broker Kafka 0.10.1.0 deployment in production. There are some applications which have Kafka Producers embedded in them which send application logs to a topic. This topic has 10 partitions with replication factor of 3. We are observing that memory usage on some of these applica

Re: Kafka producer huge memory usage (leak?)

Additionally, here's the producer config kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092 kafka.acks=0 kafka.key.serializer=org.apache.kafka.common.serialization.StringSerializer kafka.value.serializer=org.apache.kafka.common.serialization.StringSerializer kafka.max.block.ms=1000 kaf

Re: Kafka consumer offset topic deletion

> > Hello, > We have a Kafka 0.11.xx version setup. > So the system topic which is __consumer_offset, we are looking at many > such topics like __consumer_offset-1,2,4. > So one topic in particular,is now having log segments which is > contributing to 5GB of data. > I had a look at our server.p

Kafka consumer offset topic deletion

Hello, We have a Kafka 0.11.xx version setup. So the system topic which is __consumer_offset, we are looking at many such topics like __consumer_offset-1,2,4. So one topic in particular,is now having log segments which is contributing to 5GB of data. I had a look at our server.properties file b

min.insync.replicas not respected, boundary case ?

Hi, Testing out some kafka consistency guarantees I have following basic producer config: ProducerConfig.ACKS_CONFIG=all ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG=true ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION=1 ProducerConfig.RETRIES_CONFIG=3 My test setup is a 3 node kafka cluster (

Re: min.insync.replicas not respected, boundary case ?

Leader is a replica On Tue, 18 Sep 2018 at 22:52 jorg.heym...@gmail.com wrote: > Hi, > > Testing out some kafka consistency guarantees I have following basic > producer config: > > ProducerConfig.ACKS_CONFIG=all > ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG=true > ProducerConfig.MAX_IN_FLIGHT_REQUE

Question regarding consumer getting a new message on the topic subscribed poll/select

Hello, I have a basic question. Lets say a kafka consumer is subscribed to to a topic "test". When the producer publishes logs/records into that topic, does the consumer explicitly need to poll at regular intervals to find out if a new message is available and pull the new message or a notificati

Re: Completely clear out kafka brokers & zookeeper?

You can delete config files,clean up kafka data directories and zookeeper directories .That should do it. On Mon, Sep 17, 2018 at 5:31 PM Dylan Martin wrote: > I have some boxes that I'm using to test kafka configurations (and > zookeeper). What's the recommended procedure to clean them out to

Kafka stream issue : Deleting obsolete state directory

Hi, We have a stream application where we are facing a little strange problem with deleting state directory. Our application is deployed into in-house container platform and storage is backed by nfs. When app is running for a while and when it hits the cleanup delay of 60ms as defined dafaul

Re: Kafka stream issue : Deleting obsolete state directory

Hi Bhavesh, I don't see the log file attachment, can you resend it? Also, what version of Kafka Streams are you running? Thanks, Bill On Tue, Sep 18, 2018 at 12:31 PM Bhavesh Patel wrote: > Hi, > > We have a stream application where we are facing a little strange problem > with deleting state

Kafka consumer offset topic data

Hello, We have a Kafka 0.11.xx version setup. So the system topic which is __consumer_offset, we are looking at many such topics like __consumer_offset-1,2,4. So one topic in particular,is now having log segments which is contributing to 5GB of data. I had a look at our server.properties file b

Re: Kafka stream issue : Deleting obsolete state directory

Hi Bill, Apologies, Please find the error stack attached. We are using kafka 2.0. Thanks, Bhavesh Patel On Tue, Sep 18, 2018 at 8:04 PM Bill Bejeck wrote: > Hi Bhavesh, > > I don't see the log file attachment, can you resend it? > > Also, what version of Kafka Streams are you running? > > Tha

Re: Question regarding consumer getting a new message on the topic subscribed poll/select

You will need to `poll()` -- that is how Kafka works. The brokers practically don't even know that a consumer exist and thus cannot sent any notification. It's be design. -Matthias On 9/18/18 8:26 AM, Siddhartha Khaitan wrote: > Hello, > > I have a basic question. > > Lets say a kafka consume

Re: Kafka consumer offset topic deletion

Odd that the log compaction isn't working. What OS is your broker running on and can you please post your server.properties? On Wed, 19 Sep. 2018, 2:13 am Kaushik Nambiar, wrote: > > > > Hello, > > We have a Kafka 0.11.xx version setup. > > So the system topic which is __consumer_offset, we are

Re: Big Log Retention

In the particular case I'm referring to, I believe some partitions were replicated to new nodes. These new nodes had been added to the cluster as a result of human error, and were removed shortly thereafter. It's my hypothesis that this resulted in many partitions having their replicas on nodes tha

Re: Question regarding consumer getting a new message on the topic subscribed poll/select

Thanks a lot for the response. On Tue, Sep 18, 2018 at 3:21 PM, Matthias J. Sax wrote: > You will need to `poll()` -- that is how Kafka works. > > The brokers practically don't even know that a consumer exist and thus > cannot sent any notification. It's be design. > > > -Matthias > > On 9/18/18

Re: Kafka stream issue : Deleting obsolete state directory

Hi Bhavesh, Sorry, but I'm still not seeing any log file attachment, maybe it's being filtered out. Can you copy the relevant section in the body of the email? Thanks, Bill On Tue, Sep 18, 2018 at 3:21 PM Bhavesh Patel wrote: > Hi Bill, > > Apologies, Please find the error stack attached. > >

Questions about manage offset in external storage and consumer failure detect

Hi, I have two questions about Kafka consumer api: 1. I read the section "Storing Offsets Outside Kafka" in jdoc of KafkaConsumer. Is it safer or more efficient to manage offset with external storage compared to manage with Kafka(Zookeeper or topic _consumer_offset)? Or is there any benefit f

Re: Big Log Retention

Seems reasonable. Removing brokers should be done with care, and it's tricky to reassign partitions off of brokers that aren't online (i don't even try, I just bring up a broker with that ID and reassign them off). Depending on what you have running connected to kafka the reassigning may have bee

Re: Kafka consumer offset topic deletion

That's unusually large. Ours are around 32k-90mb each. Initially curious if you have log.cleaner.enable=true and what offsets.retention.minutes is set to. And yes it can affect cluster performance. We had instances of consumer outages that were caused by bugged large consumer offfset files, especi

State flush & recovery during failures

Hi All, I have KafkaStreams application (processor API) that does roughly the below steps. 1. read 2. add to state 3. check state size (count based trigger) 3.1 process 3.2 delete records from state 3.3 forward 3.4 commit (kafka internally does) 3.4.1 flush state 3.4.2 flush producer 3.4.3 commit

Re: Kafka producer huge memory usage (leak?)

Any thoughts on this matter? Someone, please help. On Tue, Sep 18, 2018 at 6:05 PM Shantanu Deshmukh wrote: > Additionally, here's the producer config > > kafka.bootstrap.servers=x.x.x.x:9092,x.x.x.x:9092,x.x.x.x:9092 > kafka.acks=0 > kafka.key.serializer=org.apache.kafka.common.serialization.St

Re: Kafka consumer offset topic deletion

Hello Bret, Thank you for your reply. For one consumer offset topic,I can see many log segments.Each log segment is around 100 mb. So due to many such log segments we are experiencing such data issues. Your views on this one Regards, Kaushik Nambiar On Wed, Sep 19, 2018, 10:37 AM Brett Rann wrot

Re: Kafka consumer offset topic deletion

Hello Bret, About the properties you mentioned earlier. I couldn't find any of these properties in my server.properties file. So we were assuming the default values would b in place. So I guess the default values for the above mentioned properties are true and 24 hours for Kafka 0.11.xx Regards, K

Re: Kafka producer huge memory usage (leak?)

Hi Shantanu, Please refer this . Hope it helps. Thanks, Subash On Wed, Sep 19, 2018 at 11:59 AM Shantanu Deshmukh wrote: > Any thoughts on this matter? Someone, please help. > > On Tue, Sep 18