Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-07-23 Thread Guozhang Wang
to link the purging rate to the rate at which records > are produced? > > Best, > > Pieter > > -Oorspronkelijk bericht- > Van: Guozhang Wang > Verzonden: Monday, 17 June 2019 22:14 > Aan: users@kafka.apache.org > Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors f

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-24 Thread Pieter Hameete
nue this work via Slack if you prefer. > > Best, > > Pieter > > -Oorspronkelijk bericht- > Van: Guozhang Wang > Verzonden: Wednesday, 12 June 2019 20:30 > Aan: users@kafka.apache.org > Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams > applications

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-17 Thread Guozhang Wang
-Oorspronkelijk bericht- > Van: Guozhang Wang > Verzonden: Wednesday, 12 June 2019 20:30 > Aan: users@kafka.apache.org > Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams > applications > > @Jonathan Actually after double checking the code I realized > http

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-13 Thread Pieter Hameete
ieter -Oorspronkelijk bericht- Van: Guozhang Wang Verzonden: Wednesday, 12 June 2019 20:30 Aan: users@kafka.apache.org Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications @Jonathan Actually after double checking the code I realized https://issues.apache.org

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-12 Thread Guozhang Wang
gt; > Best, > > Pieter > > -Oorspronkelijk bericht- > Van: Jonathan Santilli > Verzonden: Tuesday, 11 June 2019 14:14 > Aan: Kafka Users > Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams > applications > > Hello, > > we cu

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-12 Thread Pieter Hameete
ct to these default settings. Best, Pieter -Oorspronkelijk bericht- Van: Jonathan Santilli Verzonden: Tuesday, 11 June 2019 14:14 Aan: Kafka Users Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications Hello, we currently see the same WARN logs when the App

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-11 Thread Jonathan Santilli
.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer. > > > Best, > > Pieter > > -Oorspronkelijk bericht- > Van: Guozhang Wang > Verzonden: Monday, 10 June 2019 22:19 > Aan: users@kafka.apache.org > Onderwerp: Re: Repeating UNKNOWN_P

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-11 Thread Pieter Hameete
iki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer. Best, Pieter -Oorspronkelijk bericht- Van: Guozhang Wang Verzonden: Monday, 10 June 2019 22:19 Aan: users@kafka.apache.org Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams app

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-10 Thread Guozhang Wang
gt; resolve the errors in our case. > > > > Do you think the changes listed under KIP-360 > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+h > > andling+of+unknown+producer > > could solve what we are seeing here? > > > > Best, >

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-07 Thread Pieter Hameete
8:16 Aan: users@kafka.apache.org Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications Pieter, KIP-360 should be able to fix it, yes, but it won't be completed soon, and the earliest release it may get in is in 2.4. At the mean time, I thought that be increasing the

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-06 Thread Guozhang Wang
https://cwiki.apache.org/confluence/display/KAFKA/KIP-360%3A+Improve+handling+of+unknown+producer > could solve what we are seeing here? > > Best, > > Pieter > > -Oorspronkelijk bericht- > Van: Pieter Hameete > Verzonden: Thursday, 6 June 2019 13:39 > Aan: us

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-06 Thread Pieter Hameete
> Verzonden: Wednesday, 5 June 2019 11:23 > Aan: users@kafka.apache.org > Onderwerp: RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams > applications > > Hi Guozhang, > > Thanks for your reply! I noticed my original mail went out twice by > accident, sorry for that. &

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-06 Thread Pieter Hameete
e the broker throws an UNKNOWN_PRODUCER_ID exception? Best, Pieter -Oorspronkelijk bericht- Van: Guozhang Wang Verzonden: Thursday, 6 June 2019 01:39 Aan: users@kafka.apache.org Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications Hello Pieter, It

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-05 Thread Guozhang Wang
19 11:23 > Aan: users@kafka.apache.org > Onderwerp: RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams > applications > > Hi Guozhang, > > Thanks for your reply! I noticed my original mail went out twice by > accident, sorry for that. > > We currently have a small v

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-05 Thread Matthias J. Sax
ing for any other topics so far. > > Best, > > Pieter > > -Oorspronkelijk bericht- > Van: Pieter Hameete > Verzonden: Wednesday, 5 June 2019 11:23 > Aan: users@kafka.apache.org > Onderwerp: RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams > applica

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-05 Thread Pieter Hameete
users@kafka.apache.org Onderwerp: RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications Hi Guozhang, Thanks for your reply! I noticed my original mail went out twice by accident, sorry for that. We currently have a small variety of keys so not all partitions are 'actively used&#x

RE: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-05 Thread Pieter Hameete
kafka.apache.org Onderwerp: Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications Hello Pieter, If you only have one record every few seconds that may be too small given you have at least 25 partitions (as I saw you have a xxx--repartition-24 partition), which means that for a si

Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-04 Thread Pieter Hameete
Hello, Our Kafka streams applications are showing the following warning every few seconds (on each of our 3 brokers, and on each of the 2 instances of the streams application): [Producer clientId=event-rule-engine-dd71ae9b-523c-425d-a7c0-c62993315b30-StreamThread-1-1_24-producer, transaction

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-04 Thread Guozhang Wang
Hello Pieter, If you only have one record every few seconds that may be too small given you have at least 25 partitions (as I saw you have a xxx--repartition-24 partition), which means that for a single partition, it may not see any records for a long time, and in this case you may need to overrid

Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

2019-06-04 Thread Pieter Hameete
Hello, Our Kafka streams applications are showing the following warning every few seconds (on each of our 3 brokers, and on each of the 2 instances of the streams application): [Producer clientId=event-rule-engine-dd71ae9b-523c-425d-a7c0-c62993315b30-StreamThread-1-1_24-producer, transaction