Re: Kafka Producer avoid sending more records when met exception while also keeping performance

2024-03-13 Thread William Lee
t; kafka/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java at master · a0x8o/kafka <https://github.com/a0x8o/kafka/blob/master/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L538> Do you have any thoughts? Thanks and regards, William Lee Ric

Re: Kafka Producer avoid sending more records when met exception while also keeping performance

2024-03-11 Thread William Lee
a. > > Thanks, > Greg Harris > > [1] > https://kafka.apache.org/37/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html > [2] https://en.wikipedia.org/wiki/CAP_theorem > > On Mon, Mar 11, 2024 at 7:32 AM William Lee > wrote: > > > > Hi Haruki, &

Re: Kafka Producer avoid sending more records when met exception while also keeping performance

2024-03-11 Thread William Lee
Hi Haruki, Thanks for your answer. > I still don't get why you need this behavior though The reason is I have to ensure message ordering per partition strictly. Once there is an exception in the producer callback, it indicates that the exception is not a retryable exception(from kafka producer's

Kafka Producer avoid sending more records when met exception while also keeping performance

2024-03-11 Thread William Lee
Hi all, I am facing a problem when I detect an exception in kafka producer callback, I immediately stop sending more new records and stop the kafka producer, but some extra records were still sent. I found a way to resolve this issue: setting max.in.flight.requests.per.connection to 1 and closing