Hi List, Our requirement is to have the producers wait (indefinitely) until kafka broker comes up (in case of maintenance or network interruptions). We have disk caching in front of producer, when the producer waits and it drains when the producer eventually reconnects. This functionality was working fine in Kafka 0.8.2.1. The producer settings I used are.
block.on.buffer.full=true retries=MAXINT acks=all reconnect.backoff.ms=10000 max.in.flight.requests.per.connection=1 (I would like to acknowledge Gwen Shapiras slideshare http://www.slideshare.net/gwenshap/kafka-reliability-when-it-absolutely-positively-has-to-be-there on which this setting was derived for 0.8.2.1) Issue: -------- I am currently evaluating an upgrade to the latest version and what I observe is that this setting no longer works. I debugged the code a bit and observed that the changes added for https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient could be causing this. To give more details. Thread 1: BufferPool. Allocate() moreMemory.await(maxTimeToBlock, TimeUnit.MILLISECONDS) => Waiting Thread 2: Sender.run() accumulator.abortExpiredBatches(this.requestTimeout => Defaults to 30 seconds RecordAccumilator.deallocate(batch) BufferPool.deallocate() moreMem.signal() => This will wake up the first thread. So looks like the property value set by max.block.ms is overridden by the value of request.timeout.ms. I am not fully sure if this behavior is intended. If that's the case it would be good to indicate in the documentation. Also, I would be interested to know what is the ideal producer configuration setting for preventing data loss for 0.9.x. Many Thanks, Atul Soman.