Data lose prevention configuration no longer works with Kafka 9.x
Hi List, Our requirement is to have the producers wait (indefinitely) until kafka broker comes up (in case of maintenance or network interruptions). We have disk caching in front of producer, when the producer waits and it drains when the producer eventually reconnects. This functionality was working fine in Kafka 0.8.2.1. The producer settings I used are. block.on.buffer.full=true retries=MAXINT acks=all reconnect.backoff.ms=1 max.in.flight.requests.per.connection=1 (I would like to acknowledge Gwen Shapiras slideshare http://www.slideshare.net/gwenshap/kafka-reliability-when-it-absolutely-positively-has-to-be-there on which this setting was derived for 0.8.2.1) Issue: I am currently evaluating an upgrade to the latest version and what I observe is that this setting no longer works. I debugged the code a bit and observed that the changes added for https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient could be causing this. To give more details. Thread 1: BufferPool. Allocate() moreMemory.await(maxTimeToBlock, TimeUnit.MILLISECONDS) => Waiting Thread 2: Sender.run() accumulator.abortExpiredBatches(this.requestTimeout=> Defaults to 30 seconds RecordAccumilator.deallocate(batch) BufferPool.deallocate() moreMem.signal() => This will wake up the first thread. So looks like the property value set by max.block.ms is overridden by the value of request.timeout.ms. I am not fully sure if this behavior is intended. If that's the case it would be good to indicate in the documentation. Also, I would be interested to know what is the ideal producer configuration setting for preventing data loss for 0.9.x. Many Thanks, Atul Soman. Note: I am resending this mail after subscribing to dev@kafka.apache.org<mailto:dev@kafka.apache.org> since I guess my first mail didn't go through.
Data lose prevention configuration no longer works with Kafka 9.x
Hi List, Our requirement is to have the producers wait (indefinitely) until kafka broker comes up (in case of maintenance or network interruptions). We have disk caching in front of producer, when the producer waits and it drains when the producer eventually reconnects. This functionality was working fine in Kafka 0.8.2.1. The producer settings I used are. block.on.buffer.full=true retries=MAXINT acks=all reconnect.backoff.ms=1 max.in.flight.requests.per.connection=1 (I would like to acknowledge Gwen Shapiras slideshare http://www.slideshare.net/gwenshap/kafka-reliability-when-it-absolutely-positively-has-to-be-there on which this setting was derived for 0.8.2.1) Issue: I am currently evaluating an upgrade to the latest version and what I observe is that this setting no longer works. I debugged the code a bit and observed that the changes added for https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient could be causing this. To give more details. Thread 1: BufferPool. Allocate() moreMemory.await(maxTimeToBlock, TimeUnit.MILLISECONDS) => Waiting Thread 2: Sender.run() accumulator.abortExpiredBatches(this.requestTimeout=> Defaults to 30 seconds RecordAccumilator.deallocate(batch) BufferPool.deallocate() moreMem.signal() => This will wake up the first thread. So looks like the property value set by max.block.ms is overridden by the value of request.timeout.ms. I am not fully sure if this behavior is intended. If that's the case it would be good to indicate in the documentation. Also, I would be interested to know what is the ideal producer configuration setting for preventing data loss for 0.9.x. Many Thanks, Atul Soman.
[jira] [Commented] (KAFKA-3119) Adding -daemon option to zookeeper-server-start.sh USAGE
[ https://issues.apache.org/jira/browse/KAFKA-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106125#comment-15106125 ] Atul Soman commented on KAFKA-3119: --- I am unable to assign this issue to me. Is it because i do not have some permissions. Can someone guide me on what i need to do to assign this issue to myself. > Adding -daemon option to zookeeper-server-start.sh USAGE > > > Key: KAFKA-3119 > URL: https://issues.apache.org/jira/browse/KAFKA-3119 > Project: Kafka > Issue Type: Improvement > Components: config >Affects Versions: 0.9.0.0 > Environment: All >Reporter: Atul Soman >Priority: Minor > Labels: easyfix > > zookeeper-server-start.sh used to start zookeeper already have the ability to > run in background using -daemon option. But while trying to check for its > usage pattern by running zookeeper-server-start.sh, it do not display -daemon > as a viable option. > It is good to expose all used features in the USAGE: section, especially for > a feature as critical as this which will be used a lot in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3119) Adding -daemon option to zookeeper-server-start.sh USAGE
[ https://issues.apache.org/jira/browse/KAFKA-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106103#comment-15106103 ] Atul Soman commented on KAFKA-3119: --- Created pull request https://github.com/apache/kafka/pull/785 with the fix for this issue. > Adding -daemon option to zookeeper-server-start.sh USAGE > > > Key: KAFKA-3119 > URL: https://issues.apache.org/jira/browse/KAFKA-3119 > Project: Kafka > Issue Type: Improvement > Components: config >Affects Versions: 0.9.0.0 > Environment: All >Reporter: Atul Soman >Priority: Minor > Labels: easyfix > > zookeeper-server-start.sh used to start zookeeper already have the ability to > run in background using -daemon option. But while trying to check for its > usage pattern by running zookeeper-server-start.sh, it do not display -daemon > as a viable option. > It is good to expose all used features in the USAGE: section, especially for > a feature as critical as this which will be used a lot in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-3119) Adding -daemon option to zookeeper-server-start.sh USAGE
Atul Soman created KAFKA-3119: - Summary: Adding -daemon option to zookeeper-server-start.sh USAGE Key: KAFKA-3119 URL: https://issues.apache.org/jira/browse/KAFKA-3119 Project: Kafka Issue Type: Improvement Components: config Affects Versions: 0.9.0.0 Environment: All Reporter: Atul Soman Priority: Minor zookeeper-server-start.sh used to start zookeeper already have the ability to run in background using -daemon option. But while trying to check for its usage pattern by running zookeeper-server-start.sh, it do not display -daemon as a viable option. It is good to expose all used features in the USAGE: section, especially for a feature as critical as this which will be used a lot in production. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
