[jira] [Commented] (KAFKA-3085) BrokerChangeListener computes inconsistent live/dead broker list
[ https://issues.apache.org/jira/browse/KAFKA-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091625#comment-15091625 ] ASF GitHub Bot commented on KAFKA-3085: --- GitHub user dajac opened a pull request: https://github.com/apache/kafka/pull/752 KAFKA-3085: BrokerChangeListener computes inconsistent live/dead broker list You can merge this pull request into a Git repository by running: $ git pull https://github.com/dajac/kafka KAFKA-3085 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/752.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #752 commit 706cf9248769bb408ec228c11f7b717770b573d0 Author: David Jacot Date: 2016-01-11T08:51:25Z BrokerChangeListener computes inconsistent live/dead broker list > BrokerChangeListener computes inconsistent live/dead broker list > > > Key: KAFKA-3085 > URL: https://issues.apache.org/jira/browse/KAFKA-3085 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: David Jacot > > On a broker change ZK event, BrokerChangeListener gets the current broker > list from ZK. It then computes a new broker list, a dead broker list, and a > live broker list with more detailed broker info. The new and live broker list > are computed by reading the value associated with each of the current broker > twice. If a broker is de-registered in between, these two list will not be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3085) BrokerChangeListener computes inconsistent live/dead broker list
[ https://issues.apache.org/jira/browse/KAFKA-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092892#comment-15092892 ] ASF GitHub Bot commented on KAFKA-3085: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/752 > BrokerChangeListener computes inconsistent live/dead broker list > > > Key: KAFKA-3085 > URL: https://issues.apache.org/jira/browse/KAFKA-3085 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: David Jacot > > On a broker change ZK event, BrokerChangeListener gets the current broker > list from ZK. It then computes a new broker list, a dead broker list, and a > live broker list with more detailed broker info. The new and live broker list > are computed by reading the value associated with each of the current broker > twice. If a broker is de-registered in between, these two list will not be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3085) BrokerChangeListener computes inconsistent live/dead broker list
[ https://issues.apache.org/jira/browse/KAFKA-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15092923#comment-15092923 ] Jun Rao commented on KAFKA-3085: [~dajac], sorry, I overlooked this when reviewing the patch. The fix in the above PR is still not complete. So I am leaving this jira open. The issue is that we compute things like newBrokerIds based on currentBrokerList, which is the list of broker id returned from ZK. However, if a broker disappears from ZK when we try to read the broker info using the broker id, the broker essentially doesn't exist and shouldn't be included in newBrokerIds. So, perhaps the fix should be (1) compute curBrokers from currentBrokerList and only include live brokers at that time; (2) derive newBrokerIds and deadBrokerIds from curBrokers by comparing with controllerContext.liveOrShuttingDownBrokerIds. Do you want to give this another try? Thanks, > BrokerChangeListener computes inconsistent live/dead broker list > > > Key: KAFKA-3085 > URL: https://issues.apache.org/jira/browse/KAFKA-3085 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: David Jacot > > On a broker change ZK event, BrokerChangeListener gets the current broker > list from ZK. It then computes a new broker list, a dead broker list, and a > live broker list with more detailed broker info. The new and live broker list > are computed by reading the value associated with each of the current broker > twice. If a broker is de-registered in between, these two list will not be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3085) BrokerChangeListener computes inconsistent live/dead broker list
[ https://issues.apache.org/jira/browse/KAFKA-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093334#comment-15093334 ] David Jacot commented on KAFKA-3085: [~junrao] No worries. I'll update it accordingly. > BrokerChangeListener computes inconsistent live/dead broker list > > > Key: KAFKA-3085 > URL: https://issues.apache.org/jira/browse/KAFKA-3085 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: David Jacot > > On a broker change ZK event, BrokerChangeListener gets the current broker > list from ZK. It then computes a new broker list, a dead broker list, and a > live broker list with more detailed broker info. The new and live broker list > are computed by reading the value associated with each of the current broker > twice. If a broker is de-registered in between, these two list will not be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3085) BrokerChangeListener computes inconsistent live/dead broker list
[ https://issues.apache.org/jira/browse/KAFKA-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093411#comment-15093411 ] ASF GitHub Bot commented on KAFKA-3085: --- GitHub user dajac opened a pull request: https://github.com/apache/kafka/pull/756 KAFKA-3085: BrokerChangeListener computes inconsistent live/dead broker list. Follow up PR as per comments in the ticket. @junrao It should be correct now as `curBrokers` included only live brokers and live/dead brokers are computed based on it. Could you take a look when you have time? You can merge this pull request into a Git repository by running: $ git pull https://github.com/dajac/kafka KAFKA-3085 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/756.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #756 commit 94edafcf9d77a25ef5506cff9a3e9a1e76ff14ca Author: David Jacot Date: 2016-01-12T06:42:40Z BrokerChangeListener computes inconsistent live/dead broker list. > BrokerChangeListener computes inconsistent live/dead broker list > > > Key: KAFKA-3085 > URL: https://issues.apache.org/jira/browse/KAFKA-3085 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: David Jacot > > On a broker change ZK event, BrokerChangeListener gets the current broker > list from ZK. It then computes a new broker list, a dead broker list, and a > live broker list with more detailed broker info. The new and live broker list > are computed by reading the value associated with each of the current broker > twice. If a broker is de-registered in between, these two list will not be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-3085) BrokerChangeListener computes inconsistent live/dead broker list
[ https://issues.apache.org/jira/browse/KAFKA-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104191#comment-15104191 ] ASF GitHub Bot commented on KAFKA-3085: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/756 > BrokerChangeListener computes inconsistent live/dead broker list > > > Key: KAFKA-3085 > URL: https://issues.apache.org/jira/browse/KAFKA-3085 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 0.9.0.0 >Reporter: Jun Rao >Assignee: David Jacot > Fix For: 0.9.1.0 > > > On a broker change ZK event, BrokerChangeListener gets the current broker > list from ZK. It then computes a new broker list, a dead broker list, and a > live broker list with more detailed broker info. The new and live broker list > are computed by reading the value associated with each of the current broker > twice. If a broker is de-registered in between, these two list will not be > consistent. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
