[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16752726#comment-16752726 ] ASF GitHub Bot commented on KAFKA-5692: --- junrao commented on pull request #3848: KAFKA-5692: Change PreferredReplicaLeaderElectionCommand to use Admin… URL: https://github.com/apache/kafka/pull/3848 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement > Components: admin, core >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip, patch-available > Fix For: 2.2.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719228#comment-16719228 ] Jun Rao commented on KAFKA-5692: [~tombentley], that's good to know. Thanks for your help. > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip, patch-available > Fix For: 2.2.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719212#comment-16719212 ] Tom Bentley commented on KAFKA-5692: [~junrao] I should have time in the next few weeks to work on this PR again. > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip, patch-available > Fix For: 2.2.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713395#comment-16713395 ] Jun Rao commented on KAFKA-5692: [~tombentley], thanks for contributing the PR. Would be still be interested in finishing up the PR? If not, perhaps we can unassign the Jira and see if someone else wants to finish this up. > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Improvement >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: kip, patch-available > Fix For: 2.2.0 > > > The PreferredReplicaLeaderElectionCommand currently uses a direct connection > to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16164874#comment-16164874 ] ASF GitHub Bot commented on KAFKA-5692: --- GitHub user tombentley opened a pull request: https://github.com/apache/kafka/pull/3848 KAFKA-5692: Change PreferredReplicaLeaderElectionCommand to use Admin… …Client See also KIP-183. The contribution is my original work and I license the work to the project under the project's open source license. This implements the following algorithm: 1. AdminClient sends ElectPreferredLeadersRequest. 2. KafakApis receives ElectPreferredLeadersRequest and delegates to ReplicaManager.electPreferredLeaders() 3. ReplicaManager delegates to KafkaController.electPreferredLeaders() 4. KafkaController adds a PreferredReplicaLeaderElection to the EventManager, 5. ReplicaManager.electPreferredLeaders()'s callback uses the delayedElectPreferredReplicasPurgatory to wait for the results of the election to appear in the metadata cache. If there are no results because of errors, or because the preferred leaders are already leading the partitions then a response is returned immediately. In the EventManager work thread the preferred leader is elected as follows: 1. The EventManager runs PreferredReplicaLeaderElection.process() 2. process() calls KafkaController.onPreferredReplicaElectionWithResults() 3. KafkaController.onPreferredReplicaElectionWithResults() calls the PartitionStateMachine.handleStateChangesWithResults() to perform the election (asynchronously the PSM will send LeaderAndIsrRequest to the new and old leaders and UpdateMetadataRequest to all brokers) then invokes the callback. Note: the change in parameter type for CollectionUtils.groupDataByTopic(). This makes sense because the AdminClient APIs use Collection consistently, rather than List or Set. If binary compatiblity is a consideration the old version should be kept, delegating to the new version. I had to add PartitionStateMachine.handleStateChangesWithResults() in order to be able to process a set of state changes in the PartitionStateMachine *and get back individual results*. At the same time I noticed that all callers of existing handleStateChange() were destructuring a TopicAndPartition that they already had in order to call handleStateChange(), and that handleStateChange() immediately instantiated a new TopicAndPartition. Since TopicAndPartition is immutable this is pointless, so I refactored it. handleStateChange() also now returns any exception it caught, which is necessary for handleStateChangesWithResults() You can merge this pull request into a Git repository by running: $ git pull https://github.com/tombentley/kafka KAFKA-5692-elect-preferred Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/3848.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3848 commit 6b9bf178049e1eedfb5f07771cc3c595c02484d9 Author: Tom Bentley Date: 2017-09-06T14:39:24Z KAFKA-5692: Change PreferredReplicaLeaderElectionCommand to use AdminClient See also KIP-183. This implements the following algorithm: 1. AdminClient sends ElectPreferredLeadersRequest. 2. KafakApis receives ElectPreferredLeadersRequest and delegates to ReplicaManager.electPreferredLeaders() 3. ReplicaManager delegates to KafkaController.electPreferredLeaders() 4. KafkaController adds a PreferredReplicaLeaderElection to the EventManager, 5. ReplicaManager.electPreferredLeaders()'s callback uses the delayedElectPreferredReplicasPurgatory to wait for the results of the election to appear in the metadata cache. If there are no results because of errors, or because the preferred leaders are already leading the partitions then a response is returned immediately. In the EventManager work thread the preferred leader is elected as follows: 1. The EventManager runs PreferredReplicaLeaderElection.process() 2. process() calls KafkaController.onPreferredReplicaElectionWithResults() 3. KafkaController.onPreferredReplicaElectionWithResults() calls the PartitionStateMachine.handleStateChangesWithResults() to perform the election (asynchronously the PSM will send LeaderAndIsrRequest to the new and old leaders and UpdateMetadataRequest to all brokers) then invokes the callback. Note: the change in parameter type for CollectionUtils.groupDataByTopic(). This makes sense because the AdminClient APIs use Collection consistently, rather
[jira] [Commented] (KAFKA-5692) Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
[ https://issues.apache.org/jira/browse/KAFKA-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111088#comment-16111088 ] Tom Bentley commented on KAFKA-5692: This is now covered by [KIP-183|https://cwiki.apache.org/confluence/display/KAFKA/KIP-183+-+Change+PreferredReplicaLeaderElectionCommand+to+use+AdminClient] > Refactor PreferredReplicaLeaderElectionCommand to use AdminClient > - > > Key: KAFKA-5692 > URL: https://issues.apache.org/jira/browse/KAFKA-5692 > Project: Kafka > Issue Type: Bug >Reporter: Tom Bentley >Assignee: Tom Bentley >Priority: Minor > Labels: needs-kip > > The PreferredReplicaLeaderElectionCommand currently uses a direction > connection to zookeeper. The zookeeper dependency should be deprecated and an > AdminClient API created to be used instead. > This change will require a KIP. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
