[jira] [Commented] (KAFKA-2334) Prevent HW from going back during leader failover
[ https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030772#comment-16030772 ] Ismael Juma commented on KAFKA-2334: Is this still an issue? > Prevent HW from going back during leader failover > -- > > Key: KAFKA-2334 > URL: https://issues.apache.org/jira/browse/KAFKA-2334 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.2.1 >Reporter: Guozhang Wang >Assignee: Neha Narkhede > Labels: reliability > Fix For: 0.11.1.0 > > > Consider the following scenario: > 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as > the follower. > 1. A producer keep sending to Kafka with ack=-1. > 2. A consumer repeat issuing ListOffset request to Kafka. > And the following sequence: > 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2. > 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO > becomes 100) and hold the request in purgatory. > 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and > returns the 100 messages. > 3. B2 append its received message to local log (LEO becomes 100). > 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing > that B2's LEO has caught up to 100, and hence update its own HW, and > satisfying the ProduceRequest in purgatory, and sending the FetchResponse > with HW 100 back to B2 ASYNCHRONOUSLY. > 5. B1 successfully sends the ProduceResponse to the producer, and then fails, > hence the FetchResponse did not reach B2, whose HW remains 0. > From the consumer's point of view, it could first see the latest offset of > 100 (from B1), and then see the latest offset of 0 (from B2), and then the > latest offset gradually catch up to 100. > This is because we use HW to guard the ListOffset and > Fetch-from-ordinary-consumer. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KAFKA-2334) Prevent HW from going back during leader failover
[ https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15032642#comment-15032642 ] Guozhang Wang commented on KAFKA-2334: -- [~jinxing6...@126.com] The problem with this scenario is not related to LeaderAndISRRequest but in follower's FetchResponse, which will make the follower to update HW according to the returned value: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaFetcherThread.scala#L120 If the Fetch Response is not returned from the previous leader, then this follower will not update its HW and when itself becoming the new leader, the HW will effectively "go back" from the consumer point of view. > Prevent HW from going back during leader failover > -- > > Key: KAFKA-2334 > URL: https://issues.apache.org/jira/browse/KAFKA-2334 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.2.1 >Reporter: Guozhang Wang >Assignee: Neha Narkhede > Fix For: 0.10.0.0 > > > Consider the following scenario: > 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as > the follower. > 1. A producer keep sending to Kafka with ack=-1. > 2. A consumer repeat issuing ListOffset request to Kafka. > And the following sequence: > 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2. > 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO > becomes 100) and hold the request in purgatory. > 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and > returns the 100 messages. > 3. B2 append its received message to local log (LEO becomes 100). > 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing > that B2's LEO has caught up to 100, and hence update its own HW, and > satisfying the ProduceRequest in purgatory, and sending the FetchResponse > with HW 100 back to B2 ASYNCHRONOUSLY. > 5. B1 successfully sends the ProduceResponse to the producer, and then fails, > hence the FetchResponse did not reach B2, whose HW remains 0. > From the consumer's point of view, it could first see the latest offset of > 100 (from B1), and then see the latest offset of 0 (from B2), and then the > latest offset gradually catch up to 100. > This is because we use HW to guard the ListOffset and > Fetch-from-ordinary-consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2334) Prevent HW from going back during leader failover
[ https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022117#comment-15022117 ] jin xing commented on KAFKA-2334: - after receiving LeaderAndIsrRequest, Broker B2 will finally call " Partition::makeLeader", part of code is as below: ... zkVersion = leaderAndIsr.zkVersion leaderReplicaIdOpt = Some(localBrokerId) // construct the high watermark metadata for the new leader replica val newLeaderReplica = getReplica().get newLeaderReplica.convertHWToLocalOffsetMetadata() // reset log end offset for remote replicas assignedReplicas.foreach(r => if (r.brokerId != localBrokerId) r.logEndOffset = LogOffsetMetadata.UnknownOffsetMetadata) // we may need to increment high watermark since ISR could be down to 1 maybeIncrementLeaderHW(newLeaderReplica) if (topic == OffsetManager.OffsetsTopicName) offsetManager.loadOffsetsFromLog(partitionId) ... I can tell Broker B2 will first set 'leaderReplicaIdOpt = Some(localBrokerId)', and then try to update high watermark; by setting leaderReplicaIdOpt, Broker B2 will be available for consumer(if the consumer send fetchReqeust, there will be no NotLeaderForPartitionException); In the short interval which after 'leaderReplicaIdOpt = Some(localBrokerId)' and before setting up hw, what the consumer get is the "gone back" hw; If my understanding is wright, just reverse the order of setting up leaderReplicaIdOpt and updating high watermark will fix this issue; am I wrong ? > Prevent HW from going back during leader failover > -- > > Key: KAFKA-2334 > URL: https://issues.apache.org/jira/browse/KAFKA-2334 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.2.1 >Reporter: Guozhang Wang >Assignee: Neha Narkhede > Fix For: 0.10.0.0 > > > Consider the following scenario: > 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as > the follower. > 1. A producer keep sending to Kafka with ack=-1. > 2. A consumer repeat issuing ListOffset request to Kafka. > And the following sequence: > 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2. > 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO > becomes 100) and hold the request in purgatory. > 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and > returns the 100 messages. > 3. B2 append its received message to local log (LEO becomes 100). > 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing > that B2's LEO has caught up to 100, and hence update its own HW, and > satisfying the ProduceRequest in purgatory, and sending the FetchResponse > with HW 100 back to B2 ASYNCHRONOUSLY. > 5. B1 successfully sends the ProduceResponse to the producer, and then fails, > hence the FetchResponse did not reach B2, whose HW remains 0. > From the consumer's point of view, it could first see the latest offset of > 100 (from B1), and then see the latest offset of 0 (from B2), and then the > latest offset gradually catch up to 100. > This is because we use HW to guard the ListOffset and > Fetch-from-ordinary-consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2334) Prevent HW from going back during leader failover
[ https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009364#comment-15009364 ] Albert Strasheim commented on KAFKA-2334: - Hitting this issue in production with 0.8.2.1 on a large cluster. Fix appreciated. :) > Prevent HW from going back during leader failover > -- > > Key: KAFKA-2334 > URL: https://issues.apache.org/jira/browse/KAFKA-2334 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 0.8.2.1 >Reporter: Guozhang Wang >Assignee: Neha Narkhede > Fix For: 0.10.0.0 > > > Consider the following scenario: > 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as > the follower. > 1. A producer keep sending to Kafka with ack=-1. > 2. A consumer repeat issuing ListOffset request to Kafka. > And the following sequence: > 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2. > 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO > becomes 100) and hold the request in purgatory. > 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and > returns the 100 messages. > 3. B2 append its received message to local log (LEO becomes 100). > 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing > that B2's LEO has caught up to 100, and hence update its own HW, and > satisfying the ProduceRequest in purgatory, and sending the FetchResponse > with HW 100 back to B2 ASYNCHRONOUSLY. > 5. B1 successfully sends the ProduceResponse to the producer, and then fails, > hence the FetchResponse did not reach B2, whose HW remains 0. > From the consumer's point of view, it could first see the latest offset of > 100 (from B1), and then see the latest offset of 0 (from B2), and then the > latest offset gradually catch up to 100. > This is because we use HW to guard the ListOffset and > Fetch-from-ordinary-consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2334) Prevent HW from going back during leader failover
[ https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627040#comment-14627040 ] Guozhang Wang commented on KAFKA-2334: -- One possible solution to this issue is to let the new leader only become available (i.e. start accepting Produce / Fetch requests for the partition) after its HW caught up with its LEO. This will likely increase the unavailability latency a bit, in practice it should not cause much performance implication since most of the time its HW == LEO, and even not it will quickly catch up. The tricky part is how to implement it without introducing too much logic complexity on the broker side. Prevent HW from going back during leader failover -- Key: KAFKA-2334 URL: https://issues.apache.org/jira/browse/KAFKA-2334 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Fix For: 0.9.0 Consider the following scenario: 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as the follower. 1. A producer keep sending to Kafka with ack=-1. 2. A consumer repeat issuing ListOffset request to Kafka. And the following sequence: 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2. 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO becomes 100) and hold the request in purgatory. 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and returns the 100 messages. 3. B2 append its received message to local log (LEO becomes 100). 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing that B2's LEO has caught up to 100, and hence update its own HW, and satisfying the ProduceRequest in purgatory, and sending the FetchResponse with HW 100 back to B2 ASYNCHRONOUSLY. 5. B1 successfully sends the ProduceResponse to the producer, and then fails, hence the FetchResponse did not reach B2, whose HW remains 0. From the consumer's point of view, it could first see the latest offset of 100 (from B1), and then see the latest offset of 0 (from B2), and then the latest offset gradually catch up to 100. This is because we use HW to guard the ListOffset and Fetch-from-ordinary-consumer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
