[
https://issues.apache.org/jira/browse/KAFKA-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17904573#comment-17904573
]
Calvin Liu commented on KAFKA-17678:
------------------------------------
I suppose it is an expected behavior for the HWM which means the logs below it
should be persisted by all the ISR members and the size of the ISR should be
larger than the min ISR.
> Problematic new HWM increment behaviour introduced by KIP-207 and KIP-966
> -------------------------------------------------------------------------
>
> Key: KAFKA-17678
> URL: https://issues.apache.org/jira/browse/KAFKA-17678
> Project: Kafka
> Issue Type: Bug
> Components: replication
> Reporter: Francois Visconte
> Priority: Major
>
> We identified a bug/new behaviour that would lead to consumer lagging for a
> long time and ListOffsets requests failing during that time frame.
> While the ListOffsets requests failure is expected and has been introduced by
> KIP-207, the problematic behavior is more about the inability to increment
> highWatermark and the consequence of having lagging consumers.
> Here is the situation
> * We have a topic with min.isr=2
> * We have a partition on broker 16, 17 and 18
> * Leader for this partition is broker 17
> # Broker 18 failed. Partition has 2 ISRs
> # Broker 16 failed. Partition has 1 ISR (17)
> # Broker 7 has LEO higher than HWM:
> {{[Broker id=17] Leader topic-86 with topic id Some(yFhPOnPsRDiYHgfF2bR2aQ)
> starts at leader epoch 7 from offset 3067193660 with partition epoch 11, high
> watermark 3067191497, ISR [10017], adding replicas [] and removing replicas
> [] (under-min-isr). Previous leader Some(10017) and previous leader epoch was
> 6.}}
> At this point producers cannot produce to topic-86 partition because there is
> only one ISR, which is expected behavior.
> But it seems that KIP-207 prevent answering to ListOffsets requests here
>
> {code:java}
> // Only consider throwing an error if we get a client request (isolationLevel
> is defined) and the high watermark
> // is lagging behind the start offset
> val maybeOffsetsError: Option[ApiException] = leaderEpochStartOffsetOpt
> .filter(epochStart => isolationLevel.isDefined && epochStart >
> localLog.highWatermark)
> .map(epochStart => Errors.OFFSET_NOT_AVAILABLE.exception(s"Failed to fetch
> offsets for " +
> s"partition $topicPartition with leader $epochLogString as this partition's "
> +
> s"high watermark (${localLog.highWatermark}) is lagging behind the " +
> s"start offset from the beginning of this epoch ($epochStart).")){code}
> It seems that the path to get to the HWM being stuck for so long was
> introduced in preparation of KIP-966, see this ticket and PR.
> As a result:
> * The stuck HWM in the above scenario can also mean that a small part of
> messages isn't readable by consumers even though it was in the past.
> * In case of truncation, the HWM might still go backwards. This is still
> possible even with min.ISR, although it should be rare.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
