Haifeng Chen created KAFKA-21419:
------------------------------------
Summary: KRaft controller drops still-adding replicas from
addingReplicas when an in-progress reassignment is overridden
Key: KAFKA-21419
URL: https://issues.apache.org/jira/browse/KAFKA-21419
Project: Kafka
Issue Type: Bug
Components: controller
Affects Versions: 4.3.0, 4.2.0, 4.1.0, 4.0.0
Reporter: Haifeng Chen
When a partition reassignment is still in progress and a new reassignment for
the same partition is submitted (for example with
{{{}kafka-reassign-partitions.sh --execute --additional{}}}, which Cruise
Control does routinely), the KRaft controller computes the new
{{addingReplicas}} as {{{}target − currentReplicas{}}}. {{currentReplicas}}
already contains the replicas the first reassignment is still adding, so any of
them that are also in the new target are silently promoted to plain replicas
even though they have not joined the ISR yet.
The override then completes as soon as the replicas it newly adds are in sync,
leaving the partition with a replica that is out of the ISR and that no
reassignment accounts for: {{--list}} shows nothing in flight, {{--verify}}
reports success, under-replicated partitions stay non-zero until the replica
happens to catch up, and the removing replica is dropped while the partition
has fewer in-sync replicas than it looks.
Reproduction (any KRaft cluster):
# Partition on {{{}[1,2,3]{}}}; reassign to {{[1,2,4]}} and hold broker 4 out
of the ISR (e.g. 1 B/s follower throttle). {{--list}} shows {{adding: 4.
removing: 3.}}
# While 4 is still adding, run {{--execute --additional}} with
{{{}[1,2,4,5]{}}}.
# {{--list}} shows {{adding: 5. removing: 3.}} — 4 is gone from adding.
# Once 5 is in sync the reassignment completes: {{{}replicas=[1,2,4,5]
isr=[1,2,5]{}}}, no reassignment in progress.
Expected: step 3 shows {{adding: 4,5. removing: 3.}} and the reassignment stays
in progress until 4 is also in sync.
The override case is only correct today when the new adding set happens to be
empty (the builder leaves the old one in place) or when the RF-reduction guard
{{adding.size() < removing.size()}} blocks completion, which masks the bug for
shrinking overrides.
ZooKeeper-mode controllers computed the new adding set relative to the original
replica set, so this is a KRaft regression.
Fix: derive the new adding set as {{{}target − (currentReplicas −
currentAddingReplicas){}}}, so a replica that was still being added by the
previous reassignment stays in {{addingReplicas}} if the new target keeps it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
