[ 
https://issues.apache.org/jira/browse/KAFKA-9821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen updated KAFKA-9821:
-------------------------------
    Labels: need-kip  (was: )

> Partition may skip assignment with static members and incremental rebalances
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-9821
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9821
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer, streams
>            Reporter: Guozhang Wang
>            Priority: Major
>              Labels: need-kip
>
> When static membership (KIP-345) and incremental rebalancing (KIP-429) are 
> turned on at the same time, that upon failure it is possible some partitions 
> are not assigned to anyone. The event sequence is the following:
> 1. An assignment (task1) from rebalance is sent to an existing static member 
> with owned list (partition1, partition2), hence upon receiving the assignment 
> the static member is supposed to revoke partition2 and then re-join the group 
> to trigger another rebalance.
> 2. The member crashed before re-join the group, lost all of its assigned 
> partitions. However since this member is static with long session timeout, it 
> was not kicked out of the group yet at the coordinator side.
> 3. The member resumes and then re-join with a known instance.id. The 
> coordinator would not trigger a rebalance in this case and just give it the 
> previous assignment (partition1), and since the member has forgot about its 
> previous owned partitions it would just take partition1 and not re-join.
> 4. As a result the partition2 is not owned by this member any more but not 
> re-assigned to anyone; until the next rebalance it would not be fetched by 
> any member of the group.
> The key here is that today we are relying on the member's local memory to 
> calculate the added / revoked diff based on (owned, assigned). But if the 
> member crashed and lost all of its owned partition, AND if it is a static 
> member whose re-join would not trigger a new rebalance, this will break.
> After thinking about that I think 1) on the consumer side, maybe we should 
> augment the Assignment protocol with an error code indicating 
> rebalance_needed which is going to be persisted on broker’s offset topic as 
> well so that consumer member can learn about this without comparing the 
> assignment with its owned partitions.
> But 1) would affect the protocol and hence would only fix-forward future 
> versions; so 2) on the streams side, maybe we should leverage the existing 
> error field, to set REBALANCE_NEEDED for revocation as well (today it is only 
> set for version probing and a couple edge cases) so that streams can, upon 
> getting the error, force trigger a rebalance via the new consumer API. This 
> means that we can fix-backward in 2.4/2.5 as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to