Thanks Guozhang. This is a valid observation regarding the current status of the PR.
I updated the KIP to explicitly call out how the downgrade process should work in the section Compatibility, Deprecation, and Migration. Additionally, I reduced the configuration modes for the connect.protocol to only two: eager and compatible. That's because there's no way at the moment to select a protocol based on simple majority and not unanimity across at least one option for the sub-protocol. Therefore there's no way to lock a group of workers in a cooperative-only mode at the moment, if we account for accidental joins of workers running at an older version. The changes have been reflected in the KIP doc and will be reflected in the PR in a subsequent commit. Thanks, Konstantine On Thu, Mar 7, 2019 at 1:17 PM Guozhang Wang <wangg...@gmail.com> wrote: > Hi Konstantine, > > Thanks for the updated KIP and the PR as well (which is huge :) I briefly > looked through it as well as the KIP, and I have one minor comment to add > (otherwise I'm binding +1 on it as well) about the backward compatibility. > I'll use one example to illustrate the issue: > > 1) Suppose you have workerA and B on newer version and configured the > connect.protocol as "compatible", they will send both V0/V1 to the leader > (say it's workerA) who will choose V1 as the current protocol, this will be > sent back to A and B who would remember the current protocol version is > already V1. So after this rebalance everyone remembers that V1 can be used, > which means that upon prepareJoin they will not revoke all the assigned > tasks. > > 2) Now let's say a new worker joins but with old version V0 (practically > this is rare, but for illustration purposes some common scenarios may falls > into this, e.g. an existing worker being downgraded, which is essentially > as being kicked out of the group, and then rejoined as a new member on the > older version), the leader realized that at least one of the member does > not know V1 and hence would fall back to use version V0 to perform > assignment. V0 algorithm would do eager rebalance which may move some tasks > to the new comer immediately from the existing members, as it assumes that > everyone would revoke everything before join (a.k.a the sync-barrier) but > this is actually not true, since everyone other than the old versioned new > comer would still follow the behavior of V1 --- not revoking anything --- > before sending the join group request. > > This could be solvable though, e.g. when leader realized that he needs to > use V0, while the previous "currentProtocol" value is V1, instead of just > blindly follow the algorithm of V0 it could just reassign the existing > partitions without migrating anything, while at the same time tell everyone > that the currentProtocol version is downgraded to V0; and then they can > trigger another rebalance based on V0 where everything will revoke the > tasks before sending join group requests. > > > Guozhang > > On Wed, Mar 6, 2019 at 2:28 PM Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > I'd like to open the vote on KIP-415: Incremental Cooperative Rebalancing > > in Kafka Connect > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect > > > > a proposal that will allow Kafka Connect to scale significantly the > number > > of connectors and tasks it can run in a cluster of Connect workers. > > > > Thanks, > > Konstantine > > > > > -- > -- Guozhang >
