+1 to make it into V3.1. And the partial and risk-free proposal to make default to ["range", "cooperative-sticky"] in 3.0 is a brilliant idea. Sounds good to me.
I can work on it for V3.0 if there's no objections. Thank you. Luke On Wed, Jun 23, 2021 at 12:38 PM Ismael Juma <ism...@juma.me.uk> wrote: > +1 to making the switch in 3.1 with more time versus rushing it in 3.0. > > Ismael > > On Tue, Jun 22, 2021 at 8:58 PM Sophie Blee-Goldman > <sop...@confluent.io.invalid> wrote: > > > I believe we've figured out the root cause of KAFKA-12896 > > <https://issues.apache.org/jira/browse/KAFKA-12896>, and should have a > fix > > prepared shortly. See the linked issues for more details. > > > > Regarding KIP-726 itself, given that the latest proposal is fully > > compatible and does not require any breaking changes, we may or may not > > push to get this into 3.0. Cooperative rebalancing is a huge improvement > > for the majority of consumer applications, but it's already available for > > those who want it, so I don't feel too badly about letting it slip from > 3.0 > > in order to focus on other things. I would also personally feel better > > about merging it at the beginning of 3.1 so we have the entire release > > cycle to flush out any potential regressions, rather than rushing it in > at > > the last moment (even though the majority of the work has been done for a > > while). > > > > If anyone feels strongly about getting this into 3.0, please let me know, > > as it's definitely still within reach. Otherwise I'll probably aim for > 3.1 > > instead. > > > > Note (@Luke in particular): we could opt for a partial and risk-free > > improvement in 3.0, that would make it easier for users to turn on > > cooperative rebalancing without actually enabling it for them (yet). If > we > > change the default config to ["range", "cooperative-sticky"] in 3.0, then > > the RangeAssignor will remain the default but any application can upgrade > > to the CooperativeStickyAssignor with just a single rolling bounce (to > > remove the "range" assignor), rather than the usual two. We can then go > > ahead and fully make the CooperativeStickyAssignor the default assignor > in > > 3.1 by changing the default config to ["cooperative-sticky", "range"] as > > this KIP has proposed. > > > > Thoughts? > > > > On Thu, Jun 10, 2021 at 1:46 AM Luke Chen <show...@gmail.com> wrote: > > > > > Hi Ryan, > > > Thanks for your good comments. I've listed your comments in "Rejected > > > Alternatives" in KIP. > > > > > > 1. Some cooperative-sticky related defects might not free before V3.0 > > > → We've marked important defects as blocker for V3.0, ex: > > KAFKA-12896. > > > Please raise any important defect if you found any. > > > > > > 2. Cooperative-sticky assignor is also very new for C/C++ users in > > > librdkafka, so not many in that community have tried incremental > > > cooperative yet. And bugs are still recently being worked out there > too. > > > → Thanks for raising this. I checked this library and found > currently > > > only 1 cooperative-sticky related bug open, which is good (10 bugs are > > > fixed). Anyway, I think the clients can always change the assignor to > > other > > > assignors if there are still bugs in the library. > > > > > > Thank you. > > > Luke > > > > > > On Thu, Jun 10, 2021 at 12:40 PM Ryan Leslie <rles...@bloomberg.net> > > > wrote: > > > > > > > Thanks for the quick replies, Luke and Sophie. > > > > > > > > I've not voted, but I agree with accepting the KIP since it's a > > superior > > > > feature. I was just reacting mostly to this comment since it didn't > > > mention > > > > open issues: > > > > > > > > > > > Thanks Luke. We may as well get this KIP in to 3.0 so that we > can > > > > fully > > > > > > > enable cooperative rebalancing > > > > > > > by default in 3.0 if we have KAFKA-12477 done in time, and if > we > > > > don't > > > > > > then > > > > > > > there's no harm as it's > > > > > > > not going to change the behavior. > > > > > > > > But I see now, as Luke said, that the main issue is already > considered > > a > > > > blocker so it was assumed. Though, I did also wonder if any bugs that > > may > > > > have existed since several version ago should actually hold up 3.0, > > > which I > > > > know is especially about moving away from ZooKeeper. > > > > > > > > My sentiment was just that during many release cycles of Kafka since > > > > cooperative was introduced, there have been issues discovered. And > that > > > > makes sense given that the implementation was complex and quite a lot > > of > > > > code changed to make it happen. Hopefully the last of the kinks will > > have > > > > been worked out before 3.0. I just wondered if it should be a default > > in > > > > 3.0 if it hasn't yet been free of defects for a significant period of > > > time. > > > > KIP-726 doesn't list any drawbacks for cooperative-sticky, but > perhaps > > > this > > > > is one. I also appreciate that it's already successfully adopted by > > many, > > > > particularly streams / connect users. But this may also be where the > > > > feature has the most benefit due to expensive setup/teardown during > > > > rebalance, and stop-the-world can be less of a concern for many > > "regular > > > > consumers". > > > > > > > > This is probably irrelevant here, but another thing to mention is > that > > > the > > > > feature is also very new for C/C++ users in librdkafka, so not many > in > > > that > > > > community have tried incremental cooperative yet. And bugs are still > > > > recently being worked out there too. > > > > > > > > Just playing devil's advocate here, sorry to come across as a > negative > > > > nancy! > > > > > > > > On 2021/06/09 00:05:41, Sophie Blee-Goldman > > <sop...@confluent.io.INVALID > > > > > > > > wrote: > > > > > Hey Ryan, > > > > > > > > > > Yes, I believe any open bugs regarding the cooperative-sticky > > assignor > > > > > should be considered as blockers > > > > > to it being made the default, if not blockers to the release in > > > general. > > > > I > > > > > don't think they need to block the > > > > > acceptance of this KIP, though, just possibly the implementation of > > it. > > > > > > > > > >
