+1 to making the switch in 3.1 with more time versus rushing it in 3.0. Ismael
On Tue, Jun 22, 2021 at 8:58 PM Sophie Blee-Goldman <sop...@confluent.io.invalid> wrote: > I believe we've figured out the root cause of KAFKA-12896 > <https://issues.apache.org/jira/browse/KAFKA-12896>, and should have a fix > prepared shortly. See the linked issues for more details. > > Regarding KIP-726 itself, given that the latest proposal is fully > compatible and does not require any breaking changes, we may or may not > push to get this into 3.0. Cooperative rebalancing is a huge improvement > for the majority of consumer applications, but it's already available for > those who want it, so I don't feel too badly about letting it slip from 3.0 > in order to focus on other things. I would also personally feel better > about merging it at the beginning of 3.1 so we have the entire release > cycle to flush out any potential regressions, rather than rushing it in at > the last moment (even though the majority of the work has been done for a > while). > > If anyone feels strongly about getting this into 3.0, please let me know, > as it's definitely still within reach. Otherwise I'll probably aim for 3.1 > instead. > > Note (@Luke in particular): we could opt for a partial and risk-free > improvement in 3.0, that would make it easier for users to turn on > cooperative rebalancing without actually enabling it for them (yet). If we > change the default config to ["range", "cooperative-sticky"] in 3.0, then > the RangeAssignor will remain the default but any application can upgrade > to the CooperativeStickyAssignor with just a single rolling bounce (to > remove the "range" assignor), rather than the usual two. We can then go > ahead and fully make the CooperativeStickyAssignor the default assignor in > 3.1 by changing the default config to ["cooperative-sticky", "range"] as > this KIP has proposed. > > Thoughts? > > On Thu, Jun 10, 2021 at 1:46 AM Luke Chen <show...@gmail.com> wrote: > > > Hi Ryan, > > Thanks for your good comments. I've listed your comments in "Rejected > > Alternatives" in KIP. > > > > 1. Some cooperative-sticky related defects might not free before V3.0 > > → We've marked important defects as blocker for V3.0, ex: > KAFKA-12896. > > Please raise any important defect if you found any. > > > > 2. Cooperative-sticky assignor is also very new for C/C++ users in > > librdkafka, so not many in that community have tried incremental > > cooperative yet. And bugs are still recently being worked out there too. > > → Thanks for raising this. I checked this library and found currently > > only 1 cooperative-sticky related bug open, which is good (10 bugs are > > fixed). Anyway, I think the clients can always change the assignor to > other > > assignors if there are still bugs in the library. > > > > Thank you. > > Luke > > > > On Thu, Jun 10, 2021 at 12:40 PM Ryan Leslie <rles...@bloomberg.net> > > wrote: > > > > > Thanks for the quick replies, Luke and Sophie. > > > > > > I've not voted, but I agree with accepting the KIP since it's a > superior > > > feature. I was just reacting mostly to this comment since it didn't > > mention > > > open issues: > > > > > > > > > Thanks Luke. We may as well get this KIP in to 3.0 so that we can > > > fully > > > > > > enable cooperative rebalancing > > > > > > by default in 3.0 if we have KAFKA-12477 done in time, and if we > > > don't > > > > > then > > > > > > there's no harm as it's > > > > > > not going to change the behavior. > > > > > > But I see now, as Luke said, that the main issue is already considered > a > > > blocker so it was assumed. Though, I did also wonder if any bugs that > may > > > have existed since several version ago should actually hold up 3.0, > > which I > > > know is especially about moving away from ZooKeeper. > > > > > > My sentiment was just that during many release cycles of Kafka since > > > cooperative was introduced, there have been issues discovered. And that > > > makes sense given that the implementation was complex and quite a lot > of > > > code changed to make it happen. Hopefully the last of the kinks will > have > > > been worked out before 3.0. I just wondered if it should be a default > in > > > 3.0 if it hasn't yet been free of defects for a significant period of > > time. > > > KIP-726 doesn't list any drawbacks for cooperative-sticky, but perhaps > > this > > > is one. I also appreciate that it's already successfully adopted by > many, > > > particularly streams / connect users. But this may also be where the > > > feature has the most benefit due to expensive setup/teardown during > > > rebalance, and stop-the-world can be less of a concern for many > "regular > > > consumers". > > > > > > This is probably irrelevant here, but another thing to mention is that > > the > > > feature is also very new for C/C++ users in librdkafka, so not many in > > that > > > community have tried incremental cooperative yet. And bugs are still > > > recently being worked out there too. > > > > > > Just playing devil's advocate here, sorry to come across as a negative > > > nancy! > > > > > > On 2021/06/09 00:05:41, Sophie Blee-Goldman > <sop...@confluent.io.INVALID > > > > > > wrote: > > > > Hey Ryan, > > > > > > > > Yes, I believe any open bugs regarding the cooperative-sticky > assignor > > > > should be considered as blockers > > > > to it being made the default, if not blockers to the release in > > general. > > > I > > > > don't think they need to block the > > > > acceptance of this KIP, though, just possibly the implementation of > it. > > > > > >
