I'm +1 on 3.0 for the mid year release. On Tue, Feb 16, 2021 at 5:08 PM Matthias J. Sax <mj...@apache.org> wrote:
> Hi, > > given that we passed 2.8 feature freeze, I wanted to restart this > thread. Currently, `trunk` is at `2.9.0-SNAPSHOT` and I am wondering if > the decision for the 3.0 release is final and if we should bump the > version number? > > I am asking particularly because there a many Jiras with a 3.0 target > release version for breaking changes and we should ensure that we have > enough time to work on those tickets. -- As long as we don't agree that > the next release will indeed be 3.0, those tickets are effectively > blocked/pending. > > Thoughts? > > > -Matthias > > > On 10/15/20 4:28 PM, Matthias J. Sax wrote: > > Thanks for clarifying Colin. Works for me. Overall, 3.0 should be guided > > by the ZK removal progress and if we are not there yet, it's better to > > have a 2.8 first. > > > > > > -Matthias > > > > > > On 10/15/20 2:41 PM, Colin McCabe wrote: > >> Hi all, > >> > >> Just to follow up on this... since we're not quite ready for 3.0 yet, > it's probably best if we release a 2.8 next, and then go to 3.0 after > that. Sorry for any confusion. > >> > >> best, > >> Colin > >> > >> > >> On Mon, Jul 20, 2020, at 12:52, Matthias J. Sax wrote: > >>> Did we reach any conclusion on the subject? > >>> > >>> It seems we are aiming for 2.7 after 2.6 and plan the major version > bump > >>> to 3.0 after 2.7 (assuming we make progress on ZK removal as planned?) > >>> > >>> > >>> -Matthias > >>> > >>> > >>> On 5/18/20 1:11 PM, Boyang Chen wrote: > >>>> One more thing I would like to see deprecated (hopefully no one > mentioned > >>>> before) is the zk based consumer offset support. > >>>> > >>>> On Mon, May 11, 2020 at 2:15 PM Colin McCabe <cmcc...@apache.org> > wrote: > >>>> > >>>>> Hi Michael, > >>>>> > >>>>> It would be better to discuss the background behind KIP-500 in a > separate > >>>>> thread, since this thread is about the Kafka 3.0 release. As others > have > >>>>> said, your questions are answered in the KIP. For example, "what is > the > >>>>> actual goal?" is addressed in the motivation section. > >>>>> > >>>>> I agree that Kafka's usage of Apache ZooKeeper could be optimized. > But > >>>>> there are fundamental limitations to this approach compared to > storing our > >>>>> metadata internally. For example, having to contact a remote server > to > >>>>> reload all your metadata on a controller failover simply doesn't > scale past > >>>>> a certain point. > >>>>> > >>>>> Apache Curator is a nice API, and if we were starting again today we > would > >>>>> certainly consider using it. But it doesn't allow us to do anything > more > >>>>> efficiently than ZooKeeper could already do it. > >>>>> > >>>>> Finally, Kafka's core competence is logs. While our replication > protocol > >>>>> is not Raft, it shares many similarities with that protocol. So I > think > >>>>> it's a bit unfair to say that it is "catastrophic hubris" to believe > we can > >>>>> implement the protocol. > >>>>> > >>>>> best, > >>>>> Colin > >>>>> > >>>>> > >>>>> On Sun, May 10, 2020, at 11:02, Michael K. Edwards wrote: > >>>>>> Yes, I've read the KIP. But all it really says to me is "we have > never > >>>>>> gotten around to using ZooKeeper properly." To the extent that any > of > >>>>> the > >>>>>> distributed-state-maintenance problems discussed in "Metadata as an > Event > >>>>>> Log" can be solved — and some of them intrinsically can't, because > CAP > >>>>>> theorem — most of them are already implemented very effectively in > >>>>> Curator > >>>>>> recipes. (For instance, Curator's Tree Cache > >>>>>> https://curator.apache.org/curator-recipes/tree-cache.html is a > good > >>>>> fit to > >>>>>> some of the state-maintenance needs.) > >>>>>> > >>>>>> Kafka does have some usage patterns that don't map neatly onto > existing > >>>>>> Curator recipes. For instance, neither LeaderSelector nor > LeaderLatch > >>>>>> implements leader preference in the way that the existing Kafka > partition > >>>>>> leadership election procedure does. But why not handle that by > improving > >>>>>> and extending Curator? That way, other Curator users benefit, and > we get > >>>>>> additional highly experienced reviewers' eyes on the distributed > >>>>>> algorithms, which are very very tricky to get right. > >>>>>> > >>>>>> > >>>>>> On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> > >>>>> wrote: > >>>>>> > >>>>>>> Hi Michael. This is discussed in the KIP. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation > >>>>>>> > >>>>>>> Ron > >>>>>>> > >>>>>>>> On May 10, 2020, at 1:35 PM, Michael K. Edwards < > >>>>> m.k.edwa...@gmail.com> > >>>>>>> wrote: > >>>>>>>> > >>>>>>>> What is the actual goal of removing the ZooKeeper dependency? In > my > >>>>>>>> experience, if ZooKeeper is properly provisioned and deployed, > it's > >>>>>>> largely > >>>>>>>> trouble-free. (You do need to know how to use observers > properly.) > >>>>>>> There > >>>>>>>> are some subtleties about timeouts and leadership changes, but > >>>>> they're > >>>>>>>> pretty small stuff. Why go to all the trouble of building a new > >>>>>>>> distributed-consensus system that's going to have catastrophic > bugs > >>>>> for > >>>>>>>> years to come? It seems like such an act of hubris to me, as well > >>>>> as a > >>>>>>>> massive waste of engineering effort. What is there to be gained? > >>>>>>>> > >>>>>>>>> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax <mj...@apache.org > > > >>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Sure, we can compile a list for Kafka Streams. But the KIP would > be > >>>>> for > >>>>>>>>> 3.0, so I don't think it's urgent to do it now? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -Matthias > >>>>>>>>> > >>>>>>>>>> On 5/8/20 3:47 PM, Colin McCabe wrote: > >>>>>>>>>> Thanks, Guozhang-- sounds like a good plan. > >>>>>>>>>> > >>>>>>>>>> I think it would be good to have a list of deprecated streams > APIs > >>>>> that > >>>>>>>>> we want to remove in 3.0. Maybe it's easiest to do that as its > own > >>>>> KIP? > >>>>>>>>>> > >>>>>>>>>> For MirrorMaker 1, we should have a KIP to deprecate its use in > >>>>> 2.6 if > >>>>>>>>> we want to remove it in 3.0. I don't have a good sense of how > >>>>>>> practical it > >>>>>>>>> is to deprecate this now, so I will defer to others here. But > the > >>>>> KIP > >>>>>>>>> freeze for 2.6 is coming soon, so if we want to make the case, > now > >>>>> is > >>>>>>> the > >>>>>>>>> time. > >>>>>>>>>> > >>>>>>>>>> best, > >>>>>>>>>> Colin > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > >>>>>>>>>>> Hey folks, > >>>>>>>>>>> > >>>>>>>>>>> Sorry for stating that the bridge release would not break any > >>>>>>>>> compatibility > >>>>>>>>>>> before, which is incorrect and confused many people. > >>>>>>>>>>> > >>>>>>>>>>> I think one way to think about the versioning is that: > >>>>>>>>>>> > >>>>>>>>>>> 0) In a 2.x version moving ahead we would deprecate the > >>>>> ZK-dependent > >>>>>>>>> tools > >>>>>>>>>>> such as --zookeeper flags from various scripts (KIP-555) > >>>>>>>>>>> > >>>>>>>>>>> 1) In 3.0 we would at least make one incompatible change for > >>>>> example > >>>>>>> to > >>>>>>>>>>> remove the deprecated ZK flags. > >>>>>>>>>>> > >>>>>>>>>>> 2) In a future major version (e.g. 4.0) we would drop ZK > entirely, > >>>>>>>>>>> including usages such as security credentials / broker > >>>>> registration / > >>>>>>>>> etc > >>>>>>>>>>> which are via ZK today as well. > >>>>>>>>>>> > >>>>>>>>>>> Then for the bridge release(s), it can be any or all of 3.x. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> For 1), I'd love to add a few more incompatibility changes in > 3.0 > >>>>> from > >>>>>>>>>>> Kafka Streams: we evolve Streams public APIs by deprecating and > >>>>> then > >>>>>>>>> remove > >>>>>>>>>>> in major releases, and since 2.0 we've accumulated quite a few > >>>>>>>>> deprecated > >>>>>>>>>>> APIs, and I can compile a list of KIPs that contain those if > >>>>> people > >>>>>>> are > >>>>>>>>>>> interested. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Guozhang > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe < > cmcc...@apache.org> > >>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > >>>>>>>>>>>>>> In fact, we know that the bridge release will involve at > least > >>>>> one > >>>>>>>>>>>>>> incompatible change. We will need to drop support for the > >>>>>>>>> --zookeeper > >>>>>>>>>>>>>> flags in the command-line tools. > >>>>>>>>>>>>> > >>>>>>>>>>>>> If the bridge release(s) and the subsequent post-ZK release > are > >>>>>>> _both_ > >>>>>>>>>>>>> breaking changes, I think we only have one option: the 3.x > line > >>>>> are > >>>>>>>>> the > >>>>>>>>>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by > >>>>> Andrew > >>>>>>>>>>>>> Schofield. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Specifically: > >>>>>>>>>>>>> - in order to _remove_ (not merely deprecate) the --zookeeper > >>>>> args, > >>>>>>> we > >>>>>>>>>>>> will > >>>>>>>>>>>>> need a major release. > >>>>>>>>>>>>> - in oder to drop support for ZK entirely (e.g. break a > bunch of > >>>>>>>>> external > >>>>>>>>>>>>> tooling like Cruise Control), we will need a major release. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I count two major releases. > >>>>>>>>>>>> > >>>>>>>>>>>> Hi Ryanne, > >>>>>>>>>>>> > >>>>>>>>>>>> I agree that dropping ZK completely will need a new major > release > >>>>>>> after > >>>>>>>>>>>> 3.0. I think that's OK and in keeping with how we've handled > >>>>>>>>> deprecation > >>>>>>>>>>>> and removal in the past. It's important for users to have a > >>>>> smooth > >>>>>>>>> upgrade > >>>>>>>>>>>> path. > >>>>>>>>>>>> > >>>>>>>>>>>> best, > >>>>>>>>>>>> Colin > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Ryanne > >>>>>>>>>>>>> > >>>>>>>>>>>>> - > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < > >>>>> cmcc...@apache.org> > >>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > >>>>>>>>>>>>>>> Hey Colin, I think we should wait until after KIP-500's > >>>>> "bridge > >>>>>>>>>>>>>>> release" so there is a clean break from Zookeeper after > 3.0. > >>>>> The > >>>>>>>>>>>>>>> bridge release by definition is an attempt to not break > >>>>> anything, > >>>>>>> so > >>>>>>>>>>>>>>> it theoretically doesn't warrant a major release. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi Ryanne, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I think it's important to clarify this a little bit. The > >>>>> bridge > >>>>>>>>>>>> release > >>>>>>>>>>>>>> (really, releases, plural) allow you to upgrade from a > cluster > >>>>> that > >>>>>>>>> is > >>>>>>>>>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, > that > >>>>>>>>> doesn't > >>>>>>>>>>>>>> imply that the bridge release itself doesn't break anything. > >>>>>>>>> Upgrading > >>>>>>>>>>>>>> to the bridge release itself might involve some minor > >>>>>>>>> incompatibility. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Kafka does occasionally have incompatible changes. In those > >>>>> cases, > >>>>>>>>> we > >>>>>>>>>>>>>> bump the major version number. One example is that when we > >>>>> went > >>>>>>> from > >>>>>>>>>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This > is > >>>>> an > >>>>>>>>>>>>>> incompatible change. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> In fact, we know that the bridge release will involve at > least > >>>>> one > >>>>>>>>>>>>>> incompatible change. We will need to drop support for the > >>>>>>>>> --zookeeper > >>>>>>>>>>>>>> flags in the command-line tools. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We've been preparing for this change for a long time. > People > >>>>> have > >>>>>>>>>>>> spent > >>>>>>>>>>>>>> a lot of effort designing new APIs that can be used instead > of > >>>>> the > >>>>>>>>> old > >>>>>>>>>>>>>> zookeeper-based code that some of the command-line tools > >>>>> used. We > >>>>>>>>> have > >>>>>>>>>>>>>> also deprecated the old ZK-based flags. But at the end of > the > >>>>> day, > >>>>>>>>> it > >>>>>>>>>>>>>> is still an incompatible change. So it's unfortunately not > >>>>>>> possible > >>>>>>>>>>>> for > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>> bridge release to be a 2.x release. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If that's not the case (i.e. if a single "bridge release" > >>>>> turns > >>>>>>> out > >>>>>>>>>>>> to > >>>>>>>>>>>>>>> be impractical), we should consider forking 3.0 while > >>>>> maintaining > >>>>>>> a > >>>>>>>>>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can > >>>>> evolve > >>>>>>>>>>>>>>> dramatically without breaking the 2.x line. In particular, > >>>>>>> anything > >>>>>>>>>>>>>>> related to removing Zookeeper could land in pre-3.0 while > >>>>> every > >>>>>>>>> other > >>>>>>>>>>>>>>> feature targets 2.6. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Just to be super clear about this, what we want to do here > is > >>>>>>> support > >>>>>>>>>>>>>> operating in __either__ KIP-500 mode and legacy mode for a > >>>>> while. > >>>>>>> So > >>>>>>>>>>>> the > >>>>>>>>>>>>>> same branch will have support for both the old way and the > new > >>>>> way > >>>>>>> of > >>>>>>>>>>>>>> managing metadata. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This will allow us to get an "alpha" version of the KIP-500 > >>>>> mode > >>>>>>> out > >>>>>>>>>>>> early > >>>>>>>>>>>>>> for people to experiment with. It also greatly reduces the > >>>>> number > >>>>>>> of > >>>>>>>>>>>> Kafka > >>>>>>>>>>>>>> releases we have to make, and the amount of backporting we > >>>>> have to > >>>>>>>>> do. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> If you are proposing 2.6 should be the "bridge release", I > >>>>> think > >>>>>>>>> this > >>>>>>>>>>>>>>> is premature given Kafka's time-based release schedule. If > the > >>>>>>>>> bridge > >>>>>>>>>>>>>>> features happen to be merged before 2.6's feature freeze, > then > >>>>>>> sure > >>>>>>>>>>>> -- > >>>>>>>>>>>>>>> let's make that the bridge release in retrospect. And if we > >>>>> get > >>>>>>> all > >>>>>>>>>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard > >>>>> with > >>>>>>>>>>>> naming > >>>>>>>>>>>>>>> it "3.0" instead. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> That said, we should aim to remove legacy MirrorMaker > before > >>>>> 3.0 > >>>>>>> as > >>>>>>>>>>>>>>> well. I'm happy to drive that additional breaking change. > >>>>> Maybe > >>>>>>> 2.6 > >>>>>>>>>>>>>>> can be the "bridge" for MM2 as well. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I don't have a strong opinion either way about this, but if > we > >>>>> want > >>>>>>>>> to > >>>>>>>>>>>>>> remove the original MirrorMaker, we have to deprecate it > first, > >>>>>>>>>>>> right? Are > >>>>>>>>>>>>>> we ready to do that? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> best, > >>>>>>>>>>>>>> Colin > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Ryanne > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe < > cmcc...@apache.org > >>>>>> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We've had a few proposals recently for incompatible > >>>>> changes. One > >>>>>>>>>>>> of > >>>>>>>>>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the > >>>>>>> Administrative > >>>>>>>>>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK > Mutation > >>>>>>>>>>>>>>>> Protocols to the Controller. I think it's time to start > >>>>> thinking > >>>>>>>>>>>>>>>> about Kafka 3.0. Specifically, I think we should move to > 3.0 > >>>>>>> after > >>>>>>>>>>>>>>>> the 2.6 release. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like to > >>>>> make > >>>>>>>>>>>>>>>> running in a ZooKeeper-less mode possible (but not yet the > >>>>>>>>>>>> default.) > >>>>>>>>>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as > well as > >>>>>>> some > >>>>>>>>>>>>>>>> of the other KIPs we've done recently. Since it will take > >>>>> some > >>>>>>>>>>>> time > >>>>>>>>>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will > hide > >>>>> it > >>>>>>>>>>>>>>>> behind an option initially. (We'll have a KIP describing > >>>>> this all > >>>>>>>>>>>> in > >>>>>>>>>>>>>>>> detail soon.) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> What does everyone think about having Kafka 3.0 come up > next > >>>>>>> after > >>>>>>>>>>>>>>>> 2.6? Are there any other things we should change in the > 2.6 > >>>>> -> > >>>>>>> 3.0 > >>>>>>>>>>>>>>>> transition? > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> best, Colin > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> -- Guozhang > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >>> Attachments: > >>> * signature.asc >
