> Before we deprecate server side auto topic creation, we should have client side auto topic creation for the producer:
Deprecating sounds fine, but before disabling it, it might be worthwhile to wait long enough for non Java clients to catch up to this too. :) On Mon, May 11, 2020 at 4:45 PM Ismael Juma <ism...@juma.me.uk> wrote: > Before we deprecate server side auto topic creation, we should have client > side auto topic creation for the producer: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > Ismael > > On Mon, May 11, 2020 at 1:41 PM Colin McCabe <cmcc...@apache.org> wrote: > > > On Mon, May 11, 2020, at 01:19, David Jacot wrote: > > > Hi all, > > > > > > First, I agree with what has been discussed. Having 3.x as the bridge > > > releases and entirely removing ZK in 4.0 makes total sense. > > > > > > Second, what would you think about removing the auto topics creation > > > in 3.0? It is not recommended to use it anymore and that could simplify > > > a bit our path towards removing ZK. We could deprecate it in 2.6 > already > > > and remove it in 3.0. If you guys generally agree, I could draft a KIP > > and > > > kick off the discussion. > > > > > > > I like the idea of removing automatic topic creation, but it seems like > it > > would be a big step, given that it's currently on by default. > > > > What about if we changed it to be off by default and deprecated in 3.0? > > Then, people who hadn't yet changed their workflows could turn it on in > > 3.0, and we could remove it entirely in 4.0. > > > > best, > > Colin > > > > > > > > Best, > > > David > > > > > > On Sun, May 10, 2020 at 8:02 PM Michael K. Edwards < > > m.k.edwa...@gmail.com> > > > wrote: > > > > > > > Yes, I've read the KIP. But all it really says to me is "we have > never > > > > gotten around to using ZooKeeper properly." To the extent that any > of > > the > > > > distributed-state-maintenance problems discussed in "Metadata as an > > Event > > > > Log" can be solved — and some of them intrinsically can't, because > CAP > > > > theorem — most of them are already implemented very effectively in > > Curator > > > > recipes. (For instance, Curator's Tree Cache > > > > https://curator.apache.org/curator-recipes/tree-cache.html is a good > > fit > > > > to > > > > some of the state-maintenance needs.) > > > > > > > > Kafka does have some usage patterns that don't map neatly onto > existing > > > > Curator recipes. For instance, neither LeaderSelector nor > LeaderLatch > > > > implements leader preference in the way that the existing Kafka > > partition > > > > leadership election procedure does. But why not handle that by > > improving > > > > and extending Curator? That way, other Curator users benefit, and we > > get > > > > additional highly experienced reviewers' eyes on the distributed > > > > algorithms, which are very very tricky to get right. > > > > > > > > > > > > On Sun, May 10, 2020 at 10:47 AM Ron Dagostino <rndg...@gmail.com> > > wrote: > > > > > > > > > Hi Michael. This is discussed in the KIP. > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum#KIP-500:ReplaceZooKeeperwithaSelf-ManagedMetadataQuorum-Motivation > > > > > > > > > > Ron > > > > > > > > > > > On May 10, 2020, at 1:35 PM, Michael K. Edwards < > > m.k.edwa...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > What is the actual goal of removing the ZooKeeper dependency? In > > my > > > > > > experience, if ZooKeeper is properly provisioned and deployed, > it's > > > > > largely > > > > > > trouble-free. (You do need to know how to use observers > properly.) > > > > > There > > > > > > are some subtleties about timeouts and leadership changes, but > > they're > > > > > > pretty small stuff. Why go to all the trouble of building a new > > > > > > distributed-consensus system that's going to have catastrophic > > bugs for > > > > > > years to come? It seems like such an act of hubris to me, as > well > > as a > > > > > > massive waste of engineering effort. What is there to be gained? > > > > > > > > > > > >> On Fri, May 8, 2020 at 4:11 PM Matthias J. Sax < > mj...@apache.org> > > > > > wrote: > > > > > >> > > > > > >> Sure, we can compile a list for Kafka Streams. But the KIP would > > be > > > > for > > > > > >> 3.0, so I don't think it's urgent to do it now? > > > > > >> > > > > > >> > > > > > >> -Matthias > > > > > >> > > > > > >>> On 5/8/20 3:47 PM, Colin McCabe wrote: > > > > > >>> Thanks, Guozhang-- sounds like a good plan. > > > > > >>> > > > > > >>> I think it would be good to have a list of deprecated streams > > APIs > > > > that > > > > > >> we want to remove in 3.0. Maybe it's easiest to do that as its > > own > > > > KIP? > > > > > >>> > > > > > >>> For MirrorMaker 1, we should have a KIP to deprecate its use in > > 2.6 > > > > if > > > > > >> we want to remove it in 3.0. I don't have a good sense of how > > > > > practical it > > > > > >> is to deprecate this now, so I will defer to others here. But > > the KIP > > > > > >> freeze for 2.6 is coming soon, so if we want to make the case, > > now is > > > > > the > > > > > >> time. > > > > > >>> > > > > > >>> best, > > > > > >>> Colin > > > > > >>> > > > > > >>> > > > > > >>>> On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > > > > > >>>> Hey folks, > > > > > >>>> > > > > > >>>> Sorry for stating that the bridge release would not break any > > > > > >> compatibility > > > > > >>>> before, which is incorrect and confused many people. > > > > > >>>> > > > > > >>>> I think one way to think about the versioning is that: > > > > > >>>> > > > > > >>>> 0) In a 2.x version moving ahead we would deprecate the > > ZK-dependent > > > > > >> tools > > > > > >>>> such as --zookeeper flags from various scripts (KIP-555) > > > > > >>>> > > > > > >>>> 1) In 3.0 we would at least make one incompatible change for > > example > > > > > to > > > > > >>>> remove the deprecated ZK flags. > > > > > >>>> > > > > > >>>> 2) In a future major version (e.g. 4.0) we would drop ZK > > entirely, > > > > > >>>> including usages such as security credentials / broker > > registration > > > > / > > > > > >> etc > > > > > >>>> which are via ZK today as well. > > > > > >>>> > > > > > >>>> Then for the bridge release(s), it can be any or all of 3.x. > > > > > >>>> > > > > > >>>> > > > > > >>>> For 1), I'd love to add a few more incompatibility changes in > > 3.0 > > > > from > > > > > >>>> Kafka Streams: we evolve Streams public APIs by deprecating > and > > then > > > > > >> remove > > > > > >>>> in major releases, and since 2.0 we've accumulated quite a few > > > > > >> deprecated > > > > > >>>> APIs, and I can compile a list of KIPs that contain those if > > people > > > > > are > > > > > >>>> interested. > > > > > >>>> > > > > > >>>> > > > > > >>>> Guozhang > > > > > >>>> > > > > > >>>> > > > > > >>>>> On Thu, May 7, 2020 at 3:53 PM Colin McCabe < > > cmcc...@apache.org> > > > > > wrote: > > > > > >>>>> > > > > > >>>>> On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > > > > > >>>>>>> In fact, we know that the bridge release will involve at > > least > > > > one > > > > > >>>>>>> incompatible change. We will need to drop support for the > > > > > >> --zookeeper > > > > > >>>>>>> flags in the command-line tools. > > > > > >>>>>> > > > > > >>>>>> If the bridge release(s) and the subsequent post-ZK release > > are > > > > > _both_ > > > > > >>>>>> breaking changes, I think we only have one option: the 3.x > > line > > > > are > > > > > >> the > > > > > >>>>>> bridge release(s), and ZK is removed in 4.0, as suggested by > > > > Andrew > > > > > >>>>>> Schofield. > > > > > >>>>>> > > > > > >>>>>> Specifically: > > > > > >>>>>> - in order to _remove_ (not merely deprecate) the > --zookeeper > > > > args, > > > > > we > > > > > >>>>> will > > > > > >>>>>> need a major release. > > > > > >>>>>> - in oder to drop support for ZK entirely (e.g. break a > bunch > > of > > > > > >> external > > > > > >>>>>> tooling like Cruise Control), we will need a major release. > > > > > >>>>>> > > > > > >>>>>> I count two major releases. > > > > > >>>>> > > > > > >>>>> Hi Ryanne, > > > > > >>>>> > > > > > >>>>> I agree that dropping ZK completely will need a new major > > release > > > > > after > > > > > >>>>> 3.0. I think that's OK and in keeping with how we've handled > > > > > >> deprecation > > > > > >>>>> and removal in the past. It's important for users to have a > > smooth > > > > > >> upgrade > > > > > >>>>> path. > > > > > >>>>> > > > > > >>>>> best, > > > > > >>>>> Colin > > > > > >>>>> > > > > > >>>>>> > > > > > >>>>>> Ryanne > > > > > >>>>>> > > > > > >>>>>> - > > > > > >>>>>> > > > > > >>>>>> On Wed, May 6, 2020 at 10:52 PM Colin McCabe < > > cmcc...@apache.org> > > > > > >> wrote: > > > > > >>>>>> > > > > > >>>>>>>> On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > > > > > >>>>>>>> Hey Colin, I think we should wait until after KIP-500's > > "bridge > > > > > >>>>>>>> release" so there is a clean break from Zookeeper after > > 3.0. The > > > > > >>>>>>>> bridge release by definition is an attempt to not break > > > > anything, > > > > > so > > > > > >>>>>>>> it theoretically doesn't warrant a major release. > > > > > >>>>>>> > > > > > >>>>>>> Hi Ryanne, > > > > > >>>>>>> > > > > > >>>>>>> I think it's important to clarify this a little bit. The > > bridge > > > > > >>>>> release > > > > > >>>>>>> (really, releases, plural) allow you to upgrade from a > > cluster > > > > that > > > > > >> is > > > > > >>>>>>> using ZooKeeper to one that is not using ZooKeeper. But, > > that > > > > > >> doesn't > > > > > >>>>>>> imply that the bridge release itself doesn't break > anything. > > > > > >> Upgrading > > > > > >>>>>>> to the bridge release itself might involve some minor > > > > > >> incompatibility. > > > > > >>>>>>> > > > > > >>>>>>> Kafka does occasionally have incompatible changes. In > those > > > > cases, > > > > > >> we > > > > > >>>>>>> bump the major version number. One example is that when we > > went > > > > > from > > > > > >>>>>>> Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This > > is an > > > > > >>>>>>> incompatible change. > > > > > >>>>>>> > > > > > >>>>>>> In fact, we know that the bridge release will involve at > > least > > > > one > > > > > >>>>>>> incompatible change. We will need to drop support for the > > > > > >> --zookeeper > > > > > >>>>>>> flags in the command-line tools. > > > > > >>>>>>> > > > > > >>>>>>> We've been preparing for this change for a long time. > People > > > > have > > > > > >>>>> spent > > > > > >>>>>>> a lot of effort designing new APIs that can be used instead > > of > > > > the > > > > > >> old > > > > > >>>>>>> zookeeper-based code that some of the command-line tools > > used. > > > > We > > > > > >> have > > > > > >>>>>>> also deprecated the old ZK-based flags. But at the end of > > the > > > > day, > > > > > >> it > > > > > >>>>>>> is still an incompatible change. So it's unfortunately not > > > > > possible > > > > > >>>>> for > > > > > >>>>>>> the > > > > > >>>>>>> bridge release to be a 2.x release. > > > > > >>>>>>> > > > > > >>>>>>>> If that's not the case (i.e. if a single "bridge release" > > turns > > > > > out > > > > > >>>>> to > > > > > >>>>>>>> be impractical), we should consider forking 3.0 while > > > > maintaining > > > > > a > > > > > >>>>>>>> line of Zookeeper-dependent Kafka in 2.x. That way 3.x can > > > > evolve > > > > > >>>>>>>> dramatically without breaking the 2.x line. In particular, > > > > > anything > > > > > >>>>>>>> related to removing Zookeeper could land in pre-3.0 while > > every > > > > > >> other > > > > > >>>>>>>> feature targets 2.6. > > > > > >>>>>>> > > > > > >>>>>>> Just to be super clear about this, what we want to do here > is > > > > > support > > > > > >>>>>>> operating in __either__ KIP-500 mode and legacy mode for a > > while. > > > > > So > > > > > >>>>> the > > > > > >>>>>>> same branch will have support for both the old way and the > > new > > > > way > > > > > of > > > > > >>>>>>> managing metadata. > > > > > >>>>>>> > > > > > >>>>>>> This will allow us to get an "alpha" version of the KIP-500 > > mode > > > > > out > > > > > >>>>> early > > > > > >>>>>>> for people to experiment with. It also greatly reduces the > > > > number > > > > > of > > > > > >>>>> Kafka > > > > > >>>>>>> releases we have to make, and the amount of backporting we > > have > > > > to > > > > > >> do. > > > > > >>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> If you are proposing 2.6 should be the "bridge release", I > > think > > > > > >> this > > > > > >>>>>>>> is premature given Kafka's time-based release schedule. If > > the > > > > > >> bridge > > > > > >>>>>>>> features happen to be merged before 2.6's feature freeze, > > then > > > > > sure > > > > > >>>>> -- > > > > > >>>>>>>> let's make that the bridge release in retrospect. And if > we > > get > > > > > all > > > > > >>>>>>>> the post-Zookeeper features merged before 2.7, I'm onboard > > with > > > > > >>>>> naming > > > > > >>>>>>>> it "3.0" instead. > > > > > >>>>>>>> > > > > > >>>>>>>> That said, we should aim to remove legacy MirrorMaker > > before 3.0 > > > > > as > > > > > >>>>>>>> well. I'm happy to drive that additional breaking change. > > Maybe > > > > > 2.6 > > > > > >>>>>>>> can be the "bridge" for MM2 as well. > > > > > >>>>>>> > > > > > >>>>>>> I don't have a strong opinion either way about this, but if > > we > > > > want > > > > > >> to > > > > > >>>>>>> remove the original MirrorMaker, we have to deprecate it > > first, > > > > > >>>>> right? Are > > > > > >>>>>>> we ready to do that? > > > > > >>>>>>> > > > > > >>>>>>> best, > > > > > >>>>>>> Colin > > > > > >>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> Ryanne > > > > > >>>>>>>> > > > > > >>>>>>>> On Mon, May 4, 2020, 5:05 PM Colin McCabe < > > cmcc...@apache.org> > > > > > >>>>> wrote: > > > > > >>>>>>>> > > > > > >>>>>>>>> Hi all, > > > > > >>>>>>>>> > > > > > >>>>>>>>> We've had a few proposals recently for incompatible > > changes. > > > > One > > > > > >>>>> of > > > > > >>>>>>>>> them is my KIP-604: Remove ZooKeeper Flags from the > > > > > Administrative > > > > > >>>>>>>>> Tools. The other is Boyang's KIP-590: Redirect ZK > Mutation > > > > > >>>>>>>>> Protocols to the Controller. I think it's time to start > > > > thinking > > > > > >>>>>>>>> about Kafka 3.0. Specifically, I think we should move to > > 3.0 > > > > > after > > > > > >>>>>>>>> the 2.6 release. > > > > > >>>>>>>>> > > > > > >>>>>>>>> From the perspective of KIP-500, in Kafka 3.x we'd like > to > > make > > > > > >>>>>>>>> running in a ZooKeeper-less mode possible (but not yet > the > > > > > >>>>> default.) > > > > > >>>>>>>>> This is the motivation behind KIP-590 and KIP-604, as > well > > as > > > > > some > > > > > >>>>>>>>> of the other KIPs we've done recently. Since it will > take > > some > > > > > >>>>> time > > > > > >>>>>>>>> to stabilize the new ZooKeeper-free Kafka code, we will > > hide it > > > > > >>>>>>>>> behind an option initially. (We'll have a KIP describing > > this > > > > all > > > > > >>>>> in > > > > > >>>>>>>>> detail soon.) > > > > > >>>>>>>>> > > > > > >>>>>>>>> What does everyone think about having Kafka 3.0 come up > > next > > > > > after > > > > > >>>>>>>>> 2.6? Are there any other things we should change in the > > 2.6 -> > > > > > 3.0 > > > > > >>>>>>>>> transition? > > > > > >>>>>>>>> > > > > > >>>>>>>>> best, Colin > > > > > >>>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>> > > > > > >>>>>> > > > > > >>>>> > > > > > >>>> > > > > > >>>> > > > > > >>>> -- > > > > > >>>> -- Guozhang > > > > > >>>> > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
