Thanks, Guozhang-- sounds like a good plan. I think it would be good to have a list of deprecated streams APIs that we want to remove in 3.0. Maybe it's easiest to do that as its own KIP?
For MirrorMaker 1, we should have a KIP to deprecate its use in 2.6 if we want to remove it in 3.0. I don't have a good sense of how practical it is to deprecate this now, so I will defer to others here. But the KIP freeze for 2.6 is coming soon, so if we want to make the case, now is the time. best, Colin On Thu, May 7, 2020, at 16:28, Guozhang Wang wrote: > Hey folks, > > Sorry for stating that the bridge release would not break any compatibility > before, which is incorrect and confused many people. > > I think one way to think about the versioning is that: > > 0) In a 2.x version moving ahead we would deprecate the ZK-dependent tools > such as --zookeeper flags from various scripts (KIP-555) > > 1) In 3.0 we would at least make one incompatible change for example to > remove the deprecated ZK flags. > > 2) In a future major version (e.g. 4.0) we would drop ZK entirely, > including usages such as security credentials / broker registration / etc > which are via ZK today as well. > > Then for the bridge release(s), it can be any or all of 3.x. > > > For 1), I'd love to add a few more incompatibility changes in 3.0 from > Kafka Streams: we evolve Streams public APIs by deprecating and then remove > in major releases, and since 2.0 we've accumulated quite a few deprecated > APIs, and I can compile a list of KIPs that contain those if people are > interested. > > > Guozhang > > > On Thu, May 7, 2020 at 3:53 PM Colin McCabe <cmcc...@apache.org> wrote: > > > On Wed, May 6, 2020, at 21:33, Ryanne Dolan wrote: > > > > In fact, we know that the bridge release will involve at least one > > > > incompatible change. We will need to drop support for the --zookeeper > > > > flags in the command-line tools. > > > > > > If the bridge release(s) and the subsequent post-ZK release are _both_ > > > breaking changes, I think we only have one option: the 3.x line are the > > > bridge release(s), and ZK is removed in 4.0, as suggested by Andrew > > > Schofield. > > > > > > Specifically: > > > - in order to _remove_ (not merely deprecate) the --zookeeper args, we > > will > > > need a major release. > > > - in oder to drop support for ZK entirely (e.g. break a bunch of external > > > tooling like Cruise Control), we will need a major release. > > > > > > I count two major releases. > > > > Hi Ryanne, > > > > I agree that dropping ZK completely will need a new major release after > > 3.0. I think that's OK and in keeping with how we've handled deprecation > > and removal in the past. It's important for users to have a smooth upgrade > > path. > > > > best, > > Colin > > > > > > > > Ryanne > > > > > > - > > > > > > On Wed, May 6, 2020 at 10:52 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > > > On Mon, May 4, 2020, at 17:12, Ryanne Dolan wrote: > > > > > Hey Colin, I think we should wait until after KIP-500's "bridge > > > > > release" so there is a clean break from Zookeeper after 3.0. The > > > > > bridge release by definition is an attempt to not break anything, so > > > > > it theoretically doesn't warrant a major release. > > > > > > > > Hi Ryanne, > > > > > > > > I think it's important to clarify this a little bit. The bridge > > release > > > > (really, releases, plural) allow you to upgrade from a cluster that is > > > > using ZooKeeper to one that is not using ZooKeeper. But, that doesn't > > > > imply that the bridge release itself doesn't break anything. Upgrading > > > > to the bridge release itself might involve some minor incompatibility. > > > > > > > > Kafka does occasionally have incompatible changes. In those cases, we > > > > bump the major version number. One example is that when we went from > > > > Kafka 1.x to Kafka 2.0, we dropped support for JDK7. This is an > > > > incompatible change. > > > > > > > > In fact, we know that the bridge release will involve at least one > > > > incompatible change. We will need to drop support for the --zookeeper > > > > flags in the command-line tools. > > > > > > > > We've been preparing for this change for a long time. People have > > spent > > > > a lot of effort designing new APIs that can be used instead of the old > > > > zookeeper-based code that some of the command-line tools used. We have > > > > also deprecated the old ZK-based flags. But at the end of the day, it > > > > is still an incompatible change. So it's unfortunately not possible > > for > > > > the > > > > bridge release to be a 2.x release. > > > > > > > > > If that's not the case (i.e. if a single "bridge release" turns out > > to > > > > > be impractical), we should consider forking 3.0 while maintaining a > > > > > line of Zookeeper-dependent Kafka in 2.x. That way 3.x can evolve > > > > > dramatically without breaking the 2.x line. In particular, anything > > > > > related to removing Zookeeper could land in pre-3.0 while every other > > > > > feature targets 2.6. > > > > > > > > Just to be super clear about this, what we want to do here is support > > > > operating in __either__ KIP-500 mode and legacy mode for a while. So > > the > > > > same branch will have support for both the old way and the new way of > > > > managing metadata. > > > > > > > > This will allow us to get an "alpha" version of the KIP-500 mode out > > early > > > > for people to experiment with. It also greatly reduces the number of > > Kafka > > > > releases we have to make, and the amount of backporting we have to do. > > > > > > > > > > > > > > If you are proposing 2.6 should be the "bridge release", I think this > > > > > is premature given Kafka's time-based release schedule. If the bridge > > > > > features happen to be merged before 2.6's feature freeze, then sure > > -- > > > > > let's make that the bridge release in retrospect. And if we get all > > > > > the post-Zookeeper features merged before 2.7, I'm onboard with > > naming > > > > > it "3.0" instead. > > > > > > > > > > That said, we should aim to remove legacy MirrorMaker before 3.0 as > > > > > well. I'm happy to drive that additional breaking change. Maybe 2.6 > > > > > can be the "bridge" for MM2 as well. > > > > > > > > I don't have a strong opinion either way about this, but if we want to > > > > remove the original MirrorMaker, we have to deprecate it first, > > right? Are > > > > we ready to do that? > > > > > > > > best, > > > > Colin > > > > > > > > > > > > > > Ryanne > > > > > > > > > > On Mon, May 4, 2020, 5:05 PM Colin McCabe <cmcc...@apache.org> > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > We've had a few proposals recently for incompatible changes. One > > of > > > > > > them is my KIP-604: Remove ZooKeeper Flags from the Administrative > > > > > > Tools. The other is Boyang's KIP-590: Redirect ZK Mutation > > > > > > Protocols to the Controller. I think it's time to start thinking > > > > > > about Kafka 3.0. Specifically, I think we should move to 3.0 after > > > > > > the 2.6 release. > > > > > > > > > > > > From the perspective of KIP-500, in Kafka 3.x we'd like to make > > > > > > running in a ZooKeeper-less mode possible (but not yet the > > default.) > > > > > > This is the motivation behind KIP-590 and KIP-604, as well as some > > > > > > of the other KIPs we've done recently. Since it will take some > > time > > > > > > to stabilize the new ZooKeeper-free Kafka code, we will hide it > > > > > > behind an option initially. (We'll have a KIP describing this all > > in > > > > > > detail soon.) > > > > > > > > > > > > What does everyone think about having Kafka 3.0 come up next after > > > > > > 2.6? Are there any other things we should change in the 2.6 -> 3.0 > > > > > > transition? > > > > > > > > > > > > best, Colin > > > > > > > > > > > > > > > > > > > > > > > -- > -- Guozhang >
