Hi Ismael, I can volunteer to write the KIP. Unless somebody else has any objections, I'll get to write it by the end of this week.
Best, Josep Prat Open Source Engineering Director, aivenjosep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Thu, Dec 21, 2023, 17:58 Ismael Juma <m...@ismaeljuma.com> wrote: > Hi all, > > After understanding the use case Josep and Anton described in more detail, > I think it's fair to say that quorum reconfiguration is necessary for > migration of Apache Kafka users who follow this pattern. Given that, I > think we should have a 3.8 release before the 4.0 release. > > The next question is whether we should do something special when it comes > to timeline, parallel releases, etc. After careful consideration, I think > we should simply follow our usual approach: regular 3.8 release around > early May 2024 and regular 4.0 release around early September 2024. The > community will be able to start working on items specific to 4.0 after 3.8 > is branched in late March/early April - I don't think we need to deal with > the overhead of maintaining multiple long-lived branches for > feature development. > > If the proposal above sounds reasonable, I suggest we write a KIP and vote > on it. Any volunteers? > > Ismael > > On Tue, Nov 21, 2023 at 8:18 PM Ismael Juma <m...@ismaeljuma.com> wrote: > > > Hi Luke, > > > > I think we're conflating different things here. There are 3 separate > > points in your email, but only 1 of them requires 3.8: > > > > 1. JBOD may have some bugs in 3.7.0. Whatever bugs exist can be fixed in > > 3.7.x. We have already said that we will backport critical fixes to 3.7.x > > for some time. > > 2. Quorum reconfiguration is important to include in 4.0, the release > > where ZK won't be supported. This doesn't need a 3.8 release either. > > 3. Quorum reconfiguration is necessary for migration use cases and hence > > needs to be in a 3.x release. This one would require a 3.8 release if > true. > > But we should have a debate on whether it is indeed true. It's not clear > to > > me yet. > > > > Ismael > > > > On Tue, Nov 21, 2023 at 7:30 PM Luke Chen <show...@gmail.com> wrote: > > > >> Hi Colin and Jose, > >> > >> I revisited the discussion of KIP-833 here > >> <https://lists.apache.org/thread/90zkqvmmw3y8j6tkgbg3md78m7hs4yn6>, and > >> you > >> can see I'm the first one to reply to the discussion thread to express > my > >> excitement at that time. Till now, I personally still think having KRaft > >> in > >> Kafka is a good direction we have to move forward. But to move to this > >> destination, we need to make our users comfortable with this decision. > The > >> worst scenario is, we said 4.0 is ready, and ZK is removed. Then, some > >> users move to 4.0 and say, wait a minute, why does it not support xxx > >> feature? And then start to search for other alternatives to replace > Apache > >> Kafka. We all don't want to see this, right? So, that's why some > community > >> users start to express their concern to move to 4.0 too quickly, > including > >> me. > >> > >> > >> Quoting Colin: > >> > While dynamic quorum reconfiguration is a nice feature, it doesn't > block > >> anything: not migration, not deployment. > >> > >> Clearly Confluent team might deploy ZooKeeper in a particular way and > >> didn’t depend on its ability to support reconfiguration. So KRaft is > ready > >> from your point of view. But users of Apache Kafka might have come to > >> depend on some ZooKeeper functionality, such as the ability to > reconfigure > >> ZooKeeper quorums, that is not available in KRaft, yet. I don’t think > the > >> Apache Kafka documentation has ever said “do not depend on this ability > of > >> Apache Kafka or Zookeeper”, so it doesn’t seem unreasonable for users to > >> have deployed ZooKeeper in this way. In KIP-833 > >> < > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-MissingFeatures > >> >, > >> we said: “Modifying certain dynamic configurations on the standalone > KRaft > >> controller” was an important missing feature. Unfortunately it wasn’t as > >> explicit as it could have been. While no one expects KRaft to support > all > >> the features of ZooKeeper, it looks to me that users might depend on > this > >> particular feature and it’s only recently that it’s become apparent that > >> you don’t consider it a blocker. > >> > >> Quoting José: > >> > If we do a 3.8 release before 4.0 and we implement KIP-853 in 3.8, the > >> user will be able to migrate to a KRaft cluster that supports > dynamically > >> changing the set of voters and has better support for disk failures. > >> > >> Yes, KIP-853 and disk failure support are both very important missing > >> features. For the disk failure support, I don't think this is a > >> "good-to-have-feature", it should be a "must-have" IMO. We can't > announce > >> the 4.0 release without a good solution for disk failure in KRaft. > >> > >> It’s also worth thinking about how Apache Kafka users who depend on JBOD > >> might look at the risks of not having a 3.8 release. JBOD support on > KRaft > >> is planned to be added in 3.7, and is still in progress so far. So it’s > >> hard to say it’s a blocker or not. But in practice, even if the feature > is > >> made into 3.7 in time, a lot of new code for this feature is unlikely to > >> be > >> entirely bug free. We need to maintain the confidence of those users, > and > >> forcing them to migrate through 3.7 where this new code is hardly > >> battle-tested doesn’t appear to do that. > >> > >> Our goal for 4.0 should be that all the “main” features in KRaft are in > >> production ready state. To reach the goal, I think having one more > release > >> makes sense. We can have different opinions about what the “main > features” > >> in KRaft are, but we should all agree, JBOD is one of them. > >> > >> Alternatively, like Josep proposed, we can choose to have 4.0 + 3.7.x or > >> 3.8 releases in parallel to maintain these 2 releases for a defined > >> period. > >> But I think this is not a small effort to do that, especially as in > v4.0, > >> much of ZK code will be removed, thus the diff between codebases will be > >> large. In other words the additional costs of the backporting required > >> with > >> this alternative are likely to be higher than doing a 3.8 in my opinion. > >> > >> Quoting José again: > >> > What are the disadvantages of adding the 3.8 release before 4.0? This > >> would push the 4.0 release by 3-4 months. From what we can tell, it > would > >> also delay when KIP-896 can be implemented and extend how long the > >> community needs to maintain the code used by ZK mode. Is there anything > >> else? > >> > >> If we agree with previous points, I think the disadvantages will just > >> disappear. The 3-4 months delay, the maintenance effort, KIP-896, and > >> maybe > >> you can also raise scala 2.12 and java 8 removal, which are not that > >> critical compared with what I mentioned earlier that the worst case > might > >> be that the users lose their confidence to Apache Kafka. > >> > >> > >> Quoting Colin: > >> > I would not want to delay that because we want an additional feature. > >> And > >> we will always want additional features. So I am concerned we will end > up > >> in an infinite loop of people asking for "just one more feature" before > >> they migrate. > >> > >> I totally agree with you. We can keep delaying the 4.0 release forever. > >> I'd > >> also like to draw a line to it. So, in my opinion, the 3.8 release is > the > >> line. No 3.9, 3.10 releases after that. If this is the decision, will > your > >> concern about this infinite loop disappear? > >> > >> Final note: Speaking of the missing features, I can always cooperate > with > >> you and all other community contributors to make them happen, like we > have > >> discussed earlier. Just let me know. > >> > >> Thank you. > >> Luke > >> > >> On Wed, Nov 22, 2023 at 2:54 AM Colin McCabe <cmcc...@apache.org> > wrote: > >> > >> > On Tue, Nov 21, 2023, at 03:47, Josep Prat wrote: > >> > > Hi Colin, > >> > > > >> > > I think it's great that Confluent runs KRaft clusters in production, > >> > > and it means that it is production ready for Confluent and it's > users. > >> > > But luckily for Kafka, the community is bigger than this (self > managed > >> > > in the cloud or in-prem, or customers of other SaaS companies). > >> > > >> > Hi Josep, > >> > > >> > Confluent is not the only company using or developing KRaft. Most of > the > >> > big organizations developing Kafka are involved. I mentioned > Confluent's > >> > deployments because I wanted to be clear that KRaft mode is not > >> > experimental or new. Talking about software in production is a good > way > >> to > >> > clear up these misconceptions. > >> > > >> > Indeed, KRaft mode is many years old. It started around 2020, and > became > >> > production-ready in AK 3.5 in 2022. ZK mode was deprecated in AK 3.5, > >> which > >> > was released June 2023. If we release AK 4.0 around April (or maybe a > >> month > >> > or two later) then that will be almost a full year between deprecation > >> and > >> > removal of ZK mode. We've talked about this a lot, in KIPs, in Apache > >> blog > >> > posts, at conferences, and so forth. > >> > > >> > > We've heard at least from 1 SaaS company, Aiven (disclaimer, it is > my > >> > > employer) where the current feature set makes it not trivial to > >> > > migrate. This same issue might happen not only at Aiven but with any > >> > > user of Kafka who uses immutable infrastructure. > >> > > >> > Can you discuss why you feel it is "not trivial to migrate"? From the > >> > discussion above, the main gap is that we should improve the > >> documentation > >> > for handling failed disks. > >> > > >> > > Another case is for > >> > > users that have hundreds (or more) of clusters and more than 100k > >> nodes > >> > > experience node failures multiple times during a single day. In this > >> > > situation, not having KIP 853 makes these power users unable to join > >> > > the game as introducing a new error-prone manual (or needed to > >> > > automate) operation is usually a huge no-go. > >> > > >> > We have thousands of KRaft clusters in production and haven't seen > these > >> > problems, as I described above. > >> > > >> > best, > >> > Colin > >> > > >> > > > >> > > But I hear the concerns of delaying 4.0 for another 3 to 4 months. > >> > > Would it help if we would aim at shortening the timeline for 3.8.0 > and > >> > > start with the 4.0.0 a bit earlier help? > >> > > Maybe we could work on 3.8.0 almost in parallel with 4.0.0: > >> > > - Start with 3.8.0 release process > >> > > - After a small time (let's say a week) create the release branch > >> > > - Start with 4.0.0 release process as usual > >> > > - Cherry pick KRaft related issues to 3.8.0 > >> > > - Release 3.8.0 > >> > > I suspect 4.0.0 will need a bit more time than usual to ensure the > >> code > >> > > is cleaned up of deprecated classes and methods on top of the usual > >> > > work we have. For this reason I think there would be enough time > >> > > between releasing 3.8.0 and 4.0.0. > >> > > > >> > > What do you all think? > >> > > > >> > > Best, > >> > > Josep Prat > >> > > > >> > > On 2023/11/20 20:03:18 Colin McCabe wrote: > >> > >> Hi Josep, > >> > >> > >> > >> I think there is some confusion here. Quorum reconfiguration is not > >> > needed for KRaft to become production ready. Confluent runs thousands > of > >> > KRaft clusters without quorum reconfiguration, and has for years. > While > >> > dynamic quorum reconfiguration is a nice feature, it doesn't block > >> > anything: not migration, not deployment. As best as I understand it, > the > >> > use-case Aiven has isn't even reconfiguration per se, just wiping a > >> disk. > >> > There are ways to handle this -- I discussed some earlier in the > >> thread. I > >> > think it would be productive to continue that discussion -- especially > >> the > >> > part around documentation and testing of these cases. > >> > >> > >> > >> A lot of people have done a lot of work to get Kafka 4.0 ready. I > >> would > >> > not want to delay that because we want an additional feature. And we > >> will > >> > always want additional features. So I am concerned we will end up in > an > >> > infinite loop of people asking for "just one more feature" before they > >> > migrate. > >> > >> > >> > >> best, > >> > >> Colin > >> > >> > >> > >> > >> > >> On Mon, Nov 20, 2023, at 04:15, Josep Prat wrote: > >> > >> > Hi all, > >> > >> > > >> > >> > I wanted to share my opinion regarding this topic. I know some > >> > >> > discussions happened some time ago (over a year) but I believe > it's > >> > >> > wise to reflect and re-evaluate if those decisions are still > valid. > >> > >> > KRaft, as of Kafka 3.6.x and 3.7.x, has not yet feature parity > with > >> > >> > Zookeeper. By dropping Zookeeper altogether before achieving such > >> > >> > parity, we are opening the door to leaving a chunk of Apache > Kafka > >> > >> > users without an easy way to upgrade to 4.0. > >> > >> > In pro of making upgrades as smooth as possible, I propose to > have > >> a > >> > >> > Kafka version where KIP-853 is merged and Zookeeper still is > >> > supported. > >> > >> > This will enable community members who can't migrate yet to KRaft > >> to > >> > do > >> > >> > so in a safe way (rolling back is something goes wrong). > >> > Additionally, > >> > >> > this will give us more confidence on having KRaft replacing > >> > >> > successfully Zookeeper without any big problems by discovering > and > >> > >> > fixing bugs or by confirming that KRaft works as expected. > >> > >> > For this I strongly believe we should have a 3.8.x version before > >> > 4.0.x. > >> > >> > > >> > >> > What do other think in this regard? > >> > >> > > >> > >> > Best, > >> > >> > > >> > >> > On 2023/11/14 20:47:10 Colin McCabe wrote: > >> > >> >> On Tue, Nov 14, 2023, at 04:37, Anton Agestam wrote: > >> > >> >> > Hi Colin, > >> > >> >> > > >> > >> >> > Thank you for your thoughtful and comprehensive response. > >> > >> >> > > >> > >> >> >> KIP-853 is not a blocker for either 3.7 or 4.0. We discussed > >> this > >> > in > >> > >> >> >> several KIPs that happened this year and last year. The most > >> > notable was > >> > >> >> >> probably KIP-866, which was approved in May 2022. > >> > >> >> > > >> > >> >> > I understand this is the case, I'm raising my concern because > I > >> was > >> > >> >> > foreseeing some major pain points as a consequence of this > >> > decision. Just > >> > >> >> > to make it clear though: I am not asking for anyone to do work > >> for > >> > me, and > >> > >> >> > I understand the limitations of resources available to > implement > >> > features. > >> > >> >> > What I was asking is rather to consider the implications of > >> > _removing_ > >> > >> >> > features before there exists a replacement for them. > >> > >> >> > > >> > >> >> > I understand that the timeframe for 3.7 isn't feasible, and > >> > because of that > >> > >> >> > I think what I was asking is rather: can we make sure that > there > >> > are more > >> > >> >> > 3.x releases until controller quorum online resizing is > >> > implemented? > >> > >> >> > > >> > >> >> > From your response, I gather that your stance is that it's > >> > important to > >> > >> >> > drop ZK support sooner rather than later and that the > necessary > >> > pieces for > >> > >> >> > doing so are already in place. > >> > >> >> > >> > >> >> Hi Anton, > >> > >> >> > >> > >> >> Yes. I'm basically just repeating what we agreed upon in 2022 as > >> > part of KIP-833. > >> > >> >> > >> > >> >> > > >> > >> >> > --- > >> > >> >> > > >> > >> >> > I want to make sure I've understood your suggested sequence > for > >> > controller > >> > >> >> > node replacement. I hope the mentions of Kubernetes are rather > >> for > >> > examples > >> > >> >> > of how to carry things out, rather than saying "this is only > >> > supported on > >> > >> >> > Kubernetes"? > >> > >> >> > >> > >> >> Apache Kafka is supported in lots of environments, including > >> non-k8s > >> > ones. I was just pointing out that using k8s means that you control > your > >> > own DNS resolution, which simplifies matters. If you don't control DNS > >> > there are some extra steps for changing the quorum voters. > >> > >> >> > >> > >> >> > > >> > >> >> > Given we have three existing nodes as such: > >> > >> >> > > >> > >> >> > - a.local -> 192.168.0.100 > >> > >> >> > - b.local -> 192.168.0.101 > >> > >> >> > - c.local -> 192.168.0.102 > >> > >> >> > > >> > >> >> > As well as a candidate node 192.168.0.103 that we want to > >> replace > >> > for the > >> > >> >> > role of c.local. > >> > >> >> > > >> > >> >> > 1. Shut down controller process on node .102 (to make sure we > >> > don't "go > >> > >> >> > back in time"). > >> > >> >> > 2. rsync state from leader to .103. > >> > >> >> > 3. Start controller process on .103. > >> > >> >> > 4. Point the c.local entry at .103. > >> > >> >> > > >> > >> >> > I have a few questions about this sequence: > >> > >> >> > > >> > >> >> > 1. Would this sequence be safe against leadership changes? > >> > >> >> > > >> > >> >> > >> > >> >> If the leader changes, the new leader should have all of the > >> > committed entries that the old leader had. > >> > >> >> > >> > >> >> > 2. Does it work > >> > >> >> > >> > >> >> Probably the biggest issue is dealing with "torn writes" that > >> happen > >> > because you're copying the current log segment while it's being > written > >> to. > >> > The system should be robust against this. However, we don't regularly > do > >> > this, so there hasn't been a lot of testing. > >> > >> >> > >> > >> >> I think Jose had a PR for improving the handling of this which > we > >> > might want to dig up. We'd want the system to auto-truncate the > partial > >> > record at the end of the log, if there is one. > >> > >> >> > >> > >> >> > 3. By "state", do we mean `metadata.log.dir`? Something else? > >> > >> >> > >> > >> >> Yes, the state of the metadata.log.dir. Keep in mind you will > need > >> > to change the node ID in meta.properties after copying, of course. > >> > >> >> > >> > >> >> > 4. What are the effects on cluster availability? (I think this > >> is > >> > the same > >> > >> >> > as asking what happens if a or b crashes during the process, > or > >> if > >> > network > >> > >> >> > partitions occur). > >> > >> >> > >> > >> >> Cluster metadata state tends to be pretty small. typically a > >> hundred > >> > megabytes or so. Therefore, I do not think it will take more than a > >> second > >> > or two to copy from one node to another. However, if you do > experience a > >> > crash when one node out of three is down, then you will be unavailable > >> > until you can bring up a second node to regain a majority. > >> > >> >> > >> > >> >> > > >> > >> >> > --- > >> > >> >> > > >> > >> >> > If this is considered the official way of handling controller > >> node > >> > >> >> > replacements, does it make sense to improve documentation in > >> this > >> > area? Is > >> > >> >> > there already a plan for this documentation layed out in some > >> > KIPs? This is > >> > >> >> > something I'd be happy to contribute to. > >> > >> >> > > >> > >> >> > >> > >> >> Yes, I think we should have official documentation about this. > >> We'd > >> > be happy to review anything in that area. > >> > >> >> > >> > >> >> >> To circle back to KIP-853, I think it stands a good chance of > >> > making it > >> > >> >> >> into AK 4.0. > >> > >> >> > > >> > >> >> > This sounds good, but the point I was making was if we could > >> have > >> > a release > >> > >> >> > with both KRaft and ZK supporting this feature to ease the > >> > migration out of > >> > >> >> > ZK. > >> > >> >> > > >> > >> >> > >> > >> >> The problem is, supporting multiple controller implementations > is > >> a > >> > huge burden. So we don't want to extend the 3.x release past the point > >> > that's needed to complete all the must-dos (SCRAM, delegation tokens, > >> JBOD) > >> > >> >> > >> > >> >> best, > >> > >> >> Colin > >> > >> >> > >> > >> >> > >> > >> >> > BR, > >> > >> >> > Anton > >> > >> >> > > >> > >> >> > Den tors 9 nov. 2023 kl 23:04 skrev Colin McCabe < > >> > cmcc...@apache.org>: > >> > >> >> > > >> > >> >> >> Hi Anton, > >> > >> >> >> > >> > >> >> >> It rarely makes sense to scale up and down the number of > >> > controller nodes > >> > >> >> >> in the cluster. Only one controller node will be active at > any > >> > given time. > >> > >> >> >> The main reason to use 5 nodes would be to be able to > tolerate > >> 2 > >> > failures > >> > >> >> >> instead of 1. > >> > >> >> >> > >> > >> >> >> At Confluent, we generally run KRaft with 3 controllers. We > >> have > >> > not seen > >> > >> >> >> problems with this setup, even with thousands of clusters. We > >> have > >> > >> >> >> discussed using 5 node controller clusters on certain very > big > >> > clusters, > >> > >> >> >> but we haven't done that yet. This is all very similar to ZK, > >> > where most > >> > >> >> >> deployments were 3 nodes as well. > >> > >> >> >> > >> > >> >> >> KIP-853 is not a blocker for either 3.7 or 4.0. We discussed > >> this > >> > in > >> > >> >> >> several KIPs that happened this year and last year. The most > >> > notable was > >> > >> >> >> probably KIP-866, which was approved in May 2022. > >> > >> >> >> > >> > >> >> >> Many users these days run in a Kubernetes environment where > >> > Kubernetes > >> > >> >> >> actually controls the DNS. This makes changing the set of > >> voters > >> > less > >> > >> >> >> important than it was historically. > >> > >> >> >> > >> > >> >> >> For example, in a world with static DNS, you might have to > >> change > >> > the > >> > >> >> >> controller.quorum.voters setting from: > >> > >> >> >> > >> > >> >> >> 100@a.local:9073,101@b.local:9073,102@c.local:9073 > >> > >> >> >> > >> > >> >> >> to: > >> > >> >> >> > >> > >> >> >> 100@a.local:9073,101@b.local:9073,102@d.local:9073 > >> > >> >> >> > >> > >> >> >> In a world with k8s controlling the DNS, you simply remap > >> c.local > >> > to point > >> > >> >> >> ot the IP address of your new pod for controller 102, and > >> you're > >> > done. No > >> > >> >> >> need to update controller.quorum.voters. > >> > >> >> >> > >> > >> >> >> Another question is whether you re-create the pod data from > >> > scratch every > >> > >> >> >> time you add a new node. If you store the controller data on > an > >> > EBS volume > >> > >> >> >> (or cloud-specific equivalent), you really only have to > detach > >> it > >> > from the > >> > >> >> >> previous pod and re-attach it to the new pod. k8s also > handles > >> > this > >> > >> >> >> automatically, of course. > >> > >> >> >> > >> > >> >> >> If you want to reconstruct the full controller pod state each > >> > time you > >> > >> >> >> create a new pod (for example, so that you can use only > >> instance > >> > storage), > >> > >> >> >> you should be able to rsync that state from the leader. In > >> > general, the > >> > >> >> >> invariant that we want to maintain is that the state should > not > >> > "go back in > >> > >> >> >> time" -- if controller 102 promised to hold all log data up > to > >> > offset X, it > >> > >> >> >> should come back with committed data at at least that offset. > >> > >> >> >> > >> > >> >> >> There are lots of new features we'd like to implement for > >> KRaft, > >> > and Kafka > >> > >> >> >> in general. If you have some you really would like to see, I > >> > think everyone > >> > >> >> >> in the community would be happy to work with you. The flip > >> side, > >> > of course, > >> > >> >> >> is that since there are an unlimited number of features we > >> could > >> > do, we > >> > >> >> >> can't really block the release for any one feature. > >> > >> >> >> > >> > >> >> >> To circle back to KIP-853, I think it stands a good chance of > >> > making it > >> > >> >> >> into AK 4.0. Jose, Alyssa, and some other people have worked > on > >> > it. It > >> > >> >> >> definitely won't make it into 3.7, since we have only a few > >> weeks > >> > left > >> > >> >> >> before that release happens. > >> > >> >> >> > >> > >> >> >> best, > >> > >> >> >> Colin > >> > >> >> >> > >> > >> >> >> > >> > >> >> >> On Thu, Nov 9, 2023, at 00:20, Anton Agestam wrote: > >> > >> >> >> > Hi Luke, > >> > >> >> >> > > >> > >> >> >> > We have been looking into what switching from ZK to KRaft > >> will > >> > mean for > >> > >> >> >> > Aiven. > >> > >> >> >> > > >> > >> >> >> > We heavily depend on an “immutable infrastructure” model > for > >> > deployments. > >> > >> >> >> > This means that, when we perform upgrades, we introduce new > >> > nodes to our > >> > >> >> >> > clusters, scale the cluster up to incorporate the new > nodes, > >> > and then > >> > >> >> >> phase > >> > >> >> >> > the old ones out once all partitions are moved to the new > >> > generation. > >> > >> >> >> This > >> > >> >> >> > allows us, and anyone else using a similar model, to do > >> > upgrades as well > >> > >> >> >> as > >> > >> >> >> > cluster resizing with zero downtime. > >> > >> >> >> > > >> > >> >> >> > Reading up on KRaft and the ZK-to-KRaft migration path, > this > >> is > >> > somewhat > >> > >> >> >> > worrying for us. It seems like, if KIP-853 is not included > >> > prior to > >> > >> >> >> > dropping support for ZK, we will essentially have no > >> satisfying > >> > upgrade > >> > >> >> >> > path. Even if KIP-853 is included in 4.0, I’m unsure if > that > >> > would allow > >> > >> >> >> a > >> > >> >> >> > migration path for us, since a new cluster generation would > >> not > >> > be able > >> > >> >> >> to > >> > >> >> >> > use ZK during the migration step. > >> > >> >> >> > On the other hand, if KIP-853 was released in a version > prior > >> > to dropping > >> > >> >> >> > ZK support, because it allows online resizing of KRaft > >> > clusters, this > >> > >> >> >> would > >> > >> >> >> > allow us and others that use an immutable infrastructure > >> > deployment > >> > >> >> >> model, > >> > >> >> >> > to provide a zero downtime migration path. > >> > >> >> >> > > >> > >> >> >> > For that reason, we’d like to raise awareness around this > >> issue > >> > and > >> > >> >> >> > encourage considering the implementation of KIP-853 or > >> > equivalent a > >> > >> >> >> blocker > >> > >> >> >> > not only for 4.0, but for the last version prior to 4.0. > >> > >> >> >> > > >> > >> >> >> > BR, > >> > >> >> >> > Anton > >> > >> >> >> > > >> > >> >> >> > On 2023/10/11 12:17:23 Luke Chen wrote: > >> > >> >> >> >> Hi all, > >> > >> >> >> >> > >> > >> >> >> >> While Kafka 3.6.0 is released, I’d like to start the > >> > discussion for the > >> > >> >> >> >> “road to Kafka 4.0”. Based on the plan in KIP-833 > >> > >> >> >> >> < > >> > >> >> >> > > >> > >> >> >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.7 > >> > >> >> >> >>, > >> > >> >> >> >> the next release 3.7 will be the final release before > moving > >> > to Kafka > >> > >> >> >> 4.0 > >> > >> >> >> >> to remove the Zookeeper from Kafka. Before making this > major > >> > change, I'd > >> > >> >> >> >> like to get consensus on the "must-have features/fixes for > >> > Kafka 4.0", > >> > >> >> >> to > >> > >> >> >> >> avoid some users being surprised when upgrading to Kafka > >> 4.0. > >> > The intent > >> > >> >> >> > is > >> > >> >> >> >> to have a clear communication about what to expect in the > >> > following > >> > >> >> >> > months. > >> > >> >> >> >> In particular we should be signaling what features and > >> > configurations > >> > >> >> >> are > >> > >> >> >> >> not supported, or at risk (if no one is able to add > support > >> or > >> > fix known > >> > >> >> >> >> bugs). > >> > >> >> >> >> > >> > >> >> >> >> Here is the JIRA tickets list > >> > >> >> >> >> < > >> > https://issues.apache.org/jira/issues/?jql=labels%20%3D%204.0-blocker > > > >> > >> >> >> I > >> > >> >> >> >> labeled for "4.0-blocker". The criteria I labeled as > >> > “4.0-blocker” are: > >> > >> >> >> >> 1. The feature is supported in Zookeeper Mode, but not > >> > supported in > >> > >> >> >> KRaft > >> > >> >> >> >> mode, yet (ex: KIP-858: JBOD in KRaft) > >> > >> >> >> >> 2. Critical bugs in KRaft, (ex: KAFKA-15489 : split brain > in > >> > KRaft > >> > >> >> >> >> controller quorum) > >> > >> >> >> >> > >> > >> >> >> >> If you disagree with my current list, welcome to have > >> > discussion in the > >> > >> >> >> >> specific JIRA ticket. Or, if you think there are some > >> tickets > >> > I missed, > >> > >> >> >> >> welcome to start a discussion in the JIRA ticket and ping > me > >> > or other > >> > >> >> >> >> people. After we get the consensus, we can label/unlabel > it > >> > afterwards. > >> > >> >> >> >> Again, the goal is to have an open communication with the > >> > community > >> > >> >> >> about > >> > >> >> >> >> what will be coming in 4.0. > >> > >> >> >> >> > >> > >> >> >> >> Below is the high level category of the list content: > >> > >> >> >> >> > >> > >> >> >> >> 1. Recovery from disk failure > >> > >> >> >> >> KIP-856 > >> > >> >> >> >> < > >> > >> >> >> > > >> > >> >> >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-856:+KRaft+Disk+Failure+Recovery > >> > >> >> >> >>: > >> > >> >> >> >> KRaft Disk Failure Recovery > >> > >> >> >> >> > >> > >> >> >> >> 2. Prevote to support controllers more than 3 > >> > >> >> >> >> KIP-650 > >> > >> >> >> >> < > >> > >> >> >> > > >> > >> >> >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-650%3A+Enhance+Kafkaesque+Raft+semantics > >> > >> >> >> >>: > >> > >> >> >> >> Enhance Kafkaesque Raft semantics > >> > >> >> >> >> > >> > >> >> >> >> 3. JBOD support > >> > >> >> >> >> KIP-858 > >> > >> >> >> >> < > >> > >> >> >> > > >> > >> >> >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft > >> > >> >> >> >>: > >> > >> >> >> >> Handle > >> > >> >> >> >> JBOD broker disk failure in KRaft > >> > >> >> >> >> > >> > >> >> >> >> 4. Scale up/down Controllers > >> > >> >> >> >> KIP-853 > >> > >> >> >> >> < > >> > >> >> >> > > >> > >> >> >> > >> > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes > >> > >> >> >> >>: > >> > >> >> >> >> KRaft Controller Membership Changes > >> > >> >> >> >> > >> > >> >> >> >> 5. Modifying dynamic configurations on the KRaft > controller > >> > >> >> >> >> > >> > >> >> >> >> 6. Critical bugs in KRaft > >> > >> >> >> >> > >> > >> >> >> >> Does this make sense? > >> > >> >> >> >> Any feedback is welcomed. > >> > >> >> >> >> > >> > >> >> >> >> Thank you. > >> > >> >> >> >> Luke > >> > >> >> >> >> > >> > >> >> >> > >> > >> >> > >> > >> > >> > > >> > > >
