I agree the migration should be functional - it wasn't obvious if the migration issues are edge cases or not. If they are edge cases, I think 3.5.1 would be fine given the preview status.
I understand that a new RC is needed, but that doesn't mean we should let everything in. Each change carries some risk. And if we don't agree on the bar for the migration work, we may be having the same discussion next week. :) Ismael On Wed, May 24, 2023, 12:00 PM Josep Prat <josep.p...@aiven.io.invalid> wrote: > Hi there, > Is the plan described in KIP-833[1] still valid? In there it states that > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the path to > migrate to Kraft should be somewhat functional (in my opinion). If we don't > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion that > these could be fixed in subsequent patches of 3.5.x. Just my 5cts. > > [1]: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5 > Best, > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > Are all these blockers? For example, zk to kraft migration are is still > in > > preview - can we fix some of these in 3.5.1? > > > > Ismael > > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe <cmcc...@apache.org> wrote: > > > > > Hi Mickael, > > > > > > Thanks for putting together this RC. Unfortunately, we've identified > > > several blocker issues in this release candidate. > > > > > > KAFKA-15009: New ACLs are not written to ZK during migration > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in > > > migration. > > > KAFKA-15004: Topic config changes are not synced during zk to kraft > > > migration (dual-write) > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration > > > (dual-write) when partitions are changed for topic > > > KAFKA-14996: The KRaft controller should properly handle overly large > > user > > > operations > > > > > > We are working on PRs for these issues and will get them in soon, we > > think! > > > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for > another > > > RC next week. > > > > > > best, > > > Colin > > > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote: > > > > Hi David, > > > > > > > > We're already quite a bit behind schedule. If you think these fixes > > > > are really important and can be ready in the next couple of days, I'm > > > > open to backport them and build another release candidate. Let me > know > > > > once you've investigated the severity of KAFKA-15010. > > > > > > > > Thanks, > > > > Mickael > > > > > > > > > > > > On Tue, May 23, 2023 at 6:34 PM David Arthur > > > > <david.art...@confluent.io.invalid> wrote: > > > >> > > > >> Mickael, we have some migration fixes on trunk, is it okay to > > > cherry-pick > > > >> these to 3.5? > > > >> > > > >> KAFKA-15007 Use the correct MetadataVersion in MigrationPropagator > > > >> KAFKA-15009 Handle new ACLs in KRaft snapshot during migration > > > >> > > > >> There is another issue KAFKA-15010 that I'm also investigating to > > > determine > > > >> the impact and likelihood of seeing it in practice. This one may be > a > > > >> significant migration blocker > > > >> > > > >> Cheers, > > > >> David > > > >> > > > >> On Tue, May 23, 2023 at 9:57 AM Mickael Maison < > > > mickael.mai...@gmail.com> > > > >> wrote: > > > >> > > > >> > Hi Christo, > > > >> > > > > >> > Yes this is expected. This happens when nested fields also accept > > > >> > optional tagged fields. The tables list all fields, so they may > > > >> > include _tagged_fields multiple times. > > > >> > Clearly the layout of this page could be improved, if you have > ideas > > > >> > how to describe the protocol in a better way, feel free to share > > them. > > > >> > > > > >> > Thanks, > > > >> > Mickael > > > >> > > > > >> > On Tue, May 23, 2023 at 3:50 PM Mickael Maison < > > > mickael.mai...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > > Hi Josep, > > > >> > > > > > >> > > Good catch! I opened a PR to fix this: > > > >> > > https://github.com/apache/kafka-site/pull/514 > > > >> > > > > > >> > > Thanks, > > > >> > > Mickael > > > >> > > > > > >> > > > > > >> > > On Tue, May 23, 2023 at 3:36 PM Christo Lolov < > > > christolo...@gmail.com> > > > >> > wrote: > > > >> > > > > > > >> > > > Hey Mickael! > > > >> > > > > > > >> > > > I am giving a +1 (non-binding) for this candidate release. > > > >> > > > > > > >> > > > * Built from the binary tar.gz source with Java 17 and Scala > > 2.13 > > > on > > > >> > Intel > > > >> > > > (m5.4xlarge) and ARM (m6g.4xlarge) machines. > > > >> > > > * Ran unit and integration tests on Intel and ARM machines. > > > >> > > > * Ran the Quickstart in both Zookeeper and KRaft modes on > Intel > > > and ARM > > > >> > > > machines. > > > >> > > > > > > >> > > > Question: > > > >> > > > * I went through https://kafka.apache.org/35/protocol.html > and > > > there > > > >> > are > > > >> > > > quite a few repetitive __tagged_fileds fields within the same > > > >> > structures - > > > >> > > > is this expected? > > > >> > > > > > > >> > > > On Tue, 23 May 2023 at 12:01, Josep Prat > > > <josep.p...@aiven.io.invalid> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > Hi Mickael, > > > >> > > > > I just wanted to point out that I think the documentation > you > > > >> > recently > > > >> > > > > merged on Kafka site regarding the 3.5.0 version has a > problem > > > when > > > >> > it > > > >> > > > > states the version number and the sub-menu that links to > > > previous > > > >> > versions. > > > >> > > > > Left a comment here: > > > >> > > > > > > > >> > > > > > > > https://github.com/apache/kafka-site/pull/513#pullrequestreview-1438927939 > > > >> > > > > > > > >> > > > > Best, > > > >> > > > > > > > >> > > > > On Tue, May 23, 2023 at 9:29 AM Josep Prat < > > josep.p...@aiven.io > > > > > > > >> > wrote: > > > >> > > > > > > > >> > > > > > Hi Mickael, > > > >> > > > > > > > > >> > > > > > I can +1 this candidate. I verified the following: > > > >> > > > > > - Built from source with Java 17 and Scala 2.13 > > > >> > > > > > - Signatures and hashes of the artifacts generated > > > >> > > > > > - Navigated through Javadoc including links to JDK classes > > > >> > > > > > - Run the unit tests > > > >> > > > > > - Run integration tests > > > >> > > > > > - Run the quickstart in KRaft and Zookeeper mode > > > >> > > > > > > > > >> > > > > > Best, > > > >> > > > > > > > > >> > > > > > On Mon, May 22, 2023 at 5:30 PM Mickael Maison < > > > >> > mimai...@apache.org> > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > >> Hello Kafka users, developers and client-developers, > > > >> > > > > >> > > > >> > > > > >> This is the first candidate for release of Apache Kafka > > > 3.5.0. > > > >> > Some of > > > >> > > > > the > > > >> > > > > >> major features include: > > > >> > > > > >> - KIP-710: Full support for distributed mode in dedicated > > > >> > MirrorMaker > > > >> > > > > >> 2.0 clusters > > > >> > > > > >> - KIP-881: Rack-aware Partition Assignment for Kafka > > > Consumers > > > >> > > > > >> - KIP-887: Add ConfigProvider to make use of environment > > > variables > > > >> > > > > >> - KIP-889: Versioned State Stores > > > >> > > > > >> - KIP-894: Use incrementalAlterConfig for syncing topic > > > >> > configurations > > > >> > > > > >> - KIP-900: KRaft kafka-storage.sh API additions to > support > > > SCRAM > > > >> > for > > > >> > > > > >> Kafka Brokers > > > >> > > > > >> > > > >> > > > > >> Release notes for the 3.5.0 release: > > > >> > > > > >> > > > >> > > > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/RELEASE_NOTES.html > > > >> > > > > >> > > > >> > > > > >> *** Please download, test and vote by Friday, May 26, 5pm > > PT > > > >> > > > > >> > > > >> > > > > >> Kafka's KEYS file containing PGP keys we use to sign the > > > release: > > > >> > > > > >> https://kafka.apache.org/KEYS > > > >> > > > > >> > > > >> > > > > >> * Release artifacts to be voted upon (source and binary): > > > >> > > > > >> https://home.apache.org/~mimaison/kafka-3.5.0-rc0/ > > > >> > > > > >> > > > >> > > > > >> * Maven artifacts to be voted upon: > > > >> > > > > >> > > > >> > > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > >> > > > > >> > > > >> > > > > >> * Javadoc: > > > >> > > > > >> > https://home.apache.org/~mimaison/kafka-3.5.0-rc0/javadoc/ > > > >> > > > > >> > > > >> > > > > >> * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag: > > > >> > > > > >> https://github.com/apache/kafka/releases/tag/3.5.0-rc0 > > > >> > > > > >> > > > >> > > > > >> The PR adding the 35 documentation is not merged yet > > > >> > > > > >> (https://github.com/apache/kafka-site/pull/513) > > > >> > > > > >> * Documentation: > > > >> > > > > >> https://kafka.apache.org/35/documentation.html > > > >> > > > > >> * Protocol: > > > >> > > > > >> https://kafka.apache.org/35/protocol.html > > > >> > > > > >> > > > >> > > > > >> * Successful Jenkins builds for the 3.5 branch: > > > >> > > > > >> Unit/integration tests: Jenkins is not detecting the 3.5 > > > branch, > > > >> > > > > >> working with INFRA to sort it out: > > > >> > > > > >> https://issues.apache.org/jira/browse/INFRA-24577 > > > >> > > > > >> System tests: The build is still running, I'll send an > > update > > > >> > once I > > > >> > > > > >> have the results > > > >> > > > > >> > > > >> > > > > >> Thanks, > > > >> > > > > >> Mickael > > > >> > > > > >> > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > -- > > > >> > > > > > [image: Aiven] <https://www.aiven.io> > > > >> > > > > > > > > >> > > > > > *Josep Prat* > > > >> > > > > > Open Source Engineering Director, *Aiven* > > > >> > > > > > josep.p...@aiven.io | +491715557497 > > > >> > > > > > aiven.io <https://www.aiven.io> | > > > >> > > > > > <https://www.facebook.com/aivencloud> > > > >> > > > > > <https://www.linkedin.com/company/aiven/> < > > > >> > > > > https://twitter.com/aiven_io> > > > >> > > > > > *Aiven Deutschland GmbH* > > > >> > > > > > Alexanderufer 3-7, 10117 Berlin > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > -- > > > >> > > > > [image: Aiven] <https://www.aiven.io> > > > >> > > > > > > > >> > > > > *Josep Prat* > > > >> > > > > Open Source Engineering Director, *Aiven* > > > >> > > > > josep.p...@aiven.io | +491715557497 > > > >> > > > > aiven.io <https://www.aiven.io> | < > > > >> > https://www.facebook.com/aivencloud > > > >> > > > > > > > > >> > > > > <https://www.linkedin.com/company/aiven/> < > > > >> > > > > https://twitter.com/aiven_io> > > > >> > > > > *Aiven Deutschland GmbH* > > > >> > > > > Alexanderufer 3-7, 10117 Berlin > > > >> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > >> > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > >> > > > > > > > >> > > > > >> > > > >> > > > >> -- > > > >> -David > > > > > > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud > > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B >
