I have a fix for KAFKA-15968 <https://issues.apache.org/jira/browse/KAFKA-15968> here https://github.com/apache/kafka/pull/14919/. After a bit of digging, I found that this behavior has existed in the KRaft controller since the beginning, so it is not a regression.
Another thing I observed while investigating this is that MetadataLoader *does* treat CorruptRecordExceptions as fatal, which leads to the crash we want. RaftClient calls handleCommit serially for all its listeners, so if QuorumController#handleCommit is called first and does not crash, the call to MetadataLoader#handleCommit will crash. Considering these two factors, I don't strongly feel like we need to block the release for this fix. -David On Mon, Dec 4, 2023 at 10:49 AM David Arthur <david.art...@confluent.io> wrote: > Mickael, > > I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while > investigating a log corruption issue on the controller. I'm still > investigating the issue to see how far back this goes, but I think this > could be a blocker. > > Essentially, the bug is that the controller does not treat a > CorruptRecordException as fatal, so the process will continue running. If > this happens on an active controller, it could corrupt the cluster's > metadata in general (since missing a single metadata record can cause lots > of downstream problems). > > I'll update this thread by the end of day with a stronger > blocker/non-blocker opinion. > > Thanks, > David > > > On Mon, Dec 4, 2023 at 6:48 AM Luke Chen <show...@gmail.com> wrote: > >> Hi Mickael: >> >> I did: >> 1. Validated all checksums, signatures, and hashes >> 2. Ran quick start for KRaft using scala 2.12 artifacts >> 3. Spot checked the documentation and Javadoc >> 4. Validated the licence file >> >> When running the validation to scala 2.12 package, I found these libraries >> are missing: (We only include scala 2.13 libraries in licence file) >> scala-java8-compat_2.12-1.0.2 is missing in license file >> scala-library-2.12.18 is missing in license file >> scala-logging_2.12-3.9.4 is missing in license file >> scala-reflect-2.12.18 is missing in license file >> >> It looks like this issue has been there for a long time, so it won't be a >> block issue for v3.6.1. >> >> +1 (binding) from me. >> >> Thank you. >> Luke >> >> On Sat, Dec 2, 2023 at 5:46 AM Bill Bejeck <bbej...@gmail.com> wrote: >> >> > Hi Mickael, >> > >> > I did the following: >> > >> > 1. Validated all checksums, signatures, and hashes >> > 2. Built from source >> > 3. Ran all the unit tests >> > 4. Spot checked the documentation and Javadoc >> > 5. Ran the ZK, Kraft, and Kafka Streams quickstart guides >> > >> > I did notice that the `fillDotVersion` in `js/templateData.js` needs >> > updating to `3.6.1`, but this is minor and should not block the release. >> > >> > It's a +1(binding) for me, pending the successful system test run >> > >> > Thanks, >> > Bill >> > >> > On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan >> <jols...@confluent.io.invalid >> > > >> > wrote: >> > >> > > I've started a system test run on my end. >> > > >> > > Justine >> > > >> > > On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan <jols...@confluent.io> >> > > wrote: >> > > >> > > > I built from source and ran a simple transactional produce bench. I >> > ran a >> > > > handful of unit tests as well. >> > > > I scanned the docs and everything looked reasonable. >> > > > >> > > > I was wondering if we got the system test results mentioned > System >> > > > tests: Still running I'll post an update once they complete. >> > > > >> > > > Justine >> > > > >> > > > On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison < >> > mickael.mai...@gmail.com >> > > > >> > > > wrote: >> > > > >> > > >> Hi Josep, >> > > >> >> > > >> Good catch! >> > > >> If it's the only issue we find, I don't think we should block the >> > > >> release just to fix that. >> > > >> >> > > >> If we find another issue, I'll backport it before running another >> RC, >> > > >> otherwise I'll backport it once 3.6.1 is released. >> > > >> >> > > >> Thanks, >> > > >> Mickael >> > > >> >> > > >> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat >> > <josep.p...@aiven.io.invalid >> > > > >> > > >> wrote: >> > > >> > >> > > >> > Hi Mickael, >> > > >> > This PR[1] made me realize NOTICE-binary is missing the notice >> for >> > > >> > commons-io. I don't know if it's a blocker or not. I can cherry >> pick >> > > the >> > > >> > commit to the 3.6 branch if you want. >> > > >> > >> > > >> > Best, >> > > >> > >> > > >> > >> > > >> > [1]: https://github.com/apache/kafka/pull/14865 >> > > >> > >> > > >> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat <josep.p...@aiven.io >> > >> > > >> wrote: >> > > >> > >> > > >> > > Hi Mickael, >> > > >> > > Thanks for running the release. It's a +1 for me (non-binding). >> > > >> > > I did the following: >> > > >> > > - Verified artifact's signatures and hashes >> > > >> > > - Checked JavaDoc (with navigation to Oracle JavaDoc) >> > > >> > > - Compiled source code >> > > >> > > - Run unit tests and integration tests >> > > >> > > - Run getting started with ZK and KRaft >> > > >> > > >> > > >> > > Best, >> > > >> > > >> > > >> > > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash < >> > > >> > > kamal.chandraprak...@gmail.com> wrote: >> > > >> > > >> > > >> > >> +1 (non-binding) >> > > >> > >> >> > > >> > >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13 >> > > >> > >> 2. Ran all the unit and integration tests. >> > > >> > >> 3. Ran quickstart and verified the produce-consume on a 3 node >> > > >> cluster. >> > > >> > >> 4. Verified the tiered storage functionality with local-tiered >> > > >> storage. >> > > >> > >> >> > > >> > >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri < >> > > >> fedeval...@gmail.com> >> > > >> > >> wrote: >> > > >> > >> >> > > >> > >> > Hi Mickael, >> > > >> > >> > >> > > >> > >> > - Build from source (Java 17, Scala 2.13) >> > > >> > >> > - Run unit and integration tests >> > > >> > >> > - Run custom client apps using staging artifacts >> > > >> > >> > >> > > >> > >> > +1 (non binding) >> > > >> > >> > >> > > >> > >> > Thanks >> > > >> > >> > Fede >> > > >> > >> > >> > > >> > >> > >> > > >> > >> > >> > > >> > >> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz < >> ja...@scholz.cz >> > > >> > > >> wrote: >> > > >> > >> > > >> > > >> > >> > > +1 non-binding. I used the staged Scala 2.13 artifacts and >> > the >> > > >> staged >> > > >> > >> > Maven >> > > >> > >> > > repo for my tests. All seems to work fine. >> > > >> > >> > > >> > > >> > >> > > Thanks >> > > >> > >> > > Jakub >> > > >> > >> > > >> > > >> > >> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison < >> > > >> mimai...@apache.org> >> > > >> > >> > wrote: >> > > >> > >> > > >> > > >> > >> > > > Hello Kafka users, developers and client-developers, >> > > >> > >> > > > >> > > >> > >> > > > This is the first candidate for release of Apache Kafka >> > > 3.6.1. >> > > >> > >> > > > >> > > >> > >> > > > This is a bugfix release with several fixes, including >> > > >> dependency >> > > >> > >> > > > version bumps for CVEs. >> > > >> > >> > > > >> > > >> > >> > > > Release notes for the 3.6.1 release: >> > > >> > >> > > > >> > > >> > >> >> > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html >> > > >> > >> > > > >> > > >> > >> > > > *** Please download, test and vote by Friday, December 1 >> > > >> > >> > > > >> > > >> > >> > > > Kafka's KEYS file containing PGP keys we use to sign the >> > > >> release: >> > > >> > >> > > > https://kafka.apache.org/KEYS >> > > >> > >> > > > >> > > >> > >> > > > * Release artifacts to be voted upon (source and >> binary): >> > > >> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/ >> > > >> > >> > > > >> > > >> > >> > > > * Maven artifacts to be voted upon: >> > > >> > >> > > > >> > > >> > >> >> > > >> >> > https://repository.apache.org/content/groups/staging/org/apache/kafka/ >> > > >> > >> > > > >> > > >> > >> > > > * Javadoc: >> > > >> > >> > > > >> https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/ >> > > >> > >> > > > >> > > >> > >> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 >> tag: >> > > >> > >> > > > https://github.com/apache/kafka/releases/tag/3.6.1-rc0 >> > > >> > >> > > > >> > > >> > >> > > > PR for updating docs: >> > > >> > >> > > > https://github.com/apache/kafka-site/pull/568 >> > > >> > >> > > > >> > > >> > >> > > > * Documentation: >> > > >> > >> > > > https://kafka.apache.org/36/documentation.html >> > > >> > >> > > > >> > > >> > >> > > > * Protocol: >> > > >> > >> > > > https://kafka.apache.org/36/protocol.html >> > > >> > >> > > > >> > > >> > >> > > > * Successful Jenkins builds for the 3.6 branch: >> > > >> > >> > > > Unit/integration tests: We still have a lot of flaky >> tests >> > in >> > > >> the >> > > >> > >> 3.6 >> > > >> > >> > > > branch. Looking at the last few 3.6 builds in >> > > >> > >> > > > >> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ >> > it >> > > >> seems >> > > >> > >> all >> > > >> > >> > > > tests passed at least once apart from >> > > >> > >> > > > ClusterConnectionStatesTest.testSingleIP(). There's >> > > >> > >> > > > https://issues.apache.org/jira/browse/KAFKA-15762 to >> fix >> > > that >> > > >> test. >> > > >> > >> > > > System tests: Still running I'll post an update once >> they >> > > >> complete. >> > > >> > >> > > > >> > > >> > >> > > > Thanks, >> > > >> > >> > > > Mickael >> > > >> > >> > > > >> > > >> > >> > >> > > >> > >> >> > > >> > > >> > > >> > > >> > > >> > > -- >> > > >> > > [image: Aiven] <https://www.aiven.io> >> > > >> > > >> > > >> > > *Josep Prat* >> > > >> > > Open Source Engineering Director, *Aiven* >> > > >> > > josep.p...@aiven.io | +491715557497 >> > > >> > > aiven.io <https://www.aiven.io> | >> > > >> > > <https://www.facebook.com/aivencloud> >> > > >> > > <https://www.linkedin.com/company/aiven/> < >> > > >> https://twitter.com/aiven_io> >> > > >> > > *Aiven Deutschland GmbH* >> > > >> > > Alexanderufer 3-7, 10117 Berlin >> > > >> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> > > >> > > Amtsgericht Charlottenburg, HRB 209739 B >> > > >> > > >> > > >> > >> > > >> > >> > > >> > -- >> > > >> > [image: Aiven] <https://www.aiven.io> >> > > >> > >> > > >> > *Josep Prat* >> > > >> > Open Source Engineering Director, *Aiven* >> > > >> > josep.p...@aiven.io | +491715557497 >> > > >> > aiven.io <https://www.aiven.io> | < >> > > >> https://www.facebook.com/aivencloud> >> > > >> > <https://www.linkedin.com/company/aiven/> < >> > > >> https://twitter.com/aiven_io> >> > > >> > *Aiven Deutschland GmbH* >> > > >> > Alexanderufer 3-7, 10117 Berlin >> > > >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> > > >> > Amtsgericht Charlottenburg, HRB 209739 B >> > > >> >> > > > >> > > >> > >> > > > -- > -David > -- -David
