Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1875

2023-05-27 Thread Apache Jenkins Server 
See

Re: [VOTE] 3.5.0 RC0

2023-05-27 Thread David Arthur 
Mickael, after looking more closely, I definitely think KAFKA-15010 is a
blocker. It creates the case where the controller can totally miss a
metadata update and not write it back to ZK. Since things like dynamic
configs and ACLs are only read from ZK by the ZK brokers, we could have
significant problems while the brokers are being migrated (when some are
KRaft and some are ZK). E.g., ZK brokers could be totally unaware of an ACL
change while the KRaft brokers have it. I have a fix ready here
https://github.com/apache/kafka/pull/13758. I think we can get it committed
soon.

Another blocker is KAFKA-15004 which was just merged to trunk. This is
another dual-write bug where new topic/broker configs will not be written
back to ZK by the controller.

The fix for KAFKA-15010 has a few dependencies on fixes we made this past
week, so we'll need to cherry-pick a few commits. The changes are totally
contained within the migration area of code, so I think the risk in
including them is fairly low.

-David

On Thu, May 25, 2023 at 2:15 PM Greg Harris 
wrote:

> Hey all,
>
> A contributor just pointed out a small but noticeable flaw in the
> implementation of KIP-581
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> which is planned for this release.
> Impact: the feature works for root values in a record, but does not
> work for any fields within structs. Fields within structs will
> continue to have their previous, backwards-compatible behavior.
> The contributor has submitted a bug-fix PR which reports the problem
> and does not yet have a merge-able solution, but they are actively
> responding and interested in having this fixed:
> https://github.com/apache/kafka/pull/13748
> The overall fix should be a one-liner + some unit tests. While this is
> not a regression, it does make the feature largely useless, as the
> majority of use-cases will be for struct fields.
>
> Thanks!
> Greg Harris
>
> On Wed, May 24, 2023 at 7:05 PM Ismael Juma  wrote:
> >
> > I agree the migration should be functional - it wasn't obvious if the
> > migration issues are edge cases or not. If they are edge cases, I think
> > 3.5.1 would be fine given the preview status.
> >
> > I understand that a new RC is needed, but that doesn't mean we should let
> > everything in. Each change carries some risk. And if we don't agree on
> the
> > bar for the migration work, we may be having the same discussion next
> week.
> > :)
> >
> > Ismael
> >
> > On Wed, May 24, 2023, 12:00 PM Josep Prat 
> > wrote:
> >
> > > Hi there,
> > > Is the plan described in KIP-833[1] still valid? In there it states
> that
> > > 3.5.0 should aim at deprecation of Zookeeper, so conceptually, the
> path to
> > > migrate to Kraft should be somewhat functional (in my opinion). If we
> don't
> > > want to deprecate Zookeeper in 3.5.0, then I share Ismael's opinion
> that
> > > these could be fixed in subsequent patches of 3.5.x. Just my 5cts.
> > >
> > > [1]:
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833:+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.5
> > > Best,
> > >
> > > On Wed, May 24, 2023 at 8:51 PM Ismael Juma  wrote:
> > >
> > > > Are all these blockers? For example, zk to kraft migration are is
> still
> > > in
> > > > preview - can we fix some of these in 3.5.1?
> > > >
> > > > Ismael
> > > >
> > > > On Wed, May 24, 2023, 10:22 AM Colin McCabe 
> wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Thanks for putting together this RC. Unfortunately, we've
> identified
> > > > > several blocker issues in this release candidate.
> > > > >
> > > > > KAFKA-15009: New ACLs are not written to ZK during migration
> > > > > KAFKA-15007: MV is not set correctly in the MetadataPropagator in
> > > > > migration.
> > > > > KAFKA-15004: Topic config changes are not synced during zk to kraft
> > > > > migration (dual-write)
> > > > > KAFKA-15003: TopicIdReplicaAssignment is not updated in migration
> > > > > (dual-write) when partitions are changed for topic
> > > > > KAFKA-14996: The KRaft controller should properly handle overly
> large
> > > > user
> > > > > operations
> > > > >
> > > > > We are working on PRs for these issues and will get them in soon,
> we
> > > > think!
> > > > >
> > > > > So unfortunately I have to leave a -1 here for RC0. Let's aim for
> > > another
> > > > > RC next week.
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Wed, May 24, 2023, at 07:05, Mickael Maison wrote:
> > > > > > Hi David,
> > > > > >
> > > > > > We're already quite a bit behind schedule. If you think these
> fixes
> > > > > > are really important and can be ready in the next couple of
> days, I'm
> > > > > > open to backport them and build another release candidate. Let me
> > > know
> > > > > > once you've investigated the severity of KAFKA-15010.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > > On Tue,

Re: [VOTE] 3.4.1 RC3

2023-05-27 Thread Jakub Scholz 
+1 (non-binding) ... I used the staged binaries and Maven artifacts to run
my tests and all seems to work fine.

Thanks for running the release.

Jakub

On Fri, May 26, 2023 at 9:34 AM Luke Chen  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the 4th candidate for release of Apache Kafka 3.4.1.
>
> This is a bugfix release with several fixes since the release of 3.4.0. A
> few of the major issues include:
> - core
> KAFKA-14644  Process
> should stop after failure in raft IO thread
> KAFKA-14946  KRaft
> controller node shutting down while renouncing leadership
> KAFKA-14887  ZK session
> timeout can cause broker to shutdown
> - client
> KAFKA-14639  Kafka
> CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
> - connect
> KAFKA-12558  MM2 may
> not
> sync partition offsets correctly
> KAFKA-14666  MM2 should
> translate consumer group offsets behind replication flow
> - stream
> KAFKA-14172  bug: State
> stores lose state when tasks are reassigned under EOS
>
>
> Release notes for the 3.4.1 release:
> https://home.apache.org/~showuon/kafka-3.4.1-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Jun 2, 2023
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~showuon/kafka-3.4.1-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~showuon/kafka-3.4.1-rc3/javadoc/
>
> * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> https://github.com/apache/kafka/releases/tag/3.4.1-rc3
>
> * Documentation: (will be updated after released)
> https://kafka.apache.org/34/documentation.html
>
> * Protocol: (will be updated after released)
> https://kafka.apache.org/34/protocol.html
>
> The most recent build has had test failures. These all appear to be due to
> flakiness, but it would be nice if someone more familiar with the failed
> tests could confirm this. I may update this thread with passing build links
> if I can get one, or start a new release vote thread if test failures must
> be addressed beyond re-running builds until they pass.
>
> Unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/141/
>
> System tests:
> Will update the results later
>
> Thank you
> Luke
>

Re: [VOTE] 3.4.1 RC3

2023-05-27 Thread Federico Valeri 
Hi Luke,

- Checked signature, checksum, libs, and license file
- Build from source with Java 17 and Scala 2.13
- Ran full unit and integration test suites
- Ran Java app with staging Maven artifacts against ZK and KRaft
multi-node clusters

+1 (non binding)

Thanks
Fede


On Fri, May 26, 2023 at 1:42 PM Josep Prat  wrote:
>
> Hi Luke,
>
> Thanks again for running this!
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
> -- For KRaft, I looked at the process running Kafka and confirmed that the
> spamming log message is not present anymore ("Generated a metadata delta
> between...")
> - Checked that the diff in LICENSE-binary is identical than previous RC
> -- For each difference of the previous one I checked the binary on the
> scala 2.13 package had the version specified in the file
>
> This gets a +1 from me.
>
> Best
>
> On Fri, May 26, 2023 at 9:33 AM Luke Chen  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the 4th candidate for release of Apache Kafka 3.4.1.
> >
> > This is a bugfix release with several fixes since the release of 3.4.0. A
> > few of the major issues include:
> > - core
> > KAFKA-14644  Process
> > should stop after failure in raft IO thread
> > KAFKA-14946  KRaft
> > controller node shutting down while renouncing leadership
> > KAFKA-14887  ZK session
> > timeout can cause broker to shutdown
> > - client
> > KAFKA-14639  Kafka
> > CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
> > - connect
> > KAFKA-12558  MM2 may
> > not
> > sync partition offsets correctly
> > KAFKA-14666  MM2 should
> > translate consumer group offsets behind replication flow
> > - stream
> > KAFKA-14172  bug: State
> > stores lose state when tasks are reassigned under EOS
> >
> >
> > Release notes for the 3.4.1 release:
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Jun 2, 2023
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~showuon/kafka-3.4.1-rc3/javadoc/
> >
> > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> > https://github.com/apache/kafka/releases/tag/3.4.1-rc3
> >
> > * Documentation: (will be updated after released)
> > https://kafka.apache.org/34/documentation.html
> >
> > * Protocol: (will be updated after released)
> > https://kafka.apache.org/34/protocol.html
> >
> > The most recent build has had test failures. These all appear to be due to
> > flakiness, but it would be nice if someone more familiar with the failed
> > tests could confirm this. I may update this thread with passing build links
> > if I can get one, or start a new release vote thread if test failures must
> > be addressed beyond re-running builds until they pass.
> >
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/141/
> >
> > System tests:
> > Will update the results later
> >
> > Thank you
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B