[jira] [Created] (KAFKA-15011) Single step upgrade of Kafka

[Swathi Mocharla (Jira)]

Swathi Mocharla created KAFKA-15011:
---

 Summary: Single step upgrade of Kafka
 Key: KAFKA-15011
 URL: https://issues.apache.org/jira/browse/KAFKA-15011
 Project: Kafka
  Issue Type: Wish
  Components: core
Affects Versions: 3.4.0
 Environment: K8S
Reporter: Swathi Mocharla


Many customers seek a single step upgrade, while maintaining service 
reachability. Currently, with the two step upgrade, on production systems where 
the data is almost always flowing, the checks mentioned, the TAT for the 
upgrade and the number of steps involved seems large.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #135

[Apache Jenkins Server]

See 

Re: [VOTE] 3.4.1 RC0

[Luke Chen]

Hi Matthias,

Yes, I agree we should get this hotfix into 3.4.1.
I've backported into the 3.4 branch.
I'll create a new RC for 3.4.1.

Thanks.
Luke

On Mon, May 22, 2023 at 5:13 AM Matthias J. Sax  wrote:

> Hi Luke,
>
> RC0 for 3.4.1 includes a fix for
> https://issues.apache.org/jira/browse/KAFKA-14862. We recently
> discovered that tge fix itself introduces a regression. We have already
> a PR to fix-forward the regression:
> https://github.com/apache/kafka/pull/13734
>
> I think we should get the open PR merged, and back part not only to 3.5,
> but also to 3.4.1, and get a new RC for 3.4.1.
>
> Thoughts?
>
>
> -Matthias
>
>
> On 5/19/23 6:12 AM, Josep Prat wrote:
> > Hi Luke,
> > This gets a +1 from my end. I believe non-binding because if I understand
> > it correctly, binding votes for releases are only issued by PMCs (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
> > ).
> >
> > I did the following validations:
> > - Verified signatures and checksums for all the generated artifacts
> > - Built from source with Java 11 and Scala 2.13.10
> > - Run unit tests
> > - Run integration tests
> > - Run the quickstart with Zookeeper and KRaft
> >
> > Best,
> >
> > On Wed, May 17, 2023 at 2:11 PM Josep Prat  wrote:
> >
> >> Hi Luke,
> >>
> >> I ran the tests from the source package you created and I didn't get any
> >> of the test failures you had on your CI build. I got other flaky tests
> >> though, that after being run in isolation ran successfully. I'll try to
> run
> >> signature validation, and some further testing later today or later this
> >> week.
> >>
> >> Best,
> >>
> >> On Wed, May 17, 2023 at 12:43 PM Federico Valeri 
> >> wrote:
> >>
> >>> Hi Luke, thanks for running the release.
> >>>
> >>> Looks like the Maven artifacts are not in staging:
> >>>
> >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-clients/3.4.1/
> >>>
> >>> Documentation still has 3.4.0, instead of 3.4.1 (not sure if this will
> >>> be aligned later):
> >>> https://kafka.apache.org/34/documentation.html#producerapi
> >>>
> >>> Br
> >>> Fede
> >>>
> >>>
> >>> On Wed, May 17, 2023 at 5:24 AM Luke Chen  wrote:
> 
>  Hello Kafka users, developers and client-developers,
> 
>  This is the first candidate for release of Apache Kafka 3.4.1.
> 
>  This is a bugfix release with several fixes since the release of
> 3.4.0.
> >>> A
>  few of the major issues include:
>  - core
>  KAFKA-14644 
> Process
>  should stop after failure in raft IO thread
>  KAFKA-14946  KRaft
>  controller node shutting down while renouncing leadership
>  KAFKA-14887  ZK
> >>> session
>  timeout can cause broker to shutdown
>  - client
>  KAFKA-14639  Kafka
>  CooperativeStickyAssignor revokes/assigns partition in one rebalance
> >>> cycle
>  - connect
>  KAFKA-12558  MM2
> >>> may not
>  sync partition offsets correctly
>  KAFKA-14666  MM2
> >>> should
>  translate consumer group offsets behind replication flow
>  - stream
>  KAFKA-14172  bug:
> >>> State
>  stores lose state when tasks are reassigned under EOS
> 
>  Release notes for the 3.4.1 release:
>  https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html
> 
>  *** Please download, test and vote by May 24, 2023
>  Kafka's KEYS file containing PGP keys we use to sign the release:
>  https://kafka.apache.org/KEYS
> 
>  * Release artifacts to be voted upon (source and binary):
>  https://home.apache.org/~showuon/kafka-3.4.1-rc0/
> 
>  * Maven artifacts to be voted upon:
> 
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> 
>  * Javadoc:
>  https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/
> 
>  * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
>  https://github.com/apache/kafka/releases/tag/3.4.1-rc0
> 
>  * Documentation:
>  https://kafka.apache.org/34/documentation.html
> 
>  * Protocol:
>  https://kafka.apache.org/34/protocol.html
> 
>  The most recent build has had test failures. These all appear to be
> due
> >>> to
>  flakiness, but it would be nice if someone more familiar with the
> failed
>  tests could confirm this. I may update this thread with passing build
> >>> links
>  if I can get one, or start a new release vote thread if test failures
> >>> must
>  be addressed beyond re-running builds until they pass.
> 
>  Unit/integration tests:
>  https://ci-builds.apache.org/job/

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1857

[Apache Jenkins Server]

See 

Re: [VOTE] 3.4.1 RC0

[Matthias J. Sax]

Hi Luke,

RC0 for 3.4.1 includes a fix for 
https://issues.apache.org/jira/browse/KAFKA-14862. We recently 
discovered that tge fix itself introduces a regression. We have already 
a PR to fix-forward the regression: 
https://github.com/apache/kafka/pull/13734


I think we should get the open PR merged, and back part not only to 3.5, 
but also to 3.4.1, and get a new RC for 3.4.1.


Thoughts?


-Matthias


On 5/19/23 6:12 AM, Josep Prat wrote:

Hi Luke,
This gets a +1 from my end. I believe non-binding because if I understand
it correctly, binding votes for releases are only issued by PMCs (
https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
).

I did the following validations:
- Verified signatures and checksums for all the generated artifacts
- Built from source with Java 11 and Scala 2.13.10
- Run unit tests
- Run integration tests
- Run the quickstart with Zookeeper and KRaft

Best,

On Wed, May 17, 2023 at 2:11 PM Josep Prat  wrote:


Hi Luke,

I ran the tests from the source package you created and I didn't get any
of the test failures you had on your CI build. I got other flaky tests
though, that after being run in isolation ran successfully. I'll try to run
signature validation, and some further testing later today or later this
week.

Best,

On Wed, May 17, 2023 at 12:43 PM Federico Valeri 
wrote:


Hi Luke, thanks for running the release.

Looks like the Maven artifacts are not in staging:

https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-clients/3.4.1/

Documentation still has 3.4.0, instead of 3.4.1 (not sure if this will
be aligned later):
https://kafka.apache.org/34/documentation.html#producerapi

Br
Fede


On Wed, May 17, 2023 at 5:24 AM Luke Chen  wrote:


Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.4.1.

This is a bugfix release with several fixes since the release of 3.4.0.

A

few of the major issues include:
- core
KAFKA-14644  Process
should stop after failure in raft IO thread
KAFKA-14946  KRaft
controller node shutting down while renouncing leadership
KAFKA-14887  ZK

session

timeout can cause broker to shutdown
- client
KAFKA-14639  Kafka
CooperativeStickyAssignor revokes/assigns partition in one rebalance

cycle

- connect
KAFKA-12558  MM2

may not

sync partition offsets correctly
KAFKA-14666  MM2

should

translate consumer group offsets behind replication flow
- stream
KAFKA-14172  bug:

State

stores lose state when tasks are reassigned under EOS

Release notes for the 3.4.1 release:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by May 24, 2023
Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.4.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/

* Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
https://github.com/apache/kafka/releases/tag/3.4.1-rc0

* Documentation:
https://kafka.apache.org/34/documentation.html

* Protocol:
https://kafka.apache.org/34/protocol.html

The most recent build has had test failures. These all appear to be due

to

flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build

links

if I can get one, or start a new release vote thread if test failures

must

be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/133/

System tests:
Will update the results later

Thank you.
Luke





--
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |

   
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

[ShunKang Lin]

Hi all,

Thanks for voting!
Just wanted to bump this thread and see if we could get a few more votes.
Currently we have +2 non-binding and +1 binding.
Hoping we can get this approved, reviewed, and merged in time for 3.6.0.

Best,
ShunKang

Kirk True  于2023年5月11日周四 22:54写道：

> +1 (non-binding)
>
> Thanks ShunKang!
>
> On Wed, May 10, 2023, at 7:26 PM, Luke Chen wrote:
> > +1(binding) from me.
> > Thanks for the improvement!
> >
> > Luke
> >
> > On Sun, May 7, 2023 at 6:34 PM Divij Vaidya 
> wrote:
> >
> > > Vote +1 (non binding)
> > >
> > > I think that this is a nice improvement as it prevents an unnecessary
> data
> > > copy for users who are using ByteBuffer serialization on the producer.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Sun, May 7, 2023 at 9:24 AM ShunKang Lin  >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I'd like to open the vote for KIP-872, which proposes to add
> > > > Serializer#serializeToByteBuffer() to reduce memory copying.
> > > >
> > > > The proposal is here:
> > > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> > > >
> > > > The pull request is here:
> > > > https://github.com/apache/kafka/pull/12685
> > > >
> > > > Thanks to all who reviewed the proposal, and thanks in advance for
> taking
> > > > the time to vote!
> > > >
> > > > Best,
> > > > ShunKang
> > > >
> > >
> >
>

Re: [VOTE] KIP-927: Improve the kafka-metadata-quorum output

[Federico Valeri]

The KIP is accepted with 3 binding votes (Luke, Ziming, John) and 2
non binding votes (Divij, Kamal).

Thanks everyone!

Cheers
Fede

On Sat, May 20, 2023 at 10:58 PM John Roesler  wrote:
>
> I’m +1 (binding)
>
> Thanks, Federico!
>
> It looks like a nice improvement to me.
>
> -John
>
>
> On Sat, May 20, 2023, at 09:16, Kamal Chandraprakash wrote:
> > +1 (non binding)
> >
> > On Wed, May 17, 2023 at 3:22 AM Divij Vaidya 
> > wrote:
> >
> >> +1 (non binding)
> >>
> >> Divij Vaidya
> >>
> >>
> >>
> >> On Tue, May 16, 2023 at 4:35 AM ziming deng 
> >> wrote:
> >>
> >> > Thanks for this improvement, +1 from me(binging)
> >> >
> >> > —
> >> > Best,
> >> > Ziming
> >> >
> >> > > On May 16, 2023, at 00:43, Federico Valeri 
> >> wrote:
> >> > >
> >> > > Hi all,
> >> > >
> >> > > I'd like to start a vote on KIP-927: Improve the kafka-metadata-quorum
> >> > output.
> >> > >
> >> > >
> >> >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> >> > >
> >> > > Discussion thread:
> >> > > https://lists.apache.org/thread/pph59hxvz5jkk709x53p44xrpdqwv8qc
> >> > >
> >> > > Thanks
> >> > > Fede
> >> >
> >> >
> >>