Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Thanks David, I would be perfectly fine having: - Level 2: Experimental (draft and unstable might be too scary) - Level 3: Preview - Level 4: Production Ready PS: I know this is not how KIPs are usually discussed, but names are really special and this is something I feel the community needs to generally agree with or at least be comfortable with. Best, On Wed, Oct 2, 2024 at 4:53 PM David Arthur wrote: > I'm not sure why, but for some reason I cannot keep Preview and Early > Access straight. I always mix them up. > > "Early Access" -- you are getting access to something early > > "Preview" -- you are viewing something in advance > > To me, semantically, these terms are just too similar. I would prefer the > Level 2 term to indicate that the thing is not ready yet. Things like > "draft", "experimental", and "unstable" come to mind. > > In fact, a quick google search reveals that the gaming industry uses "Early > Access" and "Preview" interchangeably :) > > -David > > On Wed, Oct 2, 2024 at 8:09 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > > > Hi Josep, > > Thanks for applying the first coat of paint 🙂 > > > > Personally, I think the names you propose are good choices. We have > > precedent already and the sequence is pretty clear > > based on the names themselves. > > > > Thanks, > > Andrew > > > > > > From: Josep Prat > > Sent: 02 October 2024 09:10 > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-1081: Graduation Steps for Features > > > > Hi all, > > > > I think the discussion regarding the steps has winded down and we've > > reached a good enough consensus. With that out of the way, we can now > start > > to paint our bike shed, a.k.a. choose the names for each phase. > > As we mentioned, step number 1 is virtual and doesn't really need a name. > > Step 2's name is: "Early Access" > > Step 3's name is: "Preview" > > Step 4's name is: "Production Ready" > > > > These names are aligned with what we've been using up until now. Let's > now > > discuss the suitability of these names. > > > > Thanks! > > > > > > On Mon, Sep 16, 2024 at 5:34 PM Josep Prat wrote: > > > > > Hi all! > > > I did come around and wrote the feedback pending in the KIP itself. > > Please > > > take another read! I added a section attempting to define the term > > > "usable". Also I applied the feedback. > > > > > > Thanks! > > > > > > On Wed, Sep 4, 2024 at 1:34 AM Colin McCabe > wrote: > > > > > >> On Fri, Aug 30, 2024, at 16:40, Matthias J. Sax wrote: > > >> > Great discussion. Also wanted to follow up with a few things. > > >> > > > >> > > > >> > I believe the term "usable" is not well defined leading to > > confusion... > > >> > I agree with Viktor that "usable" in the context of level 2 should > > just > > >> > mean: I can enable the feature and it does something... not more, > not > > >> > less. It might crash; it might compute the wrong result for some > > cases, > > >> > it might have terrible performance, etc... but: I can kick the > tires. > > >> > > > >> > > >> Yeah, it would be good to clarify this, to avoid "usable" becoming too > > >> expansive. > > >> > > >> > > > >> > About the proposed "checklist" from Viktor: I think we should not > have > > >> > anything about testing in it -- that test are required goes w/o > saying > > >> > and is already covered in the KIP document itself. To me, it's the > KIP > > >> > author's / community's responsivity to decide on a case-by-case > basis > > >> > when a feature is considered ready for the next level, and what > > testing > > >> > is sufficient for each level. > > >> > > > >> > > > >> > Similar for docs, even if I agree that docs should be more or less > > >> > complete at level 3. Otherwise, users will have a hard time to > really > > >> > try the feature and thus kinda defeats the purpose of level 3. > > >> > > >> +1 > > >> > > >> > > > >> > > > >
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi all, I think the discussion regarding the steps has winded down and we've reached a good enough consensus. With that out of the way, we can now start to paint our bike shed, a.k.a. choose the names for each phase. As we mentioned, step number 1 is virtual and doesn't really need a name. Step 2's name is: "Early Access" Step 3's name is: "Preview" Step 4's name is: "Production Ready" These names are aligned with what we've been using up until now. Let's now discuss the suitability of these names. Thanks! On Mon, Sep 16, 2024 at 5:34 PM Josep Prat wrote: > Hi all! > I did come around and wrote the feedback pending in the KIP itself. Please > take another read! I added a section attempting to define the term > "usable". Also I applied the feedback. > > Thanks! > > On Wed, Sep 4, 2024 at 1:34 AM Colin McCabe wrote: > >> On Fri, Aug 30, 2024, at 16:40, Matthias J. Sax wrote: >> > Great discussion. Also wanted to follow up with a few things. >> > >> > >> > I believe the term "usable" is not well defined leading to confusion... >> > I agree with Viktor that "usable" in the context of level 2 should just >> > mean: I can enable the feature and it does something... not more, not >> > less. It might crash; it might compute the wrong result for some cases, >> > it might have terrible performance, etc... but: I can kick the tires. >> > >> >> Yeah, it would be good to clarify this, to avoid "usable" becoming too >> expansive. >> >> > >> > About the proposed "checklist" from Viktor: I think we should not have >> > anything about testing in it -- that test are required goes w/o saying >> > and is already covered in the KIP document itself. To me, it's the KIP >> > author's / community's responsivity to decide on a case-by-case basis >> > when a feature is considered ready for the next level, and what testing >> > is sufficient for each level. >> > >> > >> > Similar for docs, even if I agree that docs should be more or less >> > complete at level 3. Otherwise, users will have a hard time to really >> > try the feature and thus kinda defeats the purpose of level 3. >> >> +1 >> >> > >> > >> > Last: @Colin, yes we eventually need to pick names for the levels. But >> I >> > believe it's actually the right way to agree on the "what" first, and >> > just say "level X" for now, and only after we agree on the levels, we >> > enter the ring for the fun part: picking names. This should be the very >> > last step :popcorn: >> > >> >> Maybe this is just me, but using numbers instead of names makes it quite >> hard for me to get a handle on the discussion. I have opinions on what >> alpha / beta / production-ready mean. I don't have opinions on what "Level >> 4" means or what "manuscript" means. So I feel like we will go around and >> around until we can give a name to what we're talking about. >> >> best, >> Colin >> >> >> > >> > >> > -Matthias >> > >> > >> > >> > >> > On 8/30/24 8:57 AM, Colin McCabe wrote: >> >> On Mon, Aug 26, 2024, at 10:51, Josep Prat wrote: >> >>> Hi Colin, >> >>> >> >>> Names are in the KIP. Level 1 to 4 are never meant to be used outside >> of >> >>> this discussion. It's my, apparently successful, attempt to focus on >> what >> >>> the levels mean instead of their names: >> >>> >> >>>> Names >> >>> >> >>> "In Development" >> >>> "Early Access" >> >>> "Preview" >> >>> "Production Ready" >> >> >> >> Hi Josep, >> >> >> >> Thanks for the clarification. I think we should remove references to >> level 1, 2, 3, 4, etc. if that is not the terminology that we want to use. >> One of the big purposes of a KIP is to standardize on terminology. That's >> not achieved if different parts of the KIP use different names for the same >> things. >> >> >> >>> >> >>> Alternatively, if we want to be a bit more playful we could go with a >> theme >> >>> borrowed from the book industry (as an homage to Franz Kafka): >> >>> >> >>>
Re: [ANNOUNCE] New committer: Kamal Chandraprakash
Congrats Kamal! On Mon, Sep 30, 2024 at 2:46 PM wrote: > Congrats Kamal! > > - > Gaurav > > > On 30 Sep 2024, at 18:10, Mickael Maison > wrote: > > > > Congratulations Kamal! > > > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen wrote: > >> > >> Hi all, > >> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer, > Kamal > >> Chandraprakash. > >> > >> Kamal has been a Kafka contributor since May 2017. He has made > significant > >> contributions to the tiered storage feature (KIP-405). He authored > KIP-1018 > >> and KIP-1075 which improved tiered storage operation. He also > contributed > >> to discussing and reviewing many KIPs. > >> > >> Congratulations, Kamal! > >> > >> Thanks, > >> Luke (on behalf of the Apache Kafka PMC) > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.1 release
Hi there, I'll attempt to cut the first RC for 3.8.1 this Wednesday. If you have any bug fix that you'd like to backport to the 3.8 branch and you'd need more time, please let me know. Best, On Tue, Sep 24, 2024 at 1:15 PM Josep Prat wrote: > Hi folks! > As promised, here you have the release plan for 3.8.1: > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.1 > > I'm aiming to have a release candidate by the first week of October unless > somebody else finds a blocker before then. > > Best, > > On Fri, Sep 20, 2024 at 11:57 AM Chia-Ping Tsai > wrote: > >> +1 >> >> Luke Chen 於 2024年9月20日 週五 下午4:22寫道： >> >> > Thanks Josep! >> > +1 >> > >> > Luke >> > >> > On Fri, Sep 20, 2024 at 3:36 PM Josep Prat > > >> > wrote: >> > >> > > Hey folks, >> > > >> > > I'd like to volunteer to be the release manager for a bug fix release >> of >> > > the 3.8 series. This will be the first bug fix release and will be >> > version >> > > 3.8.1. >> > > >> > > If no one has any objections, I will send out a release plan on >> > 2024/09/23 >> > > that includes a list of all of the fixes we are targeting for 3.8.1 >> along >> > > with a timeline (aiming probably for a release happening at the >> beginning >> > > of October). >> > > >> > > Thanks! >> > > >> > > -- >> > > [image: Aiven] <https://www.aiven.io> >> > > >> > > *Josep Prat* >> > > Open Source Engineering Director, *Aiven* >> > > josep.p...@aiven.io | +491715557497 >> > > aiven.io <https://www.aiven.io> | < >> > https://www.facebook.com/aivencloud >> > > > >> > > <https://www.linkedin.com/company/aiven/> < >> > > https://twitter.com/aiven_io> >> > > *Aiven Deutschland GmbH* >> > > Alexanderufer 3-7, 10117 Berlin >> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, >> > > Anna Richardson, Kenneth Chen >> > > Amtsgericht Charlottenburg, HRB 209739 B >> > > >> > >> > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > Anna Richardson, Kenneth Chen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: So long, Jenkins 👋
I see you have the python script under "committer-tools", I guess I might need to get used to call that script instead of going to the "pulls" page. Best, On Thu, Sep 26, 2024 at 3:36 PM Josep Prat wrote: > Hi David, > I think we need a way to flag in the PR list ( > github.com/apache/kafka/pulls) the ones that are waiting for a committer > to approve the workflows. As an example: > [image: image.png] > This PR has a green checkmark where the check status usually goes. But if > one navigates to the PR in question, one can see that the CI tasks didn't > start and wait for a committer to approve and run. > [image: image.png] > Do you have another way to identify these PRs? Or should we maybe work on > auto labelling PRs from non-committers (the ones that would wait for CI to > run). > > On Thu, Sep 26, 2024 at 11:00 AM Josep Prat wrote: > >> That's what I feared >> >> On Thu, Sep 26, 2024 at 10:31 AM Chia-Ping Tsai >> wrote: >> >>> hi Josep >>> >>> > Do you see any potential impact if we backport the change to those? >>> >>> In my opinion, the main concern is that non-trunk PRs can't effectively >>> leverage the cache, meaning they require more time and resources to run >>> CI. >>> Additionally, github-ci is triggered by trunk branch only, and we have >>> not >>> tested it on non-trunk branch yet. Given that 3.9.0 and 3.8.1 releases >>> are >>> processing, we could continue using Jenkins CI to avoid the additional >>> overhead of backporting. >>> >>> By the way, we'll eventually need to backport GitHub CI to the non-trunk >>> branches once the 4.1 branch is created. >>> >>> Best, >>> Chia-Ping >>> >>> >>> >>> Chia-Ping Tsai 於 2024年9月26日 週四 下午4:15寫道： >>> >>> > Thanks to David for providing us with an improved CI! >>> > >>> > Cheers, >>> > Chia-Ping >>> > >>> > David Arthur 於 2024年9月26日 週四 上午8:51寫道： >>> > >>> >> Today, we disabled the Jenkins build on trunk. With this change, we >>> should >>> >> now be expecting all green status checks on PRs before merging. Of >>> course, >>> >> flaky tests still exist, but generally speaking we should have green >>> >> builds >>> >> (see KIP-1090 for some plans on flaky tests). >>> >> >>> >> Any committer or "collaborator" (as defined in .asf.yaml) is able to >>> >> manually re-run a GitHub Action via the UI. >>> >> >>> >> For non-committers, someone must approve the workflow. There is a >>> >> "approve-workflows.py" script in committer-tools to help with this. >>> I'm >>> >> still investigating options to improve this. >>> >> >>> >> We will keep the Jenkins build enabled for 3.9 and other release >>> branches. >>> >> >>> >> Cheers, >>> >> David A >>> >> >>> > >>> >> >> >> -- >> [image: Aiven] <https://www.aiven.io> >> >> *Josep Prat* >> Open Source Engineering Director, *Aiven* >> josep.p...@aiven.io | +491715557497 >> aiven.io <https://www.aiven.io> | >> <https://www.facebook.com/aivencloud> >> <https://www.linkedin.com/company/aiven/> >> <https://twitter.com/aiven_io> >> *Aiven Deutschland GmbH* >> Alexanderufer 3-7, 10117 Berlin >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, >> Anna Richardson, Kenneth Chen >> Amtsgericht Charlottenburg, HRB 209739 B >> > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > Anna Richardson, Kenneth Chen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: So long, Jenkins 👋
Hi David, I think we need a way to flag in the PR list (github.com/apache/kafka/pulls) the ones that are waiting for a committer to approve the workflows. As an example: [image: image.png] This PR has a green checkmark where the check status usually goes. But if one navigates to the PR in question, one can see that the CI tasks didn't start and wait for a committer to approve and run. [image: image.png] Do you have another way to identify these PRs? Or should we maybe work on auto labelling PRs from non-committers (the ones that would wait for CI to run). On Thu, Sep 26, 2024 at 11:00 AM Josep Prat wrote: > That's what I feared > > On Thu, Sep 26, 2024 at 10:31 AM Chia-Ping Tsai > wrote: > >> hi Josep >> >> > Do you see any potential impact if we backport the change to those? >> >> In my opinion, the main concern is that non-trunk PRs can't effectively >> leverage the cache, meaning they require more time and resources to run >> CI. >> Additionally, github-ci is triggered by trunk branch only, and we have not >> tested it on non-trunk branch yet. Given that 3.9.0 and 3.8.1 releases are >> processing, we could continue using Jenkins CI to avoid the additional >> overhead of backporting. >> >> By the way, we'll eventually need to backport GitHub CI to the non-trunk >> branches once the 4.1 branch is created. >> >> Best, >> Chia-Ping >> >> >> >> Chia-Ping Tsai 於 2024年9月26日 週四 下午4:15寫道： >> >> > Thanks to David for providing us with an improved CI! >> > >> > Cheers, >> > Chia-Ping >> > >> > David Arthur 於 2024年9月26日 週四 上午8:51寫道： >> > >> >> Today, we disabled the Jenkins build on trunk. With this change, we >> should >> >> now be expecting all green status checks on PRs before merging. Of >> course, >> >> flaky tests still exist, but generally speaking we should have green >> >> builds >> >> (see KIP-1090 for some plans on flaky tests). >> >> >> >> Any committer or "collaborator" (as defined in .asf.yaml) is able to >> >> manually re-run a GitHub Action via the UI. >> >> >> >> For non-committers, someone must approve the workflow. There is a >> >> "approve-workflows.py" script in committer-tools to help with this. I'm >> >> still investigating options to improve this. >> >> >> >> We will keep the Jenkins build enabled for 3.9 and other release >> branches. >> >> >> >> Cheers, >> >> David A >> >> >> > >> > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > Anna Richardson, Kenneth Chen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: So long, Jenkins 👋
That's what I feared On Thu, Sep 26, 2024 at 10:31 AM Chia-Ping Tsai wrote: > hi Josep > > > Do you see any potential impact if we backport the change to those? > > In my opinion, the main concern is that non-trunk PRs can't effectively > leverage the cache, meaning they require more time and resources to run CI. > Additionally, github-ci is triggered by trunk branch only, and we have not > tested it on non-trunk branch yet. Given that 3.9.0 and 3.8.1 releases are > processing, we could continue using Jenkins CI to avoid the additional > overhead of backporting. > > By the way, we'll eventually need to backport GitHub CI to the non-trunk > branches once the 4.1 branch is created. > > Best, > Chia-Ping > > > > Chia-Ping Tsai 於 2024年9月26日 週四 下午4:15寫道： > > > Thanks to David for providing us with an improved CI! > > > > Cheers, > > Chia-Ping > > > > David Arthur 於 2024年9月26日 週四 上午8:51寫道： > > > >> Today, we disabled the Jenkins build on trunk. With this change, we > should > >> now be expecting all green status checks on PRs before merging. Of > course, > >> flaky tests still exist, but generally speaking we should have green > >> builds > >> (see KIP-1090 for some plans on flaky tests). > >> > >> Any committer or "collaborator" (as defined in .asf.yaml) is able to > >> manually re-run a GitHub Action via the UI. > >> > >> For non-committers, someone must approve the workflow. There is a > >> "approve-workflows.py" script in committer-tools to help with this. I'm > >> still investigating options to improve this. > >> > >> We will keep the Jenkins build enabled for 3.9 and other release > branches. > >> > >> Cheers, > >> David A > >> > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: So long, Jenkins 👋
One thing that just occurred to me, David, should we backport this CI change to 3.7, 3.8 and 3.9? These are the 3 active branches at the moment (3.6 technically is still active, but will be EOL'd when 3.9 is released soon). Do you see any potential impact if we backport the change to those? Best, On Thu, Sep 26, 2024 at 8:24 AM Luke Chen wrote: > Cool! Thanks for the effort, David! > > Luke > > > On Thu, Sep 26, 2024 at 1:26 PM Josep Prat > wrote: > > > Awesome David! > > > > Best, > > -- > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Thu, Sep 26, 2024, 04:36 Kirk True wrote: > > > > > This is amazing work David! You leveled up every contributor and > > committer > > > on the project. > > > > > > Thanks > > > > > > > On Sep 25, 2024, at 6:27 PM, Chris Egerton > > > wrote: > > > > > > > > David, you're a legend. Thanks for spearheading this effort! > > > > > > > > On Wed, Sep 25, 2024, 20:51 David Arthur wrote: > > > > > > > >> Today, we disabled the Jenkins build on trunk. With this change, we > > > should > > > >> now be expecting all green status checks on PRs before merging. Of > > > course, > > > >> flaky tests still exist, but generally speaking we should have green > > > builds > > > >> (see KIP-1090 for some plans on flaky tests). > > > >> > > > >> Any committer or "collaborator" (as defined in .asf.yaml) is able to > > > >> manually re-run a GitHub Action via the UI. > > > >> > > > >> For non-committers, someone must approve the workflow. There is a > > > >> "approve-workflows.py" script in committer-tools to help with this. > > I'm > > > >> still investigating options to improve this. > > > >> > > > >> We will keep the Jenkins build enabled for 3.9 and other release > > > branches. > > > >> > > > >> Cheers, > > > >> David A > > > >> > > > > > > > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: So long, Jenkins 👋
Awesome David! Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Thu, Sep 26, 2024, 04:36 Kirk True wrote: > This is amazing work David! You leveled up every contributor and committer > on the project. > > Thanks > > > On Sep 25, 2024, at 6:27 PM, Chris Egerton > wrote: > > > > David, you're a legend. Thanks for spearheading this effort! > > > > On Wed, Sep 25, 2024, 20:51 David Arthur wrote: > > > >> Today, we disabled the Jenkins build on trunk. With this change, we > should > >> now be expecting all green status checks on PRs before merging. Of > course, > >> flaky tests still exist, but generally speaking we should have green > builds > >> (see KIP-1090 for some plans on flaky tests). > >> > >> Any committer or "collaborator" (as defined in .asf.yaml) is able to > >> manually re-run a GitHub Action via the UI. > >> > >> For non-committers, someone must approve the workflow. There is a > >> "approve-workflows.py" script in committer-tools to help with this. I'm > >> still investigating options to improve this. > >> > >> We will keep the Jenkins build enabled for 3.9 and other release > branches. > >> > >> Cheers, > >> David A > >> > >
Re: [DISCUSS] Apache Kafka 3.8.1 release
Hi folks! As promised, here you have the release plan for 3.8.1: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.1 I'm aiming to have a release candidate by the first week of October unless somebody else finds a blocker before then. Best, On Fri, Sep 20, 2024 at 11:57 AM Chia-Ping Tsai wrote: > +1 > > Luke Chen 於 2024年9月20日 週五 下午4:22寫道： > > > Thanks Josep! > > +1 > > > > Luke > > > > On Fri, Sep 20, 2024 at 3:36 PM Josep Prat > > wrote: > > > > > Hey folks, > > > > > > I'd like to volunteer to be the release manager for a bug fix release > of > > > the 3.8 series. This will be the first bug fix release and will be > > version > > > 3.8.1. > > > > > > If no one has any objections, I will send out a release plan on > > 2024/09/23 > > > that includes a list of all of the fixes we are targeting for 3.8.1 > along > > > with a timeline (aiming probably for a release happening at the > beginning > > > of October). > > > > > > Thanks! > > > > > > -- > > > [image: Aiven] <https://www.aiven.io> > > > > > > *Josep Prat* > > > Open Source Engineering Director, *Aiven* > > > josep.p...@aiven.io | +491715557497 > > > aiven.io <https://www.aiven.io> | < > > https://www.facebook.com/aivencloud > > > > > > > <https://www.linkedin.com/company/aiven/> < > > > https://twitter.com/aiven_io> > > > *Aiven Deutschland GmbH* > > > Alexanderufer 3-7, 10117 Berlin > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > > Anna Richardson, Kenneth Chen > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 4.0.0 release
Thanks David for volunteering! +1 On Mon, Sep 23, 2024 at 3:11 PM David Jacot wrote: > Hi all, > > I would like to volunteer to be the release manager driving the next > release - Apache Kafka 4.0.0. > > If there are no objections, I'll start building a release plan in the wiki > in the next couple of days. > > Best, > David > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
[DISCUSS] Apache Kafka 3.8.1 release
Hey folks, I'd like to volunteer to be the release manager for a bug fix release of the 3.8 series. This will be the first bug fix release and will be version 3.8.1. If no one has any objections, I will send out a release plan on 2024/09/23 that includes a list of all of the fixes we are targeting for 3.8.1 along with a timeline (aiming probably for a release happening at the beginning of October). Thanks! -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] 3.9.0 RC0
I can see the Stream Quickstart ones in the staging repo, but only these. When I did the 3.8 release I realized that https://repository.apache.org/#stagingRepositories had 3 different entries (repositories) every time I tried to upload to maven central (1 was kind of faulty with only togdor jar file and no metadata, and the other 2 were for StreamQuickStart jars, and all the other jars). Taking a look there now, I see only the repository for the quickstart is present. Colin, did you happen to see the other repositories as well? Best, On Wed, Sep 18, 2024 at 11:26 AM Luke Chen wrote: > Same here, no 3.9.0 artifacts in the staging repo. > > Thanks. > Luke > > > On Wed, Sep 18, 2024 at 4:49 PM Jakub Scholz wrote: > > > Hi Colin, > > > > Thanks for the RC. > > > > It looks like the Maven artifacts are missing for 3.9.0 in the staging > > repo. Are we still waiting for some staging job to complete? Or is that > > some CDN issue and I'm the only one not seeing them? Or is that some > > general problem? > > > > Thanks & Regards > > Jakub > > > > On Wed, Sep 18, 2024 at 5:07 AM Colin McCabe wrote: > > > > > Hello Kafka users, developers and client-developers, > > > > > > > > > > > > This is the first candidate for release of Apache Kafka 3.9.0. > > > > > > > > > - This is a major release, the final one in the 3.x line. (There may of > > > course be other minor releases in this line, such as 3.9.1.) > > > - Tiered storage will be considered production-ready in this release. > > > - This will be the final major release to feature the deprecated > > ZooKeeper > > > mode. > > > > > > This release includes the following KIPs: > > > - KIP-853: Support dynamically changing KRaft controller membership > > > - KIP-1057: Add remote log metadata flag to the dump log tool > > > - KIP-1049: Add config log.summary.interval.ms to Kafka Streams > > > - KIP-1040: Improve handling of nullable values in InsertField, > > > ExtractField, and other transformations > > > - KIP-1031: Control offset translation in MirrorSourceConnector > > > - KIP-1033: Add Kafka Streams exception handler for exceptions > occurring > > > during processing > > > - KIP-1017: Health check endpoint for Kafka Connect > > > - KIP-1025: Optionally URL-encode clientID and clientSecret in > > > authorization header > > > - KIP-1005: Expose EarliestLocalOffset and TieredOffset > > > - KIP-950: Tiered Storage Disablement > > > - KIP-956: Tiered Storage Quotas > > > > > > Release notes for the 3.9.0 release: > > > https://home.apache.org/~cmccabe/kafka-3.9.0-rc0/RELEASE_NOTES.html > > > > > > *** Please download, test and vote by September 23, 2024. > > > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > > https://kafka.apache.org/KEYS > > > > > > * Release artifacts to be voted upon (source and binary): > > > https://home.apache.org/~cmccabe/kafka-3.9.0-rc0/ > > > > > > * Docker release artifacts to be voted upon: > > > apache/kafka:3.9.0-rc0 > > > apache/kafka-native:3.9.0-rc0 > > > > > > * Maven artifacts to be voted upon: > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > > > * Javadoc: > > > https://home.apache.org/~cmccabe/kafka-3.9.0-rc0/javadoc/ > > > > > > * Documentation: > > > https://kafka.apache.org/39/documentation.html > > > > > > * Protocol: > > > https://kafka.apache.org/39/protocol.html > > > > > > * Tag to be voted upon (off 3.9 branch) is the 3.9.0-rc0 tag: > > > https://github.com/apache/kafka/releases/tag/3.9.0-rc0 > > > > > > I've run several successful junit and ducktape builds. While there were > > > some flaky tests, when re-running them, they passed. > > > > > > * Successful Docker Image Github Actions Pipeline for 3.8 branch: > > > Docker Build Test Pipeline (JVM): > > > https://github.com/apache/kafka/actions/runs/10912118987 > > > Docker Build Test Pipeline (Native): > > > https://github.com/apache/kafka/actions/runs/10914303515 > > > > > > Thanks to everyone who helped with this release candidate, either by > > > contributing code, testing, or documentation. > > > > > > Regards, > > > Colin > > > > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi all! I did come around and wrote the feedback pending in the KIP itself. Please take another read! I added a section attempting to define the term "usable". Also I applied the feedback. Thanks! On Wed, Sep 4, 2024 at 1:34 AM Colin McCabe wrote: > On Fri, Aug 30, 2024, at 16:40, Matthias J. Sax wrote: > > Great discussion. Also wanted to follow up with a few things. > > > > > > I believe the term "usable" is not well defined leading to confusion... > > I agree with Viktor that "usable" in the context of level 2 should just > > mean: I can enable the feature and it does something... not more, not > > less. It might crash; it might compute the wrong result for some cases, > > it might have terrible performance, etc... but: I can kick the tires. > > > > Yeah, it would be good to clarify this, to avoid "usable" becoming too > expansive. > > > > > About the proposed "checklist" from Viktor: I think we should not have > > anything about testing in it -- that test are required goes w/o saying > > and is already covered in the KIP document itself. To me, it's the KIP > > author's / community's responsivity to decide on a case-by-case basis > > when a feature is considered ready for the next level, and what testing > > is sufficient for each level. > > > > > > Similar for docs, even if I agree that docs should be more or less > > complete at level 3. Otherwise, users will have a hard time to really > > try the feature and thus kinda defeats the purpose of level 3. > > +1 > > > > > > > Last: @Colin, yes we eventually need to pick names for the levels. But I > > believe it's actually the right way to agree on the "what" first, and > > just say "level X" for now, and only after we agree on the levels, we > > enter the ring for the fun part: picking names. This should be the very > > last step :popcorn: > > > > Maybe this is just me, but using numbers instead of names makes it quite > hard for me to get a handle on the discussion. I have opinions on what > alpha / beta / production-ready mean. I don't have opinions on what "Level > 4" means or what "manuscript" means. So I feel like we will go around and > around until we can give a name to what we're talking about. > > best, > Colin > > > > > > > > -Matthias > > > > > > > > > > On 8/30/24 8:57 AM, Colin McCabe wrote: > >> On Mon, Aug 26, 2024, at 10:51, Josep Prat wrote: > >>> Hi Colin, > >>> > >>> Names are in the KIP. Level 1 to 4 are never meant to be used outside > of > >>> this discussion. It's my, apparently successful, attempt to focus on > what > >>> the levels mean instead of their names: > >>> > >>>> Names > >>> > >>> "In Development" > >>> "Early Access" > >>> "Preview" > >>> "Production Ready" > >> > >> Hi Josep, > >> > >> Thanks for the clarification. I think we should remove references to > level 1, 2, 3, 4, etc. if that is not the terminology that we want to use. > One of the big purposes of a KIP is to standardize on terminology. That's > not achieved if different parts of the KIP use different names for the same > things. > >> > >>> > >>> Alternatively, if we want to be a bit more playful we could go with a > theme > >>> borrowed from the book industry (as an homage to Franz Kafka): > >>> > >>> "In Development" > >>> "Manuscript" > >>> "Pre-print" > >>> "Published" > >>> > >>> > >> > >> The need to standardize terminology also means that, sorry, you have to > choose. :) This is actually a feedback I often give on KIPs. People like to > add sections that say "maybe we'll do X, maybe we'll do Y." But to make > progress on the KIP, you have to choose either X or Y and put the other one > in the "rejected alternatives" section. > >> > >> I think our purpose in choosing names should be clarity for users and > developers. That's why I suggested "not implemented", "alpha", "beta", > "production ready". I am curious what your thoughts are about these. > >> > >> best, > >> Colin > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: Build Updates for week of Sep 9, 2024
Thanks for the great summary David! GH CI looks indeed really good. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Thu, Sep 12, 2024, 20:17 David Arthur wrote: > A lot has been happening with the GitHub Actions build in the past few > weeks. I thought I would share some updates. > > *Build Statistics* > Now that we have all PRs builds running the test suite (see note below), we > can do a better comparison between GH and Jenkins > > Github Actions > Successful trunk builds (1): > 1h56m 5% > 1h58m avg > 2h1m 95% > > Github Actions > Successful PR builds: > 1h14m 5% > 1h35m avg > 1h59m 95% > > Jenkins > Successful trunk builds: > 1h27m 5% > 4h7m avg > 5h36m 95% > > Jenkins > Successful PR builds: > 1h22m 5% > 3h48m avg > 5h35m 95% > > It's pretty clear that the GitHub Actions build is significantly more > stable than Jenkins and actually faster on average despite running on > slower hardware. > > 1) We are seeing timeouts occasionally on GH due to a test getting stuck. > We have narrowed it down to one test class. > > *Enabling GitHub Actions by default* > In https://github.com/apache/kafka/pull/17105 we turned on the full "CI" > workflow by default for PRs. This has been running now for a few days and > so far we are well under the quota limit for GH Action Runner usage. > > *Green trunk Builds* > Most of our trunk commits have had green builds on GH Actions and Jenkins. > This has been the result of a lot of focused effort on fixing flaky tests, > which is great to see! > > On Jenkins, we are continuing to see very erratic build times presumably > due to resource contention. On Github, our trunk build times are much more > consistent (presumably due to better isolation). > > *Gradle Build Cache* > Pull Requests now can take advantage of the Gradle Build Cache. The way > this works is that trunk will write to a cache managed by GitHub Actions > and PRs will read from it. In theory, if a PR only changes some code in > ":streams", none of the ":core" tests will be run (and vica-versa). > > Here is an example PR build that cut its testing time by around 1hr > https://ge.apache.org/s/dj2svkxx2edno/timeline. > > In practice, we are still seeing a lot of cache misses since the cache will > slightly lag behind trunk. Stay tuned for improvements to this... > > *Gradle Build Scans* > We are now able to publish Gradle Build Scans for PRs from public forks. > This is very exciting as it will allow contributors (not just committers!) > to gain insights into their builds and have very nice looking test reports. > > Another improvement here is that the build scan links will be included in > the PR "Checks". This is much easier to navigate to than finding it in the > workflow run. > > *De-flaking Integration Tests* > A new "deflake" action was added to our GH Actions. It can be used to > repeatedly run a @ClusterTest in the CI environment. I wrote up some > instructions in a doc on our wiki: > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606545#FlakyTests-GitHub > "deflake"Action > > *Closing old PRs* > We have finished KAFKA-15073. Our "stale" workflow will now actually close > PRs that are inactive for more than 120 days. > > > Cheers, > David A >
Re: [DISCUSS] Regarding Old PRs
Hi David, I checked the PR, looks good! And I think the times set are adequate. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Wed, Sep 11, 2024, 19:51 David Arthur wrote: > Hey folks, I wanted to revive this old thread. > > I'd like to do the following: > > * Change our stale workflow to start with the oldest PRs and move forward > * Enable closing of stale PRs (over 120 days) > > Here's a patch with these changes: > https://github.com/apache/kafka/pull/17166 > Docs for actions/stale: https://github.com/actions/stale > > Cheers, > David A > > On Sat, Jun 10, 2023 at 2:53 AM David Jacot wrote: > > > Thanks, David. I left a few comments in the PR. > > > > -David > > > > Le ven. 9 juin 2023 à 15:31, David Arthur > .invalid> > > a écrit : > > > > > Hey all, I just wanted to bump this one more time before I merge this > PR > > > (thanks for the review, Josep!). I'll merge it at the end of the day > > today > > > unless anyone has more feedback. > > > > > > Thanks! > > > David > > > > > > On Wed, Jun 7, 2023 at 8:50 PM David Arthur wrote: > > > > > > > I filed KAFKA-15073 for this. Here is a patch > > > > https://github.com/apache/kafka/pull/13827. This simply adds a > "stale" > > > > label to PRs with no activity in the last 90 days. I figure that's a > > good > > > > starting point. > > > > > > > > As for developer workflow, the "stale" action is quite flexible in > how > > it > > > > finds candidate PRs to mark as stale. For example, we can exclude PRs > > > that > > > > have an Assignee, or a particular set of labels. Docs are here > > > > https://github.com/actions/stale > > > > > > > > -David > > > > > > > > > > > > On Wed, Jun 7, 2023 at 2:36 PM Josep Prat > <https://www.google.com/maps/search/Josep+Prat+?entry=gmail&source=g> > > > > > > > wrote: > > > > > > > > > Thanks David! > > > > > > > > > > ——— > > > > > Josep Prat > > > > > > > > > > Aiven Deutschland GmbH > > > > > > > > > > Alexanderufer 3-7, 10117 Berlin > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > m: +491715557497 > > > > > > > > > > w: aiven.io > > > > > > > > > > e: josep.p...@aiven.io > > > > > > > > > > On Wed, Jun 7, 2023, 20:28 David Arthur > > > > .invalid> > > > > > wrote: > > > > > > > > > > > Hey all, I started poking around at Github actions on my fork. > > > > > > > > > > > > https://github.com/mumrah/kafka/actions > > > > > > > > > > > > I'll post a PR if I get it working and we can discuss what kind > of > > > > > settings > > > > > > we want (or if we want it all) > > > > > > > > > > > > -David > > > > > > > > > > > > On Tue, Jun 6, 2023 at 1:18 PM Chris Egerton > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Josep, > > > > > > > > > > > > > > Thanks for bringing this up! Will try to keep things brief. > > > > > > > > > > > > > > I'm generally in favor of this initiative. A couple ideas that > I > > > > really > > > > > > > liked: requiring a component label (producer, consumer, > connect, > > > > > streams, > > > > > > > etc.) before closing, and disabling auto-close (i.e., > > automatically > > > > > > tagging > > > > > > > PRs as stale, but leaving it to a human being to actually close > > > > them). > > > > > > > > > > > > > > We might replace the "stale" label with a "close-by-" > label
Re: [ANNOUNCE] New committer: Jeff Kim
Congratulations Jeff! Best, On Mon, Sep 9, 2024 at 8:50 AM Chia-Ping Tsai wrote: > Congratulations, Jeff! thanks for you to bring the great new group > coordinator! > > Best, > Chia-Ping > > David Jacot 於 2024年9月9日 週一 下午2:44寫道： > > > Hi all, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, > Jeff > > Kim. > > > > Jeff has been a Kafka contributor since May 2020. In addition to being > > a regular contributor and reviewer, he has made significant > > contributions to the next generation of the consumer rebalance > > protocol (KIP-848) and to the new group coordinator. He authored > > KIP-915 which improved how coordinators can be downgraded. He also > > contributed multiple fixes/improvements to the fetch from follower > > feature. > > > > Congratulations, Jeff! > > > > Thanks, > > David (on behalf of the Apache Kafka PMC) > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [ANNOUNCE] New Kafka PMC Member: Josep Prat
Thanks all!! -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Sep 6, 2024, 19:45 Igor Soarez wrote: > Congratulations Josep! > > -- > Igor >
Re: Permissions to create or update KIPs
Hi Loic, Thanks for your interest in Apache Kafka! Your accounts are all set up. Let me know if you have any problems! Best, On Thu, Aug 29, 2024 at 10:04 AM Loic Greffier wrote: > Hello, > > I would like to have permissions to create or update KIPs. > > My Confluence ID and Jira ID are "loicgreffier" for both. > > Thanks! > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [ANNOUNCE] New committer: Lianet Magrans
Congrats Lianet! On Wed, Aug 28, 2024 at 5:38 PM Chia-Ping Tsai wrote: > Congratulations, Lianet!!! > > On 2024/08/28 15:35:23 David Jacot wrote: > > Hi all, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, > > Lianet Magrans. > > > > Lianet has been a Kafka contributor since June 2023. In addition to > > being a regular contributor and reviewer, she has made significant > > contributions to the next generation of the consumer rebalance > > protocol (KIP-848) and to the new consumer. She has also contributed > > to discussing and reviewing many KIPs. > > > > Congratulations, Lianet! > > > > Thanks, > > David (on behalf of the Apache Kafka PMC) > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi David, DA2.1. Agreed DA2.2. I agree with this, yes. This should be, in my opinion, KIP specific and part of the DISCUSS thread. DA4.2 I think it makes sense to have this requirement. We might need to word it properly, but I think it makes sense. DA7. Thanks for keeping the bikeshed painted :) Best, On Tue, Aug 27, 2024 at 4:55 PM David Arthur wrote: > DA2.1 re "big" and "simple" features. I think the process described in this > KIP should be for "big" features only, right? We don't want to add the > overhead of tracking the maturity/stability of a new metric or API field. > Conceptually, I understand that these would immediately be level 4 > features, but I think this should be implicit and not required by smaller > KIPs. > > DA2.2 This means we need some distinction for "big" features. What > qualifies? Maybe this is something we can decide in the discussion phase > for each KIP. > > DA3.1 ok > > DA4.1 Fair enough, we can raise a follow up KIP for backport policy. > > DA4.2 On the bugfixes, once something has reached level 4, we may want to > consider that Blocker/Critical bugs should trigger a patch release. After > all, if we say it's production ready and something turns out to be severely > broken, we should fix it promptly. That would be my expectation as a user. > This could be part of the follow up KIP > > DA5.1 Yea that's fine > > DA6.1 Ok, thanks I missed that section > > DA7. Regarding the names, "Preview" and "Early Access" are too ambiguous in > my opinion. Both convey a "before it's baked" status, but to me it's not > easy to distinguish between the two. I like the literary terms, but it > might be a bit too idiomatic. > > Here' my pick for the bikeshed color: > 2: Experimental > 3: Beta > 4. Stable (or Production) > > -David > > > > On Mon, Aug 26, 2024 at 1:51 PM Josep Prat > wrote: > > > Hi Colin, > > > > Names are in the KIP. Level 1 to 4 are never meant to be used outside of > > this discussion. It's my, apparently successful, attempt to focus on what > > the levels mean instead of their names: > > > > > Names > > > > "In Development" > > "Early Access" > > "Preview" > > "Production Ready" > > > > Alternatively, if we want to be a bit more playful we could go with a > theme > > borrowed from the book industry (as an homage to Franz Kafka): > > > > "In Development" > > "Manuscript" > > "Pre-print" > > "Published" > > > > > > Regarding level 2, it is intended to be usable up to some extent. For > > example all the common flows are implemented but some corner cases might > > not be built yet: > > > Level 2 > > > > > > The KIP is now usable, but it might not be completed yet. > > > > So the main difference between Level 1 and 2 is that in level 1 the > feature > > can't be used. In level 2 let's say the common flows are. Level 3 all > flows > > (common and corner cases) are implemented. > > > > Does that make sense? > > > > > > Best, > > > > -- > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +10491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Mon, Aug 26, 2024, 19:05 Colin McCabe wrote: > > > > > On Mon, Aug 26, 2024, at 08:17, Viktor Somogyi-Vass wrote: > > > > Hi all, > > > > > > > > A couple of additional questions/suggestions regarding the KIP: > > > > > > > > VS1. Testing: you wrote about testing only on level 4 but I think we > > > should > > > > add some testing criteria to each one to ensure that it meets > > > > those requirements specified by the given level and it is gradually > > more > > > > mature. For instance for level 1 I think we may not need to do any > > > > integration testing. For level 2 I think we should at least have > happy > > > path > > > > integration tests where applicable and do full integration testing on > > > level > > > > 3. Of course smaller KIPs may skip level 2 as you mentioned or have > > > > different testing requirements, so it may not be applicable >
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi Colin, Names are in the KIP. Level 1 to 4 are never meant to be used outside of this discussion. It's my, apparently successful, attempt to focus on what the levels mean instead of their names: > Names "In Development" "Early Access" "Preview" "Production Ready" Alternatively, if we want to be a bit more playful we could go with a theme borrowed from the book industry (as an homage to Franz Kafka): "In Development" "Manuscript" "Pre-print" "Published" Regarding level 2, it is intended to be usable up to some extent. For example all the common flows are implemented but some corner cases might not be built yet: > Level 2 > > The KIP is now usable, but it might not be completed yet. So the main difference between Level 1 and 2 is that in level 1 the feature can't be used. In level 2 let's say the common flows are. Level 3 all flows (common and corner cases) are implemented. Does that make sense? Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +10491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Mon, Aug 26, 2024, 19:05 Colin McCabe wrote: > On Mon, Aug 26, 2024, at 08:17, Viktor Somogyi-Vass wrote: > > Hi all, > > > > A couple of additional questions/suggestions regarding the KIP: > > > > VS1. Testing: you wrote about testing only on level 4 but I think we > should > > add some testing criteria to each one to ensure that it meets > > those requirements specified by the given level and it is gradually more > > mature. For instance for level 1 I think we may not need to do any > > integration testing. For level 2 I think we should at least have happy > path > > integration tests where applicable and do full integration testing on > level > > 3. Of course smaller KIPs may skip level 2 as you mentioned or have > > different testing requirements, so it may not be applicable everywhere. > > > > VS2. Documentation: I think documentation and user resources (tutorial) > is > > also a bit similar. On level 1 and maybe 2 I wouldn't expect any > > documentation but I would be happy with a feature doc on level 3 and > level > > 4 may also need to provide examples (if applicable). For instance in the > > case of a KRaft or MM2 level feature, I would only consider it level 4 > > (among other things) if it had a very extensive documentation and > > tutorials/examples for users. > > > > VS3. Checklist: I guess the bigger point of view of my previous points is > > that we should create a checklist of a few items (not too many) that KIPs > > graduating to that level should fulfill (besides the ones on the previous > > level) to ensure it is really some sort of an objective graduation. In my > > mind it looks like this: > > Level 1: > > - the KIP has to be accepted > > Level 2: > > - the feature is usable > > - has integration tests for the happy path > > - unit tests exists that cover the existing functionality > > - some minimal documentation exists for early adopters > > Hi Viktor, > > I don't think it makes sense to require that "the feature is usable" at > level 2. As I understand it, this level just means that the feature is > under devlopment. Most features are not usable on day 1 of development. > Similarly, documentation is usually the thing that gets written last. It is > not reasonable to expect it to be written during development, when the > feature might be changing from week to week. > > > Level 3: > > - stable API > > - integration tests cover all paths > > - unit tests cover added functionality > > - documentation exists for users > > Level 4: > > - extensive documentation exists for users with examples or tutorials > if > > needed > > - unit tests cover added functionality > > - integration test suites covering the KIPs functionality > > - system tests if needed > > I think we should avoid turning this into a code quality checklist. It > really should be about when the feature is ready to be used by end-users. > And a lot of KIPs don't involve code at all, like deprecating some > configuration, changing the Scala version, changing the JDK version, etc. > > We already added a section about "Testing" to each KIP. Really the > requirement to reach the last level should be that you did all the testing > that you promised to do. If that testing was insuff
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi Viktor, Thanks for your feedback! I agree with your points. I didn't originally want to write it so focused on coding changes so it fits all sorts of KIPs. For example dropping or adding a Scala or Java version, there are no tests (unit, integration or system) needed for the specific change, "just" adding or removing a CI pipeline and build config. But maybe it's worth adding this into the KIP with a big disclaimer that there are KIPs that might not need any of those and can still be at level 4 (like the Java version I mentioned). I'll update the KIP later today or tomorrow. Best, On Mon, Aug 26, 2024 at 5:20 PM Viktor Somogyi-Vass wrote: > Hi all, > > A couple of additional questions/suggestions regarding the KIP: > > VS1. Testing: you wrote about testing only on level 4 but I think we should > add some testing criteria to each one to ensure that it meets > those requirements specified by the given level and it is gradually more > mature. For instance for level 1 I think we may not need to do any > integration testing. For level 2 I think we should at least have happy path > integration tests where applicable and do full integration testing on level > 3. Of course smaller KIPs may skip level 2 as you mentioned or have > different testing requirements, so it may not be applicable everywhere. > > VS2. Documentation: I think documentation and user resources (tutorial) is > also a bit similar. On level 1 and maybe 2 I wouldn't expect any > documentation but I would be happy with a feature doc on level 3 and level > 4 may also need to provide examples (if applicable). For instance in the > case of a KRaft or MM2 level feature, I would only consider it level 4 > (among other things) if it had a very extensive documentation and > tutorials/examples for users. > > VS3. Checklist: I guess the bigger point of view of my previous points is > that we should create a checklist of a few items (not too many) that KIPs > graduating to that level should fulfill (besides the ones on the previous > level) to ensure it is really some sort of an objective graduation. In my > mind it looks like this: > Level 1: > - the KIP has to be accepted > Level 2: > - the feature is usable > - has integration tests for the happy path > - unit tests exists that cover the existing functionality > - some minimal documentation exists for early adopters > Level 3: > - stable API > - integration tests cover all paths > - unit tests cover added functionality > - documentation exists for users > Level 4: > - extensive documentation exists for users with examples or tutorials if > needed > - unit tests cover added functionality > - integration test suites covering the KIPs functionality > - system tests if needed > > PS. I like the alternative names :) > > Best, > Viktor > > On Mon, Aug 26, 2024 at 11:20 AM Josep Prat > wrote: > > > Hi David, > > > > Thanks for the feedback! > > > > DA1. I'm fine not exposing level 1, but I think it was worth having it > for > > completeness-sake as you mention. This level is probably more of a > > technicality but my state-machine brain needs the initial state. > > > > DA2. Yes, this is the key difference between level 3 and 4. Not all > > features need to go through level 3, for example, refactoring APIs or > > adding new public methods for convenience can directly go to level 4. So > I > > see level 3 as the default "rest" level for "big" features until we gain > > certainty. While "simpler" features could go up to level 4 directly. > > > > DA3. This is a good suggestion. I don't know if we can be too > prescriptive > > with this. It all would boil down to the amount and quality of feedback > > from the early adopters. Now the KIP mentions that levels can only be > > changed in minors and majors, this means that if we don't say anything > > else, the minimum "baking time" would be 1 minor release. This is the > > technical lower limit. We could mention that we encourage to gather > > feedback from the community for 2 minor releases (the one where the > feature > > was released at level 3 and the next minor release). So a feature > reaching > > level 3 in Kafka 4.0, could technically change to level 4.1, but it is > > encouraged to wait at least until 4.2. > > > > DA4. I see your point. I did some archeology to write this KIP and as I > > mention in the motivation, prior 2.8 all features were released as what > we > > could assume would be production ready. If I'm not mistaken, the policy > for > > critical bug backporting has always
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
KIP. It looks like a good proposal. A few > > comments. > > > > > > AS1: I don’t think features should be able to progress between the > > levels in > > > patch releases. Yes, there may be some bug fixes which mean that > > > the usability of a feature has progressed markedly, but given that > > > Kafka releases are so frequent, it doesn’t seem too much of a burden > > > making features wait until the next minor release to progress to the > > > next level. > > > > > > > Hi Andrew, > > > > I think that's fine going forward, but we need to clarify that it only > > applies to new features started after this KIP. One reason is because the > > plan for KIP-853 is to make it production-ready in one of the point > > releases of 3.9. > > > > > AS2: While I think that the Graduation Steps Log idea is a good > addition > > to > > > these more complicated KIPs, but I don’t like release managers having > > > to visit all of the KIPs to see what the state of play is. Each release > > has > > > a release plan on the wiki which people update to add in the KIPs. > > > I would prefer the KIP authors to use that as the way to signal to the > > > release manager that their KIPs have attained a particular level. > > > The Graduation Steps Log is more for the KIP reader to find out when > > > the interesting feature they are reading about was actually delivered > > > in a simple way. The KIP author would have to keep these in sync, > > > which is quite a small burden for something that only changes a few > times > > > at most in the life of a KIP. > > > > I think realistically, the release managers will have to check up on the > > status of KIPs... for example, by pinging the people working on them. > > Otherwise we'll have stuff stuck in an older state even after it's ready, > > because the devs forgot to update it. This is similar to the work release > > managers already do to update the release version of KIPs. I agree that > > it's good to encourage the developers to do this on their own initiative > as > > well. > > > > Also, for KIPs that don't need multiple steps, I wonder if we need > > anything different than what we have now (just a "release version" > field). > > > > best, > > Colin > > > > > > > > AS3: For KIPs which end up with some pieces being completed before > > others, > > > it would be simple to document the sub-features which were delivered > > > and graduated at different releases without being too prescriptive and > > > inventing sub-KIPs or whatever. > > > > > > AS4: I suggest adding to the KIP some statement about feature versions, > > > configurations and other switches at the various levels. For example, > > > if a Kafka feature such as “transaction.version” is used to enable a > KIP, > > > I suppose it would be introduced at level 3 or later, and switched on > > > by default at level 4. The same with unstable APIs - I supposed at > > > levels 1 and 2, if there are new RPCs defined, I need to use > > > `unstable.api.versions.enable` on the broker to enable them. > > > > > > AS5: I wonder whether we need to avoid “features” here because it > > > already has a specific meaning in Kafka (bin/kafka-features.sh and > > > so on). Maybe this KIP should be “Graduation Steps for KIPs”. > > > > > > AS6: Typo: Level 3 says “This level is optional, and a feature must not > > go > > > through it”. This should probably say “does not need to”. > > > > > > AS7: I suggest “Incubation” for level 1. > > > > > > Thanks, > > > Andrew > > > > > > > > >> On 22 Aug 2024, at 08:40, Josep Prat > > wrote: > > >> > > >> Hi Matthias, > > >> > > >> Thanks for your feedback! > > >> - Feedback for level 1: > > >> Yes, indeed it is almost superfluous, but I decided to add it for > > >> completeness. A KIP in level 1 might have some PRs that are merged, > but > > >> it's not yet functional nor complete. For example, only the interfaces > > have > > >> been added. > > >> We could also remove level 1, but then KIPs would start without a > level > > >> assigned which is implicitly a level in my opinion. > > >> The reason for level 1 in my opinion is to ease the work of the > release > > >> manager in understanding if wor
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi Colin, Thanks for your feedback! > I also don't see what the difference is from a developer's point of view. In both cases, the feature is under development. It seems that we are trying to draw a distinction between the feature being started, and the feature not being started. But this is often vague in practice. People may start refactoring things prior to working on a feature, in preparation for that feature. Does that count as "starting the feature"? The main difference between level 1 and level 2 is the fact that only when a feature reaches level 2 it makes sense to ask for early adopters to start tinkering with it. For example, KIP-853 would have been at level 1 during Kafka 3.8.0. It was not complete and not usable at all. Once Kafka 3.9.0 is out, KIP-853 would reach level 2. The feature is mostly finished ( https://issues.apache.org/jira/browse/KAFKA-14094 is done), but there some (minor) functionalities still missing ( https://issues.apache.org/jira/browse/KAFKA-17241 at the moment of writing 22 subtasks are open). The feature is now usable, but maybe some aspects are not 100% polished. We would expect users to start playing around with it (maybe not jumping to production just right yet, but start testing). Regarding the name, I specifically didn't want to go with alpha - beta, because this is usually used for software releases or products. Applying it to features might, in my opinion, cause confusion on the state of our Kafka releases. I know we don't use the concept of alpha and beta, but rather release candidate, but as these words are so embroidered in software versioning I would try to exclude them. Best, On Fri, Aug 23, 2024 at 7:01 PM Colin McCabe wrote: > Hi Josep, > > Thanks for the KIP. > > From a user's point of view, I don't see what the difference is between > level 1 and level 2. Both are not completed, not API stable, not able to be > used in production, not enabled by default. The release manager's view is > similar to the user's point of view in this regard: those two levels are > really the same. > > I also don't see what the difference is from a developer's point of view. > In both cases, the feature is under development. It seems that we are > trying to draw a distinction between the feature being started, and the > feature not being started. But this is often vague in practice. People may > start refactoring things prior to working on a feature, in preparation for > that feature. Does that count as "starting the feature"? > > Your rejected alternatives section explains why you don't like "early > availability" and "general availability." But it doesn't explain why you > don't like the "preview" or "production" labels, which we have sometimes > used in the past. > > > Using "alpha", "beta" and "gamma" has been rejected because of the > widespread use and > > understanding of what "alpha" and "beta" mean in software development. > > I assume this was intended to say "misunderstanding"? Because rejecting a > label because it's well-understood by our users seems... odd. :) > > If we get rid of the level 1 (which doesn't seem useful to me, as I've > explained above), having "alpha", "beta", and "production" seems very easy > to understand and friendly for our users. > > (Or, if you really absolutely need to have level 1, you could rename it to > "not implemented" which seems an apt description?) > > best, > Colin > > > On Mon, Aug 19, 2024, at 03:35, Josep Prat wrote: > > Hi TengYao, > > > > I get your point. I think smaller features definitely go too quickly > > through stages to even acknowledge the change. > > However, I would still think it's necessary to have these 2 levels > > separated when it comes to bigger feature initiatives. The biggest use > case > > I see right now is to signal to the release manager (and the community) > if > > a feature is usable or not yet usable. I believe the fact to become > usable > > for a feature is a big enough step to gain its own entity. > > > > > > Let's take KIP-853 as an example. This KIP was approved and initially > added > > to the release plan for Kafka 3.8. At this point the feature would be in > > Level 1. By the time of the feature freeze the feature was still on Level > > 1, so the release manager (who happened to be me) knew that the KIP would > > not make it in this release and would need to be postponed to a future > > release. After that, development on this feature continued and it was > > declared to enter level 2 r
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi Andrew, thanks for the feedback! AS1: I'm fine with it. Updated the KIP. AS2: I think both steps are needed. I agree with you that KIP authors should keep the release plan page up-to-date with the status of their KIP. However, as a user of the KIP, having a graduation steps log would help understanding the state of their feature and to which version they would need to update to gain more stability for example. AS3: Yes, I don't intend to add sub-KIPs. But rather structure your big KIP in functionalities and those functionalities are the ones that can graduate independently. AS4: I can see how a feature in level 2 has also a configuration value to enable it. Let me add a new column to the summary table and descriptions of the levels. AS5: Makes sense, updated the KIP AS6: thanks, updated the KIP. AS7: sounds good to me. Let's see if somebody else has opinions about it. As I changed the title, the new URL is: https://cwiki.apache.org/confluence/x/mYn9Eg On Thu, Aug 22, 2024 at 4:36 PM Andrew Schofield wrote: > Hi Josep, > Thanks for creating this KIP. It looks like a good proposal. A few > comments. > > AS1: I don’t think features should be able to progress between the levels > in > patch releases. Yes, there may be some bug fixes which mean that > the usability of a feature has progressed markedly, but given that > Kafka releases are so frequent, it doesn’t seem too much of a burden > making features wait until the next minor release to progress to the > next level. > > AS2: While I think that the Graduation Steps Log idea is a good addition to > these more complicated KIPs, but I don’t like release managers having > to visit all of the KIPs to see what the state of play is. Each release has > a release plan on the wiki which people update to add in the KIPs. > I would prefer the KIP authors to use that as the way to signal to the > release manager that their KIPs have attained a particular level. > The Graduation Steps Log is more for the KIP reader to find out when > the interesting feature they are reading about was actually delivered > in a simple way. The KIP author would have to keep these in sync, > which is quite a small burden for something that only changes a few times > at most in the life of a KIP. > > AS3: For KIPs which end up with some pieces being completed before others, > it would be simple to document the sub-features which were delivered > and graduated at different releases without being too prescriptive and > inventing sub-KIPs or whatever. > > AS4: I suggest adding to the KIP some statement about feature versions, > configurations and other switches at the various levels. For example, > if a Kafka feature such as “transaction.version” is used to enable a KIP, > I suppose it would be introduced at level 3 or later, and switched on > by default at level 4. The same with unstable APIs - I supposed at > levels 1 and 2, if there are new RPCs defined, I need to use > `unstable.api.versions.enable` on the broker to enable them. > > AS5: I wonder whether we need to avoid “features” here because it > already has a specific meaning in Kafka (bin/kafka-features.sh and > so on). Maybe this KIP should be “Graduation Steps for KIPs”. > > AS6: Typo: Level 3 says “This level is optional, and a feature must not go > through it”. This should probably say “does not need to”. > > AS7: I suggest “Incubation” for level 1. > > Thanks, > Andrew > > > > On 22 Aug 2024, at 08:40, Josep Prat > wrote: > > > > Hi Matthias, > > > > Thanks for your feedback! > > - Feedback for level 1: > > Yes, indeed it is almost superfluous, but I decided to add it for > > completeness. A KIP in level 1 might have some PRs that are merged, but > > it's not yet functional nor complete. For example, only the interfaces > have > > been added. > > We could also remove level 1, but then KIPs would start without a level > > assigned which is implicitly a level in my opinion. > > The reason for level 1 in my opinion is to ease the work of the release > > manager in understanding if worked KIPs are to be included in the release > > notes and in the list of released KIPs. > >> This seems to be little bit confusing. If it's not meant to be used > > by anybody, should it not be feature flagged and disable/hidden from > users > > completely? > > Yes, this is correct, most of the time they would be behind a feature > flag, > > or they will be new code paths not reachable yet. Let's take KIP-853 for > > example, at the moment of Kafka 3.8.0 release, only the first 6 tasks > under > > https://issues.apache.org/jira/browse/KAFKA-14094 were completed. The > > feature was not usable and not visible fo
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
level changes can only be triggered in minors and majors. However, I was thinking that if everything needed for the feature itself could be done in patch releases why couldn't the graduation step change in patch releases as well. For example, let's imagine a fictitious KIP that is on level 3. All public API changes have been completed already and their implementation is there. The KIP drivers wanted to seek validation in real scenarios that's why the feature is at level 3 and not 4. Let's keep imagining that they get multiple satisfactory reports from the community and a couple of reports that indicate a performance degradation occurs under circumstance X. This is fixed and released in the next patch version. Should we be able to graduate this feature to level 4 in this patch version? To be honest if people have concerns about this, I'm fine by leaving this possibly corner case out and only allow graduation changes on major and minors. Let me know what you think Best, [1]: https://www.sarvika.com/2021/06/17/mvp-mlp-mup/ contains some explanation and comparison of MVP vs MLP vs MUP. On Thu, Aug 22, 2024 at 7:07 AM Matthias J. Sax wrote: > Thanks Josep. Couple of comments/questions: > > For Level 1, the KIP says: > > > The feature is not yet stable and it is not meant to be used by end > users. > > This seems to be little bit confusing. If it's not meant to be used by > anybody, should it not be feature flagged and disable/hidden from users > completely? > > Of course, it should not be used in production, but isn't the goal that > users do try it out and provide early feedback about the feature? > However, this is described on Level 2. So I am really not sure what > Level 1 is supposed to be, or why we would need it as an public > "contract". Later the KIP say > > > Implicitly when a KIP is approved and development starts, the feature is > effectively in "Level 1" > > Which really means to me, we should drop this level from the "public > hierarchy", as it won't add much (any?) value? > > The propose name "In Development" indicates the same thing to me. In the > end it means, a release does just not contain anything about it (even if > there might be code that was merged), and the KIP should not be > mentioned in the release notes at all, and thus we don't need a name as > the feature is not graduated into any (useful) level yet. > > > I also find the term "usable" very fuzzy. Can we add more color to it? > > > Another thought: we did have cases for KS for which a KIP was only > partially implemented in a release, but the completed parts where fully > functional, and thus in stage 4. How do we intent to handle this going > forward? I would find it rather odd to only release something as stage 2 > just to follow the process... (as a matter of fact, we still have some > KIPs which never got fully completed but are pending completion for > years now, and might frankly never get completed) > > Cf > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+KIP+Overview > for examples > > > > > A feature can be promoted to a "higher" level at any release (including > patch releases) > > Not sure if this makes sense? To me, a new level can only be reached in > a major or minor release? > > > > > -Matthias > > On 8/20/24 6:21 AM, Josep Prat wrote: > > Hi all, > > > > I added a new section in the KIP to specify how to change the graduation > > levels for a feature. > > > > Best! > > > > On Mon, Aug 19, 2024 at 4:01 PM Josep Prat wrote: > > > >> Hi TengYao and Chia-Ping, > >> > >> I updated the KIP page with examples. > >> > >> Best, > >> > >> On Mon, Aug 19, 2024 at 2:39 PM TengYao Chi > wrote: > >> > >>> Hi Josep > >>> > >>> Thanks for the explanation. I see your point. > >>> It makes sense to keep these levels distinct for larger initiatives > like > >>> KIP-853. I agree with your perspective. > >>> > >>> Best regards, > >>> TengYao > >>> > >>> Josep Prat 於 2024年8月19日 週一 下午6:36寫道： > >>> > >>>> Hi TengYao, > >>>> > >>>> I get your point. I think smaller features definitely go too quickly > >>>> through stages to even acknowledge the change. > >>>> However, I would still think it's necessary to have these 2 levels > >>>> separated when it comes to bigger feature initiatives. The biggest use > >>> case > >>>> I see right now is to signal t
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi all, I added a new section in the KIP to specify how to change the graduation levels for a feature. Best! On Mon, Aug 19, 2024 at 4:01 PM Josep Prat wrote: > Hi TengYao and Chia-Ping, > > I updated the KIP page with examples. > > Best, > > On Mon, Aug 19, 2024 at 2:39 PM TengYao Chi wrote: > >> Hi Josep >> >> Thanks for the explanation. I see your point. >> It makes sense to keep these levels distinct for larger initiatives like >> KIP-853. I agree with your perspective. >> >> Best regards, >> TengYao >> >> Josep Prat 於 2024年8月19日 週一 下午6:36寫道： >> >> > Hi TengYao, >> > >> > I get your point. I think smaller features definitely go too quickly >> > through stages to even acknowledge the change. >> > However, I would still think it's necessary to have these 2 levels >> > separated when it comes to bigger feature initiatives. The biggest use >> case >> > I see right now is to signal to the release manager (and the community) >> if >> > a feature is usable or not yet usable. I believe the fact to become >> usable >> > for a feature is a big enough step to gain its own entity. >> > >> > >> > Let's take KIP-853 as an example. This KIP was approved and initially >> added >> > to the release plan for Kafka 3.8. At this point the feature would be in >> > Level 1. By the time of the feature freeze the feature was still on >> Level >> > 1, so the release manager (who happened to be me) knew that the KIP >> would >> > not make it in this release and would need to be postponed to a future >> > release. After that, development on this feature continued and it was >> > declared to enter level 2 right in time for being in Kafka 3.9. >> > >> > Let me know what you think. >> > >> > Best, >> > >> > On Mon, Aug 19, 2024 at 8:51 AM TengYao Chi >> wrote: >> > >> > > Hello Josep, >> > > I think this KIP is a great addition to the community that we now >> have a >> > > crystal-clear definition for the state of a feature. >> > > >> > > In the current proposal, I think Level 1 is defined as the stage >> where a >> > > feature is "incomplete and unusable", while Level 2 represents a >> feature >> > > that is "usable but potentially incomplete". >> > > The distinction between these two levels might not always be clear, >> > > especially during the transition of a feature from "unusable" to >> "usable >> > > but incomplete". >> > > >> > > IMHO, to simplify the process and reduce confusion for both developers >> > and >> > > users, I would suggest merging Level 1 and Level 2 into a single >> unified >> > > level. >> > > This merged level could cover the entire phase from when a feature is >> > > unstable to when it becomes usable but incomplete. >> > > >> > > WYDT? >> > > >> > > Best regards, >> > > TengYao >> > > >> > > Josep Prat 於 2024年8月19日 週一 上午2:58寫道： >> > > >> > > > Hi Chia-Ping, >> > > > >> > > > As far as I can tell, Tiered Storage is still at level 3. I think >> the >> > > > intention would be to declare it level 4 in 4.0.0. >> > > > KIP-848 was in level 2 in Kafka 3.7. and it went level 3 in Kafka >> 3.8. >> > > > Level 4 features would be for example MirrorMaker2 for example. As >> far >> > > as I >> > > > understand the Docker image is level 4. >> > > > >> > > > Does that make sense? If so I can update the KIP with those >> examples. >> > > > >> > > > Best, >> > > > >> > > > -- >> > > > Josep Prat >> > > > Open Source Engineering Director, Aiven >> > > > josep.p...@aiven.io | +491715557497 | aiven.io >> > > > Aiven Deutschland GmbH >> > > > Alexanderufer 3-7, 10117 Berlin >> > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, >> > > > Anna Richardson, Kenneth Chen >> > > > Amtsgericht Charlottenburg, HRB 209739 B >> > > > >> > > > On Sun, Aug 18, 2024, 21:46 Chia-Ping Tsai >> wrote: >> > > > >> > > > > hi Josep >> > > > > &g
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi TengYao and Chia-Ping, I updated the KIP page with examples. Best, On Mon, Aug 19, 2024 at 2:39 PM TengYao Chi wrote: > Hi Josep > > Thanks for the explanation. I see your point. > It makes sense to keep these levels distinct for larger initiatives like > KIP-853. I agree with your perspective. > > Best regards, > TengYao > > Josep Prat 於 2024年8月19日 週一 下午6:36寫道： > > > Hi TengYao, > > > > I get your point. I think smaller features definitely go too quickly > > through stages to even acknowledge the change. > > However, I would still think it's necessary to have these 2 levels > > separated when it comes to bigger feature initiatives. The biggest use > case > > I see right now is to signal to the release manager (and the community) > if > > a feature is usable or not yet usable. I believe the fact to become > usable > > for a feature is a big enough step to gain its own entity. > > > > > > Let's take KIP-853 as an example. This KIP was approved and initially > added > > to the release plan for Kafka 3.8. At this point the feature would be in > > Level 1. By the time of the feature freeze the feature was still on Level > > 1, so the release manager (who happened to be me) knew that the KIP would > > not make it in this release and would need to be postponed to a future > > release. After that, development on this feature continued and it was > > declared to enter level 2 right in time for being in Kafka 3.9. > > > > Let me know what you think. > > > > Best, > > > > On Mon, Aug 19, 2024 at 8:51 AM TengYao Chi wrote: > > > > > Hello Josep, > > > I think this KIP is a great addition to the community that we now have > a > > > crystal-clear definition for the state of a feature. > > > > > > In the current proposal, I think Level 1 is defined as the stage where > a > > > feature is "incomplete and unusable", while Level 2 represents a > feature > > > that is "usable but potentially incomplete". > > > The distinction between these two levels might not always be clear, > > > especially during the transition of a feature from "unusable" to > "usable > > > but incomplete". > > > > > > IMHO, to simplify the process and reduce confusion for both developers > > and > > > users, I would suggest merging Level 1 and Level 2 into a single > unified > > > level. > > > This merged level could cover the entire phase from when a feature is > > > unstable to when it becomes usable but incomplete. > > > > > > WYDT? > > > > > > Best regards, > > > TengYao > > > > > > Josep Prat 於 2024年8月19日 週一 上午2:58寫道： > > > > > > > Hi Chia-Ping, > > > > > > > > As far as I can tell, Tiered Storage is still at level 3. I think the > > > > intention would be to declare it level 4 in 4.0.0. > > > > KIP-848 was in level 2 in Kafka 3.7. and it went level 3 in Kafka > 3.8. > > > > Level 4 features would be for example MirrorMaker2 for example. As > far > > > as I > > > > understand the Docker image is level 4. > > > > > > > > Does that make sense? If so I can update the KIP with those examples. > > > > > > > > Best, > > > > > > > > -- > > > > Josep Prat > > > > Open Source Engineering Director, Aiven > > > > josep.p...@aiven.io | +491715557497 | aiven.io > > > > Aiven Deutschland GmbH > > > > Alexanderufer 3-7, 10117 Berlin > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > > > Anna Richardson, Kenneth Chen > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > On Sun, Aug 18, 2024, 21:46 Chia-Ping Tsai > wrote: > > > > > > > > > hi Josep > > > > > > > > > > Although I didn't join the discussion before, the KIP is > interesting > > > and > > > > > great to me. > > > > > > > > > > one small comment: > > > > > > > > > > Could you please add existent features as an example to each level > > for > > > > the > > > > > readers who have poor reading (like me) ? For instance, I guess the > > new > > > > > coordinator is level 3? tiered storage is level 4? > > > > > > > > > > Best, > > > > > Chia-
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi TengYao, I get your point. I think smaller features definitely go too quickly through stages to even acknowledge the change. However, I would still think it's necessary to have these 2 levels separated when it comes to bigger feature initiatives. The biggest use case I see right now is to signal to the release manager (and the community) if a feature is usable or not yet usable. I believe the fact to become usable for a feature is a big enough step to gain its own entity. Let's take KIP-853 as an example. This KIP was approved and initially added to the release plan for Kafka 3.8. At this point the feature would be in Level 1. By the time of the feature freeze the feature was still on Level 1, so the release manager (who happened to be me) knew that the KIP would not make it in this release and would need to be postponed to a future release. After that, development on this feature continued and it was declared to enter level 2 right in time for being in Kafka 3.9. Let me know what you think. Best, On Mon, Aug 19, 2024 at 8:51 AM TengYao Chi wrote: > Hello Josep, > I think this KIP is a great addition to the community that we now have a > crystal-clear definition for the state of a feature. > > In the current proposal, I think Level 1 is defined as the stage where a > feature is "incomplete and unusable", while Level 2 represents a feature > that is "usable but potentially incomplete". > The distinction between these two levels might not always be clear, > especially during the transition of a feature from "unusable" to "usable > but incomplete". > > IMHO, to simplify the process and reduce confusion for both developers and > users, I would suggest merging Level 1 and Level 2 into a single unified > level. > This merged level could cover the entire phase from when a feature is > unstable to when it becomes usable but incomplete. > > WYDT? > > Best regards, > TengYao > > Josep Prat 於 2024年8月19日 週一 上午2:58寫道： > > > Hi Chia-Ping, > > > > As far as I can tell, Tiered Storage is still at level 3. I think the > > intention would be to declare it level 4 in 4.0.0. > > KIP-848 was in level 2 in Kafka 3.7. and it went level 3 in Kafka 3.8. > > Level 4 features would be for example MirrorMaker2 for example. As far > as I > > understand the Docker image is level 4. > > > > Does that make sense? If so I can update the KIP with those examples. > > > > Best, > > > > -- > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Sun, Aug 18, 2024, 21:46 Chia-Ping Tsai wrote: > > > > > hi Josep > > > > > > Although I didn't join the discussion before, the KIP is interesting > and > > > great to me. > > > > > > one small comment: > > > > > > Could you please add existent features as an example to each level for > > the > > > readers who have poor reading (like me) ? For instance, I guess the new > > > coordinator is level 3? tiered storage is level 4? > > > > > > Best, > > > Chia-Ping > > > > > > > > > > > > Josep Prat 於 2024年8月19日 週一 上午2:13寫道： > > > > > > > Hi all, > > > > I want to start a discussion for KIP-1081: Graduation Steps for > > Features. > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1081%3A+Graduation+Steps+for+Features > > > > > > > > We already had a bit of a discussion here > > > > https://lists.apache.org/thread/5z6rxvs9m0bro5ssmtg8qcgdk40882bz and > > > that > > > > materialized into this KIP. > > > > > > > > I deliberately defined the graduation steps without giving them a > name, > > > so > > > > we don't go bike-shedding there. There is a separate section for the > > > names > > > > of each step. Also an alternative set of names. I'd like to get some > > > > feedback on the steps, and also on the names for the steps. > > > > > > > > Looking forward to your opinions, and hopefully only a tiny bit of > > > > bike-shedding :) > > > > > > > > Best, > > > > > > > > -- > > > > [image: Aiven] <https://www.aiven.io> > > > &g
Re: [DISCUSS] KIP-1081: Graduation Steps for Features
Hi Chia-Ping, As far as I can tell, Tiered Storage is still at level 3. I think the intention would be to declare it level 4 in 4.0.0. KIP-848 was in level 2 in Kafka 3.7. and it went level 3 in Kafka 3.8. Level 4 features would be for example MirrorMaker2 for example. As far as I understand the Docker image is level 4. Does that make sense? If so I can update the KIP with those examples. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Sun, Aug 18, 2024, 21:46 Chia-Ping Tsai wrote: > hi Josep > > Although I didn't join the discussion before, the KIP is interesting and > great to me. > > one small comment: > > Could you please add existent features as an example to each level for the > readers who have poor reading (like me) ? For instance, I guess the new > coordinator is level 3? tiered storage is level 4? > > Best, > Chia-Ping > > > > Josep Prat 於 2024年8月19日 週一 上午2:13寫道： > > > Hi all, > > I want to start a discussion for KIP-1081: Graduation Steps for Features. > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1081%3A+Graduation+Steps+for+Features > > > > We already had a bit of a discussion here > > https://lists.apache.org/thread/5z6rxvs9m0bro5ssmtg8qcgdk40882bz and > that > > materialized into this KIP. > > > > I deliberately defined the graduation steps without giving them a name, > so > > we don't go bike-shedding there. There is a separate section for the > names > > of each step. Also an alternative set of names. I'd like to get some > > feedback on the steps, and also on the names for the steps. > > > > Looking forward to your opinions, and hopefully only a tiny bit of > > bike-shedding :) > > > > Best, > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
[DISCUSS] KIP-1081: Graduation Steps for Features
Hi all, I want to start a discussion for KIP-1081: Graduation Steps for Features. https://cwiki.apache.org/confluence/display/KAFKA/KIP-1081%3A+Graduation+Steps+for+Features We already had a bit of a discussion here https://lists.apache.org/thread/5z6rxvs9m0bro5ssmtg8qcgdk40882bz and that materialized into this KIP. I deliberately defined the graduation steps without giving them a name, so we don't go bike-shedding there. There is a separate section for the names of each step. Also an alternative set of names. I'd like to get some feedback on the steps, and also on the names for the steps. Looking forward to your opinions, and hopefully only a tiny bit of bike-shedding :) Best, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] GitHub CI
Hi David, One of the enhancements we can have with this change (it's easier to do with GH actions) is to write back the result of the CI run as a comment on the PR itself. I believe not needing to periodically check CI to see if the run finished would be a great win. By having CI commenting on the PR everyone watching the PR (author and reviewers) will get notified when it's done. Best, On Fri, Aug 16, 2024 at 10:50 AM TengYao Chi wrote: > Hello David, > > I think this proposal is awesome. Personally, I find the UI of Jenkins not > very friendly for newcomers. > When I first contributed to Kafka and wanted to see the result of my PR > build, it took me some time to understand each part of Jenkins. So, for me, > the modern and user-friendly UI of GitHub Actions is very attractive. And > most important, if we could stay on the same platform and complete most of > the tasks, it would be great. > I think the main concern is the test suite’s execution time, but as you > mentioned, ASF plans to provide compatible machines for GitHub Actions, so > that might not be an issue. > > I would like to upvote this proposal. > Thank you. > > Best Regards, > TengYao > > David Arthur 於 2024年8月16日 週五 上午2:18寫道： > > > Hey everyone, > > > > Over the past several months (years, maybe?) I've tinkered around with > > GitHub Actions as a possible alternative to Jenkins for Apache Kafka CI. > I > > think it is time to actually give it an earnest try. > > > > We have already done some work with GH Actions. Namely the Docker build > and > > the "stale PR" workflow. I would like to add a new workflow that will run > > the JUnit tests in a GH Action. > > > > Here is an example PR on my personal fork that is using an Action > > > > https://github.com/mumrah/kafka/pull/5 > > > > For the full test suite, it took 1h41m. A random Jenkins run I found took > > 1h17m. A difference of 24m. This is simply because the Jenkins hardware > is > > beefier than the GH Actions public runners. > > > > ASF has been evaluating the use of larger runners as well as ASF-hosted > > runners on beefier hardware. I think eventually, the compute capacity > will > > be comparable. > > > > There are many benefits to GH Actions compared to Jenkins. To name a few: > > > > * Significantly better UI > > * Wide availability of plugins from the GitHub Actions Marketplace > > * Better/easier integration with Pull Requests > > * Easier to customize workflows based on different GitHub events > > * Ability to write custom actions that utilize the `gh` GitHub CLI > > > > Another nice thing (and the original motivation for my inquiry) is that > GH > > Actions has caching as a built-in concept. This means we can leverage the > > Gradle cache and potentially speed up build times on PRs significantly. > > > > I'd like to run both Jenkins and GH Actions side by side for a few weeks > so > > we can gather data to make an informed determination. > > > > What do folks in the community think about this? > > > > Cheers, > > David A > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
[jira] [Created] (KAFKA-17328) Update Release.py script to point to dist.apache.org/dev
Josep Prat created KAFKA-17328: -- Summary: Update Release.py script to point to dist.apache.org/dev Key: KAFKA-17328 URL: https://issues.apache.org/jira/browse/KAFKA-17328 Project: Kafka Issue Type: Task Reporter: Josep Prat Infra communicated the decommission of the home.apache.org box. This box was used by the Release manager to store the RC candidates up for vote. The release.py script is written to push the artifacts in the RM's home box. We need to update this process and use the dist.apache.org/dev space. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: New release branch 3.9
Hi Colin, Just to make sure you (as a Release Manager) are aware of the change. ASF Infra just communicated the removal of the home.apache.org machines (in 4 weeks they will be gone). This means the release.py script wouldn't work if executed after that date and would need to be modified to store RC artifacts under dist.apache.org/dev instead. Best, On Fri, Aug 9, 2024 at 11:38 PM Colin McCabe wrote: > Updates: > > KIP-1007: I have removed this PR from the Apache Kafka 3.9 release plan. > Since the KIP was discarded, I did not add it in the "postponed to > subsequent release" section. > > KIP-1025: Marked 3.9 status as Done > > KIP-1023: Postponed to subsequent release > > KIP-1005: Marked 3.9 status as Done > > KIP-996: Postponed to subsequent release > > KIP-994: Postponed to subsequent release > > KIP-997: Postponed to subsequent release > > KIP-966: Postponed to subsequent release > > KIP-956: Postponed to subsequent release > > KIP-950: Marked 3.9 status as Done > > best, > Colin > > On Fri, Aug 9, 2024, at 14:13, Colin McCabe wrote: > > Hi Chia-Ping Tsai, > > > > Thank you! And thanks for your own contributions, especially the many > > code reviews (not only of my PRs, but everyone's). > > > > best, > > Colin > > > > On Thu, Aug 8, 2024, at 19:14, Chia-Ping Tsai wrote: > >>> - We have completed all the feature work for KIP-853: KRaft controller > membership changes. (Please note that there are still a few bug fixes > remaining, and the docs JIRA, KAFKA-17048, but those are all > post-feature-freeze tasks.) > >> > >> Sorry to make meaningless reply on this mail thread, but big thanks to > >> Colin, Jose, and other contributors to ship KIP-853 to 3.9! > >> > >> Best, > >> Chia-Ping > >> > >> > >>> > >>> Colin McCabe 於 2024年8月9日 上午9:47 寫道： > >>> > >>> Hi all, > >>> > >>> I think it's time to transition the 3.9 branch to feature freeze. > Please note that this is 2 weeks past the original planned date. I > apologize for this; our planning wasn't perfect. > >>> > >>> A few updates on features that are in or out of Apache Kafka 3.9: > >>> > >>> - We have completed all the feature work for KIP-853: KRaft controller > membership changes. (Please note that there are still a few bug fixes > remaining, and the docs JIRA, KAFKA-17048, but those are all > post-feature-freeze tasks.) > >>> > >>> - After speaking with Calvin, I moved KIP-966 out of 3.9 and into 4.0. > I do feel bad about this but I just don't think we'll be able to get it > into this short release. > >>> > >>> There are a few other KIPs that probably need to be moved to 4.0, such > as KIP-1023 and KIP-1025. I will do a more thorough review tomorrow and > reach out to people if there are gray areas. If you have a feature that you > have questions about, please reach out. > >>> > >>> I would like to extend code freeze for Kafka 3.9 to August 29th. My > reasoning is that code freeze was 2 weeks later than expected, and also, I > will be out of the office most of next week. > >>> > >>> I have also created a PR to mark 3.9-IV0 as stable, so look for that > shortly in trunk and 3.9. > >>> > >>> As previously mentioned, please refrain from changing the Java version > in trunk (kafka 4.0) until we get an RC out. (However, please do continue > working on your 4.0 features and other refactors in trunk, even if they > don't apply to 3.9!) > >>> > >>> Thanks to everyone who has worked on this release, and thanks for your > patience. I do believe we will deliver on the promise of a > much-shorter-than-usual release cycle, even with the 2 week delay. > >>> > >>> best, > >>> Colin > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
[jira] [Resolved] (KAFKA-12830) Remove Deprecated constructor in TimeWindowedDeserializer and TimeWindowedSerde
[ https://issues.apache.org/jira/browse/KAFKA-12830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat resolved KAFKA-12830. Resolution: Fixed > Remove Deprecated constructor in TimeWindowedDeserializer and > TimeWindowedSerde > --- > > Key: KAFKA-12830 > URL: https://issues.apache.org/jira/browse/KAFKA-12830 > Project: Kafka > Issue Type: Sub-task > Components: streams > Reporter: Josep Prat >Assignee: PoAn Yang >Priority: Blocker > Fix For: 4.0.0 > > > The single argument constructor and a factory method of the following classes > were deprecated in version 2.8: > * > org.apache.kafka.streams.kstream.TimeWindowedDeserializer#TimeWindowedDeserializer(org.apache.kafka.common.serialization.Deserializer) > * > org.apache.kafka.streams.kstream.WindowedSerdes.TimeWindowedSerde#TimeWindowedSerde(org.apache.kafka.common.serialization.Serde) > * > org.apache.kafka.streams.kstream.WindowedSerdes#timeWindowedSerdeFrom(java.lang.Class) > > > See KAFKA-10366 & KAFKA-9649 and KIP-659 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12832) Remove Deprecated methods under RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter
[ https://issues.apache.org/jira/browse/KAFKA-12832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat resolved KAFKA-12832. Resolution: Fixed > Remove Deprecated methods under > RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter > -- > > Key: KAFKA-12832 > URL: https://issues.apache.org/jira/browse/KAFKA-12832 > Project: Kafka > Issue Type: Sub-task > Components: streams > Reporter: Josep Prat >Assignee: 黃竣陽 >Priority: Blocker > Fix For: 4.0.0 > > > The following methods under were deprecated in version 3.0.0 > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setBaseBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundCompactions > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#maxBackgroundFlushes > * > org.apache.kafka.streams.state.internals.RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapter#setMaxBackgroundFlushes > > See KAFKA-8897 and KIP-471 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Graduation steps for Features
> At the same time, even if something is in Preview and we would guarantee backward compatibility, a major release would still allow us to break it? I think so. The way I see it is the following. We have 3 levels of graduation: 1, 2 and 3 (naming is hard and don't want to bike shed on this). Level 1 is the first level and the "lower" in terms of production readiness. In this level there are no guarantees, just best of intentions. Compatibility might be broken between minor releases and why not between patches. Maybe the feature is not even complete. Level 2 is one level "up". We need to provide some more guarantees. Probably the same level of compatibility as with any other feature. But what we can't probably assure is any type of SLA/SLO, but the community is encouraged to test in production or staging environments. In plain words: "it should work but we are not completely sure it works under different situations or deployment models" for example. In this level, feature must be complete. Finally, we have level 3, the final one. This is the stage where we are confident that the feature works in all normal conditions (of course bugs can occur). A new feature can go through all 3 levels, or start on number 2 or jump directly to number 3. Also a feature can be killed in level 1. After level 1, a feature should undergo a deprecation process. I think we have enough feedback and general consensus to start drafting a KIP. I'll do in the following days. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Thu, Aug 1, 2024, 23:44 Matthias J. Sax wrote: > Interesting point. To me, EA could be the same as "experimental". I > don't think we should introduce too many categories. > > From a naming point of view, the difference between EA and Preview is > not super intuitive though. If we would go with "experimental" instead > of EA it might be easier to understand? > > > At the same time, even if something is in Preview and we would guarantee > backward compatibility, a major release would still allow us to break it? > > > > -Matthias > > > On 8/1/24 12:32 PM, Colin McCabe wrote: > > Hi all, > > > > Sorry, I mis-spoke earlier. For KRaft, we did EA -> Preview -> GA. For > some reason I remembered the order differently, but I was wrong. > > > > EA was 2.8; preview was 3.0, GA was 3.3. > > > > The main thing I remember was that we broke compatibility between KRaft > 2.8 and everything else. That release really was just a demo, in some > sense. In the 3.x line, we tried much harder to keep backwards > compatibility (and to the best of my recollection, did). > > > > From a user's point of view, understanding whether compatibility could > be broken in the future is probably the most important thing. Unfortunately > we don't seem to be consistent on this; some EA features moved smoothly to > GA with no compatibility breaks, while others did drop compat. It could be > useful to have a term for features that for which upgrade is not supported. > Perhaps "experimental"? > > > > best, > > Colin > > > > > > On Wed, Jul 31, 2024, at 20:50, Chris Egerton wrote: > >> I only raised the idea of barrier to entry regarding the template. I did > >> not state or even really imply that it should affect this initiative in > >> general. Please do not feel obligated to discuss that concept further if > >> we're all on the same page RE not adding to the template. > >> > >> On Wed, Jul 31, 2024, 22:02 Matthias J. Sax wrote: > >> > >>> Hi, > >>> > >>> thanks for starting this discussion. It's for sure useful. > >>> > >>> I don't care too much about the naming personally, it just should be > >>> well defined. > >>> > >>> I agree to Josep, that I don't think we would need to change the KIP > >>> template. A KIP is a design doc for the full feature, and should not > >>> concern itself with what state is covered in which release. (As a > matter > >>> of fact, most KIPs don't even include their release version -- we only > >>> track it in the top-level KIP overview page...) > >>> > >>> It might be useful though, to add information about different stages in > >>> the KIP overview page, which atm only says "released in version 3.8" > >>> w
Re: New release branch 3.9
Hi Colin, Looking at the JIRAs of the new umbrella issue ( https://issues.apache.org/jira/browse/KAFKA-17241) it seems to me that the adding metrics one is the only one that is different. As you mention most of the tasks are refactors, optimizations and tests. However, having proper metrics I think it's part of having being able to operate Kafka environments successfully. This is why I think if it doesn't make it to 3.9.0 (as it seems) it should definitely be in 3.9.1. The reason I'm asking is because adding metrics in a patch version is something not something we usually do. Best, On Thu, Aug 1, 2024 at 9:24 PM Colin McCabe wrote: > Hi all, > > I've moved the optional and follow-on JIRAs for KIP-853 (many of which > were "refactors," "optimizations", "tests", etc.) to KAFKA-17241 to > hopefully clear up the confusion a bit. I think a lot of these would be > good newbie JIRAs (although probably not the "refactor" ones :) > > As I said, we'll shoot for getting the remaining feature tasks for KIP-853 > done in a few days. Probably by Monday. > > best, > Colin > > > On Thu, Aug 1, 2024, at 12:18, Colin McCabe wrote: > > Hi Josep, > > > > I don't have a strong opinion on whether KAFKA-16524 should land in > > 3.9.1. Let's discuss that more on the mailing list when it's time for > > 3.9.1. > > > > best, > > Colin > > > > > > On Thu, Aug 1, 2024, at 12:07, Josep Prat wrote: > >> Hi Colin, > >> Will JIRAs like adding metrics ( > >> https://issues.apache.org/jira/browse/KAFKA-16524) make it in patch > version > >> of 3.9? > >> > >> Best, > >> > >> On Thu, Aug 1, 2024 at 8:49 PM Mickael Maison > > >> wrote: > >> > >>> Hi Colin, > >>> > >>> Thanks for the clarification. This makes much more sense now. > >>> > >>> Mickael > >>> > >>> On Thu, Aug 1, 2024 at 8:43 PM Colin McCabe > wrote: > >>> > > >>> > Hi Chris, > >>> > > >>> > I don't see why this shouldn't go to 3.9. Would you call this a > bugfix? > >>> > > >>> > best, > >>> > Colin > >>> > > >>> > > >>> > On Wed, Jul 31, 2024, at 13:54, Chris Egerton wrote: > >>> > > Hi Colin, > >>> > > > >>> > > Would it be alright if I cherry-picked > >>> > > https://github.com/apache/kafka/pull/16678 onto 3.9? This isn't a > >>> > > regression but it's a low-risk fix that I'd like to get into the > next > >>> > > release if possible. > >>> > > > >>> > > Cheers, > >>> > > > >>> > > Chris > >>> > > > >>> > > On Wed, Jul 31, 2024 at 12:29 AM Ismael Juma > >>> wrote: > >>> > > > >>> > >> I would recommend against large refactorings in trunk until the > first > >>> RC > >>> > >> for 3.9 - that will reduce cherry-pick friction. Once we have the > >>> first RC, > >>> > >> subsequent changes to 3.9 should be limited in scope. > >>> > >> > >>> > >> Ismael > >>> > >> > >>> > >> On Tue, Jul 30, 2024 at 4:31 PM Colin McCabe > >>> wrote: > >>> > >> > >>> > >> > Yeah, please go ahead. I know a lot of people are waiting for > 4.0. > >>> > >> > > >>> > >> > best, > >>> > >> > Colin > >>> > >> > > >>> > >> > > >>> > >> > On Tue, Jul 30, 2024, at 16:05, Matthias J. Sax wrote: > >>> > >> > > Thanks for clarifying Colin. So my assumptions were actually > >>> correct. > >>> > >> > > > >>> > >> > > We have a lot of contributors waiting to pick-up 4.0 tickets, > and > >>> I'll > >>> > >> > > go ahead a tell them that we are ready and they can start to > pick > >>> them > >>> > >> > up. > >>> > >> > > > >>> > >> > > Thanks. > >>> > >> > > > >>> > >> > > > >>> > >> > > -Matthias > >>> > >> > >
Re: New release branch 3.9
t;>> <https://issues.apache.org/jira/browse/KAFKA-16518> > > >> > >>>> 3. kafka-metadata-quorum describe changes for KIP-853 > > >> > >>>> <https://issues.apache.org/jira/browse/KAFKA-16521> > > >> > >>>> 4. kafka-metadata-quorum add voter and remove voter changes > > >> > >>>> <https://issues.apache.org/jira/browse/KAFKA-16523> > > >> > >>>> 5. Sending UpdateVoter request and response handling > > >> > >>>> <https://issues.apache.org/jira/browse/KAFKA-16534> > > >> > >>>> > > >> > >>>> Can we cherry pick them to the release branch 3.9.0 when they > get > > >> > merged to > > >> > >>>> trunk? They have a small impact as they shouldn't affect the > rest of > > >> > Kafka > > >> > >>>> and only affect the kraft controller membership change > feature. I > > >> > expected > > >> > >>>> them to get merged to the trunk branch in the coming days. > > >> > >>>> > > >> > >>>> Thanks, > > >> > >>>> > > >> > >>>> On Mon, Jul 29, 2024 at 7:02 PM Colin McCabe < > cmcc...@apache.org> > > >> > wrote: > > >> > >>>> > > >> > >>>>> Hi Kafka developers and friends, > > >> > >>>>> > > >> > >>>>> As promised, we now have a release branch for the upcoming > 3.9.0 > > >> > release. > > >> > >>>>> Trunk has been bumped to 4.0.0-SNAPSHOT. > > >> > >>>>> > > >> > >>>>> I'll be going over the JIRAs to move every non-blocker from > this > > >> > release > > >> > >>>> to > > >> > >>>>> the next release. > > >> > >>>>> > > >> > >>>>> From this point, most changes should go to trunk. > > >> > >>>>> *Blockers (existing and new that we discover while testing the > > >> > release) > > >> > >>>>> will be double-committed. *Please discuss with your reviewer > > >> whether > > >> > your > > >> > >>>>> PR should go to trunk or to trunk+release so they can merge > > >> > accordingly. > > >> > >>>>> > > >> > >>>>> *Please help us test the release! * > > >> > >>>>> > > >> > >>>>> best, > > >> > >>>>> Colin > > >> > >>>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> -- > > >> > >>>> -José > > >> > >>>> > > >> > > > >> > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Graduation steps for Features
Hi, I think we can keep the KIP template as is and have the extra process in the Jira area. EA, Preview and Production Ready (I also kind of like this better) feel to me more project management while the KIP document itself is more about the nature of the changes and describing the end result. Except for some KIPs that inherently describe a process delivered in stages. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Wed, Jul 31, 2024, 21:12 Andrew Schofield wrote: > Hi, > I would at least like to see agreement on the various stages. I’m used to > Early Access, then Preview, and finally General Availability, but I notice > that > at least one other person on this thread had the order different. Also, > General Availability sounds a bit wrong for an open-source project, so > maybe > Production Ready is better. > > I’d also like to see some guidelines for what to expect for features in the > various stages. For example, prior to Production Ready, I would not expect > a feature to be turned on by default for a new broker. > > The point about barrier to entry is fair, but I think this really only > applies to > a handful of large KIPs which take several releases to come to fruition. > I’m sure we didn’t know when KRaft began which release would be the first > in which it would be production ready. I’m sure we decided to release some > early > code long before that and having a proper name for that stage seems > appropriate. > > Thanks, > Andrew > > > On 31 Jul 2024, at 18:30, Chris Egerton wrote: > > > > I'd be hesitant to add too much to the KIP template. We should be careful > > to keep the barrier to entry low for new users. > > > > On Wed, Jul 31, 2024, 13:24 Kirk True wrote: > > > >> Hi Josep, > >> > >> Thanks for starting this conversation! > >> > >> Yes, for KIPs that span multiple releases, clearly defining the KIP’s > >> state for each release is very important. Perhaps the KIP template > could be > >> updated to include a “Release Plan” section that includes entries that > >> include: > >> > >> - Version: future Apache Kafka release > >> - Status: one of Preview, Early access, Production, etc. > >> - Jira: umbrella Jira under which other Jiras relevant to that release > >> are kept > >> - Notes: Free-form notes as needed by KIP author > >> > >> Using KIP-848 as an example, the “Release Plan” section could look like > >> this: > >> > >> - 3.7 > >> - Status: Early access > >> - Jira: KAFKA-123 > >> - Notes: includes support for user-assigned partitions but not > >> subscriptions, server-side regex not included > >> - 3.8 > >> - Status: Preview > >> - Jira: KAFKA-234 > >> - Notes: includes support for subscriptions and lots of bug fixes > >> - 4.0 > >> - Status: Production > >> - Jira: KAFKA-345 > >> - Notes: includes feature parity with “CLASSIC” consumer groups > >> > >> Of course, the contents of the “Release Plan” section may need to change > >> over time, and it’s important that the KIP author(s) update it > accordingly. > >> > >> Just a thought. > >> > >> Thanks, > >> Kirk > >> > >>> On Jul 31, 2024, at 9:43 AM, Josep Prat > >> wrote: > >>> > >>> Hi Colin, > >>> > >>> I'm not attached to any names, I'm happy to change them. > >>> But from what I can guess from KIP-848 the order is EA -> Preview -> GA > >>> Maybe I'm mistaken. > >>> > >>> This is why this discussion is important so we all agree on the number > of > >>> steps, order and names. > >>> > >>> I know finding a universal solution is maybe impossible but if we can > >> find > >>> one that fits most of the feature releases we had and we are planning > >>> currently would be great. > >>> > >>> Best, > >>> > >>> -- > >>> Josep Prat > >>> Open Source Engineering Director, Aiven > >>> josep.p...@aiven.io | +491715557497 | aiven.io > >>> Aiven Deutschland GmbH > >>> Alexanderufer 3-7, 10117 Berlin > >>> Geschäftsführer: Oskari Saarenmaa, Ha
Re: [DISCUSS] Graduation steps for Features
Hi Colin, I'm not attached to any names, I'm happy to change them. But from what I can guess from KIP-848 the order is EA -> Preview -> GA Maybe I'm mistaken. This is why this discussion is important so we all agree on the number of steps, order and names. I know finding a universal solution is maybe impossible but if we can find one that fits most of the feature releases we had and we are planning currently would be great. Best, ------ Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Wed, Jul 31, 2024, 18:31 Colin McCabe wrote: > Hi Josep, > > Thanks for bringing up this discussion. > > My impression was that "preview" was earlier than early access. > > In my mind, "preview" somewhat implies that the feature could be > incomplete or partial. For example, if someone said their game was a > "preview" I would expect kind of a demo. > > Early access (EA) means people can "access" it but it's "early". In other > words, use at your own risk. > > What you are calling "production ready," I think we've sometimes called > generally available (GA) in the past. I don't have a strong preference on > terms. > > I think no matter how much standardization we do, there will always be a > lot of discretion in these terms :) But maybe this helps clarify how > they've been used in the past? > > best, > Colin > > > On Wed, Jul 31, 2024, at 02:03, Josep Prat wrote: > > Hi Kafka devs, > > > > Lately we started using "early access", "production ready" and also > > "preview" to determine the grade of "production readiness" of the > features > > we deliver to our community. > > However, as far as I know, there is no official definition from the > Apache > > Kafka side on which are the graduation steps for features and what type > of > > "guarantees" each of these offer. > > > > I think we should agree on which terms we should use and what each of > these > > exactly mean in terms of reliability. So far it seems we have this > > graduation steps: > > - Early Access: Feature is just complete but not yet fully polished and > > maybe not used in production in many environments > > - Preview: Feature was early access before and it underwent at least a > > cycle of improvements and fixes and it's used in some production > > environments maybe > > - Production ready: Feature is officially released and it fulfills the > > expected initial needs > > > > Note that we don't offer any guarantees or SLA/SLO in the classical term. > > > > Is this something we can agree on? What do those terms mean to you? Do we > > need more steps? Or do we need less steps? > > > > Best, > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud> > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B >
Re: [DISCUSS] Graduation steps for Features
That sounds like a good idea. It will definitely clarify upfront what it means "Early Access", "Preview" and "Production Ready" for each KIP. I was looking at JIra and it seems we only have a 2 level hierarchy: Task(different types of) and Sub-Task. So we would need to change this. I was also thinking if we could use epics for this, but Epics seem to be only able to contain tasks and not sub-tasks. Obviously this would only make sense for big KIPs. Another thing that we would need is how the release manager can easily see which KIPs change graduation step in the given release. The easy way would be use the milestone Jiras (if we can have them) containing the KIP number in the title to be easily visible. Best, On Wed, Jul 31, 2024 at 5:08 PM David Jacot wrote: > We could also use a hierarchie: KIP Parent Jira > Milestones Jiras > Tasks. > > Best, > David > > On Wed, Jul 31, 2024 at 4:22 PM Josep Prat > wrote: > > > Hi David, > > > > One of the problems I see is that the KIP index page has a 1-to-many > > relationship between KIP and release. I guess we might want to turn this > to > > a many-to-many qualified relationship. Otherwise it might be complicated > > for the release manager or the KIP driver(s) to keep the Release Plan > page > > up-to-date for the different steps. > > > > Another alternative would be to have special sub-tasks in JIRA that would > > indicate the state of the KIP, then using the "fixed version" label > they'll > > be included in the release notes and the Release Manager can look for > these > > special ones when writing announcements or making sure the release notes > > are up-to-date. > > > > Best, > > > > On Wed, Jul 31, 2024 at 3:54 PM David Jacot > > > wrote: > > > > > Hi Josep, > > > > > > Thanks for starting the discussion. > > > > > > We used Early Access, Preview and GA (or Production Ready) for KIP-848 > > and > > > I find it pretty nice. We could add the tentative release plan to the > > KIP's > > > header and it could be used as the source of truth. > > > > > > Best, > > > David > > > > > > On Wed, Jul 31, 2024 at 11:53 AM Josep Prat > > > > > > wrote: > > > > > > > Hi Andrew, > > > > I can definitely write a KIP, but before doing so I'd like to gather > > some > > > > feedback from the community around these steps and how they are > > perceived > > > > by different groups of people. > > > > > > > > On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield < > > > > andrew_schofi...@live.com> > > > > wrote: > > > > > > > > > Hi Josep, > > > > > I think it’s high time that this was tackled. I suggest that it > would > > > be > > > > > best > > > > > handled as a KIP because then we have a document which can be > > discussed > > > > > and improved, followed by a formal vote. > > > > > > > > > > A standard set of terms with agreed meanings would be very helpful > > for > > > > > some of the larger KIPs which take many releases to be properly > ready > > > for > > > > > prime time. Most KIPs don’t need this, but a handful definitely do. > > > > > > > > > > Personally, I like the sequence that KIP-848 has taken, moving from > > > Early > > > > > Access, to Preview, and finally complete. I intend to follow the > same > > > > > sequence > > > > > for KIP-932. > > > > > > > > > > Thanks, > > > > > Andrew > > > > > > > > > > > On 31 Jul 2024, at 10:15, Josep Prat > > > > > > wrote: > > > > > > > > > > > > Also as part of this discussion I would like to flag that we need > > to > > > be > > > > > > able to know how we can flag this properly so it's known for the > > > > Release > > > > > > Manager. > > > > > > For example, a KIP is approved, the Jira associated with it is > > being > > > > > worked > > > > > > on. Release happens, Jira is still open, how can we flag that > this > > > KIP > > > > is > > > > > > in early access, or preview? > > > > > > > > > > > > Best, > > > > > > > > > &g
Re: [DISCUSS] Graduation steps for Features
Hi David, One of the problems I see is that the KIP index page has a 1-to-many relationship between KIP and release. I guess we might want to turn this to a many-to-many qualified relationship. Otherwise it might be complicated for the release manager or the KIP driver(s) to keep the Release Plan page up-to-date for the different steps. Another alternative would be to have special sub-tasks in JIRA that would indicate the state of the KIP, then using the "fixed version" label they'll be included in the release notes and the Release Manager can look for these special ones when writing announcements or making sure the release notes are up-to-date. Best, On Wed, Jul 31, 2024 at 3:54 PM David Jacot wrote: > Hi Josep, > > Thanks for starting the discussion. > > We used Early Access, Preview and GA (or Production Ready) for KIP-848 and > I find it pretty nice. We could add the tentative release plan to the KIP's > header and it could be used as the source of truth. > > Best, > David > > On Wed, Jul 31, 2024 at 11:53 AM Josep Prat > wrote: > > > Hi Andrew, > > I can definitely write a KIP, but before doing so I'd like to gather some > > feedback from the community around these steps and how they are perceived > > by different groups of people. > > > > On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield < > > andrew_schofi...@live.com> > > wrote: > > > > > Hi Josep, > > > I think it’s high time that this was tackled. I suggest that it would > be > > > best > > > handled as a KIP because then we have a document which can be discussed > > > and improved, followed by a formal vote. > > > > > > A standard set of terms with agreed meanings would be very helpful for > > > some of the larger KIPs which take many releases to be properly ready > for > > > prime time. Most KIPs don’t need this, but a handful definitely do. > > > > > > Personally, I like the sequence that KIP-848 has taken, moving from > Early > > > Access, to Preview, and finally complete. I intend to follow the same > > > sequence > > > for KIP-932. > > > > > > Thanks, > > > Andrew > > > > > > > On 31 Jul 2024, at 10:15, Josep Prat > > > wrote: > > > > > > > > Also as part of this discussion I would like to flag that we need to > be > > > > able to know how we can flag this properly so it's known for the > > Release > > > > Manager. > > > > For example, a KIP is approved, the Jira associated with it is being > > > worked > > > > on. Release happens, Jira is still open, how can we flag that this > KIP > > is > > > > in early access, or preview? > > > > > > > > Best, > > > > > > > > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat > > wrote: > > > > > > > >> Hi Kafka devs, > > > >> > > > >> Lately we started using "early access", "production ready" and also > > > >> "preview" to determine the grade of "production readiness" of the > > > features > > > >> we deliver to our community. > > > >> However, as far as I know, there is no official definition from the > > > Apache > > > >> Kafka side on which are the graduation steps for features and what > > type > > > of > > > >> "guarantees" each of these offer. > > > >> > > > >> I think we should agree on which terms we should use and what each > of > > > >> these exactly mean in terms of reliability. So far it seems we have > > this > > > >> graduation steps: > > > >> - Early Access: Feature is just complete but not yet fully polished > > and > > > >> maybe not used in production in many environments > > > >> - Preview: Feature was early access before and it underwent at > least a > > > >> cycle of improvements and fixes and it's used in some production > > > >> environments maybe > > > >> - Production ready: Feature is officially released and it fulfills > the > > > >> expected initial needs > > > >> > > > >> Note that we don't offer any guarantees or SLA/SLO in the classical > > > term. > > > >> > > > >> Is this something we can agree on? What do those terms mean to you? > Do > > > we > > > >> need more steps? Or do we need less st
Re: [DISCUSS] Docker Artifact Generation during Release
Thanks Vedarth! On Wed, Jul 31, 2024 at 12:24 PM Vedarth Sharma wrote: > Hi Josep, > > I agree with the suggestions. We can consider updating the pipelines to > generate both artifacts from a single job and make the pipelines driven by > RC number. > I have created this jira for tracking the same > https://issues.apache.org/jira/browse/KAFKA-17226 > > Thanks and regards, > Vedarth > > On Wed, Jul 31, 2024 at 12:59 PM Josep Prat > wrote: > > > Hi Kafka Devs, > > In Kafka 3.7 we added the JVM Docker images in the release process. Then, > > in 3.8 we added the Native ones. > > After having done 4 release candidates for 3.8 I collected some > experience > > on the process and I want to start a discussion on how we can optimize > > this. > > > > Currently, for each image we need to first build the image and then once > > it's finished, promote it as RC. This needs to be done independently for > > the JVM and for the Native images. The promotion workflow needs 3 > different > > fields, 2 of which are really linked (type of the image and the name of > > it). > > An example run would be: > > - type: jvm > > - RC image: apache/kafka:3.8.0-rc3 > > - Kafka url: > > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz > > While for native it would be: > > - type: native > > - RC image: apache/kafka-native:3.8.0-rc3 > > - Kafka url: > > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz > > > > It is really easy to make a mistake and select a native one with the > wrong > > image name (or vice versa). > > > > The changes that I think would make sense to perform to the workflows > are: > > - Have a combined flow for generating both images at the same time (we > > always need to generate both) > > -- This should internally call the 2 workflows > > - Have a combined flow for promoting images to RC > > -- This should take then the RC number instead of the full name as the > > name of the image is fixed > > > > A note on the process as well, the generation of the Native image takes > > almost an hour, so for future release managers, take some time when you > > generate them. > > > > Any thoughts about it? > > > > Best, > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Graduation steps for Features
Hi Andrew, I can definitely write a KIP, but before doing so I'd like to gather some feedback from the community around these steps and how they are perceived by different groups of people. On Wed, Jul 31, 2024 at 11:50 AM Andrew Schofield wrote: > Hi Josep, > I think it’s high time that this was tackled. I suggest that it would be > best > handled as a KIP because then we have a document which can be discussed > and improved, followed by a formal vote. > > A standard set of terms with agreed meanings would be very helpful for > some of the larger KIPs which take many releases to be properly ready for > prime time. Most KIPs don’t need this, but a handful definitely do. > > Personally, I like the sequence that KIP-848 has taken, moving from Early > Access, to Preview, and finally complete. I intend to follow the same > sequence > for KIP-932. > > Thanks, > Andrew > > > On 31 Jul 2024, at 10:15, Josep Prat > wrote: > > > > Also as part of this discussion I would like to flag that we need to be > > able to know how we can flag this properly so it's known for the Release > > Manager. > > For example, a KIP is approved, the Jira associated with it is being > worked > > on. Release happens, Jira is still open, how can we flag that this KIP is > > in early access, or preview? > > > > Best, > > > > On Wed, Jul 31, 2024 at 11:03 AM Josep Prat wrote: > > > >> Hi Kafka devs, > >> > >> Lately we started using "early access", "production ready" and also > >> "preview" to determine the grade of "production readiness" of the > features > >> we deliver to our community. > >> However, as far as I know, there is no official definition from the > Apache > >> Kafka side on which are the graduation steps for features and what type > of > >> "guarantees" each of these offer. > >> > >> I think we should agree on which terms we should use and what each of > >> these exactly mean in terms of reliability. So far it seems we have this > >> graduation steps: > >> - Early Access: Feature is just complete but not yet fully polished and > >> maybe not used in production in many environments > >> - Preview: Feature was early access before and it underwent at least a > >> cycle of improvements and fixes and it's used in some production > >> environments maybe > >> - Production ready: Feature is officially released and it fulfills the > >> expected initial needs > >> > >> Note that we don't offer any guarantees or SLA/SLO in the classical > term. > >> > >> Is this something we can agree on? What do those terms mean to you? Do > we > >> need more steps? Or do we need less steps? > >> > >> Best, > >> -- > >> [image: Aiven] <https://www.aiven.io/> > >> > >> *Josep Prat* > >> Open Source Engineering Director, *Aiven* > >> josep.p...@aiven.io | +491715557497 > >> aiven.io <https://www.aiven.io/> | > >> <https://www.facebook.com/aivencloud> > >> <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > >> *Aiven Deutschland GmbH* > >> Alexanderufer 3-7, 10117 Berlin > >> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >> Anna Richardson, Kenneth Chen > >> Amtsgericht Charlottenburg, HRB 209739 B > >> > > > > > > -- > > [image: Aiven] <https://www.aiven.io/> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io/> | < > https://www.facebook.com/aivencloud> > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Graduation steps for Features
Also as part of this discussion I would like to flag that we need to be able to know how we can flag this properly so it's known for the Release Manager. For example, a KIP is approved, the Jira associated with it is being worked on. Release happens, Jira is still open, how can we flag that this KIP is in early access, or preview? Best, On Wed, Jul 31, 2024 at 11:03 AM Josep Prat wrote: > Hi Kafka devs, > > Lately we started using "early access", "production ready" and also > "preview" to determine the grade of "production readiness" of the features > we deliver to our community. > However, as far as I know, there is no official definition from the Apache > Kafka side on which are the graduation steps for features and what type of > "guarantees" each of these offer. > > I think we should agree on which terms we should use and what each of > these exactly mean in terms of reliability. So far it seems we have this > graduation steps: > - Early Access: Feature is just complete but not yet fully polished and > maybe not used in production in many environments > - Preview: Feature was early access before and it underwent at least a > cycle of improvements and fixes and it's used in some production > environments maybe > - Production ready: Feature is officially released and it fulfills the > expected initial needs > > Note that we don't offer any guarantees or SLA/SLO in the classical term. > > Is this something we can agree on? What do those terms mean to you? Do we > need more steps? Or do we need less steps? > > Best, > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > Anna Richardson, Kenneth Chen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
[DISCUSS] Graduation steps for Features
Hi Kafka devs, Lately we started using "early access", "production ready" and also "preview" to determine the grade of "production readiness" of the features we deliver to our community. However, as far as I know, there is no official definition from the Apache Kafka side on which are the graduation steps for features and what type of "guarantees" each of these offer. I think we should agree on which terms we should use and what each of these exactly mean in terms of reliability. So far it seems we have this graduation steps: - Early Access: Feature is just complete but not yet fully polished and maybe not used in production in many environments - Preview: Feature was early access before and it underwent at least a cycle of improvements and fixes and it's used in some production environments maybe - Production ready: Feature is officially released and it fulfills the expected initial needs Note that we don't offer any guarantees or SLA/SLO in the classical term. Is this something we can agree on? What do those terms mean to you? Do we need more steps? Or do we need less steps? Best, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
[DISCUSS] Docker Artifact Generation during Release
Hi Kafka Devs, In Kafka 3.7 we added the JVM Docker images in the release process. Then, in 3.8 we added the Native ones. After having done 4 release candidates for 3.8 I collected some experience on the process and I want to start a discussion on how we can optimize this. Currently, for each image we need to first build the image and then once it's finished, promote it as RC. This needs to be done independently for the JVM and for the Native images. The promotion workflow needs 3 different fields, 2 of which are really linked (type of the image and the name of it). An example run would be: - type: jvm - RC image: apache/kafka:3.8.0-rc3 - Kafka url: https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz While for native it would be: - type: native - RC image: apache/kafka-native:3.8.0-rc3 - Kafka url: https://home.apache.org/~jlprat/kafka-3.8.0-rc3/kafka_2.13-3.8.0.tgz It is really easy to make a mistake and select a native one with the wrong image name (or vice versa). The changes that I think would make sense to perform to the workflows are: - Have a combined flow for generating both images at the same time (we always need to generate both) -- This should internally call the 2 workflows - Have a combined flow for promoting images to RC -- This should take then the RC number instead of the full name as the name of the image is fixed A note on the process as well, the generation of the Native image takes almost an hour, so for future release managers, take some time when you generate them. Any thoughts about it? Best, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: New release branch 3.9
Hi Matthias, Note that is KIP-853 the one with the feature parity with ZK. KIP-1012 was the one were we agreed to stay on 3.x until feature parity. The reason to have a 3.10 is to have a safe way to upgrade to KRaft Kafkas while ZK is still around. We try to explain this in KIP-1012 -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Tue, Jul 30, 2024, 22:10 Matthias J. Sax wrote: > Thanks for the input. However, I am wondering if releasing 3.9 makes > sense if KIP-1012 won't make it? > > My understanding was we do 3.9 only to not delay 3.8 and not delay 4.0. > > If we would go with a 3.10, would it also be an "intermediate" release > like 3.9? Or would it replace 4.0? For the first case, why not just > delay 3.9 until everything is ready? And for the latest case, why not > just do the 3.9 release in Nov and drop an intermediate release all > together? > > What do we gain by a 3.10 release? > > > -Matthias > > On 7/30/24 10:26 AM, Josep Prat wrote: > > +1 Greg. I'd be really happy to bump trunk to 4.0.0, but only once we > know > > we can safely do so. > > > > On Tue, Jul 30, 2024 at 7:24 PM Greg Harris > > > wrote: > > > >> Hi all, > >> > >> I agree that we are not yet ready for breaking changes on trunk, so I > >> opened a PR to bump to 3.10.0-SNAPSHOT: > >> https://github.com/apache/kafka/pull/16732 > >> > >> When KIP-853 is feature complete, we can bump to 4.0.0-SNAPSHOT. > >> > >> Thanks, > >> Greg > >> > >> On Tue, Jul 30, 2024 at 10:01 AM Josep Prat > > >> wrote: > >> > >>> Hi all, > >>> As per KIP-1012[1] we can't yet say if the next release will be 3.10.0 > or > >>> 4.0.0. It will come down to the state of KIP-853 in 3.9.0. > >>> > >>> So, in my opinion we should still wait before committing breaking > changes > >>> on trunk until we know for sure that KIP-853 will make it. > >>> Maybe Jose can share more about the chances of this. > >>> > >>> [1] > >>> > >>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release > >>> > >>> Best > >>> > >>> > >>> -- > >>> Josep Prat > >>> Open Source Engineering Director, Aiven > >>> josep.p...@aiven.io | +491715557497 | aiven.io > >>> Aiven Deutschland GmbH > >>> Alexanderufer 3-7, 10117 Berlin > >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > >>> Anna Richardson, Kenneth Chen > >>> Amtsgericht Charlottenburg, HRB 209739 B > >>> > >>> On Tue, Jul 30, 2024, 18:52 Matthias J. Sax wrote: > >>> > >>>> Thanks for cutting the release branch. > >>>> > >>>> It's great to see `trunk` being bumped to 4.0-SNAPSHOT, and I wanted > to > >>>> follow up on this: > >>>> > >>>> We have a bunch of tickets that we can only ship with 4.0 release, and > >>>> these tickets were blocked so far. I wanted to get confirmation that > we > >>>> will stick with 4.0 coming after 3.9, and that we can start to work on > >>>> these tickets? Or is there any reason why we should still hold off to > >>>> pick them up? We don't want to delay them unnecessary to make sure we > >>>> can them all into 4.0 release, but of course also don't want to work > on > >>>> them prematurely (to avoid that we have to revert them after merging). > >>>> > >>>> > >>>> -Matthias > >>>> > >>>> On 7/30/24 9:07 AM, Chia-Ping Tsai wrote: > >>>>> hi Colin, > >>>>> > >>>>> Could you please consider adding > >>>>> https://issues.apache.org/jira/browse/KAFKA-1 to 3.9.0 > >>>>> > >>>>> The issue is used to deprecate the formatters in core module. Also, > >> it > >>>>> implements the replacements for them. > >>>>> > >>>>> In order to follow the deprecation rules, it would be nice to have > >>>>> KAFKA-1 in 3.9.0 > >>>>> > >>
Re: New release branch 3.9
+1 Greg. I'd be really happy to bump trunk to 4.0.0, but only once we know we can safely do so. On Tue, Jul 30, 2024 at 7:24 PM Greg Harris wrote: > Hi all, > > I agree that we are not yet ready for breaking changes on trunk, so I > opened a PR to bump to 3.10.0-SNAPSHOT: > https://github.com/apache/kafka/pull/16732 > > When KIP-853 is feature complete, we can bump to 4.0.0-SNAPSHOT. > > Thanks, > Greg > > On Tue, Jul 30, 2024 at 10:01 AM Josep Prat > wrote: > > > Hi all, > > As per KIP-1012[1] we can't yet say if the next release will be 3.10.0 or > > 4.0.0. It will come down to the state of KIP-853 in 3.9.0. > > > > So, in my opinion we should still wait before committing breaking changes > > on trunk until we know for sure that KIP-853 will make it. > > Maybe Jose can share more about the chances of this. > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release > > > > Best > > > > > > -- > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Tue, Jul 30, 2024, 18:52 Matthias J. Sax wrote: > > > > > Thanks for cutting the release branch. > > > > > > It's great to see `trunk` being bumped to 4.0-SNAPSHOT, and I wanted to > > > follow up on this: > > > > > > We have a bunch of tickets that we can only ship with 4.0 release, and > > > these tickets were blocked so far. I wanted to get confirmation that we > > > will stick with 4.0 coming after 3.9, and that we can start to work on > > > these tickets? Or is there any reason why we should still hold off to > > > pick them up? We don't want to delay them unnecessary to make sure we > > > can them all into 4.0 release, but of course also don't want to work on > > > them prematurely (to avoid that we have to revert them after merging). > > > > > > > > > -Matthias > > > > > > On 7/30/24 9:07 AM, Chia-Ping Tsai wrote: > > > > hi Colin, > > > > > > > > Could you please consider adding > > > > https://issues.apache.org/jira/browse/KAFKA-1 to 3.9.0 > > > > > > > > The issue is used to deprecate the formatters in core module. Also, > it > > > > implements the replacements for them. > > > > > > > > In order to follow the deprecation rules, it would be nice to have > > > > KAFKA-1 in 3.9.0 > > > > > > > > If you agree to have them in 3.9.0, I will cherry-pick them into > 3.9.0 > > > when > > > > they get merged to trunk. > > > > > > > > Best, > > > > Chia-Ping > > > > > > > > > > > > José Armando García Sancio 於 > 2024年7月30日 > > > 週二 > > > > 下午11:59寫道： > > > > > > > >> Thanks Colin. > > > >> > > > >> For KIP-853 (KRaft Controller Membership Changes), we still have the > > > >> following features that are in progress. > > > >> > > > >> 1. UpdateVoter RPC and request handling > > > >> <https://issues.apache.org/jira/browse/KAFKA-16533> > > > >> 2. Storage tool changes for KIP-853 > > > >> <https://issues.apache.org/jira/browse/KAFKA-16518> > > > >> 3. kafka-metadata-quorum describe changes for KIP-853 > > > >> <https://issues.apache.org/jira/browse/KAFKA-16521> > > > >> 4. kafka-metadata-quorum add voter and remove voter changes > > > >> <https://issues.apache.org/jira/browse/KAFKA-16523> > > > >> 5. Sending UpdateVoter request and response handling > > > >> <https://issues.apache.org/jira/browse/KAFKA-16534> > > > >> > > > >> Can we cherry pick them to the release branch 3.9.0 when they get > > > merged to > > > >> trunk? They have a small impact as they shouldn't affect the rest of > > > Kafka > > > >> and only affect the kraft controller membership change feature. I > > > expected > > > >> them to get merged to the trunk branch in the coming days. > > > >> > > > >> Thanks, &g
Re: New release branch 3.9
Hi all, As per KIP-1012[1] we can't yet say if the next release will be 3.10.0 or 4.0.0. It will come down to the state of KIP-853 in 3.9.0. So, in my opinion we should still wait before committing breaking changes on trunk until we know for sure that KIP-853 will make it. Maybe Jose can share more about the chances of this. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release Best -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B On Tue, Jul 30, 2024, 18:52 Matthias J. Sax wrote: > Thanks for cutting the release branch. > > It's great to see `trunk` being bumped to 4.0-SNAPSHOT, and I wanted to > follow up on this: > > We have a bunch of tickets that we can only ship with 4.0 release, and > these tickets were blocked so far. I wanted to get confirmation that we > will stick with 4.0 coming after 3.9, and that we can start to work on > these tickets? Or is there any reason why we should still hold off to > pick them up? We don't want to delay them unnecessary to make sure we > can them all into 4.0 release, but of course also don't want to work on > them prematurely (to avoid that we have to revert them after merging). > > > -Matthias > > On 7/30/24 9:07 AM, Chia-Ping Tsai wrote: > > hi Colin, > > > > Could you please consider adding > > https://issues.apache.org/jira/browse/KAFKA-1 to 3.9.0 > > > > The issue is used to deprecate the formatters in core module. Also, it > > implements the replacements for them. > > > > In order to follow the deprecation rules, it would be nice to have > > KAFKA-1 in 3.9.0 > > > > If you agree to have them in 3.9.0, I will cherry-pick them into 3.9.0 > when > > they get merged to trunk. > > > > Best, > > Chia-Ping > > > > > > José Armando García Sancio 於 2024年7月30日 > 週二 > > 下午11:59寫道： > > > >> Thanks Colin. > >> > >> For KIP-853 (KRaft Controller Membership Changes), we still have the > >> following features that are in progress. > >> > >> 1. UpdateVoter RPC and request handling > >> <https://issues.apache.org/jira/browse/KAFKA-16533> > >> 2. Storage tool changes for KIP-853 > >> <https://issues.apache.org/jira/browse/KAFKA-16518> > >> 3. kafka-metadata-quorum describe changes for KIP-853 > >> <https://issues.apache.org/jira/browse/KAFKA-16521> > >> 4. kafka-metadata-quorum add voter and remove voter changes > >> <https://issues.apache.org/jira/browse/KAFKA-16523> > >> 5. Sending UpdateVoter request and response handling > >> <https://issues.apache.org/jira/browse/KAFKA-16534> > >> > >> Can we cherry pick them to the release branch 3.9.0 when they get > merged to > >> trunk? They have a small impact as they shouldn't affect the rest of > Kafka > >> and only affect the kraft controller membership change feature. I > expected > >> them to get merged to the trunk branch in the coming days. > >> > >> Thanks, > >> > >> On Mon, Jul 29, 2024 at 7:02 PM Colin McCabe > wrote: > >> > >>> Hi Kafka developers and friends, > >>> > >>> As promised, we now have a release branch for the upcoming 3.9.0 > release. > >>> Trunk has been bumped to 4.0.0-SNAPSHOT. > >>> > >>> I'll be going over the JIRAs to move every non-blocker from this > release > >> to > >>> the next release. > >>> > >>> From this point, most changes should go to trunk. > >>> *Blockers (existing and new that we discover while testing the release) > >>> will be double-committed. *Please discuss with your reviewer whether > your > >>> PR should go to trunk or to trunk+release so they can merge > accordingly. > >>> > >>> *Please help us test the release! * > >>> > >>> best, > >>> Colin > >>> > >> > >> > >> -- > >> -José > >> > > >
[jira] [Created] (KAFKA-17214) Add 3.8.0 Streams and Core to system tests
Josep Prat created KAFKA-17214: -- Summary: Add 3.8.0 Streams and Core to system tests Key: KAFKA-17214 URL: https://issues.apache.org/jira/browse/KAFKA-17214 Project: Kafka Issue Type: Bug Reporter: Josep Prat As per Release Instructions we should add 3.8.0 version to system tests. Example PRs: * Broker and clients: [https://github.com/apache/kafka/pull/12210] * Streams: [https://github.com/apache/kafka/pull/12209] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[ANNOUNCE] Apache Kafka 3.8.0
The Apache Kafka community is pleased to announce the release for Apache Kafka 3.8.0 This is a minor release and it includes fixes and improvements from 456 JIRAs. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.8.0/RELEASE_NOTES.html An overview of the release can be found in our announcement blog post: https://kafka.apache.org/blog#apache_kafka_380_release_announcement You can download the source and binary release (Scala 2.12 and Scala 2.13) from: https://kafka.apache.org/downloads#3.8.0 --- Apache Kafka is a distributed streaming platform with four core APIs: ** The Producer API allows an application to publish a stream of records to one or more Kafka topics. ** The Consumer API allows an application to subscribe to one or more topics and process the stream of records produced to them. ** The Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. ** The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. With these APIs, Kafka can be used for two broad classes of application: ** Building real-time streaming data pipelines that reliably get data between systems or applications. ** Building real-time streaming applications that transform or react to the streams of data. Apache Kafka is in use at large and small companies worldwide, including Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 202 contributors to this release! (Please report an unintended omission) Aadithya Chandra, Abhijeet Kumar, Abhinav Dixit, Adrian Preston, Afshin Moazami, Ahmed Najiub, Ahmed Sobeh, Akhilesh Chaganti, Almog Gavra, Alok Thatikunta, Alyssa Huang, Anatoly Popov, Andras Katona, Andrew Schofield, Anna Sophie Blee-Goldman, Antoine Pourchet, Anton Agestam, Anton Liauchuk, Anuj Sharma, Apoorv Mittal, Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj, Ayoub Omari, Bruno Cadonna, Calvin Liu, Cameron Redpath, charliecheng630, Cheng-Kai, Zhang, Cheryl Simmons, Chia Chuan Yu, Chia-Ping Tsai, ChickenchickenLove, Chris Egerton, Chris Holland, Christo Lolov, Christopher Webb, Colin P. McCabe, Colt McNealy, cooper.ts...@suse.com, Vedarth Sharma, Crispin Bernier, Daan Gerits, David Arthur, David Jacot, David Mao, dengziming, Divij Vaidya, DL1231, Dmitry Werner, Dongnuo Lyu, Drawxy, Dung Ha, Edoardo Comar, Eduwer Camacaro, Emanuele Sabellico, Erik van Oosten, Eugene Mitskevich, Fan Yang, Federico Valeri, Fiore Mario Vitale, flashmouse, Florin Akermann, Frederik Rouleau, Gantigmaa Selenge, Gaurav Narula, ghostspiders, gongxuanzhang, Greg Harris, Gyeongwon Do, Hailey Ni, Hao Li, Hector Geraldino, highluck, hudeqi, Hy (하이), IBeyondy, Iblis Lin, Igor Soarez, ilyazr, Ismael Juma, Ivan Vaskevych, Ivan Yurchenko, James Faulkner, Jamie Holmes, Jason Gustafson, Jeff Kim, jiangyuan, Jim Galasyn, Jinyong Choi, Joel Hamill, John Doe zh2725284...@gmail.com, John Roesler, John Yu, Johnny Hsu, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando García Sancio, Jun Rao, Justine Olshan, Kalpesh Patel, Kamal Chandraprakash, Ken Huang, Kirk True, Kohei Nozaki, Krishna Agarwal, KrishVora01, Kuan-Po (Cooper) Tseng, Kvicii, Lee Dongjin, Leonardo Silva, Lianet Magrans, LiangliangSui, Linu Shibu, lixinyang, Lokesh Kumar, Loïc GREFFIER, Lucas Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, mannoopj, Manyanda Chitimbo, Mario Pareja, Matthew de Detrich, Matthias Berndt, Matthias J. Sax, Matthias Sax, Max Riedel, Mayank Shekhar Narula, Michael Edgar, Michael Westerby, Mickael Maison, Mike Lloyd, Minha, Jeong, Murali Basani, n.izhikov, Nick Telford, Nikhil Ramakrishnan, Nikolay, Octavian Ciubotaru, Okada Haruki, Omnia G.H Ibrahim, Ori Hoch, Owen Leung, Paolo Patierno, Philip Nee, Phuc-Hong-Tran, PoAn Yang, Proven Provenzano, Qichao Chu, Ramin Gharib, Ritika Reddy, Rittika Adhikari, Rohan, Ron Dagostino, runom, rykovsi, Sagar Rao, Said Boudjelda, sanepal, Sanskar Jhajharia, Satish Duggana, Sean Quah, Sebastian Marsching, Sebastien Viale, Sergio Troiano, Sid Yagnik, Stanislav Kozlovski, Stig Døssing, Sudesh Wasnik, TaiJuWu, TapDang, testn, TingIāu "Ting" Kì, vamossagar12, Vedarth Sharma, Victor van den Hoven, Vikas Balani, Viktor Somogyi-Vass, Vincent Rose, Walker Carlson, wernerdv, Yang Yu, Yash Mayya, yicheny, Yu-Chen Lai, yuz10, Zhifeng Chen, Zihao Lin, Ziming Deng, 谭九鼎 We welcome your help and feedback. For more information on how to repor
Re: Help for contributing
Oops! Funny enough, both links work for me... On Mon, Jul 29, 2024 at 6:25 PM Chia-Ping Tsai wrote: > It seems the link offered by Josep has a small typo. Let's me fix it: > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20KAFKA%20AND%20status%20%3D%20Open%20AND%20labels%20%3D%20newbie%20AND%20assignee%20in%20(EMPTY) > > Best, > Chia-Ping > > Josep Prat 於 2024年7月29日 週一 下午11:01寫道： > > > Hello there, > > You can also take a look at this filter in Jira: > > > > > https://issues.apache.org/jira/browse/KAFKA-17201?jql=project+%3D+KAFKA+AND+status+%3D+Open+AND+labels+%3D+newbie+AND+assignee+in+%28EMPTY%29 > > This shows you all unassigned issues in Kafka that have the label > "newbie". > > Issues under this label are usually good first issues. > > > > Best, > > > > On Sun, Jul 28, 2024 at 3:27 PM jiang dou wrote: > > > > > Hello： > > > > > > You can find out how to contribute here： > > > https://kafka.apache.org/contributing > > > > > > Thank you > > > > > > meisam jafari 于2024年7月28日周日 13:50写道： > > > > > > > Hello there, > > > > > > > > I am very enthusiastic to contributing to the kafka, recently I read > > the > > > > kafka definitive guide book and got insights how kafka works, then I > > got > > > > kafka source code and built it and ran some unit tests. Now I want to > > > work > > > > on some real issues to start my gurney of becoming a kafka comitter. > > > Could > > > > you please help me how to take issues in jira to get started?. > > > > > > > > Thanks > > > > > > > > > > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, > > Anna Richardson, Kenneth Chen > > Amtsgericht Charlottenburg, HRB 209739 B > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: Help for contributing
Hello there, You can also take a look at this filter in Jira: https://issues.apache.org/jira/browse/KAFKA-17201?jql=project+%3D+KAFKA+AND+status+%3D+Open+AND+labels+%3D+newbie+AND+assignee+in+%28EMPTY%29 This shows you all unassigned issues in Kafka that have the label "newbie". Issues under this label are usually good first issues. Best, On Sun, Jul 28, 2024 at 3:27 PM jiang dou wrote: > Hello： > > You can find out how to contribute here： > https://kafka.apache.org/contributing > > Thank you > > meisam jafari 于2024年7月28日周日 13:50写道： > > > Hello there, > > > > I am very enthusiastic to contributing to the kafka, recently I read the > > kafka definitive guide book and got insights how kafka works, then I got > > kafka source code and built it and ran some unit tests. Now I want to > work > > on some real issues to start my gurney of becoming a kafka comitter. > Could > > you please help me how to take issues in jira to get started?. > > > > Thanks > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] 3.8.0 RC3
Thank you all who reviewed this release candidate and the previous ones.
This vote passes with 7 +1 votes (3 bindings) and no 0 or -1 votes.
+1 votes
PMC Members:
* Chris Egerton
* Greg Harris
* Mickael Maison
Committers:
* Igor Soarez
Community:
* Murali Basani
* Jakub Scholz
* Krishna Agarwal
0 votes
* No votes
-1 votes
* No votes
I'll continue with the release process and the release announcement will
follow in the next few days.
Best,
On Thu, Jul 25, 2024 at 7:44 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:
> Hi Josep,
>
> Thanks for the release candidate.
>
> +1 (non-binding)
>
> I tested and verified the docker image artifact
> apache/kafka-native:3.8.0-rc3
> - verified create topic, produce messages and consume messages flow.
> - verified the html documentation for docker image.
>
> I also ran the System tests by bringing up Kafka in the native mode.
> Following tests failed in the run.
>
> -
> >
> tests/kafkatest/tests/tools/replica_verification_test.py::ReplicaVerificationToolTest.test_replica_lags@
> > {"metadata_quorum":"ISOLATED_KRAFT"}
> > -
> >
> tests/kafkatest/tests/tools/replica_verification_test.py::ReplicaVerificationToolTest.test_replica_lags@
> > {"metadata_quorum":"ZK"}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"client-id","old_client_throttling_behavior":true}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"(user,client-id)","override_quota":true}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"client-id","old_broker_throttling_behavior":true}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"client-id","consumer_num":2}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"(user,client-id)","override_quota":false}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"client-id","override_quota":false}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"user","override_quota":true}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"client-id","override_quota":true}
> > - tests/kafkatest/tests/client/quota_test.py::QuotaTest.test_quota@
> > {"quota_type":"user","override_quota":false}
> > -
> >
> tests/kafkatest/tests/connect/connect_distributed_test.py::ConnectDistributedTest.test_dynamic_logging
> > -
> >
> tests/kafkatest/tests/connect/connect_distributed_test.py::ConnectDistributedTest.test_pause_and_resume_sink@
> >
> {"connect_protocol":"sessioned","metadata_quorum":"ISOLATED_KRAFT","use_new_coordinator":true,"group_protocol":"consumer"}
> >
>
> On re-running the failed tests locally, *all tests passed except*
>
> `tests/kafkatest/tests/tools/replica_verification_test.py::ReplicaVerificationToolTest.test_replica_lags@
> {"metadata_quorum":"ZK"}`(timeout
> error).
>
> All looks good for this docker image artifact!
>
> Regards,
> Krishna
>
> On Wed, Jul 24, 2024 at 10:46 PM Mickael Maison
> wrote:
>
> > Hi Josep,
> >
> > I have:
> > - checked signatures and checksums
> > - run ZK and KRaft quickstarts with the Scala 2.13 binaries and with
> > both the kafka and kafka-native images
> > - built and run tests from source with Java 17
> >
> > +1 (binding)
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jul 24, 2024 at 4:22 PM Igor Soarez wrote:
> > >
> > > Hi Josep,
> > >
> > > That makes sense to me, thanks for clarifying.
> > >
> > > +1 non-binding from me then.
> > >
> > > --
> > > Igor
> >
>
--
[image: Aiven] <https://www.aiven.io>
*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io | +491715557497
aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
Anna Richardson, Kenneth Chen
Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] 3.8.0 RC3
Hi Igor, You are right, the corrections we are doing on the open pull request are not in the doc artifact. But as far as I know, this shouldn't be considered a blocker for the release as the site will contain the latest and most correct information for the 3.8.x version. Thank you and also Chris and Greg for reviewing the release candidate. On Wed, Jul 24, 2024 at 12:01 PM Igor Soarez wrote: > Hi Josep, > > Thank you once again for running this release. > > I noticed that site-docs/upgrade.html in kafka_2.13-3.8.0-site-docs.tgz > does not include the corrections from > https://github.com/apache/kafka-site/pull/614 > namely the replacement of version 3.6 with 3.7 in the upgrade notes header, > and the two notices on the early access status for Tiered Storage and > JBOD. > > I also performed the following checks: > > - Verified signatures and checksums for all artifacts in > - Built from source from kafka-3.8.0-src.tgz > - Basic workload against a KRaft cluster using kafka_2.13-3.8.0.tgz > > Best, > > -- > Igor > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen, Anna Richardson, Kenneth Chen Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] 3.8.0 RC3
Here is the link to the system tests: https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-22--001.ffbb03b2-61f4-4ebb-ae1f-af5c753682fb--1721733000--confluentinc--3.8--9a2b34b68c/report.html The Quota tests are known to fail in this CI system. Regarding the other tests, they run successfully in the past and they are now timeouting. Best, On Tue, Jul 23, 2024 at 12:07 PM Josep Prat wrote: > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for release of Apache Kafka 3.8.0. > > Some of the major features included in this release are: > * KIP-1028: Docker Official Image for Apache Kafka > * KIP-974: Docker Image for GraalVM based Native Kafka Broker > * KIP-1036: Extend RecordDeserializationException exception > * KIP-1019: Expose method to determine Metric Measurability > * KIP-1004: Enforce tasks.max property in Kafka Connect > * KIP-989: Improved StateStore Iterator metrics for detecting leaks > * KIP-993: Allow restricting files accessed by File and Directory > ConfigProviders > * KIP-924: customizable task assignment for Streams > * KIP-813: Shareable State Stores > * KIP-719: Deprecate Log4J Appender > * KIP-390: Support Compression Level > * KIP-1018: Introduce max remote fetch timeout config for > DelayedRemoteFetch requests > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace > kafka.serializer.Decoder > * KIP-899: Allow producer and consumer clients to rebootstrap > > Release notes for the 3.8.0 release: > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/RELEASE_NOTES.html > > Please download, test and vote by Friday, July 26, 9am PT* > > > Kafka's KEYS file containing PGP keys we use to sign the release: > https://kafka.apache.org/KEYS > > * Release artifacts to be voted upon (source and binary): > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/ > > * Docker release artifact to be voted upon: > apache/kafka:3.8.0-rc3 > apache/kafka-native:3.8.0-rc3 > > * Maven artifacts to be voted upon: > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > * Javadoc: > https://home.apache.org/~jlprat/kafka-3.8.0-rc3/javadoc/ > > * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: > https://github.com/apache/kafka/releases/tag/3.8.0-rc3 > > * Documentation: > https://kafka.apache.org/38/documentation.html > > * Protocol: > https://kafka.apache.org/38/protocol.html > > * Successful Jenkins builds for the 3.8 branch: > Unit/integration tests: > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.8/73/ > System tests: Still running > > * Successful Docker Image Github Actions Pipeline for 3.8 branch: > Docker Build Test Pipeline (JVM): > https://github.com/apache/kafka/actions/runs/10055827182 > Docker Build Test Pipeline (Native): > https://github.com/apache/kafka/actions/runs/10055829295 > > > Thanks, > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
[VOTE] 3.8.0 RC3
Hello Kafka users, developers and client-developers, This is the fourth candidate for release of Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036: Extend RecordDeserializationException exception * KIP-1019: Expose method to determine Metric Measurability * KIP-1004: Enforce tasks.max property in Kafka Connect * KIP-989: Improved StateStore Iterator metrics for detecting leaks * KIP-993: Allow restricting files accessed by File and Directory ConfigProviders * KIP-924: customizable task assignment for Streams * KIP-813: Shareable State Stores * KIP-719: Deprecate Log4J Appender * KIP-390: Support Compression Level * KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder * KIP-899: Allow producer and consumer clients to rebootstrap Release notes for the 3.8.0 release: https://home.apache.org/~jlprat/kafka-3.8.0-rc3/RELEASE_NOTES.html Please download, test and vote by Friday, July 26, 9am PT* Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~jlprat/kafka-3.8.0-rc3/ * Docker release artifact to be voted upon: apache/kafka:3.8.0-rc3 apache/kafka-native:3.8.0-rc3 * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~jlprat/kafka-3.8.0-rc3/javadoc/ * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: https://github.com/apache/kafka/releases/tag/3.8.0-rc3 * Documentation: https://kafka.apache.org/38/documentation.html * Protocol: https://kafka.apache.org/38/protocol.html * Successful Jenkins builds for the 3.8 branch: Unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.8/73/ System tests: Still running * Successful Docker Image Github Actions Pipeline for 3.8 branch: Docker Build Test Pipeline (JVM): https://github.com/apache/kafka/actions/runs/10055827182 Docker Build Test Pipeline (Native): https://github.com/apache/kafka/actions/runs/10055829295 Thanks, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] 3.8.0 RC2
Thanks Chia-Ping, I'm cancelling this Vote thread. I'll be generating the next RC tomorrow morning CEST. Best, ------ Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Mon, Jul 22, 2024, 19:25 Chia-Ping Tsai wrote: > hi Josep, > > Sorry for bringing this news. There is a correctness issue ( > https://github.com/apache/kafka/pull/16641) and the fix should be > backport to 3.8. > > Thanks, > Chia-Ping > > On 2024/07/18 09:19:06 Josep Prat wrote: > > Hello Kafka users, developers and client-developers, > > > > This is the third candidate for release of Apache Kafka 3.8.0. > > > > Some of the major features included in this release are: > > * KIP-1028: Docker Official Image for Apache Kafka > > * KIP-974: Docker Image for GraalVM based Native Kafka Broker > > * KIP-1036: Extend RecordDeserializationException exception > > * KIP-1019: Expose method to determine Metric Measurability > > * KIP-1004: Enforce tasks.max property in Kafka Connect > > * KIP-989: Improved StateStore Iterator metrics for detecting leaks > > * KIP-993: Allow restricting files accessed by File and Directory > > ConfigProviders > > * KIP-924: customizable task assignment for Streams > > * KIP-813: Shareable State Stores > > * KIP-719: Deprecate Log4J Appender > > * KIP-390: Support Compression Level > > * KIP-1018: Introduce max remote fetch timeout config for > > DelayedRemoteFetch requests > > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission > > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace > > kafka.serializer.Decoder > > * KIP-899: Allow producer and consumer clients to rebootstrap > > > > Release notes for the 3.8.0 release: > > https://home.apache.org/~jlprat/kafka-3.8.0-rc2/RELEASE_NOTES.html > > > > Please download, test and vote by July > > 21, 9am PT>* > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~jlprat/kafka-3.8.0-rc2/ > > > > > > * Docker release artifact to be voted upon(apache/kafka-native is > supported > > from 3.8+ release.): > > apache/kafka:3.8.0-rc2 > > apache/kafka-native:3.8.0-rc2 > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: > > https://home.apache.org/~jlprat/kafka-3.8.0-rc2/javadoc/ > > > > * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: > > https://github.com/apache/kafka/releases/tag/3.8.0-rc2 > > > > * Documentation: > > https://kafka.apache.org/38/documentation.html > > > > * Protocol: > > https://kafka.apache.org/38/protocol.html > > > > * Successful Jenkins builds for the 3.8 branch: > > Unit/integration tests: > > > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/70 > > (latest one is still building, but PR passed without related failures) > > System tests: (Same as before) > > > https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html > > > > * Successful Docker Image Github Actions Pipeline for 3.8 branch: > > Docker Build Test Pipeline (JVM): > > https://github.com/apache/kafka/actions/runs/9987484582 > > Docker Build Test Pipeline (Native): > > https://github.com/apache/kafka/actions/runs/9987487177 > > > > > > > > Thanks, > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud> > > <https://www.linkedin.com/company/aiven/> < > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
[VOTE] 3.8.0 RC2
Hello Kafka users, developers and client-developers, This is the third candidate for release of Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036: Extend RecordDeserializationException exception * KIP-1019: Expose method to determine Metric Measurability * KIP-1004: Enforce tasks.max property in Kafka Connect * KIP-989: Improved StateStore Iterator metrics for detecting leaks * KIP-993: Allow restricting files accessed by File and Directory ConfigProviders * KIP-924: customizable task assignment for Streams * KIP-813: Shareable State Stores * KIP-719: Deprecate Log4J Appender * KIP-390: Support Compression Level * KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder * KIP-899: Allow producer and consumer clients to rebootstrap Release notes for the 3.8.0 release: https://home.apache.org/~jlprat/kafka-3.8.0-rc2/RELEASE_NOTES.html Please download, test and vote by * Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~jlprat/kafka-3.8.0-rc2/ * Docker release artifact to be voted upon(apache/kafka-native is supported from 3.8+ release.): apache/kafka:3.8.0-rc2 apache/kafka-native:3.8.0-rc2 * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~jlprat/kafka-3.8.0-rc2/javadoc/ * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: https://github.com/apache/kafka/releases/tag/3.8.0-rc2 * Documentation: https://kafka.apache.org/38/documentation.html * Protocol: https://kafka.apache.org/38/protocol.html * Successful Jenkins builds for the 3.8 branch: Unit/integration tests: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/70 (latest one is still building, but PR passed without related failures) System tests: (Same as before) https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html * Successful Docker Image Github Actions Pipeline for 3.8 branch: Docker Build Test Pipeline (JVM): https://github.com/apache/kafka/actions/runs/9987484582 Docker Build Test Pipeline (Native): https://github.com/apache/kafka/actions/runs/9987487177 Thanks, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.0 release
I'm starting with the process now. Best, On Thu, Jul 18, 2024 at 2:12 AM Greg Harris wrote: > Hi all, > > We found a blocker in 3.8.0-rc1 > https://issues.apache.org/jira/browse/KAFKA-17150 that is now merged to > trunk and backported to 3.8. > Additionally I've merged and backported the lower-severity > https://issues.apache.org/jira/browse/KAFKA-17148 to trunk, 3.8, and 3.7 > because Dmitry Werner provided a very prompt fix. > Thanks to Chia-Ping, Chris, and Josep for prompt reviews. At this time I am > not aware of any more blockers for 3.8.0 and we can proceed with rc2. > > Thanks! > Greg > > On Wed, Jul 10, 2024 at 10:49 PM Josep Prat > wrote: > > > Hi Greg, > > > > I'll make sure the PR with the fix ( > > https://github.com/apache/kafka/pull/16565) gets merged today. Thanks > for > > bringing this up! > > > > Best, > > > > -- > > Josep Prat > > Open Source Engineering Director, Aiven > > josep.p...@aiven.io | +491715557497 | aiven.io > > Aiven Deutschland GmbH > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > On Wed, Jul 10, 2024, 22:55 Greg Harris > > wrote: > > > > > Hi Josep, > > > > > > A contributor just raised a regression [1] that I think should be > > addressed > > > in 3.8.0 prior to the release. > > > > > > Summary: This change [2] causes multiple ERROR logs to appear during > > worker > > > startup when operators install other, unrelated plugins that package > the > > > Jackson library. > > > Severity: Workers will continue to operate normally and the errors will > > be > > > cosmetic. Operators and automatic systems that watch logs may roll-back > > > upgrades due to the perception of a severe problem. > > > Impact: I found 12 out of 250 third-party plugins that package the > > Jackson > > > library and trigger this error on upgrade. This will almost certainly > > > affect several users upon upgrades, and does not require obscure > setups. > > > Risk: The contributor has opened a simple fix PR [3], and I have > verified > > > that it addresses the problem, and can be merged tomorrow. As an > > > alternative, we can revert the performance change completely [4] but I > > want > > > to avoid this. > > > > > > With the above, what would you like to do for this release? Merge the > > fix, > > > revert, or leave as-is? > > > > > > Thanks, > > > Greg > > > > > > [1] https://issues.apache.org/jira/browse/KAFKA-17111 > > > [2] https://issues.apache.org/jira/browse/KAFKA-15996 > > > [3] https://github.com/apache/kafka/pull/16565 > > > [4] https://github.com/apache/kafka/pull/16568 > > > > > > On Wed, Jul 10, 2024 at 7:37 AM Mickael Maison < > mickael.mai...@gmail.com > > > > > > wrote: > > > > > > > Hi Dongjin, > > > > > > > > It's great to see you back! > > > > I hope what we did with KIP-390 matches your expectations. Looking > > > > forward to seeing the reboot of KIP-780. > > > > > > > > Thanks, > > > > Mickael > > > > > > > > On Wed, Jul 10, 2024 at 4:21 PM Dongjin Lee > > wrote: > > > > > > > > > > Hi Josep, > > > > > > > > > > OMG, what happened while I could not be involved with the Kafka > > > > community? > > > > > Thanks for digging down the whole situation. > > > > > > > > > > @Mickael I greatly appreciate your effort in finalizing the > feature. > > It > > > > > seems like I only have to re-boot the KIP-780. > > > > > > > > > > Thanks, > > > > > Dongjin > > > > > > > > > > On Tue, Jul 9, 2024 at 12:52 AM Josep Prat > > > > > > > > > > wrote: > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > KIP-390 is part of the 3.8 release because the JIRA associated > with > > > it: > > > > > > https://issues.apache.org/jira/browse/KAFKA-7632 is closed as > > > > resolved, > > > > > > hence the KIP is declared done and ready. I did some digging, > and I > > > saw > > > > > > that Mickael was the one do
Re: [VOTE] 3.8.0 RC1
Hi all, I'm cancelling this VOTE thread, and I'll generate a new RC once the fix is merged. Best, On Wed, Jul 17, 2024 at 8:55 AM Josep Prat wrote: > Thanks Greg for taking a look and providing a fix! > > Best, > > -- > Josep Prat > Open Source Engineering Director, Aiven > josep.p...@aiven.io | +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Wed, Jul 17, 2024, 00:40 Greg Harris > wrote: > >> Hi Josep, >> >> I found this blocker regression: >> https://issues.apache.org/jira/browse/KAFKA-17150 >> Summary: Connector configurations that specified converters with aliases >> (e.g. JsonConverter instead of >> org.apache.kafka.connect.json.JsonConverter) >> previously worked, and in this RC they throw validation exceptions and are >> unable to start/run. >> Severity: This would take tasks offline completely until the workers are >> downgraded or the connectors are reconfigured with full class names >> Impact: Aliases are commonly used in demos, and slightly less commonly >> used >> in production environments. It is by no means a demo-only feature, and I >> would expect this to break at least one production workload. >> Risk: I've opened a PR to resolve it here: >> https://github.com/apache/kafka/pull/16608 . We've already solved this >> bug >> once before, so it seems low-risk to widen the scope of the original fix >> to >> cover this new case. >> >> I found this apparent bug, but as it doesn't appear to be a regression and >> has very low severity, I don't think it's a blocker on its own: >> https://issues.apache.org/jira/browse/KAFKA-17148 >> If we're rolling a new RC, hopefully we can include this if it's resolved >> quickly. >> >> I also performed the following successful verifications: >> 1. Verified that protected members don't appear in the generated javadocs >> (KAFKA-14839) >> 2. Verified that Connect Distributed can start against a Kraft cluster >> 3. Verified that plugin scanning doesn't throw errors with jackson >> (KAFKA-17111) >> 4. Verified that the allowed.paths configuration works for >> DirectoryConfigProvider (KIP-993) >> >> Unfortunately due to the blocker regression, I think I'll have to -1 >> (binding) this RC. Sorry! >> >> Thanks, >> Greg >> >> On Tue, Jul 16, 2024 at 1:32 PM Jakub Scholz wrote: >> >> > +1 (non-binding). I used the staged Scala 2.13 binaries and the staged >> > Maven artifacts. All seems to work fine. Thanks! >> > >> > Jakub >> > >> > On Mon, Jul 15, 2024 at 5:53 PM Josep Prat > > >> > wrote: >> > >> > > Hello Kafka users, developers and client-developers, >> > > >> > > This is the second release candidate for Apache Kafka 3.8.0. >> > > >> > > Some of the major features included in this release are: >> > > * KIP-1028: Docker Official Image for Apache Kafka >> > > * KIP-974: Docker Image for GraalVM based Native Kafka Broker >> > > * KIP-1036: Extend RecordDeserializationException exception >> > > * KIP-1019: Expose method to determine Metric Measurability >> > > * KIP-1004: Enforce tasks.max property in Kafka Connect >> > > * KIP-989: Improved StateStore Iterator metrics for detecting leaks >> > > * KIP-993: Allow restricting files accessed by File and Directory >> > > ConfigProviders >> > > * KIP-924: customizable task assignment for Streams >> > > * KIP-813: Shareable State Stores >> > > * KIP-719: Deprecate Log4J Appender >> > > * KIP-390: Support Compression Level >> > > * KIP-1018: Introduce max remote fetch timeout config for >> > > DelayedRemoteFetch requests >> > > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission >> > > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace >> > > kafka.serializer.Decoder >> > > * KIP-899: Allow producer and consumer clients to rebootstrap >> > > >> > > Release notes for the 3.8.0 release: >> > > https://home.apache.org/~jlprat/kafka-3.8.0-rc1/RELEASE_NOTES.html >> > > >> > > Please download, test and vote by Thursday, July 18th, 12pm PT* >> > > >> > > Kafka's KEYS file containing PGP keys we use to sign the release: >
Re: [VOTE] 3.8.0 RC1
Thanks Greg for taking a look and providing a fix! Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Wed, Jul 17, 2024, 00:40 Greg Harris wrote: > Hi Josep, > > I found this blocker regression: > https://issues.apache.org/jira/browse/KAFKA-17150 > Summary: Connector configurations that specified converters with aliases > (e.g. JsonConverter instead of org.apache.kafka.connect.json.JsonConverter) > previously worked, and in this RC they throw validation exceptions and are > unable to start/run. > Severity: This would take tasks offline completely until the workers are > downgraded or the connectors are reconfigured with full class names > Impact: Aliases are commonly used in demos, and slightly less commonly used > in production environments. It is by no means a demo-only feature, and I > would expect this to break at least one production workload. > Risk: I've opened a PR to resolve it here: > https://github.com/apache/kafka/pull/16608 . We've already solved this bug > once before, so it seems low-risk to widen the scope of the original fix to > cover this new case. > > I found this apparent bug, but as it doesn't appear to be a regression and > has very low severity, I don't think it's a blocker on its own: > https://issues.apache.org/jira/browse/KAFKA-17148 > If we're rolling a new RC, hopefully we can include this if it's resolved > quickly. > > I also performed the following successful verifications: > 1. Verified that protected members don't appear in the generated javadocs > (KAFKA-14839) > 2. Verified that Connect Distributed can start against a Kraft cluster > 3. Verified that plugin scanning doesn't throw errors with jackson > (KAFKA-17111) > 4. Verified that the allowed.paths configuration works for > DirectoryConfigProvider (KIP-993) > > Unfortunately due to the blocker regression, I think I'll have to -1 > (binding) this RC. Sorry! > > Thanks, > Greg > > On Tue, Jul 16, 2024 at 1:32 PM Jakub Scholz wrote: > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the staged > > Maven artifacts. All seems to work fine. Thanks! > > > > Jakub > > > > On Mon, Jul 15, 2024 at 5:53 PM Josep Prat > > wrote: > > > > > Hello Kafka users, developers and client-developers, > > > > > > This is the second release candidate for Apache Kafka 3.8.0. > > > > > > Some of the major features included in this release are: > > > * KIP-1028: Docker Official Image for Apache Kafka > > > * KIP-974: Docker Image for GraalVM based Native Kafka Broker > > > * KIP-1036: Extend RecordDeserializationException exception > > > * KIP-1019: Expose method to determine Metric Measurability > > > * KIP-1004: Enforce tasks.max property in Kafka Connect > > > * KIP-989: Improved StateStore Iterator metrics for detecting leaks > > > * KIP-993: Allow restricting files accessed by File and Directory > > > ConfigProviders > > > * KIP-924: customizable task assignment for Streams > > > * KIP-813: Shareable State Stores > > > * KIP-719: Deprecate Log4J Appender > > > * KIP-390: Support Compression Level > > > * KIP-1018: Introduce max remote fetch timeout config for > > > DelayedRemoteFetch requests > > > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission > > > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace > > > kafka.serializer.Decoder > > > * KIP-899: Allow producer and consumer clients to rebootstrap > > > > > > Release notes for the 3.8.0 release: > > > https://home.apache.org/~jlprat/kafka-3.8.0-rc1/RELEASE_NOTES.html > > > > > > Please download, test and vote by Thursday, July 18th, 12pm PT* > > > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > > https://kafka.apache.org/KEYS > > > > > > * Release artifacts to be voted upon (source and binary): > > > https://home.apache.org/~jlprat/kafka-3.8.0-rc1/ > > > > > > * Docker release artifact to be voted upon(apache/kafka-native is > > supported > > > from 3.8+ release.): > > > apache/kafka:3.8.0-rc1 > > > apache/kafka-native:3.8.0-rc1 > > > > > > * Maven artifacts to be voted upon: > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > >
[VOTE] 3.8.0 RC1
Hello Kafka users, developers and client-developers, This is the second release candidate for Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036: Extend RecordDeserializationException exception * KIP-1019: Expose method to determine Metric Measurability * KIP-1004: Enforce tasks.max property in Kafka Connect * KIP-989: Improved StateStore Iterator metrics for detecting leaks * KIP-993: Allow restricting files accessed by File and Directory ConfigProviders * KIP-924: customizable task assignment for Streams * KIP-813: Shareable State Stores * KIP-719: Deprecate Log4J Appender * KIP-390: Support Compression Level * KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder * KIP-899: Allow producer and consumer clients to rebootstrap Release notes for the 3.8.0 release: https://home.apache.org/~jlprat/kafka-3.8.0-rc1/RELEASE_NOTES.html Please download, test and vote by Thursday, July 18th, 12pm PT* Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~jlprat/kafka-3.8.0-rc1/ * Docker release artifact to be voted upon(apache/kafka-native is supported from 3.8+ release.): apache/kafka:3.8.0-rc1 apache/kafka-native:3.8.0-rc1 * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~jlprat/kafka-3.8.0-rc1/javadoc/ * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: https://github.com/apache/kafka/releases/tag/3.8.0-rc1 Once https://github.com/apache/kafka-site/pull/608 is merged. You will be able to find the proper documentation under kafka.apache.org. * Documentation: https://kafka.apache.org/38/documentation.html * Protocol: https://kafka.apache.org/38/protocol.html * Successful Jenkins builds for the 3.8 branch:Unit/integration tests: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/67/ (Some known flaky tests and builds from 64 to 68 show tests passing at least once). Additionally, this is the CI run for the changes between RC0 and RC1: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16593/1/pipeline/ System tests: (Same as before) https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html * Successful Docker Image Github Actions Pipeline for 3.8 branch: Docker Build Test Pipeline (JVM): https://github.com/apache/kafka/actions/runs/9941446092 Docker Build Test Pipeline (Native): https://github.com/apache/kafka/actions/runs/9941449561 Best, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [VOTE] 3.8.0 RC0
Hi all, I'm cancelling the VOTE thread for 3.8.0-RC0. I submitted a PR with the backport https://github.com/apache/kafka/pull/16593 and I'll generate a new RC as soon as it's merged. Best, On Sat, Jul 13, 2024 at 7:09 PM Josep Prat wrote: > Thanks for reviewing the RC Jakub, > > If you can open a PR with this fix pointing to the 3.8 branch I could cut > another RC. > > Best! > > -- > Josep Prat > Open Source Engineering Director, Aiven > josep.p...@aiven.io | +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Sat, Jul 13, 2024, 16:13 Jakub Scholz wrote: > >> Hi Josep, >> >> Thanks for the RC. >> >> I gave it a quick try and ran into issues with an application using the >> Kafka Admin API that looks like this issue: >> https://issues.apache.org/jira/browse/KAFKA-16905 ... given that this >> breaks what was working fine with Kafka 3.7, can the fix be backported to >> 3.8.0 as well? If needed, I tried to create a simple reproducer for the >> issue: https://github.com/scholzj/kafka-admin-api-async-issue-reproducer. >> >> Thanks & Regards >> Jakub >> >> On Fri, Jul 12, 2024 at 11:46 AM Josep Prat >> wrote: >> >> > Hello Kafka users, developers and client-developers, >> > >> > This is the first candidate for release of Apache Kafka 3.8.0. >> > Some of the major features included in this release are: >> > * KIP-1028: Docker Official Image for Apache Kafka >> > * KIP-974: Docker Image for GraalVM based Native Kafka Broker >> > * KIP-1036: Extend RecordDeserializationException exception >> > * KIP-1019: Expose method to determine Metric Measurability >> > * KIP-1004: Enforce tasks.max property in Kafka Connect >> > * KIP-989: Improved StateStore Iterator metrics for detecting leaks >> > * KIP-993: Allow restricting files accessed by File and Directory >> > ConfigProviders >> > * KIP-924: customizable task assignment for Streams >> > * KIP-813: Shareable State Stores >> > * KIP-719: Deprecate Log4J Appender >> > * KIP-390: Support Compression Level >> > * KIP-1018: Introduce max remote fetch timeout config for >> > DelayedRemoteFetch requests >> > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission >> > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace >> > kafka.serializer.Decoder >> > * KIP-899: Allow producer and consumer clients to rebootstrap >> > >> > Release notes for the 3.8.0 release: >> > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/RELEASE_NOTES.html >> > >> > *** Please download, test and vote by Monday, July 15, 12pm PT >> > >> > Kafka's KEYS file containing PGP keys we use to sign the release: >> > https://kafka.apache.org/KEYS >> > >> > * Release artifacts to be voted upon (source and binary): >> > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/ >> > >> > * Docker release artifacts to be voted upon: >> > apache/kafka:3.8.0-rc0 >> > apache/kafka-native:3.8.0-rc0 >> > >> > * Maven artifacts to be voted upon: >> > https://repository.apache.org/content/groups/staging/org/apache/kafka/ >> > >> > * Javadoc: >> > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/javadoc/ >> > >> > * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: >> > https://github.com/apache/kafka/releases/tag/3.8.0-rc0 >> > >> > >> > Once https://github.com/apache/kafka-site/pull/608 is merged. You will >> be >> > able to find the proper documentation under kafka.apache.org. >> > * Documentation: >> > https://kafka.apache.org/38/documentation.html >> > >> > * Protocol: >> > https://kafka.apache.org/38/protocol.html >> > >> > >> > * Successful Jenkins builds for the 3.8 branch: >> > Unit/integration tests: >> > >> > >> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/67/ >> > (Some known flaky tests and builds from 64 to 68 show tests passing at >> > least once) >> > System tests: >> > >> > >> https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html >>
Re: [VOTE] 3.8.0 RC0
Thanks for reviewing the RC Jakub, If you can open a PR with this fix pointing to the 3.8 branch I could cut another RC. Best! -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Sat, Jul 13, 2024, 16:13 Jakub Scholz wrote: > Hi Josep, > > Thanks for the RC. > > I gave it a quick try and ran into issues with an application using the > Kafka Admin API that looks like this issue: > https://issues.apache.org/jira/browse/KAFKA-16905 ... given that this > breaks what was working fine with Kafka 3.7, can the fix be backported to > 3.8.0 as well? If needed, I tried to create a simple reproducer for the > issue: https://github.com/scholzj/kafka-admin-api-async-issue-reproducer. > > Thanks & Regards > Jakub > > On Fri, Jul 12, 2024 at 11:46 AM Josep Prat > wrote: > > > Hello Kafka users, developers and client-developers, > > > > This is the first candidate for release of Apache Kafka 3.8.0. > > Some of the major features included in this release are: > > * KIP-1028: Docker Official Image for Apache Kafka > > * KIP-974: Docker Image for GraalVM based Native Kafka Broker > > * KIP-1036: Extend RecordDeserializationException exception > > * KIP-1019: Expose method to determine Metric Measurability > > * KIP-1004: Enforce tasks.max property in Kafka Connect > > * KIP-989: Improved StateStore Iterator metrics for detecting leaks > > * KIP-993: Allow restricting files accessed by File and Directory > > ConfigProviders > > * KIP-924: customizable task assignment for Streams > > * KIP-813: Shareable State Stores > > * KIP-719: Deprecate Log4J Appender > > * KIP-390: Support Compression Level > > * KIP-1018: Introduce max remote fetch timeout config for > > DelayedRemoteFetch requests > > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission > > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace > > kafka.serializer.Decoder > > * KIP-899: Allow producer and consumer clients to rebootstrap > > > > Release notes for the 3.8.0 release: > > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/RELEASE_NOTES.html > > > > *** Please download, test and vote by Monday, July 15, 12pm PT > > > > Kafka's KEYS file containing PGP keys we use to sign the release: > > https://kafka.apache.org/KEYS > > > > * Release artifacts to be voted upon (source and binary): > > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/ > > > > * Docker release artifacts to be voted upon: > > apache/kafka:3.8.0-rc0 > > apache/kafka-native:3.8.0-rc0 > > > > * Maven artifacts to be voted upon: > > https://repository.apache.org/content/groups/staging/org/apache/kafka/ > > > > * Javadoc: > > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/javadoc/ > > > > * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: > > https://github.com/apache/kafka/releases/tag/3.8.0-rc0 > > > > > > Once https://github.com/apache/kafka-site/pull/608 is merged. You will > be > > able to find the proper documentation under kafka.apache.org. > > * Documentation: > > https://kafka.apache.org/38/documentation.html > > > > * Protocol: > > https://kafka.apache.org/38/protocol.html > > > > > > * Successful Jenkins builds for the 3.8 branch: > > Unit/integration tests: > > > > > https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/67/ > > (Some known flaky tests and builds from 64 to 68 show tests passing at > > least once) > > System tests: > > > > > https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html > > > > Note that the system tests can't always successfully run in CI, > `QuotaTest` > > have been reportedly running successfully locally by several Kafka > > community members. Other flaky tests were tracked under > > https://issues.apache.org/jira/browse/KAFKA-17083, > > https://issues.apache.org/jira/browse/KAFKA-17084 and > > https://issues.apache.org/jira/browse/KAFKA-17085. All of them run > > successfully on other CI environments. > > > > * Successful Docker Image Github Actions Pipeline for 3.8 branch: > > Docker Build Test Pipeline (JVM): > > https://github.com/apache/kafka/actions/runs/9905437603 > >
[VOTE] 3.8.0 RC0
Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.8.0. Some of the major features included in this release are: * KIP-1028: Docker Official Image for Apache Kafka * KIP-974: Docker Image for GraalVM based Native Kafka Broker * KIP-1036: Extend RecordDeserializationException exception * KIP-1019: Expose method to determine Metric Measurability * KIP-1004: Enforce tasks.max property in Kafka Connect * KIP-989: Improved StateStore Iterator metrics for detecting leaks * KIP-993: Allow restricting files accessed by File and Directory ConfigProviders * KIP-924: customizable task assignment for Streams * KIP-813: Shareable State Stores * KIP-719: Deprecate Log4J Appender * KIP-390: Support Compression Level * KIP-1018: Introduce max remote fetch timeout config for DelayedRemoteFetch requests * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder * KIP-899: Allow producer and consumer clients to rebootstrap Release notes for the 3.8.0 release: https://home.apache.org/~jlprat/kafka-3.8.0-rc0/RELEASE_NOTES.html *** Please download, test and vote by Monday, July 15, 12pm PT Kafka's KEYS file containing PGP keys we use to sign the release: https://kafka.apache.org/KEYS * Release artifacts to be voted upon (source and binary): https://home.apache.org/~jlprat/kafka-3.8.0-rc0/ * Docker release artifacts to be voted upon: apache/kafka:3.8.0-rc0 apache/kafka-native:3.8.0-rc0 * Maven artifacts to be voted upon: https://repository.apache.org/content/groups/staging/org/apache/kafka/ * Javadoc: https://home.apache.org/~jlprat/kafka-3.8.0-rc0/javadoc/ * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag: https://github.com/apache/kafka/releases/tag/3.8.0-rc0 Once https://github.com/apache/kafka-site/pull/608 is merged. You will be able to find the proper documentation under kafka.apache.org. * Documentation: https://kafka.apache.org/38/documentation.html * Protocol: https://kafka.apache.org/38/protocol.html * Successful Jenkins builds for the 3.8 branch: Unit/integration tests: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/67/ (Some known flaky tests and builds from 64 to 68 show tests passing at least once) System tests: https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-10--001.f1f05b43-3574-45cb-836e-8968f02d722f--1720631619--apache--3.8--4ecbb75c1f/report.html Note that the system tests can't always successfully run in CI, `QuotaTest` have been reportedly running successfully locally by several Kafka community members. Other flaky tests were tracked under https://issues.apache.org/jira/browse/KAFKA-17083, https://issues.apache.org/jira/browse/KAFKA-17084 and https://issues.apache.org/jira/browse/KAFKA-17085. All of them run successfully on other CI environments. * Successful Docker Image Github Actions Pipeline for 3.8 branch: Docker Build Test Pipeline (JVM): https://github.com/apache/kafka/actions/runs/9905437603 Docker Build Test Pipeline (Native): https://github.com/apache/kafka/actions/runs/9905490565 Thanks, -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Greg, I'll make sure the PR with the fix ( https://github.com/apache/kafka/pull/16565) gets merged today. Thanks for bringing this up! Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Wed, Jul 10, 2024, 22:55 Greg Harris wrote: > Hi Josep, > > A contributor just raised a regression [1] that I think should be addressed > in 3.8.0 prior to the release. > > Summary: This change [2] causes multiple ERROR logs to appear during worker > startup when operators install other, unrelated plugins that package the > Jackson library. > Severity: Workers will continue to operate normally and the errors will be > cosmetic. Operators and automatic systems that watch logs may roll-back > upgrades due to the perception of a severe problem. > Impact: I found 12 out of 250 third-party plugins that package the Jackson > library and trigger this error on upgrade. This will almost certainly > affect several users upon upgrades, and does not require obscure setups. > Risk: The contributor has opened a simple fix PR [3], and I have verified > that it addresses the problem, and can be merged tomorrow. As an > alternative, we can revert the performance change completely [4] but I want > to avoid this. > > With the above, what would you like to do for this release? Merge the fix, > revert, or leave as-is? > > Thanks, > Greg > > [1] https://issues.apache.org/jira/browse/KAFKA-17111 > [2] https://issues.apache.org/jira/browse/KAFKA-15996 > [3] https://github.com/apache/kafka/pull/16565 > [4] https://github.com/apache/kafka/pull/16568 > > On Wed, Jul 10, 2024 at 7:37 AM Mickael Maison > wrote: > > > Hi Dongjin, > > > > It's great to see you back! > > I hope what we did with KIP-390 matches your expectations. Looking > > forward to seeing the reboot of KIP-780. > > > > Thanks, > > Mickael > > > > On Wed, Jul 10, 2024 at 4:21 PM Dongjin Lee wrote: > > > > > > Hi Josep, > > > > > > OMG, what happened while I could not be involved with the Kafka > > community? > > > Thanks for digging down the whole situation. > > > > > > @Mickael I greatly appreciate your effort in finalizing the feature. It > > > seems like I only have to re-boot the KIP-780. > > > > > > Thanks, > > > Dongjin > > > > > > On Tue, Jul 9, 2024 at 12:52 AM Josep Prat > > > > wrote: > > > > > > > Hi Dongjin, > > > > > > > > KIP-390 is part of the 3.8 release because the JIRA associated with > it: > > > > https://issues.apache.org/jira/browse/KAFKA-7632 is closed as > > resolved, > > > > hence the KIP is declared done and ready. I did some digging, and I > saw > > > > that Mickael was the one doing the PR that closed the JIRA ticket: > > > > https://github.com/apache/kafka/pull/15516 > > > > This means that the KIP work is merged and unfortunately it is now > > quite > > > > late to perform a rollback for this feature. > > > > > > > > @Mickael Maison let me know if anything I > > > > mentioned is not accurate (as you were the one bringing the KIP to > > > > completion). > > > > > > > > Best, > > > > > > > > On Mon, Jul 8, 2024 at 5:38 PM Dongjin Lee > wrote: > > > > > > > > > Hi Josep, > > > > > > > > > > Thanks for managing the 3.8 release. I have a request: could you > > please > > > > > move the KIP-390 into the 3.9 release? > > > > > > > > > > Here is the background: KIP-390 was adopted first but hasn't been > > > > released > > > > > for a long time. After some time, I proposed KIP-780 with further > > > > > improvements and also corrected an obvious design error > > > > > (`compression.level` → `compression.(gzip|lz4|zstd). level`), but > it > > > > hasn't > > > > > been adopted due to the community's lack of response, my changing > > job, > > > > > focusing the in-house fork, etc. And last weekend, I found that > > KIP-380 > > > > has > > > > > been included in the 3.8 release plan. > > > > > > > > > > - KIP-390: > > > > > > > > > > > > > > > > > https://cwiki.a
Re: [DISCUSS] Apache Kafka 3.8.0 release
Thanks for the update, I'm looking closely to this PR. I'll start working on the first RC shortly after it's merged. Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Tue, Jul 9, 2024, 17:57 Matthias J. Sax wrote: > The KS system test did surface a bug. Bruno put a PR up for it, which is > already approved. > > We should get it into 3.8 asap to unblock the release from our side. > > > -Matthias > > On 7/8/24 8:51 AM, Josep Prat wrote: > > Hi Dongjin, > > > > KIP-390 is part of the 3.8 release because the JIRA associated with it: > > https://issues.apache.org/jira/browse/KAFKA-7632 is closed as resolved, > > hence the KIP is declared done and ready. I did some digging, and I saw > > that Mickael was the one doing the PR that closed the JIRA ticket: > > https://github.com/apache/kafka/pull/15516 > > This means that the KIP work is merged and unfortunately it is now quite > > late to perform a rollback for this feature. > > > > @Mickael Maison let me know if anything I > > mentioned is not accurate (as you were the one bringing the KIP to > > completion). > > > > Best, > > > > On Mon, Jul 8, 2024 at 5:38 PM Dongjin Lee wrote: > > > >> Hi Josep, > >> > >> Thanks for managing the 3.8 release. I have a request: could you please > >> move the KIP-390 into the 3.9 release? > >> > >> Here is the background: KIP-390 was adopted first but hasn't been > released > >> for a long time. After some time, I proposed KIP-780 with further > >> improvements and also corrected an obvious design error > >> (`compression.level` → `compression.(gzip|lz4|zstd). level`), but it > hasn't > >> been adopted due to the community's lack of response, my changing job, > >> focusing the in-house fork, etc. And last weekend, I found that KIP-380 > has > >> been included in the 3.8 release plan. > >> > >> - KIP-390: > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level > >> - KIP-780: > >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options > >> > >> However, shipping those two features at once has the following benefits: > >> > >> 1. Full functionality without design error. > >> > >> We can provide full functionality, particularly useful with tiered > storage > >> feature at once. I found that several users of tiered storage use > >> server-side recompression and want to improve the compression > efficiency. > >> Of course, it does not include any design errors :) > >> > >> 2. More chance of testing. > >> > >> Currently, I am managing an in-house fork of Apache Kafka and Cruise > >> Control[^1], running on thousands of clusters on k8s. With our ongoing > work > >> on the tiered storage plugin, we can test both KIPs at once. Since we > are > >> planning to move the terabytes of logs from thousands of microservices > into > >> the object storage, some of them can be ideal testbeds. > >> > >> If you are okay, I will re-initiate the discussion of KIP-780 and rework > >> KIP-380 on the latest trunk. > >> > >> Thanks, > >> Dongjin > >> > >> [^1]: For example: https://github.com/linkedin/cruise-control/pull/2145 > >> > >> On Mon, Feb 26, 2024 at 8:38 PM Josep Prat > > >> wrote: > >> > >>> Hi all, > >>> > >>> I'd like to volunteer as release manager for the Apache Kafka 3.8.0 > >>> release. > >>> If there are no objections, I'll start building a release plan (or > >> adapting > >>> the one Colin made some weeks ago) in the wiki in the next days. > >>> > >>> Thank you. > >>> > >>> -- > >>> [image: Aiven] <https://www.aiven.io> > >>> > >>> *Josep Prat* > >>> Open Source Engineering Director, *Aiven* > >>> josep.p...@aiven.io | +491715557497 > >>> aiven.io <https://www.aiven.io> | < > >> https://www.facebook.com/aivencloud > >>>> > >>><https://www.linkedin.com/company/aiven/> < > >>> https://twitter.com/aiven_io> > >>> *Aiven Deutschland GmbH* > >>> Alexanderufer 3-7, 10117 Berlin > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > >>> Amtsgericht Charlottenburg, HRB 209739 B > >>> > >> > >> > >> -- > >> *Dongjin Lee* > >> > >> *A hitchhiker in the mathematical world.* > >> > >> > >> > >> *github: <http://goog_969573159/>github.com/dongjinleekr > >> <https://github.com/dongjinleekr>keybase: > https://keybase.io/dongjinleekr > >> <https://keybase.io/dongjinleekr>linkedin: > kr.linkedin.com/in/dongjinleekr > >> <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > >> speakerdeck.com/dongjin > >> <https://speakerdeck.com/dongjin>* > >> > > > > >
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Dongjin, KIP-390 is part of the 3.8 release because the JIRA associated with it: https://issues.apache.org/jira/browse/KAFKA-7632 is closed as resolved, hence the KIP is declared done and ready. I did some digging, and I saw that Mickael was the one doing the PR that closed the JIRA ticket: https://github.com/apache/kafka/pull/15516 This means that the KIP work is merged and unfortunately it is now quite late to perform a rollback for this feature. @Mickael Maison let me know if anything I mentioned is not accurate (as you were the one bringing the KIP to completion). Best, On Mon, Jul 8, 2024 at 5:38 PM Dongjin Lee wrote: > Hi Josep, > > Thanks for managing the 3.8 release. I have a request: could you please > move the KIP-390 into the 3.9 release? > > Here is the background: KIP-390 was adopted first but hasn't been released > for a long time. After some time, I proposed KIP-780 with further > improvements and also corrected an obvious design error > (`compression.level` → `compression.(gzip|lz4|zstd). level`), but it hasn't > been adopted due to the community's lack of response, my changing job, > focusing the in-house fork, etc. And last weekend, I found that KIP-380 has > been included in the 3.8 release plan. > > - KIP-390: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-390%3A+Support+Compression+Level > - KIP-780: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-780%3A+Support+fine-grained+compression+options > > However, shipping those two features at once has the following benefits: > > 1. Full functionality without design error. > > We can provide full functionality, particularly useful with tiered storage > feature at once. I found that several users of tiered storage use > server-side recompression and want to improve the compression efficiency. > Of course, it does not include any design errors :) > > 2. More chance of testing. > > Currently, I am managing an in-house fork of Apache Kafka and Cruise > Control[^1], running on thousands of clusters on k8s. With our ongoing work > on the tiered storage plugin, we can test both KIPs at once. Since we are > planning to move the terabytes of logs from thousands of microservices into > the object storage, some of them can be ideal testbeds. > > If you are okay, I will re-initiate the discussion of KIP-780 and rework > KIP-380 on the latest trunk. > > Thanks, > Dongjin > > [^1]: For example: https://github.com/linkedin/cruise-control/pull/2145 > > On Mon, Feb 26, 2024 at 8:38 PM Josep Prat > wrote: > > > Hi all, > > > > I'd like to volunteer as release manager for the Apache Kafka 3.8.0 > > release. > > If there are no objections, I'll start building a release plan (or > adapting > > the one Colin made some weeks ago) in the wiki in the next days. > > > > Thank you. > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>* > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Bruno, I'd be in for merging the fix into 3.8. Given the solution is already found and it seems easy enough to reason and with very limited scope, I believe it is fairly safe to merge this one. Let's work towards merging the fix and see if the system tests improve and await for your soak testing. Best, On Mon, Jul 8, 2024 at 9:39 AM Bruno Cadonna wrote: > Hi Josep, > > we found a bug in Streams. The bug is in the state updater that we > enabled by default for 3.8. > > I have already created a fix for it: > https://github.com/apache/kafka/pull/16545 > > As you can see the fix is small and limited to a location in the code, > so I do not think that it is risky. > > The bug is not strictly a regression since the state updater was > disabled in previous releases. However, for this release we soaked > mainly with state updater enabled. So we need to soak Streams for a bit > (couple of days? one week?) with state updater disabled to be confident > that disabling does not introduce a real regression since the two code > paths are interleaved. > > Reverting the PRs of the state updater is not an option since they are > many and they were merged over quite some time which makes reverts at > least very risky and work intensive. Reverting would delay the release > for sure. > > In addition to the fix, I also opened a PR to disable the state updater: > https://github.com/apache/kafka/pull/16542 > > I have already started soak testing for both PRs. > > We did not find this bug before, because it manifests in a specific > situation that occurs rather seldomly even in our soak where we inject > various errors and infrastructure events. > > The state updater is a long awaited improvement that besides other > improvements solves a long-standing timeout issue with exactly-once > processing: https://issues.apache.org/jira/browse/KAFKA-13295 > > Let me know, whether you prefer to disable the state updater or whether > I should merge the fix into 3.8. > > Best, > Bruno > > On 7/5/24 12:01 PM, Josep Prat wrote: > > Hi all, > > Unfortunately, after 4 runs of the systems tests, we still can't have a > > combined run with no errors. I created the JIRAs linked below to track > > these. > > I would think these are blockers for the release, but I'd be extremely > > happy to be corrected! > > > > KRaft Upgrade Failures: > https://issues.apache.org/jira/browse/KAFKA-17083 > > Network Degrade Failures: > https://issues.apache.org/jira/browse/KAFKA-17084 > > Streams Cooperative Rebalance Upgrade Failures > > https://issues.apache.org/jira/browse/KAFKA-17085. > > These system tests above fail consistently on CI and on my machine. If > > anyone has the means to run system tests and can make these pass, please > > let me know. > > > > These add up to the existing > > https://issues.apache.org/jira/browse/KAFKA-16138 (discovered during > 3.7) > > for the Quota test failures that can pass locally. > > > > The status of the test runs as well as the logs of the runs can be found > > here: > > > https://docs.google.com/document/d/1wbcyzO6GM2SYQaqTMITBTBjHgZgM7mmiAt7TUfh1xt8/edit > > > > Best, > > > > On Thu, Jul 4, 2024 at 3:27 PM Josep Prat wrote: > > > >> Thanks Luke! > >> > >> -- > >> Josep Prat > >> Open Source Engineering Director, Aiven > >> josep.p...@aiven.io | +491715557497 | aiven.io > >> Aiven Deutschland GmbH > >> Alexanderufer 3-7, 10117 Berlin > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > >> Amtsgericht Charlottenburg, HRB 209739 B > >> > >> On Thu, Jul 4, 2024, 14:04 Luke Chen wrote: > >> > >>> Hi Josep, > >>> > >>> I had run tests for tests/kafkatest/tests/client/quota_test.py based on > >>> 3.8 > >>> branch, and they all passed. > >>> > >>> *19:54:24* > >>> > *19:54:24* > >>> SESSION REPORT (ALL TESTS)*19:54:24* ducktape version: > >>> 0.11.4*19:54:24* session_id: 2024-07-04--001*19:54:24* run > >>> time: 12 minutes 39.940 seconds*19:54:24* tests run: > >>> 9*19:54:24* passed: 9*19:54:24* flaky: > >>> 0*19:54:24* failed: 0*19:54:24* ignored: > >>> 0*19:54:24* > >>> > *19:54:24* > >>> test_id: > &g
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, Unfortunately, after 4 runs of the systems tests, we still can't have a combined run with no errors. I created the JIRAs linked below to track these. I would think these are blockers for the release, but I'd be extremely happy to be corrected! KRaft Upgrade Failures: https://issues.apache.org/jira/browse/KAFKA-17083 Network Degrade Failures: https://issues.apache.org/jira/browse/KAFKA-17084 Streams Cooperative Rebalance Upgrade Failures https://issues.apache.org/jira/browse/KAFKA-17085. These system tests above fail consistently on CI and on my machine. If anyone has the means to run system tests and can make these pass, please let me know. These add up to the existing https://issues.apache.org/jira/browse/KAFKA-16138 (discovered during 3.7) for the Quota test failures that can pass locally. The status of the test runs as well as the logs of the runs can be found here: https://docs.google.com/document/d/1wbcyzO6GM2SYQaqTMITBTBjHgZgM7mmiAt7TUfh1xt8/edit Best, On Thu, Jul 4, 2024 at 3:27 PM Josep Prat wrote: > Thanks Luke! > > ------ > Josep Prat > Open Source Engineering Director, Aiven > josep.p...@aiven.io | +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Thu, Jul 4, 2024, 14:04 Luke Chen wrote: > >> Hi Josep, >> >> I had run tests for tests/kafkatest/tests/client/quota_test.py based on >> 3.8 >> branch, and they all passed. >> >> *19:54:24* >> *19:54:24* >> SESSION REPORT (ALL TESTS)*19:54:24* ducktape version: >> 0.11.4*19:54:24* session_id: 2024-07-04--001*19:54:24* run >> time: 12 minutes 39.940 seconds*19:54:24* tests run: >> 9*19:54:24* passed: 9*19:54:24* flaky: >> 0*19:54:24* failed: 0*19:54:24* ignored: >> 0*19:54:24* >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.consumer_num=2*19:54:24* >> status: PASS*19:54:24* run time: 3 minutes 51.280 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=.user.client-id.override_quota=True*19:54:24* >> status: PASS*19:54:24* run time: 4 minutes 21.082 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=.user.client-id.override_quota=False*19:54:24* >> status: PASS*19:54:24* run time: 5 minutes 14.854 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.old_broker_throttling_behavior=True*19:54:24* >> status: PASS*19:54:24* run time: 3 minutes 0.505 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.old_client_throttling_behavior=True*19:54:24* >> status: PASS*19:54:24* run time: 3 minutes 19.629 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.override_quota=False*19:54:24* >> status: PASS*19:54:24* run time: 4 minutes 11.296 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.override_quota=True*19:54:24* >> status: PASS*19:54:24* run time: 4 minutes 10.578 >> seconds*19:54:24* >> >> *19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=user.override_quota=False*19:54:24* >> status: PASS*19:54:24* run time: 4 minutes 19.187 >> seconds*19:54:24* >> >> ----*19:54:24* >> test_id: >> kafkatest.tests.client.quota_test.QuotaTest.test_q
[jira] [Created] (KAFKA-17085) Streams Cooperative Rebalance Upgrade Test fails in System Tests
Josep Prat created KAFKA-17085:
--
Summary: Streams Cooperative Rebalance Upgrade Test fails in
System Tests
Key: KAFKA-17085
URL: https://issues.apache.org/jira/browse/KAFKA-17085
Project: Kafka
Issue Type: Bug
Components: system tests
Affects Versions: 3.8.0
Reporter: Josep Prat
StreamsCooperativeRebalanceUpgradeTest fails on system tests when upgrading
from: 2.1.1, 2.2.2 and 2.3.1.
Tests that fail:
{noformat}
Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
Class: StreamsCooperativeRebalanceUpgradeTest
Method: test_upgrade_to_cooperative_rebalance
Arguments:
{
"upgrade_from_version": "2.1.1"
}
{noformat}
and
{noformat}
Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
Class: StreamsCooperativeRebalanceUpgradeTest
Method: test_upgrade_to_cooperative_rebalance
Arguments:
{
"upgrade_from_version": "2.2.2"
}
{noformat}
and
{noformat}
Module: kafkatest.tests.streams.streams_cooperative_rebalance_upgrade_test
Class: StreamsCooperativeRebalanceUpgradeTest
Method: test_upgrade_to_cooperative_rebalance
Arguments:
{
"upgrade_from_version": "2.3.1"
}
{noformat}
Failure for 2.1.1 is:
{noformat}
TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far'
message ubuntu@worker28")
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py",
line 101, in test_upgrade_to_cooperative_rebalance
self.maybe_upgrade_rolling_bounce_and_verify(processors,
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py",
line 182, in maybe_upgrade_rolling_bounce_and_verify
stdout_monitor.wait_until(verify_processing_msg,
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 736, in wait_until
return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
(self.offset + 1, self.log, pattern),
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py",
line 58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
ducktape.errors.TimeoutError: Never saw 'first_bounce_phase-Processed [0-9]*
records so far' message ubuntu@worker28{noformat}
Failure for 2.2.2 is:
{noformat}
TimeoutError("Never saw 'first_bounce_phase-Processed [0-9]* records so far'
message ubuntu@worker5")
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py",
line 101, in test_upgrade_to_cooperative_rebalance
self.maybe_upgrade_rolling_bounce_and_verify(processors,
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/streams/streams_cooperative_rebalance_upgrade_test.py",
line 182, in maybe_upgrade_rolling_bounce_and_verify
stdout_monitor.wait_until(verify_processing_msg,
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 736, in wait_until
return wait_until(lambda: self.acct.ssh("tail -c +%d %s | grep '%s'" %
(self.offset + 1, self.log, pattern),
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/utils/util.py",
line 58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from
last_exception
d
[jira] [Created] (KAFKA-17084) Network Degrade Test fails in System Tests
Josep Prat created KAFKA-17084:
--
Summary: Network Degrade Test fails in System Tests
Key: KAFKA-17084
URL: https://issues.apache.org/jira/browse/KAFKA-17084
Project: Kafka
Issue Type: Bug
Components: system tests
Affects Versions: 3.8.0
Reporter: Josep Prat
Tests for NetworkDegradeTest fail consistently on the 3.8 branch.
Tests failing are:
{noformat}
Module: kafkatest.tests.core.network_degrade_test
Class: NetworkDegradeTest
Method: test_latency
Arguments:
{
"device_name": "eth0",
"latency_ms": 50,
"rate_limit_kbit": 1000,
"task_name": "latency-100-rate-1000"
}
{noformat}
and
{noformat}
Module: kafkatest.tests.core.network_degrade_test
Class: NetworkDegradeTest
Method: test_latency
Arguments:
{
"device_name": "eth0",
"latency_ms": 50,
"rate_limit_kbit": 0,
"task_name": "latency-100"
}
{noformat}
Failure for the first one is:
{noformat}
RemoteCommandError({'ssh_config': {'host': 'worker30', 'hostname':
'10.140.34.105', 'user': 'ubuntu', 'port': 22, 'password': None,
'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'},
'hostname': 'worker30', 'ssh_hostname': '10.140.34.105', 'user': 'ubuntu',
'externally_routable_ip': '10.140.34.105', '_logger': , 'os': 'linux', '_ssh_client': , '_sftp_client': , '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20
worker21', 1, b'')
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/network_degrade_test.py",
line 66, in test_latency
for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" %
zk1.account.hostname):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 680, in next
return next(self.iter_obj)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 347, in output_generator
raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker30: Command
'ping -i 1 -c 20 worker21' returned non-zero exit status 1.{noformat}
And for the second one is:
{noformat}
RemoteCommandError({'ssh_config': {'host': 'worker28', 'hostname':
'10.140.41.79', 'user': 'ubuntu', 'port': 22, 'password': None, 'identityfile':
'/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 'hostname':
'worker28', 'ssh_hostname': '10.140.41.79', 'user': 'ubuntu',
'externally_routable_ip': '10.140.41.79', '_logger': , 'os': 'linux', '_ssh_client': , '_sftp_client': , '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20
worker27', 1, b'')
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/network_degrade_test.py",
line 66, in test_latency
for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" %
zk1.account.hostname):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 680, in next
return next(self.iter_obj)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 347, in output_generator
raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker28: Command
'ping -i 1 -c 20 worker27' returned non-zero exit status 1.{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-17083) KRaft Upgrade Failures in SystemTests
Josep Prat created KAFKA-17083:
--
Summary: KRaft Upgrade Failures in SystemTests
Key: KAFKA-17083
URL: https://issues.apache.org/jira/browse/KAFKA-17083
Project: Kafka
Issue Type: Bug
Components: system tests
Affects Versions: 3.8.0
Reporter: Josep Prat
2 System tests for "TestKRaftUpgrade are consistently failing on 3.8 in the
system tests.
{noformat}
Module: kafkatest.tests.core.kraft_upgrade_test
Class: TestKRaftUpgrade
Method: test_isolated_mode_upgrade
Arguments:
{
"from_kafka_version": "dev",
"metadata_quorum": "ISOLATED_KRAFT"
}
{noformat}
and
{code:java}
Module: kafkatest.tests.core.kraft_upgrade_test
Class: TestKRaftUpgrade
Method: test_combined_mode_upgrade
Arguments:
{
"from_kafka_version": "dev",
"metadata_quorum": "COMBINED_KRAFT"
}
{code}
Failure for Isolated is:
{noformat}
RemoteCommandError({'ssh_config': {'host': 'worker15', 'hostname':
'10.140.39.207', 'user': 'ubuntu', 'port': 22, 'password': None,
'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'},
'hostname': 'worker15', 'ssh_hostname': '10.140.39.207', 'user': 'ubuntu',
'externally_routable_ip': '10.140.39.207', '_logger': , 'os': 'linux', '_ssh_client': , '_sftp_client': , '_custom_ssh_exception_checks': None},
'/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server
worker15:9092,worker16:9092,worker17:9092 upgrade --metadata 3.7', 1, b'SLF4J:
Class path contains multiple SLF4J bindings.\nSLF4J: Found binding in
[jar:file:/vagrant/tools/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
Found binding in
[jar:file:/vagrant/trogdor/build/dependant-libs-2.13.14/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]\nSLF4J:
See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.\nSLF4J: Actual binding is of type
[org.slf4j.impl.Reload4jLoggerFactory]\n1 out of 1 operation(s) failed.\n')
Traceback (most recent call last):
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 121, in test_isolated_mode_upgrade
self.run_upgrade(from_kafka_version)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 105, in run_upgrade
self.run_produce_consume_validate(core_test_action=lambda:
self.perform_version_change(from_kafka_version))
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/produce_consume_validate.py",
line 105, in run_produce_consume_validate
core_test_action(*args)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 105, in
self.run_produce_consume_validate(core_test_action=lambda:
self.perform_version_change(from_kafka_version))
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/kraft_upgrade_test.py",
line 75, in perform_version_change
self.kafka.upgrade_metadata_version(LATEST_STABLE_METADATA_VERSION)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py",
line 920, in upgrade_metadata_version
self.run_features_command("upgrade", new_version)
File
"/home/semaphore/kafka-overlay/kafka/tests/kafkatest/services/kafka/kafka.py",
line 930, in run_features_command
self.nodes[0].account.ssh(cmd)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 35, in wrapper
return method(self, *args, **kwargs)
File
"/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
line 293, in ssh
raise RemoteCommandError(self, cmd, exit_status, stderr.read())
ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker15: Command
'/opt/kafka-dev/bin/kafka-features.sh --bootstrap-server
wor
Re: [DISCUSS] Apache Kafka 3.8.0 release
Thanks Luke!
--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io | +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B
On Thu, Jul 4, 2024, 14:04 Luke Chen wrote:
> Hi Josep,
>
> I had run tests for tests/kafkatest/tests/client/quota_test.py based on 3.8
> branch, and they all passed.
>
> *19:54:24*
> *19:54:24*
> SESSION REPORT (ALL TESTS)*19:54:24* ducktape version:
> 0.11.4*19:54:24* session_id: 2024-07-04--001*19:54:24* run
> time: 12 minutes 39.940 seconds*19:54:24* tests run:
> 9*19:54:24* passed: 9*19:54:24* flaky:
> 0*19:54:24* failed: 0*19:54:24* ignored:
> 0*19:54:24*
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.consumer_num=2*19:54:24*
> status: PASS*19:54:24* run time: 3 minutes 51.280
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=.user.client-id.override_quota=True*19:54:24*
> status: PASS*19:54:24* run time: 4 minutes 21.082
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=.user.client-id.override_quota=False*19:54:24*
> status: PASS*19:54:24* run time: 5 minutes 14.854
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.old_broker_throttling_behavior=True*19:54:24*
> status: PASS*19:54:24* run time: 3 minutes 0.505
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.old_client_throttling_behavior=True*19:54:24*
> status: PASS*19:54:24* run time: 3 minutes 19.629
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.override_quota=False*19:54:24*
> status: PASS*19:54:24* run time: 4 minutes 11.296
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=client-id.override_quota=True*19:54:24*
> status: PASS*19:54:24* run time: 4 minutes 10.578
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=user.override_quota=False*19:54:24*
> status: PASS*19:54:24* run time: 4 minutes 19.187
> seconds*19:54:24*
>
> *19:54:24*
> test_id:
> kafkatest.tests.client.quota_test.QuotaTest.test_quota.quota_type=user.override_quota=True*19:54:24*
> status: PASS*19:54:24* run time: 3 minutes 13.666
> seconds*19:54:24*
>
>
>
>
> Thanks.
> Luke
>
> On Thu, Jul 4, 2024 at 6:01 PM Josep Prat
> wrote:
>
> > Hi Luke,
> >
> > Thanks for the pointer, if you have an environment where you can run the
> > tests I would highly appreciate it!
> >
> > I managed to run this test suite locally and currently only this one
> fails
> > consistently, the rest pass:
> >
> > Module: kafkatest.tests.client.quota_test
> > Class: QuotaTest
> > Method: test_quota
> > Arguments:
> > {
> > "old_client_throttling_behavior": true,
> > "quota_type": "client-id"
> > }
> >
> > Failure:
> > TimeoutError("Timed out waiting 600 seconds for service nodes to
> > finish. These nodes are still alive:
> > ['ProducerPerformanceService-0-140496695824336 node 1 on worker3']")
> > Traceback (most recent call last):
> > File
> >
> "/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/tests/runner_client.py&q
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Luke,
Thanks for the pointer, if you have an environment where you can run the
tests I would highly appreciate it!
I managed to run this test suite locally and currently only this one fails
consistently, the rest pass:
Module: kafkatest.tests.client.quota_test
Class: QuotaTest
Method: test_quota
Arguments:
{
"old_client_throttling_behavior": true,
"quota_type": "client-id"
}
Failure:
TimeoutError("Timed out waiting 600 seconds for service nodes to
finish. These nodes are still alive:
['ProducerPerformanceService-0-140496695824336 node 1 on worker3']")
Traceback (most recent call last):
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/tests/runner_client.py",
line 184, in _do_run
data = self.run_test()
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/tests/runner_client.py",
line 262, in run_test
return self.test_context.function(self.test)
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/mark/_mark.py",
line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/home/jlprat/projects/kafka/tests/kafkatest/tests/client/quota_test.py",
line 157, in test_quota
producer.run()
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/service.py",
line 345, in run
self.wait()
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/background_thread.py",
line 72, in wait
super(BackgroundThreadService, self).wait(timeout_sec)
File
"/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/service.py",
line 293, in wait
raise TimeoutError("Timed out waiting %s seconds for service nodes
to finish. " % str(timeout_sec)
ducktape.errors.TimeoutError: Timed out waiting 600 seconds for
service nodes to finish. These nodes are still alive:
['ProducerPerformanceService-0-140496695824336 node 1 on worker3']
On Thu, Jul 4, 2024 at 11:57 AM Luke Chen wrote:
> Hi Josep,
>
> For this
> - QuotaTest --> speaking with Bruno we suspect there is a problem with the
> test setup, failed with "ValueError: max() arg is an empty sequence"
>
> It's a known issue: KAFKA-16138
> <https://issues.apache.org/jira/browse/KAFKA-16138> .
> It should be passed with local specific tests run.
> Do you want me help verify it by running it in my environment?
>
> Thanks.
> Luke
>
>
>
> On Thu, Jul 4, 2024 at 4:03 PM Josep Prat
> wrote:
>
> > Hi all,
> >
> > We have had 2[1][2] runs of the system tests since the last blocker was
> > merged on 3.8. So far we have 19 tests that failed on both runs. I've
> > compiled them in this list[3].
> >
> > There seems to these different categories of failing tests:
> > - QuotaTest --> speaking with Bruno we suspect there is a problem with
> the
> > test setup, failed with "ValueError: max() arg is an empty sequence"
> > - Streams cooperative rebalance upgrade --> It fails on versions 2.3.1 or
> > older, failed with Timeout
> > - KRaft Upgrade --> from dev with Isolated and combined KRaft, failed
> with
> > RemoteCommandError
> > - Network degrade test -> failed with RemoteCommandError
> > - Replica verification tool test --> Timeout for KRaft, but ZK failed on
> > the first run but worked on the second
> >
> > If someone has further ideas on what could be causing these failures,
> > please let me know. Given holidays in the US, the possible test setup
> > problem might not be able to be fixed today.
> >
> > [1]:
> >
> >
> https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-02--001.05d6b151-356a-47e5-b724-6fcd79493422--1719991984--confluentinc--3.8--49d2ee3db9/report.html
> > [2]:
> >
> >
> https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-03--001.4803d99b-52df-4f6d-82c2-3f050a6207fa--1720038529--apache--3.8--2fbe32ecb9/report.html
> > [3]:
> >
> >
> https://docs.google.com/document/d/1wbcyzO6GM2SYQaqTMITBTBjHgZgM7mmiAt7TUfh1xt8/edit
> >
> > Best,
> >
> > On Tue, Jul 2, 2024 at 7:29 PM Josep Prat wrote:
> >
> > > Hi all,
> > > Thanks for reviewing and merging the latest blockers for 3.8.0.
> Tomorrow,
> > > I will start with the process to get the f
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, We have had 2[1][2] runs of the system tests since the last blocker was merged on 3.8. So far we have 19 tests that failed on both runs. I've compiled them in this list[3]. There seems to these different categories of failing tests: - QuotaTest --> speaking with Bruno we suspect there is a problem with the test setup, failed with "ValueError: max() arg is an empty sequence" - Streams cooperative rebalance upgrade --> It fails on versions 2.3.1 or older, failed with Timeout - KRaft Upgrade --> from dev with Isolated and combined KRaft, failed with RemoteCommandError - Network degrade test -> failed with RemoteCommandError - Replica verification tool test --> Timeout for KRaft, but ZK failed on the first run but worked on the second If someone has further ideas on what could be causing these failures, please let me know. Given holidays in the US, the possible test setup problem might not be able to be fixed today. [1]: https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-02--001.05d6b151-356a-47e5-b724-6fcd79493422--1719991984--confluentinc--3.8--49d2ee3db9/report.html [2]: https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-03--001.4803d99b-52df-4f6d-82c2-3f050a6207fa--1720038529--apache--3.8--2fbe32ecb9/report.html [3]: https://docs.google.com/document/d/1wbcyzO6GM2SYQaqTMITBTBjHgZgM7mmiAt7TUfh1xt8/edit Best, On Tue, Jul 2, 2024 at 7:29 PM Josep Prat wrote: > Hi all, > Thanks for reviewing and merging the latest blockers for 3.8.0. Tomorrow, > I will start with the process to get the first RC out. > > Best! > > On Sat, Jun 29, 2024 at 9:04 PM Josep Prat wrote: > >> Hi Justine, >> >> Marking MV 3.8-IV0 as latest >> production MV is done in this PR (I did both together) >> https://github.com/apache/kafka/pull/16400 >> >> Best, >> >> -- >> Josep Prat >> Open Source Engineering Director, Aiven >> josep.p...@aiven.io | +491715557497 | aiven.io >> Aiven Deutschland GmbH >> Alexanderufer 3-7, 10117 Berlin >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> Amtsgericht Charlottenburg, HRB 209739 B >> >> On Sat, Jun 29, 2024, 00:52 Justine Olshan >> wrote: >> >>> The PR is merged. I lowered the severity of the blocker ticket as we >>> still >>> have the change in trunk to merge. However, the 3.8 release is no longer >>> blocked by KAFKA-17050. >>> I think that was the remaining blocker. The other ones are either already >>> fixed for 3.8 (KAFKA-17011) or diverted to 3.9 (KAFKA-16840) >>> >>> I think there was one more needed change to mark MV 3.8-IV0 as latest >>> production MV. I will follow up with that. >>> >>> Justine >>> >>> On Thu, Jun 27, 2024 at 2:34 PM Justine Olshan >>> wrote: >>> >>> > Here is the PR: https://github.com/apache/kafka/pull/16478 >>> > >>> > Justine >>> > >>> > On Thu, Jun 27, 2024 at 2:21 PM Justine Olshan >>> > wrote: >>> > >>> >> Hey all, >>> >> Thanks for your patience. After some discussion, we decided to revert >>> >> group version from 3.8 since there were too many complexities >>> associated >>> >> with getting it to work. >>> >> I've downgraded the severity of KAFKA-17011 to not be a blocker and >>> >> opened a ticket (https://issues.apache.org/jira/browse/KAFKA-17050) >>> to >>> >> revert from 3.8 (and 3.9) as a blocker instead. I hope to get the PR >>> out >>> >> shortly. >>> >> This one should be less controversial and merged quickly. >>> >> >>> >> Thanks again, >>> >> Justine >>> >> >>> >> On Thu, Jun 27, 2024 at 1:22 AM Josep Prat >>> >>> >> wrote: >>> >> >>> >>> Hi all, >>> >>> >>> >>> I just wanted to ask again for your help in reviewing these 2 last >>> >>> blockers >>> >>> for the 3.8.0 release: >>> >>> https://github.com/apache/kafka/pull/16400 >>> >>> https://github.com/apache/kafka/pull/16420 >>> >>> >>> >>> Thanks! >>> >>> >>> >>> >>> >>> On Mon, Jun 24, 2024 at 9:27 AM Josep Prat >>> wrote: >>> >>> >>> >>> > Hi all, >>> >>> > >>> >>> > We currently have
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, Thanks for reviewing and merging the latest blockers for 3.8.0. Tomorrow, I will start with the process to get the first RC out. Best! On Sat, Jun 29, 2024 at 9:04 PM Josep Prat wrote: > Hi Justine, > > Marking MV 3.8-IV0 as latest > production MV is done in this PR (I did both together) > https://github.com/apache/kafka/pull/16400 > > Best, > > -- > Josep Prat > Open Source Engineering Director, Aiven > josep.p...@aiven.io | +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Sat, Jun 29, 2024, 00:52 Justine Olshan > wrote: > >> The PR is merged. I lowered the severity of the blocker ticket as we still >> have the change in trunk to merge. However, the 3.8 release is no longer >> blocked by KAFKA-17050. >> I think that was the remaining blocker. The other ones are either already >> fixed for 3.8 (KAFKA-17011) or diverted to 3.9 (KAFKA-16840) >> >> I think there was one more needed change to mark MV 3.8-IV0 as latest >> production MV. I will follow up with that. >> >> Justine >> >> On Thu, Jun 27, 2024 at 2:34 PM Justine Olshan >> wrote: >> >> > Here is the PR: https://github.com/apache/kafka/pull/16478 >> > >> > Justine >> > >> > On Thu, Jun 27, 2024 at 2:21 PM Justine Olshan >> > wrote: >> > >> >> Hey all, >> >> Thanks for your patience. After some discussion, we decided to revert >> >> group version from 3.8 since there were too many complexities >> associated >> >> with getting it to work. >> >> I've downgraded the severity of KAFKA-17011 to not be a blocker and >> >> opened a ticket (https://issues.apache.org/jira/browse/KAFKA-17050) to >> >> revert from 3.8 (and 3.9) as a blocker instead. I hope to get the PR >> out >> >> shortly. >> >> This one should be less controversial and merged quickly. >> >> >> >> Thanks again, >> >> Justine >> >> >> >> On Thu, Jun 27, 2024 at 1:22 AM Josep Prat > > >> >> wrote: >> >> >> >>> Hi all, >> >>> >> >>> I just wanted to ask again for your help in reviewing these 2 last >> >>> blockers >> >>> for the 3.8.0 release: >> >>> https://github.com/apache/kafka/pull/16400 >> >>> https://github.com/apache/kafka/pull/16420 >> >>> >> >>> Thanks! >> >>> >> >>> >> >>> On Mon, Jun 24, 2024 at 9:27 AM Josep Prat >> wrote: >> >>> >> >>> > Hi all, >> >>> > >> >>> > We currently have a couple of blockers for the 3.8.0 release. These >> are >> >>> > the following: >> >>> > - Reverting commit KAFKA-16154 and mark latest production metadata >> as >> >>> > 3.8.0: https://github.com/apache/kafka/pull/16400 >> >>> > - Fix some failing system tests: >> >>> > https://github.com/apache/kafka/pull/16420 >> >>> > Can we get some eyes on these 2 PRs? Thanks! >> >>> > >> >>> > To easily track this in feature releases, I created a new label >> called >> >>> > "Blocker" the idea is to mark PRs that are solving an Issue marked >> as >> >>> > "blocker". This might increase visibility and help getting those >> >>> reviewed >> >>> > promptly. Here is the link to the PRs with this label: >> >>> > https://github.com/apache/kafka/labels/Blocker >> >>> > >> >>> > Best, >> >>> > >> >>> > On Thu, Jun 20, 2024 at 7:09 PM Josep Prat >> >>> wrote: >> >>> > >> >>> >> Thanks for the heads up Justine! >> >>> >> >> >>> >> On Thu, Jun 20, 2024 at 5:54 PM Justine Olshan >> >>> >> wrote: >> >>> >> >> >>> >>> Sorry to derail this conversation, but just wanted to share we >> have a >> >>> >>> system test blocker with >> >>> >>> https://issues.apache.org/jira/browse/KAFKA-16990. >> >>> >>> Hopefully we can fix this in the next day or so. >> >>> >>> >> >>>
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Justine, Marking MV 3.8-IV0 as latest production MV is done in this PR (I did both together) https://github.com/apache/kafka/pull/16400 Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Sat, Jun 29, 2024, 00:52 Justine Olshan wrote: > The PR is merged. I lowered the severity of the blocker ticket as we still > have the change in trunk to merge. However, the 3.8 release is no longer > blocked by KAFKA-17050. > I think that was the remaining blocker. The other ones are either already > fixed for 3.8 (KAFKA-17011) or diverted to 3.9 (KAFKA-16840) > > I think there was one more needed change to mark MV 3.8-IV0 as latest > production MV. I will follow up with that. > > Justine > > On Thu, Jun 27, 2024 at 2:34 PM Justine Olshan > wrote: > > > Here is the PR: https://github.com/apache/kafka/pull/16478 > > > > Justine > > > > On Thu, Jun 27, 2024 at 2:21 PM Justine Olshan > > wrote: > > > >> Hey all, > >> Thanks for your patience. After some discussion, we decided to revert > >> group version from 3.8 since there were too many complexities associated > >> with getting it to work. > >> I've downgraded the severity of KAFKA-17011 to not be a blocker and > >> opened a ticket (https://issues.apache.org/jira/browse/KAFKA-17050) to > >> revert from 3.8 (and 3.9) as a blocker instead. I hope to get the PR out > >> shortly. > >> This one should be less controversial and merged quickly. > >> > >> Thanks again, > >> Justine > >> > >> On Thu, Jun 27, 2024 at 1:22 AM Josep Prat > > >> wrote: > >> > >>> Hi all, > >>> > >>> I just wanted to ask again for your help in reviewing these 2 last > >>> blockers > >>> for the 3.8.0 release: > >>> https://github.com/apache/kafka/pull/16400 > >>> https://github.com/apache/kafka/pull/16420 > >>> > >>> Thanks! > >>> > >>> > >>> On Mon, Jun 24, 2024 at 9:27 AM Josep Prat > wrote: > >>> > >>> > Hi all, > >>> > > >>> > We currently have a couple of blockers for the 3.8.0 release. These > are > >>> > the following: > >>> > - Reverting commit KAFKA-16154 and mark latest production metadata as > >>> > 3.8.0: https://github.com/apache/kafka/pull/16400 > >>> > - Fix some failing system tests: > >>> > https://github.com/apache/kafka/pull/16420 > >>> > Can we get some eyes on these 2 PRs? Thanks! > >>> > > >>> > To easily track this in feature releases, I created a new label > called > >>> > "Blocker" the idea is to mark PRs that are solving an Issue marked as > >>> > "blocker". This might increase visibility and help getting those > >>> reviewed > >>> > promptly. Here is the link to the PRs with this label: > >>> > https://github.com/apache/kafka/labels/Blocker > >>> > > >>> > Best, > >>> > > >>> > On Thu, Jun 20, 2024 at 7:09 PM Josep Prat > >>> wrote: > >>> > > >>> >> Thanks for the heads up Justine! > >>> >> > >>> >> On Thu, Jun 20, 2024 at 5:54 PM Justine Olshan > >>> >> wrote: > >>> >> > >>> >>> Sorry to derail this conversation, but just wanted to share we > have a > >>> >>> system test blocker with > >>> >>> https://issues.apache.org/jira/browse/KAFKA-16990. > >>> >>> Hopefully we can fix this in the next day or so. > >>> >>> > >>> >>> Justine > >>> >>> > >>> >>> On Mon, Jun 17, 2024 at 12:19 PM David Jacot < > david.ja...@gmail.com> > >>> >>> wrote: > >>> >>> > >>> >>> > I meant it from a time perspective, not from a branching point > >>> >>> perspective. > >>> >>> > Sorry for the confusion. As said in the other thread, doing it > four > >>> >>> months > >>> >>> > after 3.9 is desirable for KIP-848 as I expect that we will need > >>> time > >>> >>> to > >&g
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, I just wanted to ask again for your help in reviewing these 2 last blockers for the 3.8.0 release: https://github.com/apache/kafka/pull/16400 https://github.com/apache/kafka/pull/16420 Thanks! On Mon, Jun 24, 2024 at 9:27 AM Josep Prat wrote: > Hi all, > > We currently have a couple of blockers for the 3.8.0 release. These are > the following: > - Reverting commit KAFKA-16154 and mark latest production metadata as > 3.8.0: https://github.com/apache/kafka/pull/16400 > - Fix some failing system tests: > https://github.com/apache/kafka/pull/16420 > Can we get some eyes on these 2 PRs? Thanks! > > To easily track this in feature releases, I created a new label called > "Blocker" the idea is to mark PRs that are solving an Issue marked as > "blocker". This might increase visibility and help getting those reviewed > promptly. Here is the link to the PRs with this label: > https://github.com/apache/kafka/labels/Blocker > > Best, > > On Thu, Jun 20, 2024 at 7:09 PM Josep Prat wrote: > >> Thanks for the heads up Justine! >> >> On Thu, Jun 20, 2024 at 5:54 PM Justine Olshan >> wrote: >> >>> Sorry to derail this conversation, but just wanted to share we have a >>> system test blocker with >>> https://issues.apache.org/jira/browse/KAFKA-16990. >>> Hopefully we can fix this in the next day or so. >>> >>> Justine >>> >>> On Mon, Jun 17, 2024 at 12:19 PM David Jacot >>> wrote: >>> >>> > I meant it from a time perspective, not from a branching point >>> perspective. >>> > Sorry for the confusion. As said in the other thread, doing it four >>> months >>> > after 3.9 is desirable for KIP-848 as I expect that we will need time >>> to >>> > stabilize everything after switching all the default configs once 3.9 >>> is >>> > cut. >>> > >>> > David >>> > >>> > Le lun. 17 juin 2024 à 19:33, Matthias J. Sax a >>> écrit : >>> > >>> > > Why would 4.0 be based on 3.8? My understanding is, that it will be >>> > > based on 3.9. >>> > > >>> > > -Matthias >>> > > >>> > > On 6/14/24 11:22 PM, David Jacot wrote: >>> > > > I agree that we should keep 4.0 time-based. My question is based on >>> > which >>> > > > release. If I understand you, you would like to keep it based on >>> 3.8. >>> > > This >>> > > > means that 4.0 would be released in October. It would be helpful >>> to fix >>> > > the >>> > > > dates so we can plan accordingly. I will start a separate thread on >>> > > Monday. >>> > > > >>> > > > David >>> > > > >>> > > > Le sam. 15 juin 2024 à 00:44, Colin McCabe a >>> > écrit >>> > > : >>> > > > >>> > > >> +1. I think it would be good to keep 4.0 time-based. Most of the >>> > > refactors >>> > > >> we want to do are optional in some sense and can be descoped if >>> time >>> > > runs >>> > > >> out. For example, we can drop support for JDK 8 without >>> immediately >>> > > >> refactoring everything that could benefit from the improvements in >>> > > JDK9+. >>> > > >> >>> > > >> best, >>> > > >> Colin >>> > > >> >>> > > >> >>> > > >> On Fri, Jun 14, 2024, at 15:37, Matthias J. Sax wrote: >>> > > >>> That's my understanding, and I would advocate strongly to keep >>> the >>> > 4.0 >>> > > >>> release schedule as planed originally. >>> > > >>> >>> > > >>> The 3.9 one should really be an additional "out of schedule" >>> release >>> > > not >>> > > >>> impacting any other releases. >>> > > >>> >>> > > >>> >>> > > >>> -Matthias >>> > > >>> >>> > > >>> On 6/14/24 1:29 PM, David Jacot wrote: >>> > > >>>> The plan sounds good to me. I suppose that we will follow our >>> > regular >>> > > >>>> cadence for 4.0 and release it four months after 3.9 (in >>> November?). >&g
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, We currently have a couple of blockers for the 3.8.0 release. These are the following: - Reverting commit KAFKA-16154 and mark latest production metadata as 3.8.0: https://github.com/apache/kafka/pull/16400 - Fix some failing system tests: https://github.com/apache/kafka/pull/16420 Can we get some eyes on these 2 PRs? Thanks! To easily track this in feature releases, I created a new label called "Blocker" the idea is to mark PRs that are solving an Issue marked as "blocker". This might increase visibility and help getting those reviewed promptly. Here is the link to the PRs with this label: https://github.com/apache/kafka/labels/Blocker Best, On Thu, Jun 20, 2024 at 7:09 PM Josep Prat wrote: > Thanks for the heads up Justine! > > On Thu, Jun 20, 2024 at 5:54 PM Justine Olshan > wrote: > >> Sorry to derail this conversation, but just wanted to share we have a >> system test blocker with >> https://issues.apache.org/jira/browse/KAFKA-16990. >> Hopefully we can fix this in the next day or so. >> >> Justine >> >> On Mon, Jun 17, 2024 at 12:19 PM David Jacot >> wrote: >> >> > I meant it from a time perspective, not from a branching point >> perspective. >> > Sorry for the confusion. As said in the other thread, doing it four >> months >> > after 3.9 is desirable for KIP-848 as I expect that we will need time to >> > stabilize everything after switching all the default configs once 3.9 is >> > cut. >> > >> > David >> > >> > Le lun. 17 juin 2024 à 19:33, Matthias J. Sax a >> écrit : >> > >> > > Why would 4.0 be based on 3.8? My understanding is, that it will be >> > > based on 3.9. >> > > >> > > -Matthias >> > > >> > > On 6/14/24 11:22 PM, David Jacot wrote: >> > > > I agree that we should keep 4.0 time-based. My question is based on >> > which >> > > > release. If I understand you, you would like to keep it based on >> 3.8. >> > > This >> > > > means that 4.0 would be released in October. It would be helpful to >> fix >> > > the >> > > > dates so we can plan accordingly. I will start a separate thread on >> > > Monday. >> > > > >> > > > David >> > > > >> > > > Le sam. 15 juin 2024 à 00:44, Colin McCabe a >> > écrit >> > > : >> > > > >> > > >> +1. I think it would be good to keep 4.0 time-based. Most of the >> > > refactors >> > > >> we want to do are optional in some sense and can be descoped if >> time >> > > runs >> > > >> out. For example, we can drop support for JDK 8 without immediately >> > > >> refactoring everything that could benefit from the improvements in >> > > JDK9+. >> > > >> >> > > >> best, >> > > >> Colin >> > > >> >> > > >> >> > > >> On Fri, Jun 14, 2024, at 15:37, Matthias J. Sax wrote: >> > > >>> That's my understanding, and I would advocate strongly to keep the >> > 4.0 >> > > >>> release schedule as planed originally. >> > > >>> >> > > >>> The 3.9 one should really be an additional "out of schedule" >> release >> > > not >> > > >>> impacting any other releases. >> > > >>> >> > > >>> >> > > >>> -Matthias >> > > >>> >> > > >>> On 6/14/24 1:29 PM, David Jacot wrote: >> > > >>>> The plan sounds good to me. I suppose that we will follow our >> > regular >> > > >>>> cadence for 4.0 and release it four months after 3.9 (in >> November?). >> > > Is >> > > >>>> this correct? >> > > >>>> >> > > >>>> Best, >> > > >>>> David >> > > >>>> >> > > >>>> Le ven. 14 juin 2024 à 21:57, José Armando García Sancio >> > > >>>> a écrit : >> > > >>>> >> > > >>>>> +1 on the proposed release plan for 3.8. >> > > >>>>> >> > > >>>>> Thanks! >> > > >>>>> >> > > >>>>> On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma >> > > wrote: >> > > &
Re: [DISCUSS] Apache Kafka 3.8.0 release
Thanks for the heads up Justine! On Thu, Jun 20, 2024 at 5:54 PM Justine Olshan wrote: > Sorry to derail this conversation, but just wanted to share we have a > system test blocker with https://issues.apache.org/jira/browse/KAFKA-16990 > . > Hopefully we can fix this in the next day or so. > > Justine > > On Mon, Jun 17, 2024 at 12:19 PM David Jacot > wrote: > > > I meant it from a time perspective, not from a branching point > perspective. > > Sorry for the confusion. As said in the other thread, doing it four > months > > after 3.9 is desirable for KIP-848 as I expect that we will need time to > > stabilize everything after switching all the default configs once 3.9 is > > cut. > > > > David > > > > Le lun. 17 juin 2024 à 19:33, Matthias J. Sax a > écrit : > > > > > Why would 4.0 be based on 3.8? My understanding is, that it will be > > > based on 3.9. > > > > > > -Matthias > > > > > > On 6/14/24 11:22 PM, David Jacot wrote: > > > > I agree that we should keep 4.0 time-based. My question is based on > > which > > > > release. If I understand you, you would like to keep it based on 3.8. > > > This > > > > means that 4.0 would be released in October. It would be helpful to > fix > > > the > > > > dates so we can plan accordingly. I will start a separate thread on > > > Monday. > > > > > > > > David > > > > > > > > Le sam. 15 juin 2024 à 00:44, Colin McCabe a > > écrit > > > : > > > > > > > >> +1. I think it would be good to keep 4.0 time-based. Most of the > > > refactors > > > >> we want to do are optional in some sense and can be descoped if time > > > runs > > > >> out. For example, we can drop support for JDK 8 without immediately > > > >> refactoring everything that could benefit from the improvements in > > > JDK9+. > > > >> > > > >> best, > > > >> Colin > > > >> > > > >> > > > >> On Fri, Jun 14, 2024, at 15:37, Matthias J. Sax wrote: > > > >>> That's my understanding, and I would advocate strongly to keep the > > 4.0 > > > >>> release schedule as planed originally. > > > >>> > > > >>> The 3.9 one should really be an additional "out of schedule" > release > > > not > > > >>> impacting any other releases. > > > >>> > > > >>> > > > >>> -Matthias > > > >>> > > > >>> On 6/14/24 1:29 PM, David Jacot wrote: > > > >>>> The plan sounds good to me. I suppose that we will follow our > > regular > > > >>>> cadence for 4.0 and release it four months after 3.9 (in > November?). > > > Is > > > >>>> this correct? > > > >>>> > > > >>>> Best, > > > >>>> David > > > >>>> > > > >>>> Le ven. 14 juin 2024 à 21:57, José Armando García Sancio > > > >>>> a écrit : > > > >>>> > > > >>>>> +1 on the proposed release plan for 3.8. > > > >>>>> > > > >>>>> Thanks! > > > >>>>> > > > >>>>> On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma > > > wrote: > > > >>>>>> > > > >>>>>> +1 to the plan we converged on in this thread. > > > >>>>>> > > > >>>>>> Ismael > > > >>>>>> > > > >>>>>> On Fri, Jun 14, 2024 at 10:46 AM Josep Prat > > > >> > > >>>>>> > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Hi all, > > > >>>>>>> > > > >>>>>>> Thanks Colin, yes go ahead. > > > >>>>>>> > > > >>>>>>> As we are now past code freeze I would like to ask everyone > > > involved > > > >>>>> in a > > > >>>>>>> KIP that is not yet complete, to verify if what landed on the > 3.8 > > > >>>>> branch > > > >>>>>>> needs to be reverted or if it can stay. Additionally, I'll be > > > pinging &
[jira] [Resolved] (KAFKA-15045) Move Streams task assignor to public configs
[ https://issues.apache.org/jira/browse/KAFKA-15045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat resolved KAFKA-15045. Resolution: Fixed > Move Streams task assignor to public configs > > > Key: KAFKA-15045 > URL: https://issues.apache.org/jira/browse/KAFKA-15045 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: A. Sophie Blee-Goldman >Assignee: A. Sophie Blee-Goldman >Priority: Major > Labels: kip > Fix For: 3.8.0 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-924%3A+customizable+task+assignment+for+Streams -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-16988) InsufficientResourcesError in ConnectDistributedTest system test
[
https://issues.apache.org/jira/browse/KAFKA-16988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josep Prat resolved KAFKA-16988.
Resolution: Fixed
> InsufficientResourcesError in ConnectDistributedTest system test
>
>
> Key: KAFKA-16988
> URL: https://issues.apache.org/jira/browse/KAFKA-16988
> Project: Kafka
> Issue Type: Bug
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.8.0, 3.7.1
>
>
> Saw InsufficientResourcesError when running
> `ConnectDistributedTest#test_exactly_once_source` system test.
>
> {code:java}
> name="test_exactly_once_source_clean=False_connect_protocol=compatible_metadata_quorum=ZK_use_new_coordinator=False"
> classname="kafkatest.tests.connect.connect_distributed_test"
> time="403.812"> requested: 1. linux nodes available: 0')"
> type="exception">InsufficientResourcesError('linux nodes requested: 1. linux
> nodes available: 0') Traceback (most recent call last): File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run data = self.run_test() File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test return self.test_context.function(self.test) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in
> wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/connect/connect_distributed_test.py",
> line 928, in test_exactly_once_source consumer_validator =
> ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic,
> consumer_timeout_ms=1000, print_key=True) File
> "/opt/kafka-dev/tests/kafkatest/services/console_consumer.py", line 97, in
> __init__ BackgroundThreadService.__init__(self, context, num_nodes) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/services/background_thread.py",
> line 26, in __init__ super(BackgroundThreadService, self).__init__(context,
> num_nodes, cluster_spec, *args, **kwargs) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/services/service.py", line
> 107, in __init__ self.allocate_nodes() File
> "/usr/local/lib/python3.9/dist-packages/ducktape/services/service.py", line
> 217, in allocate_nodes self.nodes = self.cluster.alloc(self.cluster_spec)
> File "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/cluster.py",
> line 54, in alloc allocated = self.do_alloc(cluster_spec) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/finite_subcluster.py",
> line 37, in do_alloc good_nodes, bad_nodes =
> self._available_nodes.remove_spec(cluster_spec) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/cluster/node_container.py",
> line 131, in remove_spec raise InsufficientResourcesError(err)
> ducktape.cluster.node_container.InsufficientResourcesError: linux nodes
> requested: 1. linux nodes available: 0
> name="test_exactly_once_source_clean=False_connect_protocol=sessioned_metadata_quorum=ZK_use_new_coordinator=False"
> classname="kafkatest.tests.connect.connect_distributed_test"
> time="376.160"> requested: 1. linux nodes available: 0')"
> type="exception">InsufficientResourcesError('linux nodes requested: 1. linux
> nodes available: 0') Traceback (most recent call last): File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 186, in _do_run data = self.run_test() File
> "/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py",
> line 246, in run_test return self.test_context.function(self.test) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in
> wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File
> "/opt/kafka-dev/tests/kafkatest/tests/connect/connect_distributed_test.py",
> line 928, in test_exactly_once_source consumer_validator =
> ConsoleConsumer(self.test_context, 1, self.kafka, self.source.topic,
> consumer_timeout_ms=1000, print_key=True) File
> "/opt/kafka-dev/tests/kafkatest/services/console_consumer.py", line 97, in
> __init__ BackgroundThreadService.__init__(self, context, num_nodes) File
> "/usr/local/lib/python3.9/dist-packages/ducktape/services/background_thread.py",
> line 26, in __ini
[jira] [Resolved] (KAFKA-16373) Docker Official Image for Apache Kafka
[ https://issues.apache.org/jira/browse/KAFKA-16373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josep Prat resolved KAFKA-16373. Resolution: Fixed > Docker Official Image for Apache Kafka > -- > > Key: KAFKA-16373 > URL: https://issues.apache.org/jira/browse/KAFKA-16373 > Project: Kafka > Issue Type: New Feature >Affects Versions: 3.8.0 >Reporter: Krish Vora >Assignee: Krish Vora >Priority: Major > Labels: KIP-1028 > Fix For: 3.8.0 > > > KIP-1028: Docker Official Image for Apache Kafka: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-1028%3A+Docker+Official+Image+for+Apache+Kafka] > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Apache Kafka 3.9.0 release
+1 from me. Thanks Colin! Best, -- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Jun 14, 2024, 20:00 Alyssa Huang wrote: > Thanks Colin, +1 from me. > > On Fri, Jun 14, 2024 at 10:55 AM Colin McCabe wrote: > > > Hi all, > > > > Based on the discussion in the 3.8 release thread, we will need a 3.9 > > release. I guess I should create a new email thread to discuss this. > > > > I'd like to volunteer as release manager for the Apache Kafka 3.9.0 > > release. There's a release plan here: > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0 > > > > best, > > Colin > > >
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, Thanks Colin, yes go ahead. As we are now past code freeze I would like to ask everyone involved in a KIP that is not yet complete, to verify if what landed on the 3.8 branch needs to be reverted or if it can stay. Additionally, I'll be pinging KIPs and Jira reporters asking for their status as some Jiras seem to have all related GitHub PRs merged but their status is still Open or In Progress. I'll be checking all the open blockers and check if they are really a blocker or can be pushed. Regarding timeline, I'll attempt to generate the first RC on Wednesday or Thursday, so please revert any changes you deem necessary by then. If you need more time, please ping me. Best, ----- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Fri, Jun 14, 2024, 19:25 Colin McCabe wrote: > Hi all, > > We have had many delays with releases this year. We should try to get back > on schedule. > > I agree with the idea that was proposed a few times in this thread of > drawing a line under 3.8 now, and doing a short 3.9 release. I posted a 3.9 > release plan here: > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0 > > I think we could start doing RCs for 3.8.0 as early as next week. There > are a few things that need to be reverted first (anything related to > KIP-853 or KIP-966). > > Josep, if you agree, I will update KIP-1012 to reflect that these things > are landing in 3.9 rather than 3.8. And we can start doing all the normal > release stuff. The main blocker JIRA I'm aware of is KAFKA-16946, which is > a very simple fix. > > best, > Colin > > > On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote: > > +1 on going with 3.8 release with the existing plan and targeting the > > required features in 3.9 timelines. 4.0 will be targeted in the usual > > cycle(4 months) after 3.9 is released. > > > > > > On Fri, 14 Jun 2024 at 15:19, Edoardo Comar > wrote: > >> > >> Josep, > >> past the deadline sorry but I can't see reasons not to cherry-pick this > >> https://github.com/apache/kafka/pull/16326 > >> > >> On Wed, 12 Jun 2024 at 17:14, Josep Prat > wrote: > >> > > >> > Hi Edoardo, > >> > > >> > Correct, you can still cherry-pick this one. > >> > > >> > I'll send an email tomorrow morning (CEST) asking maintainers to stop > >> > cherry picking commits unless we discuss it beforehand. > >> > > >> > Best, > >> > > >> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar > wrote: > >> > > >> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0 > >> > > https://github.com/apache/kafka/pull/16229 > >> > > > >> > > right? > >> > > thanks > >> > > > >> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko > wrote: > >> > > > > >> > > > Hi, > >> > > > > >> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner > today, > >> > > but please proceed with the release if I don't manage. > >> > > > > >> > > > Ivan > >> > > > > >> > > > [1] https://github.com/apache/kafka/pull/13277 > >> > > > > >> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote: > >> > > > > Hi Luke, > >> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0 > (but he > >> > > can > >> > > > > confirm this). My question now would be, given that it seems we > would > >> > > need > >> > > > > a v3.9.0, do you think it's important to include > >> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0? > >> > > > > > >> > > > > Best, > >> > > > > > >> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen > wrote: > >> > > > > > >> > > > > > Hi Josep > >> > > > > > > >> > > > > > For KIP-966, I think Calvin had mentioned he won't complete in > >> > > v3.8.0. > >> > > > > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw > >> > &
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Justine, I know we discarded parallel branching, but it was under the scope of 3.8.0 and with the KIPs no yet approved. We could also not do a parallel release, but rather "quick" 3.9 and then start with 4.0. Best ----- Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Thu, Jun 13, 2024, 22:08 Josep Prat wrote: > Hi Sophie, > > I have a call tomorrow with José to clarify the estimates for KIP-853. > I also wouldn't like to delay the release for a month or more. > > Regarding your proposal, I find it would be a good way forward, +1 from my > side. > > > I also find this release and what should include is a hot topic. > What do others think? > > Best, > > > Josep Prat > Open Source Engineering Director, Aiven josep.p...@aiven.io | > +491715557497 | aiven.io > Aiven Deutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Thu, Jun 13, 2024, 21:23 Sophie Blee-Goldman > wrote: > >> Hey all -- was just wondering where we currently stand Re: delaying 3. for >> the KRaft KIPs vs doing a 3.9 release >> >> I know we don't want to have to wait for a whole release cycle to ship >> these KRaft features, but delaying 3.8 up to month is also rather >> difficult >> to swallow. I just wanted to throw an unusual idea out there and see if >> this might be a possible compromise, or is out of the question: >> >> What if we proceed with the 3.8 release as-is, without these KIPs, but >> rather than doing 3.9 in another 3 months we continue on with the plan to >> ship 4.0 in the fall and simply do a very small 3.9 release just for the >> KRaft KIPs once they are finished? >> >> I know this breaks our usual release cycle and may be confusing to some >> users, but many of us are waiting on things being shipped in 3.8 and >> delaying it by another month (perhaps more) feels unfair. At the same >> time, >> I understand the need to ship these KRaft KIPs before 4.0 and that there >> are others who would consider it unfair if the KRaft KIPs were delayed >> until the next release cycle. So it seems like an easy win-win to make >> everyone happy by shipping 3.8 now and shipping the KRaft KIPs whenever >> they are ready. This also removes the pressure on these KIPs to rush >> everything in or cut scope, and would give them some breathing room to >> take >> the time they need. >> >> Thoughts? >> >> On Wed, Jun 12, 2024 at 11:44 PM Josep Prat >> wrote: >> >> > Hi all, >> > >> > We are now past the code freeze for the 3.8.0 release. If you think a >> > commit should be backported to the 3.8 branch, please ping me in the PR >> > (@jlprat). >> > >> > Thanks! >> > >> > On Wed, Jun 12, 2024 at 7:22 PM José Armando García Sancio >> > wrote: >> > >> > > Hi Josep, >> > > >> > > See my comment below. >> > > >> > > On Wed, Jun 12, 2024 at 1:17 PM Josep Prat >> >> > > wrote: >> > > > How long do you think it will take to bring KIP-853 to completion? >> > > >> > > We are still missing a few issues/jiras that need to get implemented >> > > for the feature to be usable. I would say a few more weeks. May be >> > > early July or mid July. >> > > >> > > Thanks, >> > > -- >> > > -José >> > > >> > >> > >> > -- >> > [image: Aiven] <https://www.aiven.io> >> > >> > *Josep Prat* >> > Open Source Engineering Director, *Aiven* >> > josep.p...@aiven.io | +491715557497 >> > aiven.io <https://www.aiven.io> | < >> https://www.facebook.com/aivencloud >> > > >> > <https://www.linkedin.com/company/aiven/> < >> > https://twitter.com/aiven_io> >> > *Aiven Deutschland GmbH* >> > Alexanderufer 3-7, 10117 Berlin >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> > Amtsgericht Charlottenburg, HRB 209739 B >> > >> >
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Sophie, I have a call tomorrow with José to clarify the estimates for KIP-853. I also wouldn't like to delay the release for a month or more. Regarding your proposal, I find it would be a good way forward, +1 from my side. I also find this release and what should include is a hot topic. What do others think? Best, Josep Prat Open Source Engineering Director, Aiven josep.p...@aiven.io | +491715557497 | aiven.io Aiven Deutschland GmbH Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B On Thu, Jun 13, 2024, 21:23 Sophie Blee-Goldman wrote: > Hey all -- was just wondering where we currently stand Re: delaying 3. for > the KRaft KIPs vs doing a 3.9 release > > I know we don't want to have to wait for a whole release cycle to ship > these KRaft features, but delaying 3.8 up to month is also rather difficult > to swallow. I just wanted to throw an unusual idea out there and see if > this might be a possible compromise, or is out of the question: > > What if we proceed with the 3.8 release as-is, without these KIPs, but > rather than doing 3.9 in another 3 months we continue on with the plan to > ship 4.0 in the fall and simply do a very small 3.9 release just for the > KRaft KIPs once they are finished? > > I know this breaks our usual release cycle and may be confusing to some > users, but many of us are waiting on things being shipped in 3.8 and > delaying it by another month (perhaps more) feels unfair. At the same time, > I understand the need to ship these KRaft KIPs before 4.0 and that there > are others who would consider it unfair if the KRaft KIPs were delayed > until the next release cycle. So it seems like an easy win-win to make > everyone happy by shipping 3.8 now and shipping the KRaft KIPs whenever > they are ready. This also removes the pressure on these KIPs to rush > everything in or cut scope, and would give them some breathing room to take > the time they need. > > Thoughts? > > On Wed, Jun 12, 2024 at 11:44 PM Josep Prat > wrote: > > > Hi all, > > > > We are now past the code freeze for the 3.8.0 release. If you think a > > commit should be backported to the 3.8 branch, please ping me in the PR > > (@jlprat). > > > > Thanks! > > > > On Wed, Jun 12, 2024 at 7:22 PM José Armando García Sancio > > wrote: > > > > > Hi Josep, > > > > > > See my comment below. > > > > > > On Wed, Jun 12, 2024 at 1:17 PM Josep Prat > > > > wrote: > > > > How long do you think it will take to bring KIP-853 to completion? > > > > > > We are still missing a few issues/jiras that need to get implemented > > > for the feature to be usable. I would say a few more weeks. May be > > > early July or mid July. > > > > > > Thanks, > > > -- > > > -José > > > > > > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > >
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, We are now past the code freeze for the 3.8.0 release. If you think a commit should be backported to the 3.8 branch, please ping me in the PR (@jlprat). Thanks! On Wed, Jun 12, 2024 at 7:22 PM José Armando García Sancio wrote: > Hi Josep, > > See my comment below. > > On Wed, Jun 12, 2024 at 1:17 PM Josep Prat > wrote: > > How long do you think it will take to bring KIP-853 to completion? > > We are still missing a few issues/jiras that need to get implemented > for the feature to be usable. I would say a few more weeks. May be > early July or mid July. > > Thanks, > -- > -José > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Colin, How long do you think it will take to bring KIP-853 to completion? Best, On Wed, Jun 12, 2024 at 6:42 PM Colin McCabe wrote: > Hi Josep, > > KIP-966 is not needed for feature parity with ZK. Indeed, KIP-966 will not > be implemented in ZK at all (and never was). > > There is a PR related to honoring dynamic configs for > unclean.leader.election which we did agree to do for 3.8. It's under review > and we hope to get it in soon. > > We still plan on having KIP-853 in 3.8. In fact, it will be the main > feature of that release. > > best, > Colin > > On Wed, Jun 12, 2024, at 01:50, Josep Prat wrote: > > Hi all, > > > > We are now really close to the planned code freeze deadline (today EOD). > > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we > > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs > > identified that would achieve this are: KIP-853 [2] and KIP-966 [3]. > > At the moment of writing this email both KIPs are not completed. My > > question to the people driving both KIPs would be, how much more time do > > you think it's needed to bring these KIPs to completion? > > > > - If the time needed would be short, we could still include these 2 KIPs > in > > the release. > > - However, if the time needed would be on the scale of weeks, we should > > continue with the release plan for 3.8 and after start working on the 3.9 > > release. > > > > What are your thoughts? > > > > > > [1]: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release > > [2]: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes > > [3]: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat wrote: > > > >> Hi Rajini, > >> Yes, we could backport this one to the 3.8 branch. Would you be able to > do > >> this once you merge this PR? > >> > >> Thanks > >> > >> On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram < > rajinisiva...@gmail.com> > >> wrote: > >> > >>> Hi Josep, > >>> > >>> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks > ready > >>> to be merged (waiting for the PR build).The PR changes several files, > but > >>> is relatively straightforward and not risky. Also the changes are > under a > >>> config that is not enabled by default. Since the KIP was approved > before > >>> KIP freeze, will it be ok to include in 3.8.0? > >>> > >>> Thank you, > >>> > >>> Rajini > >>> > >>> > >>> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat > > >>> wrote: > >>> > >>> > Hi all, > >>> > > >>> > I just want to remind everybody that the code freeze deadline is > >>> > approaching. June 12th EOD is the deadline. > >>> > > >>> > Please do not automatically backport any commit to the 3.8 branch > after > >>> > June 12th EOD. Ping me if you think the commit should be backported > >>> (fixes > >>> > failures in the branch or critical bug fixes). > >>> > > >>> > Thanks all! > >>> > > >>> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio > >>> > wrote: > >>> > > >>> > > Hi Josep, > >>> > > > >>> > > See my comments below. > >>> > > > >>> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat > >>> >>> > > > >>> > > wrote: > >>> > > > So I would propose to leave the deadlines as they are and > manually > >>> > cherry > >>> > > > pick the commits related to KIP-853 and KIP-966. > >>> > > > >>> > > Your proposal sounds good to me. I suspect that will be doing > feature > >>> > > development for KIP-853 past the feature freeze and code freeze > date. > >>> > > Maybe feature development will be finished around the end of June. > >>> > > > >>> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch > >>> > > once we have one. > >>> > > > >>> > > Thanks, > >
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Edoardo, Correct, you can still cherry-pick this one. I'll send an email tomorrow morning (CEST) asking maintainers to stop cherry picking commits unless we discuss it beforehand. Best, On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar wrote: > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0 > https://github.com/apache/kafka/pull/16229 > > right? > thanks > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko wrote: > > > > Hi, > > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner today, > but please proceed with the release if I don't manage. > > > > Ivan > > > > [1] https://github.com/apache/kafka/pull/13277 > > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote: > > > Hi Luke, > > > I think Jose, also mentioned that it won't be ready for v3.8.0 (but he > can > > > confirm this). My question now would be, given that it seems we would > need > > > a v3.9.0, do you think it's important to include > > > https://github.com/apache/kafka/pull/16284 in v3.8.0? > > > > > > Best, > > > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen wrote: > > > > > > > Hi Josep > > > > > > > > For KIP-966, I think Calvin had mentioned he won't complete in > v3.8.0. > > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw > > > > > > > > For unclean leader election, all we need is this PR: > > > > https://github.com/apache/kafka/pull/16284 > > > > For this PR, I think it needs one more week to be completed. > > > > > > > > Thanks. > > > > Luke > > > > > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > We are now really close to the planned code freeze deadline (today > EOD). > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch > until we > > > > > achieve feature parity regarding Zookeeper and KRaft. The two main > KIPs > > > > > identified that would achieve this are: KIP-853 [2] and KIP-966 > [3]. > > > > > At the moment of writing this email both KIPs are not completed. My > > > > > question to the people driving both KIPs would be, how much more > time do > > > > > you think it's needed to bring these KIPs to completion? > > > > > > > > > > - If the time needed would be short, we could still include these > 2 KIPs > > > > in > > > > > the release. > > > > > - However, if the time needed would be on the scale of weeks, we > should > > > > > continue with the release plan for 3.8 and after start working on > the 3.9 > > > > > release. > > > > > > > > > > What are your thoughts? > > > > > > > > > > > > > > > [1]: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release > > > > > [2]: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes > > > > > [3]: > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas > > > > > > > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat > wrote: > > > > > > > > > > > Hi Rajini, > > > > > > Yes, we could backport this one to the 3.8 branch. Would you be > able to > > > > > do > > > > > > this once you merge this PR? > > > > > > > > > > > > Thanks > > > > > > > > > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram < > > > > rajinisiva...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hi Josep, > > > > > >> > > > > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 > looks > > > > > ready > > > > > >> to be merged (waiting for the PR build).The PR changes several > files, > > > > > but > > > > > >> is relatively straightforward and not risky
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Luke, I think Jose, also mentioned that it won't be ready for v3.8.0 (but he can confirm this). My question now would be, given that it seems we would need a v3.9.0, do you think it's important to include https://github.com/apache/kafka/pull/16284 in v3.8.0? Best, On Wed, Jun 12, 2024 at 11:40 AM Luke Chen wrote: > Hi Josep > > For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0. > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw > > For unclean leader election, all we need is this PR: > https://github.com/apache/kafka/pull/16284 > For this PR, I think it needs one more week to be completed. > > Thanks. > Luke > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat > wrote: > > > Hi all, > > > > We are now really close to the planned code freeze deadline (today EOD). > > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we > > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs > > identified that would achieve this are: KIP-853 [2] and KIP-966 [3]. > > At the moment of writing this email both KIPs are not completed. My > > question to the people driving both KIPs would be, how much more time do > > you think it's needed to bring these KIPs to completion? > > > > - If the time needed would be short, we could still include these 2 KIPs > in > > the release. > > - However, if the time needed would be on the scale of weeks, we should > > continue with the release plan for 3.8 and after start working on the 3.9 > > release. > > > > What are your thoughts? > > > > > > [1]: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release > > [2]: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes > > [3]: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat wrote: > > > > > Hi Rajini, > > > Yes, we could backport this one to the 3.8 branch. Would you be able to > > do > > > this once you merge this PR? > > > > > > Thanks > > > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram < > rajinisiva...@gmail.com > > > > > > wrote: > > > > > >> Hi Josep, > > >> > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks > > ready > > >> to be merged (waiting for the PR build).The PR changes several files, > > but > > >> is relatively straightforward and not risky. Also the changes are > under > > a > > >> config that is not enabled by default. Since the KIP was approved > before > > >> KIP freeze, will it be ok to include in 3.8.0? > > >> > > >> Thank you, > > >> > > >> Rajini > > >> > > >> > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat > > > > > >> wrote: > > >> > > >> > Hi all, > > >> > > > >> > I just want to remind everybody that the code freeze deadline is > > >> > approaching. June 12th EOD is the deadline. > > >> > > > >> > Please do not automatically backport any commit to the 3.8 branch > > after > > >> > June 12th EOD. Ping me if you think the commit should be backported > > >> (fixes > > >> > failures in the branch or critical bug fixes). > > >> > > > >> > Thanks all! > > >> > > > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio > > >> > wrote: > > >> > > > >> > > Hi Josep, > > >> > > > > >> > > See my comments below. > > >> > > > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat > > >> > >> > > > > >> > > wrote: > > >> > > > So I would propose to leave the deadlines as they are and > manually > > >> > cherry > > >> > > > pick the commits related to KIP-853 and KIP-966. > > >> > > > > >> > > Your proposal sounds good to me. I suspect that will be doing > > feature > > >> > > development for KIP-853 past the feature freeze and code freeze > > date. > > >> > > Maybe feature development will be finished around the end of June. >
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, We are now really close to the planned code freeze deadline (today EOD). According to KIP-1012 [1] we agreed to stay in the 3.x branch until we achieve feature parity regarding Zookeeper and KRaft. The two main KIPs identified that would achieve this are: KIP-853 [2] and KIP-966 [3]. At the moment of writing this email both KIPs are not completed. My question to the people driving both KIPs would be, how much more time do you think it's needed to bring these KIPs to completion? - If the time needed would be short, we could still include these 2 KIPs in the release. - However, if the time needed would be on the scale of weeks, we should continue with the release plan for 3.8 and after start working on the 3.9 release. What are your thoughts? [1]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release [2]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes [3]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas On Wed, Jun 12, 2024 at 10:40 AM Josep Prat wrote: > Hi Rajini, > Yes, we could backport this one to the 3.8 branch. Would you be able to do > this once you merge this PR? > > Thanks > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram > wrote: > >> Hi Josep, >> >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks ready >> to be merged (waiting for the PR build).The PR changes several files, but >> is relatively straightforward and not risky. Also the changes are under a >> config that is not enabled by default. Since the KIP was approved before >> KIP freeze, will it be ok to include in 3.8.0? >> >> Thank you, >> >> Rajini >> >> >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat >> wrote: >> >> > Hi all, >> > >> > I just want to remind everybody that the code freeze deadline is >> > approaching. June 12th EOD is the deadline. >> > >> > Please do not automatically backport any commit to the 3.8 branch after >> > June 12th EOD. Ping me if you think the commit should be backported >> (fixes >> > failures in the branch or critical bug fixes). >> > >> > Thanks all! >> > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio >> > wrote: >> > >> > > Hi Josep, >> > > >> > > See my comments below. >> > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat >> > > > >> > > wrote: >> > > > So I would propose to leave the deadlines as they are and manually >> > cherry >> > > > pick the commits related to KIP-853 and KIP-966. >> > > >> > > Your proposal sounds good to me. I suspect that will be doing feature >> > > development for KIP-853 past the feature freeze and code freeze date. >> > > Maybe feature development will be finished around the end of June. >> > > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch >> > > once we have one. >> > > >> > > Thanks, >> > > -- >> > > -José >> > > >> > >> > >> > -- >> > [image: Aiven] <https://www.aiven.io> >> > >> > *Josep Prat* >> > Open Source Engineering Director, *Aiven* >> > josep.p...@aiven.io | +491715557497 >> > aiven.io <https://www.aiven.io> | < >> https://www.facebook.com/aivencloud >> > > >> > <https://www.linkedin.com/company/aiven/> < >> > https://twitter.com/aiven_io> >> > *Aiven Deutschland GmbH* >> > Alexanderufer 3-7, 10117 Berlin >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> > Amtsgericht Charlottenburg, HRB 209739 B >> > >> > > > -- > [image: Aiven] <https://www.aiven.io> > > *Josep Prat* > Open Source Engineering Director, *Aiven* > josep.p...@aiven.io | +491715557497 > aiven.io <https://www.aiven.io> | > <https://www.facebook.com/aivencloud> > <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> > *Aiven Deutschland GmbH* > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi Rajini, Yes, we could backport this one to the 3.8 branch. Would you be able to do this once you merge this PR? Thanks On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram wrote: > Hi Josep, > > The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks ready > to be merged (waiting for the PR build).The PR changes several files, but > is relatively straightforward and not risky. Also the changes are under a > config that is not enabled by default. Since the KIP was approved before > KIP freeze, will it be ok to include in 3.8.0? > > Thank you, > > Rajini > > > On Tue, Jun 11, 2024 at 9:35 AM Josep Prat > wrote: > > > Hi all, > > > > I just want to remind everybody that the code freeze deadline is > > approaching. June 12th EOD is the deadline. > > > > Please do not automatically backport any commit to the 3.8 branch after > > June 12th EOD. Ping me if you think the commit should be backported > (fixes > > failures in the branch or critical bug fixes). > > > > Thanks all! > > > > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio > > wrote: > > > > > Hi Josep, > > > > > > See my comments below. > > > > > > On Wed, May 29, 2024 at 10:52 AM Josep Prat > > > > > > wrote: > > > > So I would propose to leave the deadlines as they are and manually > > cherry > > > > pick the commits related to KIP-853 and KIP-966. > > > > > > Your proposal sounds good to me. I suspect that will be doing feature > > > development for KIP-853 past the feature freeze and code freeze date. > > > Maybe feature development will be finished around the end of June. > > > > > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch > > > once we have one. > > > > > > Thanks, > > > -- > > > -José > > > > > > > > > -- > > [image: Aiven] <https://www.aiven.io> > > > > *Josep Prat* > > Open Source Engineering Director, *Aiven* > > josep.p...@aiven.io | +491715557497 > > aiven.io <https://www.aiven.io> | < > https://www.facebook.com/aivencloud > > > > > <https://www.linkedin.com/company/aiven/> < > > https://twitter.com/aiven_io> > > *Aiven Deutschland GmbH* > > Alexanderufer 3-7, 10117 Berlin > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > Amtsgericht Charlottenburg, HRB 209739 B > > > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: [DISCUSS] Apache Kafka 3.8.0 release
Hi all, I just want to remind everybody that the code freeze deadline is approaching. June 12th EOD is the deadline. Please do not automatically backport any commit to the 3.8 branch after June 12th EOD. Ping me if you think the commit should be backported (fixes failures in the branch or critical bug fixes). Thanks all! On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio wrote: > Hi Josep, > > See my comments below. > > On Wed, May 29, 2024 at 10:52 AM Josep Prat > wrote: > > So I would propose to leave the deadlines as they are and manually cherry > > pick the commits related to KIP-853 and KIP-966. > > Your proposal sounds good to me. I suspect that will be doing feature > development for KIP-853 past the feature freeze and code freeze date. > Maybe feature development will be finished around the end of June. > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch > once we have one. > > Thanks, > -- > -José > -- [image: Aiven] <https://www.aiven.io> *Josep Prat* Open Source Engineering Director, *Aiven* josep.p...@aiven.io | +491715557497 aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io> *Aiven Deutschland GmbH* Alexanderufer 3-7, 10117 Berlin Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen Amtsgericht Charlottenburg, HRB 209739 B
Re: Access to Apache Pony Mail
Hi Eric, If I understand correctly your problem I think you might be all set. Apache Pony mail is mostly used as an archive for emails (yes, Apache members can log in and send emails through it) but for the purpose of a KIP you don't need any account there. In case this is what you might be having troubles with, let me detail how a KIP process works step by step: - Write the KIP in the wiki - Once you are happy with the content, open a DISCUSS thread in the mailing list - This is achieved by sending an email to "dev@kafka.apache.org" and the title should follow this format: [DISCUSS] KIP-xxx: Title of the kip. - Wait some minutes and get the link from Pony mail and paste it to your KIP wiki page (it can take 5 minutes or more for Pony mail to show your email) - You hopefully get feedback on the KIP - Once the feedback is addressed or some days have passed since the DISCUSS email has been sent, you can open a vote thread - Similarly to the discuss thread, this is achieved by sending an email to " dev@kafka.apache.org" with a title like: [VOTE] KIP-xxx: Title of the kip - Wait for votes - Once you have 3 +1 binding votes (from maintainers) and 72 hours have passed (also no -1 votes) you can close the vote - You have your KIP approved Let me know if this is the type of answer you were looking for. Best,
