[jira] [Created] (KAFKA-16382) Kafka Streams drop NULL values after reset

[Stanislav Spiridonov (Jira)]

Stanislav Spiridonov created KAFKA-16382:


 Summary: Kafka Streams drop NULL values after reset
 Key: KAFKA-16382
 URL: https://issues.apache.org/jira/browse/KAFKA-16382
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.6.1
Reporter: Stanislav Spiridonov


Kafka Streams (KTable) drops null values after full reset.

See 
[https://github.com/foal/Null-Issue/blob/main/src/main/java/NullProblemExample.java]
 for sample topology

Step to reproduce (req NULL-IN, NULL-IN-AUX, NULL-OUT topics)
 # Start example - 1st round
 # Send to NULL-IN "A1:a" -> NULL-OUT "A1:anull"
 # Send to NULL-IN-AUX "A1:b" -> NULL-OUT "A1:anull, A1:ab"
 # Stop application 
 # Run kafka-streams-application-reset 
{code:java}
call bin/windows/kafka-streams-application-reset --application-id 
nullproblem-example^
 --input-topics "NULL-IN,NULL-IN-AUX"^
 --bootstrap-server "localhost:9092"
{code}

 # Send to NULL-IN-AUX "A1:" -> NULL-OUT "A1:anull, A1:ab" - it is Ok (no app 
running yet)
 # Start example - 2nd round
 # After initialization -> NULL-OUT *still contains* "A1:anull, A1:ab"
 # Expected output "A1:anull, A1:ab, A1:"

The issue is NOT reproduced if application just restarted (skip step 5). 

The issue is NOT reproduced if internal cache is disabled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Manikumar]

Hi,

I have backported a couple of CVE related fixes to the 3.6 branch.
https://issues.apache.org/jira/browse/KAFKA-16210
https://issues.apache.org/jira/browse/KAFKA-16322


We have one non-blocker issue to be resolved.
https://issues.apache.org/jira/browse/KAFKA-16073

Thanks,

On Fri, Mar 15, 2024 at 10:18 AM Ismael Juma  wrote:

> Hi team,
>
> We should focus on regressions and cve fixes for releases like this one
> (second bug fix and it's not the most recently released version). The idea
> is to reduce the risk of regressions since it may well be the last official
> release on this branch.
>
> Ismael
>
> On Thu, Mar 14, 2024, 1:39 AM Divij Vaidya 
> wrote:
>
> > Hi Manikumar,
> >
> > 1. Can you please take a look at
> > https://github.com/apache/kafka/pull/15490
> > which is a bug fix specific to the 3.6.x branch?
> > 2. Should we do a one-time update of all dependencies in 3.6.x branch
> > before releasing 3.6.2?
> > 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> > backporting is needed to make the release qualification easier.
> > 4. There are a large number of bugs reported as impacting 3.6.1 [1] Some
> of
> > them have attached PRs and pending review. Maybe we can request all
> > committers to take a look at the ones which have a PR attached and see if
> > we can close them in the next few days before 3.6.2. Note that this will
> be
> > on a best-effort basis and won't block release of 3.6.2.
> > 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged
> whether
> > something needs to be backported? Usually it is the responsibility of the
> > reviewer but I have observed that sometimes we forget to backport
> important
> > onces as well. I can help with this one early next week.
> >
> > [1]
> >
> >
> https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> >
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Thu, Mar 14, 2024 at 7:55 AM Manikumar 
> > wrote:
> >
> > > Hi all,
> > >
> > > Here is the release plan for 3.6.2:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> > >
> > > Currently there is one open non-blocker issue. I plan to generate the
> > first
> > > release candidate
> > > once the issue is resolved and no other issues are raised in the
> > meantime.
> > >
> > > Thanks,
> > > Manikumar
> > >
> > > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana <
> satish.dugg...@gmail.com
> > >
> > > wrote:
> > >
> > > > +1, Thanks Mani for volunteering.
> > > >
> > > > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > > > >
> > > > > +1, Thanks Manikumar!
> > > > >
> > > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > > > wrote:
> > > > >
> > > > > > Thanks Manikumar!
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > Best,
> > > > > > Bruno
> > > > > >
> > > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > > +1 thanks for volunteering!
> > > > > > >
> > > > > > > Best
> > > > > > > ---
> > > > > > >
> > > > > > > Josep Prat
> > > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > > > +491715557497 | aiven.io
> > > > > > > Aiven Deutschland GmbH
> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > >
> > > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> > divijvaidy...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > >> +1
> > > > > > >>
> > > > > > >> Thank you for volunteering.
> > > > > > >>
> > > > > > >> --
> > > > > > >> Divij Vaidya
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > > >> 
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> Thanks Manikumar!
> > > > > > >>> +1 from me
> > > > > > >>>
> > > > > > >>> Justine
> > > > > > >>>
> > > > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > > > manikumar.re...@gmail.com>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > >  Hi,
> > > > > > 
> > > > > >  I'd like to volunteer to be the release manager for a bug
> fix
> > > > release
> > > > > > >> of
> > > > > >  the 3.6 line.
> > > > > >  If there are no objections, I'll send out the release plan
> > soon.
> > > > > > 
> > > > > >  Thanks,
> > > > > >  Manikumar
> > > > > > 
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Edoardo Comar]

Hi Manikumar,
https://issues.apache.org/jira/browse/KAFKA-16369
is merged in trunk now.
can you please cherry-pick it to 3.6.2 ?

I didn't see it included in the plan, would you like me to add it ?
I'd consider it a blocker since Kafka may stay up and running but useless ...

On Mon, 18 Mar 2024 at 09:52, Manikumar  wrote:
>
> Hi,
>
> I have backported a couple of CVE related fixes to the 3.6 branch.
> https://issues.apache.org/jira/browse/KAFKA-16210
> https://issues.apache.org/jira/browse/KAFKA-16322
>
>
> We have one non-blocker issue to be resolved.
> https://issues.apache.org/jira/browse/KAFKA-16073
>
> Thanks,
>
> On Fri, Mar 15, 2024 at 10:18 AM Ismael Juma  wrote:
>
> > Hi team,
> >
> > We should focus on regressions and cve fixes for releases like this one
> > (second bug fix and it's not the most recently released version). The idea
> > is to reduce the risk of regressions since it may well be the last official
> > release on this branch.
> >
> > Ismael
> >
> > On Thu, Mar 14, 2024, 1:39 AM Divij Vaidya 
> > wrote:
> >
> > > Hi Manikumar,
> > >
> > > 1. Can you please take a look at
> > > https://github.com/apache/kafka/pull/15490
> > > which is a bug fix specific to the 3.6.x branch?
> > > 2. Should we do a one-time update of all dependencies in 3.6.x branch
> > > before releasing 3.6.2?
> > > 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> > > backporting is needed to make the release qualification easier.
> > > 4. There are a large number of bugs reported as impacting 3.6.1 [1] Some
> > of
> > > them have attached PRs and pending review. Maybe we can request all
> > > committers to take a look at the ones which have a PR attached and see if
> > > we can close them in the next few days before 3.6.2. Note that this will
> > be
> > > on a best-effort basis and won't block release of 3.6.2.
> > > 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged
> > whether
> > > something needs to be backported? Usually it is the responsibility of the
> > > reviewer but I have observed that sometimes we forget to backport
> > important
> > > onces as well. I can help with this one early next week.
> > >
> > > [1]
> > >
> > >
> > https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > >
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Thu, Mar 14, 2024 at 7:55 AM Manikumar 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Here is the release plan for 3.6.2:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> > > >
> > > > Currently there is one open non-blocker issue. I plan to generate the
> > > first
> > > > release candidate
> > > > once the issue is resolved and no other issues are raised in the
> > > meantime.
> > > >
> > > > Thanks,
> > > > Manikumar
> > > >
> > > > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > +1, Thanks Mani for volunteering.
> > > > >
> > > > > On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
> > > > > >
> > > > > > +1, Thanks Manikumar!
> > > > > >
> > > > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna 
> > > > > wrote:
> > > > > >
> > > > > > > Thanks Manikumar!
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Best,
> > > > > > > Bruno
> > > > > > >
> > > > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > > > +1 thanks for volunteering!
> > > > > > > >
> > > > > > > > Best
> > > > > > > > ---
> > > > > > > >
> > > > > > > > Josep Prat
> > > > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > > > > +491715557497 | aiven.io
> > > > > > > > Aiven Deutschland GmbH
> > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >
> > > > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> +1
> > > > > > > >>
> > > > > > > >> Thank you for volunteering.
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Divij Vaidya
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > > > >> 
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Thanks Manikumar!
> > > > > > > >>> +1 from me
> > > > > > > >>>
> > > > > > > >>> Justine
> > > > > > > >>>
> > > > > > > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar <
> > > > > manikumar.re...@gmail.com>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > >  Hi,
> > > > > > > 
> > > > > > >  I'd like to volunteer to be the release manager for a bug
> > fix
> > > > > release
> > > > > > > >> of
> > > > > > >  the 3.6 line.
> > > > > > >  If there are no objections, I'll send out the rel

Re: [VOTE] KIP-932: Queues for Kafka

[Lianet M.]

Hi Andrew,

Happy to see this KIP unleashing the capabilities of Kafka, from the core
up, to elegantly bring in a new, long wanted consumption pattern.

+1 (non-binding)

Lianet


On Sun, Mar 17, 2024, 7:06 p.m. Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi,
> I’ve been working to complete KIP-932 over the past few months and
> discussions have quietened down.
>
> I’d like to open the voting for KIP-932:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>
> Thanks,
> Andrew

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #156

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-932: Queues for Kafka

[Apoorv Mittal]

Hi Andrew,
Thanks for writing the KIP. This is indeed going to be a valuable addition
to the Kafka, excited to see the KIP.

+ 1 (Non-Binding)

Regards,
Apoorv Mittal
+44 7721681581


On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi,
> I’ve been working to complete KIP-932 over the past few months and
> discussions have quietened down.
>
> I’d like to open the voting for KIP-932:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>
> Thanks,
> Andrew

[jira] [Created] (KAFKA-16383) fix flaky test IdentityReplicationIntegrationTest.testReplicateFromLatest()

[Johnny Hsu (Jira)]

Johnny Hsu created KAFKA-16383:
--

 Summary: fix flaky test 
IdentityReplicationIntegrationTest.testReplicateFromLatest()
 Key: KAFKA-16383
 URL: https://issues.apache.org/jira/browse/KAFKA-16383
 Project: Kafka
  Issue Type: Bug
Reporter: Johnny Hsu
Assignee: Johnny Hsu


Build link: 
[https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-15463/4/testReport/junit/org.apache.kafka.connect.mirror.integration/IdentityReplicationIntegrationTest/Build___JDK_11_and_Scala_2_13___testReplicateFromLatest__/]

 

This test failed in build in several PR, which is flaky



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Christo Lolov]

Heya,

I do have some time to put into this, but to be honest I am still after
reviews of the KIP itself :)

After the latest changes it ought to be detailing both a Zookeeper approach
and a KRaft approach.

Do you have any thoughts on how it could be improved or should I start a
voting thread?

Best,
Christo

On Thu, 14 Mar 2024 at 06:12, Luke Chen  wrote:

> Hi Christo,
>
> Any update with this KIP?
> If you don't have time to complete it, I can collaborate with you to work
> on it.
>
> Thanks.
> Luke
>
> On Wed, Jan 17, 2024 at 11:38 PM Satish Duggana 
> wrote:
>
> > Hi Christo,
> > Thanks for volunteering to contribute to the KIP discussion. I suggest
> > considering this KIP for both ZK and KRaft as it will be helpful for
> > this feature to be available in 3.8.0 running with ZK clusters.
> >
> > Thanks,
> > Satish.
> >
> > On Wed, 17 Jan 2024 at 19:04, Christo Lolov 
> > wrote:
> > >
> > > Hello!
> > >
> > > I volunteer to get this KIP moving forward and implemented in Apache
> > Kafka
> > > 3.8.
> > >
> > > I have caught up with Mehari offline and we have agreed that given
> Apache
> > > Kafka 4.0 being around the corner we would like to propose this feature
> > > only for KRaft clusters.
> > >
> > > Any and all reviews and comments are welcome!
> > >
> > > Best,
> > > Christo
> > >
> > > On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
> > > wrote:
> > >
> > > > Hi everyone, any progress on the status of this KIP? Overall looks
> > good to
> > > > me but I wonder whether we still need to support it for Zookeeper
> mode
> > > > given that it will be deprecated in the next 3 months.
> > > >
> > > > On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> > > > > Hi everyone,
> > > > > I would like to start a discussion on KIP-950: Tiered Storage
> > Disablement
> > > > (
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> > > > ).
> > > > >
> > > > > This KIP proposes adding the ability to disable and re-enable
> tiered
> > > > storage on a topic.
> > > > >
> > > > > Thanks,
> > > > > Mehari
> > > > >
> > > >
> >
>

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Hey folks,

I didn't have a strong preference for requesting the versions via the tool
only or getting it from the server. Colin seemed to suggest it was "for the
best" to request from the server to make the tool lightweight.
I guess the argument against that is having to build and support another
API.

It also seems like there may be some confusion -- partially on my part. For
the tools, I had a question about the feature upgrade tool. So it seems
like we already support multiple features via the `--feature` flag, we
simply rely on the server side to throw errors now?

Justine

On Fri, Mar 15, 2024 at 3:38 PM José Armando García Sancio
 wrote:

> Hi Justine,
>
> Thanks for the update. Some comments below.
>
> On Wed, Mar 13, 2024 at 10:53 AM Justine Olshan
>  wrote:
> > 4. Include an API to list the features for a given metadata version
>
> I am not opposed to designing and implementing this. I am just
> wondering if this is strictly required?
>
> Would having auto-generated documentation address the issue of not
> knowing which feature versions are associated with a given release
> version?
>
> Does it need to be a Kafka API (RPC)? Or can this be strictly
> implemented in the tool? The latest tool, which is needed to parse
> release version to feature version, can print this mapping. Is this
> what you mean by API?
>
> > 5. I'm going back and forth on whether we should support the
> > --release-version flag still. If we do, we need to include validation so
> we
> > only upgrade on upgrade.
>
> I am not sure how this is different from supporting multiple --feature
> flags. The user can run an upgrade command where some of the features
> specified are greater than what the cluster has finalized and some of
> the features specified are less than what the cluster has finalized.
>
> In other words, the KRaft controller and kafka-storage tool are going
> to have to implement this validation even if you don't implement
> --release-version in the tools.
> Thanks,
> --
> -José
>

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Manikumar]

Hi Edoardo,

sure, pls go ahead and cherry-pick the changes to 3.7 and 3.6 branches.

Thanks,

On Mon, Mar 18, 2024 at 3:53 PM Edoardo Comar  wrote:

> Hi Manikumar,
> https://issues.apache.org/jira/browse/KAFKA-16369
> is merged in trunk now.
> can you please cherry-pick it to 3.6.2 ?
>
> I didn't see it included in the plan, would you like me to add it ?
> I'd consider it a blocker since Kafka may stay up and running but useless
> ...
>
> On Mon, 18 Mar 2024 at 09:52, Manikumar  wrote:
> >
> > Hi,
> >
> > I have backported a couple of CVE related fixes to the 3.6 branch.
> > https://issues.apache.org/jira/browse/KAFKA-16210
> > https://issues.apache.org/jira/browse/KAFKA-16322
> >
> >
> > We have one non-blocker issue to be resolved.
> > https://issues.apache.org/jira/browse/KAFKA-16073
> >
> > Thanks,
> >
> > On Fri, Mar 15, 2024 at 10:18 AM Ismael Juma  wrote:
> >
> > > Hi team,
> > >
> > > We should focus on regressions and cve fixes for releases like this one
> > > (second bug fix and it's not the most recently released version). The
> idea
> > > is to reduce the risk of regressions since it may well be the last
> official
> > > release on this branch.
> > >
> > > Ismael
> > >
> > > On Thu, Mar 14, 2024, 1:39 AM Divij Vaidya 
> > > wrote:
> > >
> > > > Hi Manikumar,
> > > >
> > > > 1. Can you please take a look at
> > > > https://github.com/apache/kafka/pull/15490
> > > > which is a bug fix specific to the 3.6.x branch?
> > > > 2. Should we do a one-time update of all dependencies in 3.6.x branch
> > > > before releasing 3.6.2?
> > > > 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> > > > backporting is needed to make the release qualification easier.
> > > > 4. There are a large number of bugs reported as impacting 3.6.1 [1]
> Some
> > > of
> > > > them have attached PRs and pending review. Maybe we can request all
> > > > committers to take a look at the ones which have a PR attached and
> see if
> > > > we can close them in the next few days before 3.6.2. Note that this
> will
> > > be
> > > > on a best-effort basis and won't block release of 3.6.2.
> > > > 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged
> > > whether
> > > > something needs to be backported? Usually it is the responsibility
> of the
> > > > reviewer but I have observed that sometimes we forget to backport
> > > important
> > > > onces as well. I can help with this one early next week.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > >
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Thu, Mar 14, 2024 at 7:55 AM Manikumar  >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Here is the release plan for 3.6.2:
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> > > > >
> > > > > Currently there is one open non-blocker issue. I plan to generate
> the
> > > > first
> > > > > release candidate
> > > > > once the issue is resolved and no other issues are raised in the
> > > > meantime.
> > > > >
> > > > > Thanks,
> > > > > Manikumar
> > > > >
> > > > > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana <
> > > satish.dugg...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > +1, Thanks Mani for volunteering.
> > > > > >
> > > > > > On Thu, 14 Mar 2024 at 06:01, Luke Chen 
> wrote:
> > > > > > >
> > > > > > > +1, Thanks Manikumar!
> > > > > > >
> > > > > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna <
> cado...@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Manikumar!
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Bruno
> > > > > > > >
> > > > > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > > > > +1 thanks for volunteering!
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > ---
> > > > > > > > >
> > > > > > > > > Josep Prat
> > > > > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io
>  |
> > > > > > > > > +491715557497 | aiven.io
> > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > >
> > > > > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> +1
> > > > > > > > >>
> > > > > > > > >> Thank you for volunteering.
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Divij Vaidya
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > > > > > > > >> 
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> Thanks Manikumar!
> > > > > > > > >>> +1 

Re: [VOTE] KIP-932: Queues for Kafka

[Kenneth Eversole]

Hi Andrew

+ 1 (Non-Binding)

This will be great addition to Kafka

On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal 
wrote:

> Hi Andrew,
> Thanks for writing the KIP. This is indeed going to be a valuable addition
> to the Kafka, excited to see the KIP.
>
> + 1 (Non-Binding)
>
> Regards,
> Apoorv Mittal
> +44 7721681581
>
>
> On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
>
> > Hi,
> > I’ve been working to complete KIP-932 over the past few months and
> > discussions have quietened down.
> >
> > I’d like to open the voting for KIP-932:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> >
> > Thanks,
> > Andrew
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2729

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Edoardo Comar]

Thanks Manikumar, done and marked the issue as resolved

On Mon, 18 Mar 2024 at 16:30, Manikumar  wrote:
>
> Hi Edoardo,
>
> sure, pls go ahead and cherry-pick the changes to 3.7 and 3.6 branches.
>
> Thanks,
>
> On Mon, Mar 18, 2024 at 3:53 PM Edoardo Comar  wrote:
>
> > Hi Manikumar,
> > https://issues.apache.org/jira/browse/KAFKA-16369
> > is merged in trunk now.
> > can you please cherry-pick it to 3.6.2 ?
> >
> > I didn't see it included in the plan, would you like me to add it ?
> > I'd consider it a blocker since Kafka may stay up and running but useless
> > ...
> >
> > On Mon, 18 Mar 2024 at 09:52, Manikumar  wrote:
> > >
> > > Hi,
> > >
> > > I have backported a couple of CVE related fixes to the 3.6 branch.
> > > https://issues.apache.org/jira/browse/KAFKA-16210
> > > https://issues.apache.org/jira/browse/KAFKA-16322
> > >
> > >
> > > We have one non-blocker issue to be resolved.
> > > https://issues.apache.org/jira/browse/KAFKA-16073
> > >
> > > Thanks,
> > >
> > > On Fri, Mar 15, 2024 at 10:18 AM Ismael Juma  wrote:
> > >
> > > > Hi team,
> > > >
> > > > We should focus on regressions and cve fixes for releases like this one
> > > > (second bug fix and it's not the most recently released version). The
> > idea
> > > > is to reduce the risk of regressions since it may well be the last
> > official
> > > > release on this branch.
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Mar 14, 2024, 1:39 AM Divij Vaidya 
> > > > wrote:
> > > >
> > > > > Hi Manikumar,
> > > > >
> > > > > 1. Can you please take a look at
> > > > > https://github.com/apache/kafka/pull/15490
> > > > > which is a bug fix specific to the 3.6.x branch?
> > > > > 2. Should we do a one-time update of all dependencies in 3.6.x branch
> > > > > before releasing 3.6.2?
> > > > > 3. We fixed quite a lot of flaky tests in 3.7.x. I will see if any
> > > > > backporting is needed to make the release qualification easier.
> > > > > 4. There are a large number of bugs reported as impacting 3.6.1 [1]
> > Some
> > > > of
> > > > > them have attached PRs and pending review. Maybe we can request all
> > > > > committers to take a look at the ones which have a PR attached and
> > see if
> > > > > we can close them in the next few days before 3.6.2. Note that this
> > will
> > > > be
> > > > > on a best-effort basis and won't block release of 3.6.2.
> > > > > 5. Have you looked at the JIRA marked as "bugs" in 3.7 and triaged
> > > > whether
> > > > > something needs to be backported? Usually it is the responsibility
> > of the
> > > > > reviewer but I have observed that sometimes we forget to backport
> > > > important
> > > > > onces as well. I can help with this one early next week.
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > https://issues.apache.org/jira/browse/KAFKA-16222?jql=project%20%3D%20KAFKA%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > >
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 14, 2024 at 7:55 AM Manikumar  > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Here is the release plan for 3.6.2:
> > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.2
> > > > > >
> > > > > > Currently there is one open non-blocker issue. I plan to generate
> > the
> > > > > first
> > > > > > release candidate
> > > > > > once the issue is resolved and no other issues are raised in the
> > > > > meantime.
> > > > > >
> > > > > > Thanks,
> > > > > > Manikumar
> > > > > >
> > > > > > On Thu, Mar 14, 2024 at 6:24 AM Satish Duggana <
> > > > satish.dugg...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1, Thanks Mani for volunteering.
> > > > > > >
> > > > > > > On Thu, 14 Mar 2024 at 06:01, Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > +1, Thanks Manikumar!
> > > > > > > >
> > > > > > > > On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna <
> > cado...@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Manikumar!
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Bruno
> > > > > > > > >
> > > > > > > > > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > > > > > > > > +1 thanks for volunteering!
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > ---
> > > > > > > > > >
> > > > > > > > > > Josep Prat
> > > > > > > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io
> >  |
> > > > > > > > > > +491715557497 | aiven.io
> > > > > > > > > > Aiven Deutschland GmbH
> > > > > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > > > >
> > > > > > > > > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya <
> > > > > divijvaidy...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > >

Re: [VOTE] KIP-932: Queues for Kafka

[Edoardo Comar]

Thanks Andrew,

+1 (binding)

Edo

On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
 wrote:
>
> Hi Andrew
>
> + 1 (Non-Binding)
>
> This will be great addition to Kafka
>
> On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal 
> wrote:
>
> > Hi Andrew,
> > Thanks for writing the KIP. This is indeed going to be a valuable addition
> > to the Kafka, excited to see the KIP.
> >
> > + 1 (Non-Binding)
> >
> > Regards,
> > Apoorv Mittal
> > +44 7721681581
> >
> >
> > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> > > Hi,
> > > I’ve been working to complete KIP-932 over the past few months and
> > > discussions have quietened down.
> > >
> > > I’d like to open the voting for KIP-932:
> > >
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > >
> > > Thanks,
> > > Andrew
> >

Re: [DISCUSS] KIP-1020: Deprecate `window.size.ms` in `StreamsConfig`

[Lucia Cerchie]

Thanks for the discussion!

I've updated the KIP here with what I believe are the relevant pieces,
please let me know if anything is missing:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=290982804

On Sun, Mar 17, 2024 at 7:09 PM Sophie Blee-Goldman 
wrote:

> Sounds good!
>
> @Lucia when you have a moment can you update the KIP with
> the new proposal, including the details that Matthias pointed
> out in his last response? After that's done I think you can go
> ahead and call for a vote whenever you're ready!
>
> On Sat, Mar 16, 2024 at 7:35 PM Matthias J. Sax  wrote:
>
> > Thanks for the summary. Sounds right to me. That is what I would propose.
> >
> > As you pointed out, we of course still need to support the current
> > confis, and we should log a warning when in use (even if the new one is
> > in use IMHO) -- but that's more an implementation detail.
> >
> > I agree that the new config should take preference in case both are
> > specified. This should be pointed out in the KIP, as it's an important
> > contract the user needs to understand.
> >
> >
> > -Matthias
> >
> > On 3/14/24 6:18 PM, Sophie Blee-Goldman wrote:
> > >>
> > >> Should we change it do `.serializer` and `.deserialize`?
> > >
> > > That's a good point -- if we're going to split this up by defining the
> > > config
> > > in both the TimeWindowedSerializer and TimeWindowedDeserializer,
> > > then it makes perfect sense to go a step further and actually define
> > > only the relevant de/serializer class instead of the full serde
> > >
> > > Just to put this all together, it sounds like the proposal is to:
> > >
> > > 1) Deprecate both these configs where they appear in StreamsConfig
> > > (as per the original plan in the KIP, just reiterating it here)
> > >
> > > 2) Don't "define" either config in any specific client's Config class,
> > > but just define a String variable with the config name in the relevant
> > > de/serializer class, and maybe point people to it in the docs somewhere
> > >
> > > 3) We would add three new public String variables for three different
> > > configs across two classes, specifically:
> > >
> > > In TimeWindowedSerializer:
> > >- define a constant for "windowed.inner.serializer.class"
> > > In TimeWindowedDeserializer:
> > >- define a constant for "windowed.inner.deserializer.class"
> > >- define a constant for "window.size.ms"
> > >
> > > 4) Lastly, we would update the windowed de/serializer implementations
> > > to check for the new configs (ie "windowed.inner.de/serializer.class")
> > > and use the provided de/serializer class, if one was given. If the new
> > > configs are not present, they would fall back to the original/current
> > > logic (ie that based on the old "windowed.inner.serde.class" config)
> > >
> > > I think that's everything. Does this sound about right for where we
> want
> > > to go with these configs?
> > >
> > > On Thu, Mar 14, 2024 at 4:58 PM Matthias J. Sax 
> > wrote:
> > >
> >  By "don't add them" do you just mean we would not have any actual
> >  variables defined anywhere for these configs (eg WINDOW_SIZE_MS)
> >  and simply document -- somewhere -- that one can use the string
> >  "window.size.ms" when configuring a command-line client with a
> >  windowed serde?
> > >>
> > >> Yes. That's the idea.
> > >>
> > >>
> > >>
> > >>> I assume you aren't proposing
> >  to remove the ability to use and understand this config from the
> >  implementations themselves, but correct me if that's wrong.
> > >>
> > >> No, that would effectively break what we fixed with the original KIP
> :)
> > >>
> > >>
> > >>
> >  Are there any other configs in similar situations that we could look
> >  to for precedent?
> > >>
> > >> Not aware of any others, either.
> > >>
> > >>
> > >>
> >  If these are truly the first/only of their kind, I would vote to
> just
> > >> stick
> >  them in the appropriate class. As for which class to put them in, I
> >  think I'm convinced that "window.size.ms" should only go in the
> >  TimeWindowedDeserializer rather than sticking them both in the
> >  TimeWindowedSerde as I originally suggested. However, I would
> >  even go a step further and not place the "inner.window.class.serde"
> >  in the TimeWindowedSerde class either. To me, it actually makes
> >  the most sense to define it in both the TimeWindowedSerializer
> >  and the TimeWindowedDeserializer.
> > >>
> > >> Not sure either. What you propose is fine with me. However, I am
> > >> wondering about the config names... Why is it `serde` for this case?
> > >> Should we change it do `.serializer` and `.deserialize`?
> > >>
> > >>
> > >>
> > >> -Matthias
> > >>
> > >>
> > >> On 3/13/24 8:19 PM, Sophie Blee-Goldman wrote:
> > >>> By "don't add them" do you just mean we would not have any actual
> > >>> variables defined anywhere for these configs (eg WINDOW_SIZE_MS)
> > >>> and simply document -- somewhere -- that one c

Re: [DISCUSS] KIP-1025: Optionally URL-encode clientID and clientSecret in authorization header

[Kirk True]

Hi Nelson,

Thank you for writing up the KIP! My apologies for the delay in response :(

Questions:

1. Is the long-term plan to keep the configuration default set to “false"? I 
understand the short-term benefits, but in general, configuration defaults 
should prefer compliance with standards (e.g. RFCs).
2. Can we change “sasl.oauthbearer.header.urlencode.enable” to be a little 
shorter? Maybe “sasl.oauthbearer.header.urlencode” or even 
“sasl.oauthbearer.urlencode”? I’m looking at the configuration names that I 
introduced in KIP-768 with a bit of cringe at their length :) This is a total 
nit, so I won’t make a stink about it if everyone else is cool with it :)

Thanks,
Kirk

> On Mar 13, 2024, at 5:31 AM, Nelson B.  wrote:
> 
> Hi all,
> 
> I just wanted to bump up this thread.
> 
> The KIP introduces a really small change and PR is already ready and only
> waiting for this KIP to get approved to be merged.
> 
> Thanks,
> 
> On Wed, Mar 6, 2024 at 12:26 PM Nelson B.  wrote:
> 
>> Hi all,
>> 
>> I would like to start a discussion on KIP-1025, which would optionally
>> URL-encode clientID and clientSecret in the authorization header
>> 
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1025%3A+Optionally+URL-encode+clientID+and+clientSecret+in+authorization+header
>> 
>> Best,
>> Nelson B.
>> 

Re: [DISCUSS] KIP-932: Queues for Kafka

[Justine Olshan]

Hey Andrew,

I noticed you started the voting thread, but there seems to be a few
questions that were not answered. One was Jun's about memory usage
> How much additional heap memory will the server use? Do we need to cache
records in heap? If so, is the cache bounded?

Your response was
> This area needs more work. Using a share group surely gets the broker to
do
more manipulation of the data that it fetches than a regular consumer. I
want to minimise
this and need to research before providing a comprehensive answer. I
suspect zero-copy
is lost and that we do not cache records in heap. I will confirm later on.

I am also concerned about memory usage from my producer state management
work, so I want to make sure we have thought about it here -- not just in
the case Jun mentioned but generally.

Likewise, we have seen issues with large consumer groups and too many
producer IDs. Are there any concerns with an analogous situation with too
many share group members or share groups? Are they any ways we try to
handle this or mitigate risks with respect to memory usage and client
connections (wrt to rebalances for example)

Thanks,

Justine

On Fri, Mar 8, 2024 at 12:51 AM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Manikumar,
> Thanks for your queries.
>
> 1) Delivery count is added to the ConsumerRecord class so that a consumer
> can
> tell how often a record has been processed. I imagine that some
> applications might
> want to take different actions based on whether a record has previously
> failed. This
> enables richer error handling for bad records. In the future, I plan
> another KIP to
> enhance error handling.
>
> 2) It is only possible to delete a share group which is empty. As a
> result, all
> well-behaved consumers will have closed their share sessions. After a
> short while,
> the share-partition leaders will discard the share-partition information
> from memory.
>
> In the presence of badly behaved consumers, a consumer would have to
> pretend to
> be a member of a share group. There are several cases:
>
> a) If the share-partition leader still has in-memory state for the deleted
> share-group, it will
> continue to fetch records but it will be fenced by the share coordinator
> when it attempts to
> write its persistent state.
>
> b) If the share-partition leader does not have in-memory state, it will
> attempt to read it
> from the share coordinator and this will fail.
>
> 3) I will add metrics for the share coordinator today. This was an
> omission. Thanks for catching it.
>
> Thanks,
> Andrew
>
> > On 6 Mar 2024, at 17:53, Manikumar  wrote:
> >
> > Hi Andrew,
> >
> > Thanks for the updated KIP. Few queries below:
> >
> > 1. What is the use-case of deliveryCount in ShareFetchResponse?
> > 2. During delete share groups, Do we need to clean any in-memory state
> from
> > share-partition leaders?
> > 3. Any metrics for the share-coordinator?
> >
> > Thanks
> > Manikumar
> >
> > On Wed, Feb 21, 2024 at 12:11 AM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> >> Hi Manikumar,
> >> Thanks for your comments.
> >>
> >> 1. I believe that in general, there are not situations in which a
> dynamic
> >> config
> >> change is prevented because of the existence of a resource. So, if we
> >> prevented
> >> setting config `group.type=consumer` on resource G of GROUP type
> >> if there was a share group G in existence, it would be a bit weird.
> >>
> >> I wonder whether changing the config name to `new.group.type` would
> help.
> >> It’s
> >> ensuring the type of a new group created.
> >>
> >> 2. The behaviour for a DEAD share group is intended to be the same as a
> >> DEAD
> >> consumer group. The group cannot be “reused” again as such, but the
> group
> >> ID
> >> can be used by a new group.
> >>
> >> 3. Yes. AlterShareGroupOffsets will cause a new SHARE_CHECKPOINT.
> >>
> >> 4. In common with Admin.deleteConsumerGroups, the underlying Kafka RPC
> >> for Admin.deleteShareGroups is DeleteGroups. This is handled by the
> group
> >> coordinator and it does this by writing control records (a tombstone in
> >> this case).
> >> The KIP doesn’t say anything about this because it’s the same as
> consumer
> >> groups.
> >> Perhaps it would be sensible to add a GroupType to DeleteGroupsRequest
> so
> >> we can
> >> make sure we are deleting the correct type of group. The fact that there
> >> is not a specific
> >> RPC for DeleteShareGroups seems correct to me.
> >>
> >> 5. I prefer using “o.a.k.clients.consumer” because it’s already a public
> >> package and
> >> many of the classes and interfaces such as ConsumerRecord are in that
> >> package.
> >>
> >> I definitely need to add more information about how the Admin operations
> >> work.
> >> I will add a section to the KIP in the next version, later today. This
> >> will fill in details for
> >> your questions (3) and (4).
> >>
> >> Thanks,
> >> Andrew
> >>
> >>> On 14 Feb 2024, at 18:04, Manikumar  wrote:
> >>>
> >>> Hi And

Re: [DISCUSS] KIP-1025: Optionally URL-encode clientID and clientSecret in authorization header

[Nelson B.]

Hi Kirk,

Thanks for your comments!

1. I think we can use KIP-1030

as
an opportunity to update the default value to "true" starting from version
4.0.
2. I've updated the config name to "sasl.oauthbearer.header.urlencode" in
the KIP, I'm gonna update PR once KIP is accepted.

Thanks,

On Tue, Mar 19, 2024 at 3:45 AM Kirk True  wrote:

> Hi Nelson,
>
> Thank you for writing up the KIP! My apologies for the delay in response :(
>
> Questions:
>
> 1. Is the long-term plan to keep the configuration default set to “false"?
> I understand the short-term benefits, but in general, configuration
> defaults should prefer compliance with standards (e.g. RFCs).
> 2. Can we change “sasl.oauthbearer.header.urlencode.enable” to be a little
> shorter? Maybe “sasl.oauthbearer.header.urlencode” or even
> “sasl.oauthbearer.urlencode”? I’m looking at the configuration names that I
> introduced in KIP-768 with a bit of cringe at their length :) This is a
> total nit, so I won’t make a stink about it if everyone else is cool with
> it :)
>
> Thanks,
> Kirk
>
> > On Mar 13, 2024, at 5:31 AM, Nelson B.  wrote:
> >
> > Hi all,
> >
> > I just wanted to bump up this thread.
> >
> > The KIP introduces a really small change and PR is already ready and only
> > waiting for this KIP to get approved to be merged.
> >
> > Thanks,
> >
> > On Wed, Mar 6, 2024 at 12:26 PM Nelson B. 
> wrote:
> >
> >> Hi all,
> >>
> >> I would like to start a discussion on KIP-1025, which would optionally
> >> URL-encode clientID and clientSecret in the authorization header
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1025%3A+Optionally+URL-encode+clientID+and+clientSecret+in+authorization+header
> >>
> >> Best,
> >> Nelson B.
> >>
>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #157

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #113

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2730

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-890 Server Side Defense

[Justine Olshan]

Hey there -- small update to the KIP,

The KIP mentioned introducing ABORTABLE_ERROR and bumping TxnOffsetCommit
and Produce requests. I've changed the name in the KIP to
ABORTABLE_TRANSACTION and the corresponding exception
AbortableTransactionException to match the pattern we had for other errors.
I also mentioned bumping all 6 transactional APIs so we can future
proof/support the error on the client going forward. If a future change
wants to have an error scenario that requires us to abort the transaction,
we can rely on the 3.8+ clients to support it. We ran into issues finding
good/generic error codes that older clients could support while working on
this KIP, so this should help in the future.

The features discussion is still ongoing in KIP-1022. Will update again
here when that concludes.

Justine

On Tue, Feb 6, 2024 at 8:39 AM Justine Olshan  wrote:

> I don't think AddPartitions is a good example since we currenly don't gate
> the version on TV or MV. (We only set a different flag depending on the TV)
>
> Even if we did want to gate it on TV, I think the idea is to move away
> from MV gating inter broker protocols. Ideally we can get to a state where
> MV is just used for metadata changes.
>
> I think some of this discussion might fit more with the feature version
> KIP, so I can try to open that up soon. Until we settle that, some of the
> work in KIP-890 is blocked.
>
> Justine
>
> On Mon, Feb 5, 2024 at 5:38 PM Jun Rao  wrote:
>
>> Hi, Justine,
>>
>> Thanks for the reply.
>>
>> Since AddPartitions is an inter broker request, will its version be gated
>> only by TV or other features like MV too? For example, if we need to
>> change
>> the protocol for AddPartitions for reasons other than txn verification in
>> the future, will the new version be gated by a new MV? If so, does
>> downgrading a TV imply potential downgrade of MV too?
>>
>> Jun
>>
>>
>>
>> On Mon, Feb 5, 2024 at 5:07 PM Justine Olshan
>> 
>> wrote:
>>
>> > One TV gates the flexible feature version (no rpcs involved, only the
>> > transactional records that should only be gated by TV)
>> > Another TV gates the ability to turn on kip-890 part 2. This would gate
>> the
>> > version of Produce and EndTxn (likely only used by transactions), and
>> > specifies a flag in AddPartitionsToTxn though the version is already
>> used
>> > without TV.
>> >
>> > I think the only concern is the Produce request and we could consider
>> work
>> > arounds similar to the AddPartitionsToTxn call.
>> >
>> > Justine
>> >
>> > On Mon, Feb 5, 2024 at 4:56 PM Jun Rao 
>> wrote:
>> >
>> > > Hi, Justine,
>> > >
>> > > Which PRC/record protocols will TV guard? Going forward, will those
>> > > PRC/record protocols only be guarded by TV and not by other features
>> like
>> > > MV?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Mon, Feb 5, 2024 at 2:41 PM Justine Olshan
>> > > > > >
>> > > wrote:
>> > >
>> > > > Hi Jun,
>> > > >
>> > > > Sorry I think I misunderstood your question or answered incorrectly.
>> > The
>> > > TV
>> > > > version should ideally be fully independent from MV.
>> > > > At least for the changes I proposed, TV should not affect MV and MV
>> > > should
>> > > > not affect TV/
>> > > >
>> > > > I think if we downgrade TV, only that feature should downgrade.
>> > Likewise
>> > > > the same with MV. The finalizedFeatures should just reflect the
>> feature
>> > > > downgrade we made.
>> > > >
>> > > > I also plan to write a new KIP for managing the disk format and
>> upgrade
>> > > > tool as we will need new flags to support these features. That
>> should
>> > > help
>> > > > clarify some things.
>> > > >
>> > > > Justine
>> > > >
>> > > > On Mon, Feb 5, 2024 at 11:03 AM Jun Rao 
>> > > wrote:
>> > > >
>> > > > > Hi, Justine,
>> > > > >
>> > > > > Thanks for the reply.
>> > > > >
>> > > > > So, if we downgrade TV, we could implicitly downgrade another
>> feature
>> > > > (say
>> > > > > MV) that has dependency (e.g. RPC). What would we return for
>> > > > > FinalizedFeatures for MV in ApiVersionsResponse in that case?
>> > > > >
>> > > > > Thanks,
>> > > > >
>> > > > > Jun
>> > > > >
>> > > > > On Fri, Feb 2, 2024 at 1:06 PM Justine Olshan
>> > > > > > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hey Jun,
>> > > > > >
>> > > > > > Yes, the idea is that if we downgrade TV (transaction version)
>> we
>> > > will
>> > > > > stop
>> > > > > > using the add partitions to txn optimization and stop writing
>> the
>> > > > > flexible
>> > > > > > feature version of the log.
>> > > > > > In the compatibility section I included some explanations on how
>> > this
>> > > > is
>> > > > > > done.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Justine
>> > > > > >
>> > > > > > On Fri, Feb 2, 2024 at 11:12 AM Jun Rao
>> 
>> > > > > wrote:
>> > > > > >
>> > > > > > > Hi, Justine,
>> > > > > > >
>> > > > > > > Thanks for the update.
>> > > > > > >
>> > > > > > > If we ever downgrade the transaction feature, any feature
>> > depending
>> > > > on
>

[jira] [Created] (KAFKA-16384) KRaft controller number recommendation

[Peter (Jira)]

Peter created KAFKA-16384:
-

 Summary: KRaft controller number recommendation
 Key: KAFKA-16384
 URL: https://issues.apache.org/jira/browse/KAFKA-16384
 Project: Kafka
  Issue Type: Bug
  Components: controller, kraft
Reporter: Peter


There seems to be some conflicting information about how many controllers 
should be used for a KRaft cluster. The first section listed mentions 3 or 5 
controllers may be used, but the second section mentions no more than 3 should 
be used at the moment.

https://kafka.apache.org/documentation/#kraft_voter

> A Kafka admin will typically select 3 or 5 servers for this role, depending 
> on factors like cost and the number of concurrent failures your system should 
> withstand without availability impact. A majority of the controllers must be 
> alive in order to maintain availability. With 3 controllers, the cluster can 
> tolerate 1 controller failure; with 5 controllers, the cluster can tolerate 2 
> controller failures.

https://kafka.apache.org/documentation/#kraft_deployment

> For redundancy, a Kafka cluster should use 3 controllers. More than 3 
> controllers is not recommended in critical environments. In the rare case of 
> a partial network failure it is possible for the cluster metadata quorum to 
> become unavailable. This limitation will be addressed in a future release of 
> Kafka.

 

Is 3 still the recommended number and is there more information on what the 
network issues are that could cause issues when using 5 controllers?

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1026: Handling producer snapshot when upgrading from < v2.8.0 for Tiered Storage

[Arpit Goyal]

@Luke Chen @Kamal Chandraprakash   @Greg
Harris Any suggestion on the above two possible approaches.
On Sun, Mar 17, 2024, 13:36 Arpit Goyal  wrote:

>
>>>  In summary , There are two possible solution to handle the above
> scenario when producer snapshot file found to be null
>
> 1. *Generate empty producer snapshot file at run time when copying
> LogSegment*
>
>
>- This will not require any backward compatibility dependencies with
>the plugin.
>- It preserves the contract i.e producerSnapshot files should be
>mandatory.
>- We could have a metric which helps us to assess how many times empty
>snapshot files have been created.
>
> 2.*  Make producerSnapshot files optional *
>
>- This would break the contract with the plugin and would require
>defining a set of approaches to handle it which is mentioned earlier in the
>thread.
>- If we make producer Snapshot optional , We would   not be  handling
>the error which @LukeChen mentioned when producerSnapshot
>accidentally deleted a given segment. But this holds true for
>TransactionalIndex.
>- The other question is do we really need to make the field optional.
>The only case where this problem can occur is only when the topic migrated
>from < 2.8 version.
>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2731

[Apache Jenkins Server]

See