Re: Contributor permissions request

[Luke Chen]

Hi Jimmy,

Your accounts are all set.

Thanks.
Luke

On Fri, Jan 5, 2024 at 11:39 AM zw  wrote:

> Hi all,I kindly request the permission for being a contributor to
> Kafka project.
>
>
> Jira username: JimmyWang
> GitHub username: JimmyWang6
> Wiki: JimmyWang
>
>
>
>
> Best,
> Jimmy Wang

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #48

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2540

[Apache Jenkins Server]

See 

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hi Ismael,
I added this automatically approach to “Rejected alternatives” concerning that 
we need to unify the semantics between alterConfigs and 
incrementalAlterConfigs, so I choose to give this privilege to users.

After reviewing these code and doing some tests I found that they following the 
similar approach, I think the simplest way is to let the client choose the best 
method heuristically.

Thank you for pointing out this, I will change the KIP later.

Best,
Ziming

> On Jan 4, 2024, at 17:28, Ismael Juma  wrote:
> 
> Hi Ziming,
> 
> Why is the flag required at all? Can we use incremental and fallback 
> automatically if it's not supported by the broker? At this point, the vast 
> majority of clusters should support it.
> 
> Ismael
> 
> On Mon, Dec 18, 2023 at 7:58 PM ziming deng  > wrote:
>> 
>> Hello, I want to start a discussion on KIP-1011, to make the broker config 
>> change path unified with that of user/topic/client-metrics and avoid some 
>> bugs.
>> 
>> Here is the link: 
>> 
>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>> cwiki.apache.org
>> 
>>  
>> KIP-1011:
>>  Use incrementalAlterConfigs when updating broker configs by 
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
>> 
>> cwiki.apache.org 
>> 
>>  
>> 
>> 
>> Best, 
>> Ziming.

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Rowland Smith]

It is probably me. I copied the original message subject into a new email.
Perhaps that is not enough to link them.

It was not my understanding from reading KIP-939 that we are doing away
with any transactional timeout in the Kafka broker. As I understand it, we
are allowing the application to set the transaction timeout to a value that
exceeds the *transaction.max.timeout.ms
* setting
on the broker, and having no timeout if the application does not set
*transaction.timeout.ms
* on the producer. The KIP says that the
semantics of *transaction.timeout.ms * are
not being changed, so I take that to mean that the broker will continue to
enforce a timeout if provided, and abort transactions that exceed it. From
the KIP:

Client Configuration Changes

*transaction.two.phase.commit.enable* The default would be ‘false’.  If set
to ‘true’, then the broker is informed that the client is participating in
two phase commit protocol and can set transaction timeout to values that
exceed *transaction.max.timeout.ms * setting
on the broker (if the timeout is not set explicitly on the client and the
two phase commit is set to ‘true’ then the transaction never expires).

*transaction.timeout.ms * The semantics is
not changed, but it can be set to values that exceed
*transaction.max.timeout.ms
* if two.phase.commit.enable is set to
‘true’.


Thinking about this more I believe we would also have a possible race
condition if the broker is unaware that a transaction has been prepared.
The application might call prepare and get a positive response, but the
broker might have already aborted the transaction for exceeding the
timeout. It is a general rule of 2PC that once a transaction has been
prepared it must be possible for it to be committed or aborted. It seems in
this case a prepared transaction might already be aborted by the broker, so
it would be impossible to commit.

I hope this is making sense and I am not misunderstanding the KIP. Please
let me know if I am.

- Rowland


On Thu, Jan 4, 2024 at 12:56 PM Justine Olshan 
wrote:

> Hey Rowland,
>
> Not sure why this message showed up in a different thread from the other
> KIP-939 discussion (is it just me?)
>
> In KIP-939, we do away with having any transactional timeout on the Kafka
> side. The external coordinator is fully responsible for controlling whether
> the transaction completes.
>
> While I think there is some use in having a prepare stage, I just wanted to
> clarify what the current KIP is proposing.
>
> Thanks,
> Justine
>
> On Wed, Jan 3, 2024 at 7:49 PM Rowland Smith  wrote:
>
> > Hi Artem,
> >
> > I saw your response in the thread I started discussing Kafka distributed
> > transaction support and the XA interface. I would like to work with you
> to
> > add XA support to Kafka on top of the excellent foundational work that
> you
> > have started with KIP-939. I agree that explicit XA support should not be
> > included in the Kafka codebase as long as the right set of basic
> operations
> > are provided. I will begin pulling together a KIP to follow KIP-939.
> >
> > I did have one comment on KIP-939 itself. I see that you considered an
> > explicit "prepare" RPC, but decided not to add it. If I understand your
> > design correctly, that would mean that a 2PC transaction would have a
> > single timeout that would need to be long enough to ensure that prepared
> > transactions are not aborted when an external coordinator fails. However,
> > this also means that an unprepared transaction would not be aborted
> without
> > waiting for the same timeout. Since long running transactions block
> > transactional consumers, having a long timeout for all transactions could
> > be disruptive. An explicit "prepare " RPC would allow the server to abort
> > unprepared transactions after a relatively short timeout, and apply a
> much
> > longer timeout only to prepared transactions. The explicit "prepare" RPC
> > would make Kafka server more resilient to client failure at the cost of
> an
> > extra synchronous RPC call. I think its worth reconsidering this.
> >
> > With an XA implementation this might become a more significant issue
> since
> > the transaction coordinator has no memory of unprepared transactions
> across
> > restarts. Such transactions would need to be cleared by hand through the
> > admin client even when the transaction coordinator restarts successfully.
> >
> > - Rowland
> >
>

[jira] [Resolved] (KAFKA-16079) Fix leak in LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16079.
---
Fix Version/s: 3.7.0
   Resolution: Fixed

> Fix leak in 
> LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest
> --
>
> Key: KAFKA-16079
> URL: https://issues.apache.org/jira/browse/KAFKA-16079
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.7.0
>
>
> Fix leak in 
> LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2539

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

[Colin McCabe]

Hi all,

Since this has been open for a few weeks, are there any objections to starting 
the vote? What do you think, Josep?

Since 3.8 is going to be the next release (according to the KIP) we should 
really vote this in as soon as possible.

Also, I created a wiki page about the 3.8 release with a tentative schedule.
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0

Please let me know if these dates make sense -- they're just proposals right 
now.

best,
Colin


On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote:
> On Thu, Dec 28, 2023, at 18:17, Justine Olshan wrote:
>> Hey Colin,
>>
>> Some folks were concerned about the lack of automatic unclean leader
>> election. I mentioned that KIP-966 would actually be better with its
>> aggressive recovery option.
>> I think folks were hoping for some availability over durability solution
>> for KRaft, so if we don't do KIP-966 we should provide an alternative or be
>> able to convince ourselves it is not needed.
>
> Hi Justine,
>
> That's a fair point. We should specify in KIP-1012 that we need to have 
> some way to configure the system to automatically do unclean leader 
> election. If we run out of time implementing KIP-966, this could be 
> something quite simple, like honoring the static 
> unclean.leader.election = true configuration.
>
>>
>> I think while many folks decided KIP-853 was a blocker, there were a lot of
>> other features that many folks were expecting so I don't think we can say
>> definitively the only must-have is KIP-853 (and hence the discussion thread
>> here :) )
>>
>> Also as an aside, I filed a ticket to remove ZK from the top of the
>> quickstart guide. https://issues.apache.org/jira/browse/KAFKA-15975
>>
>
> Yeah. There is a bunch of docs and quickstart cleanup that we should 
> do. I don't think any of it is a blocker for 3.8 or 4.0, but the new 
> year is a good time to clean things up.
>
> best,
> Colin
>
>
>> Justine
>>
>> On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe  wrote:
>>
>>> Hi Josep,
>>>
>>> Thanks for the KIP. Based on the discussions we had previously, I agree
>>> that we need a 3.8.
>>>
>>> It would be good to link to KIP-833 in the motivation section, since this
>>> KIP builds on that one.
>>>
>>> Also, I think we should mention in KIP-1012 that 3.8 will be a
>>> general-purpose release that may add some new features. This was something
>>> that we were on the fence about previously, so it would be good to clarify
>>> it here.
>>>
>>> On another note. I don't think KIP-966 is a "must-have" for Kafka 3.8, as
>>> the KIP currently states. I certainly hope that it makes it for 3.8, but if
>>> it doesn't, it can go into 4.0. It's not needed for migration, so it could
>>> just as easily go into 4.0 as 3.8. The only thing that KIP-966 really needs
>>> is "KIP-853: KRaft Controller Membership Changes."
>>>
>>> Along these lines, I think we should drop the language about "strategic
>>> feature parity with Zookeeper." Kafka isn't competing with ZooKeeper, and
>>> doesn't need feature parity with it. For example, ZK implemented
>>> Netty-TcNative OpenSSL Support, but we don't have that (and probably won't
>>> in 3.8). We probably won't add this -- or if we do, it won't be so that we
>>> can have "parity with ZK." Really the only must-have in 3.8 is KIP-853, and
>>> we should be clear about that.
>>>
>>> I think we should start issuing a deprecation log message at ERROR  level
>>> when brokers start up in ZK mode. This message could point out that some
>>> safety mechanisms and new features will not be available in ZK mode, and
>>> give a link to our documentation about migration.
>>>
>>> We should probably also move the example configurations for kraft from
>>> config/kraft to config. And move the zk ones into config/zk. Or maybe even
>>> drop the ZK ones altogether, since they're not needed for migration or
>>> upgrade.
>>>
>>> best,
>>> Colin
>>>
>>>
>>> On Fri, Dec 22, 2023, at 04:37, Josep Prat wrote:
>>> > On this note, I'd like to add that I would volunteer to be the release
>>> > manager of such release 3.8.0.
>>> >
>>> > Best,
>>> >
>>> > On Fri, Dec 22, 2023 at 1:31 PM Josep Prat  wrote:
>>> >
>>> >> Hi all!
>>> >> As agreed on the "Road to Kafka 4.0" email thread, I created KIP-1012 to
>>> >> discuss and I'd like to open it up for discussion:
>>> >>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>>> >>
>>> >> Let's use this KIP to:
>>> >> a) Leave a papertrail agreement for the need of a 3.8 version
>>> >> b) Define which KIPs are the must-haves in regards to KRaft that should
>>> be
>>> >> included there.
>>> >>
>>> >> Please let me know your feedback and suggestions.
>>> >>
>>> >> Best,
>>> >>
>>> >> --
>>> >> [image: Aiven] 
>>> >>
>>> >> *Josep Prat*
>>> >> Open Source Engineering Director, *Aiven*
>>> >> josep.p...@aiven.io   |   +491715557497
>>> >> aiven.io    |
>>> >> 

Re: [DISCUSS] Road to Kafka 4.0

[Colin McCabe]

Hi Mickael,

Thanks for bringing this up.

The main motivation given in KIP-653 for moving to log4j 2.x is that log4j 1.x 
is no longer supported. But since we moved to reload4j, which is still 
supported, that isn't a concern any longer.

To be clear, I'm not saying we shouldn't upgrade, but I'm just trying to 
explain why I think there hasn't been as much interest in this lately. I see 
this as a "cool feature" rather than as a must-do.

If we still want to do this for 4.0, it would be good to understand whether 
there's any work that has to land in 3.8. Do we have to get KIP-719 into 3.8 so 
that we have a reasonable deprecation period?

Also, if we do upgrade, I agree with Ismael that we should consider going to 
log4j3. Assuming they have a non-beta release by the time 4.0 is ready.

best,
Colin

On Thu, Jan 4, 2024, at 03:08, Mickael Maison wrote:
> Hi Ismael,
>
> Yes both KIPs have been voted.
> My point, which admittedly wasn't clear, was to gauge the interest in
> getting them done and if so identifying people to drive these tasks.
>
> KIP-719 shouldn't require too much more work to complete. There's a PR
> [0] which is relatively straightforward. I pinged Lee Dongjin.
> KIP-653 is more involved and depends on KIP-719. There's also a PR [1]
> which is pretty large.
>
> Yes log4j3 was on my mind as it's expected to be compatible with
> log4j2 and bring significant improvements.
>
> 0: https://github.com/apache/kafka/pull/10244
> 1: https://github.com/apache/kafka/pull/7898
>
> Thanks,
> Mickael
>
> On Thu, Jan 4, 2024 at 11:34 AM Ismael Juma  wrote:
>>
>> Hi Mickael,
>>
>> Given that KIP-653 was accepted, the current position is that we would move
>> to log4j2 - provided that someone is available to drive that. It's also
>> worth noting that log4j3 is now a thing (but not yet final):
>>
>> https://logging.apache.org/log4j/3.x/
>>
>> Ismael
>>
>> On Thu, Jan 4, 2024 at 2:15 AM Mickael Maison 
>> wrote:
>>
>> > Hi,
>> >
>> > I've not seen replies about log4j2.
>> > The plan was to deprecated the appender (KIP-719) and switch to log4j2
>> > (KIP-653).
>> >
>> > While reload4j works well, I'd still be in favor of switching to
>> > log4j2 in Kafka 4.0.
>> >
>> > Thanks,
>> > Mickael
>> >
>> > On Fri, Dec 29, 2023 at 2:19 AM Colin McCabe  wrote:
>> > >
>> > > Hi all,
>> > >
>> > > Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for
>> > a Kafka 3.8.x release" email thread.
>> > >
>> > > Colin
>> > >
>> > >
>> > > On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote:
>> > > > Hi Divij,
>> > > >
>> > > > Thanks for the feedback. I agree that having a 3.8 release is
>> > > > beneficial but some of the comments in this message are inaccurate and
>> > > > could mislead the community and users.
>> > > >
>> > > > On Thu, Dec 21, 2023 at 7:00 AM Divij Vaidya 
>> > wrote:
>> > > >> 1\ Durability/availability bugs in kraft - Even though kraft has been
>> > > >> around for a while, we keep finding bugs that impact availability and
>> > data
>> > > >> durability in it almost with every release [1] [2]. It's a complex
>> > feature
>> > > >> and such bugs are expected during the stabilization phase. But we
>> > can't
>> > > >> remove the alternative until we see stabilization in kraft i.e. no new
>> > > >> stability/durability bugs for at least 2 releases.
>> > > >
>> > > > I took a look at both of these issues and neither of them are bugs
>> > > > that affect KRaft's durability and availability.
>> > > >
>> > > >> [1] https://issues.apache.org/jira/browse/KAFKA-15495
>> > > >
>> > > > This issue is not specific to KRaft and has been an issue in Apache
>> > > > Kafka since the ISR leader election and replication algorithm was
>> > > > added to Apache Kafka. I acknowledge that this misunderstanding is
>> > > > partially due to the Jira description which insinuates that this only
>> > > > applies to KRaft which is not true.
>> > > >
>> > > >> [2] https://issues.apache.org/jira/browse/KAFKA-15489
>> > > >
>> > > > First, technically this issue was not first discovered in some recent
>> > > > release. This issue was identified by me back in January of 2022:
>> > > > https://issues.apache.org/jira/browse/KAFKA-13621. I decided to lower
>> > > > the priority as it requires a very specific network partition where
>> > > > the controllers are partitioned from the current leader but the
>> > > > brokers are not.
>> > > >
>> > > > This is not a durability bug as the KRaft cluster metadata partition
>> > > > leader will not be able to advance the HWM and hence commit records.
>> > > >
>> > > > Regarding availability, The KRaft's cluster metadata partition favors
>> > > > consistency and partition tolerance versus availability from CAP. This
>> > > > is by design and not a bug in the protocol or implementation.
>> > > >
>> > > >> 2\ Parity with Zk - There are also pending bugs [3] which are in the
>> > > >> category of Zk parity. Removing Zk from Kafka without having full
>> > feature
>> > > >> 

Re: Apache Kafka 3.7.0 Release

[Stanislav Kozlovski]

Thanks Apoorv, I was going to update the mailing thread as well.

Major kudos to Apoorv for the thorough work debugging and getting to the
bottom of this tricky publishing issue, a subtle regression from the work
he did in making the kafka-clients jar shadowed.

On Thu, Jan 4, 2024 at 5:09 PM Apoorv Mittal 
wrote:

> Hi Stan,
> I have opened the minor PR: https://github.com/apache/kafka/pull/15127 to
> fix publishing the dependency. Once discussed and merged in trunk, I'll
> update the 3.7 branch as well.
>
> Regards,
> Apoorv Mittal
> +44 7721681581
>
>
> On Thu, Jan 4, 2024 at 12:49 PM Matthias J. Sax  wrote:
>
> > We found a blocker for 3.7:
> > https://issues.apache.org/jira/browse/KAFKA-16077
> >
> > Already having a PR under review to fix it.
> >
> >
> > -Matthias
> >
> > On 1/3/24 10:43 AM, Stanislav Kozlovski wrote:
> > > Hey all, happy new year.
> > >
> > > Thanks for the heads up Almog. Makes sense.
> > >
> > > To give an update - I haven't been able to resolve the gradlewAll
> publish
> > > failure, and as such haven't been able to release an RC.
> > > As a minor barrier, I have to also update the year in the NOTICE file,
> > > otherwise the release script won't let me continue -
> > > https://github.com/apache/kafka/pull/15111
> > >
> > > Me and Apoorv synced offline and ran a few tests to debug the issue
> > > regarding the clients build. I successfully executed `publish` when
> > > pointing toward a custom jfrog repo with both JDK 8 and 17. Inspecting
> > the
> > > debug logs, the task that previously failed
> > > `:clients:publishMavenJavaPublicationToMavenRepository'` passed
> > > successfully. Here's a sample of the logs -
> > >
> >
> https://gist.github.com/stanislavkozlovski/841060cb467ec1d179cc9f293c8702e7
> > >
> > > Having read the release.py script a few times, I am not able to see
> what
> > is
> > > different in the setup there. It simply clones the repo anew, gets the
> > 3.7
> > > branch and runs the same command.
> > >
> > > At this point, I am contemplating pushing a commit to 3.7 that modifies
> > the
> > > release.py file that enables debug on the command:
> > > diff --git a/release.py b/release.py
> > > index 43c5809861..e299e10e74 100755
> > > --- a/release.py
> > > +++ b/release.py
> > > @@ -675,7 +675,7 @@ with
> > > open(os.path.expanduser("~/.gradle/gradle.properties")) as f:
> > >   contents = f.read()
> > >   if not user_ok("Going to build and upload mvn artifacts based on
> these
> > > settings:\n" + contents + '\nOK (y/n)?: '):
> > >   fail("Retry again later")
> > > -cmd("Building and uploading archives", "./gradlewAll publish",
> > > cwd=kafka_dir, env=jdk8_env, shell=True)
> > > +cmd("Building and uploading archives", "./gradlewAll publish --debug",
> > > cwd=kafka_dir, env=jdk8_env, shell=True)
> > >   cmd("Building and uploading archives", "mvn deploy -Pgpg-signing",
> > > cwd=streams_quickstart_dir, env=jdk8_env, shell=True)
> > >
> > >   release_notification_props = { 'release_version': release_version,
> > > (END)
> > >
> > > and continuing to debug through that.
> > >
> > > Since the release.py script grabs a new copy of origin, we have to
> modify
> > > upstream. An alternative is for me to use my local github Kafka repo,
> but
> > > that may result in the script pushing a build of that into the remote
> > > servers.
> > >
> > > On Tue, Jan 2, 2024 at 8:17 PM Almog Gavra 
> > wrote:
> > >
> > >> Hello Stan,
> > >>
> > >> I wanted to give you a heads up that
> > >> https://github.com/apache/kafka/pull/15073 (
> > >> https://issues.apache.org/jira/browse/KAFKA-16046) was identified as
> a
> > >> blocker regression and should be merged to trunk by EOD.
> > >>
> > >> Cheers,
> > >> Almog
> > >>
> > >> On Tue, Jan 2, 2024 at 4:20 AM Stanislav Kozlovski
> > >>  wrote:
> > >>
> > >>> Hi Apoorv,
> > >>>
> > >>> Thanks for taking ownership and looking into this! One more caveat is
> > >> that
> > >>> I believe this first publish is ran with JDK 8, as the release.py
> runs
> > >> with
> > >>> both JDK 8 and (if I recall correctly) 17 versions. This seems to
> fail
> > on
> > >>> the first one - so JDK 8.
> > >>> Not sure if that is related in any way. And I'm also not sure if it
> > >> should
> > >>> be kafka-clients or just clients.
> > >>>
> > >>> On Sat, Dec 30, 2023 at 10:48 AM Apoorv Mittal <
> > apoorvmitta...@gmail.com
> > >>>
> > >>> wrote:
> > >>>
> >  Hi Stan,
> >  Thanks for looking into the release. I worked with `./gradlewAll
> >  publishToMavenLocal` which generates the respective
> > `kafka-clients.jar`
> >  and deploys to maven local, I believed that `./gradlewAll publish`
> > >> should
> >  just publish the artifacts to remote repository and hence should
> > always
> >  work as jars successfully gets deployed to local maven.
> > 
> >  Though now I set up the remote private maven repository for myself
> (on
> >  jfrog) and tried `./gradlewAll publish` on the 3.7 branch and
> >  successfully completed 

[jira] [Created] (KAFKA-16084) Simplify and deduplicate StandaloneHerderTest mocking

[Greg Harris (Jira)]

Greg Harris created KAFKA-16084:
---

 Summary: Simplify and deduplicate StandaloneHerderTest mocking
 Key: KAFKA-16084
 URL: https://issues.apache.org/jira/browse/KAFKA-16084
 Project: Kafka
  Issue Type: Test
  Components: connect
Reporter: Greg Harris


The StandaloneHerderTest has some cruft that can be cleaned up. What i've found:

* The `connector` field is written in nearly every test, but only read by one 
test, and looks to be nearly irrelevant.
* `expectConfigValidation` has two ways of specifying consecutive validations. 
1. The boolean shouldCreateConnector which is true in the first invocation and 
false in subsequent invocations. 2. by passing multiple configurations via 
varargs.
* The class uses a mix of Mock annotations and mock(Class) invocations
* The test doesn't stop the thread pool created inside the herder and might 
leak threads
* Mocking for Worker#startConnector is 6 lines which are duplicated 8 times 
throughout the test
* Some waits are 1000 ms and others are 1000 s, and could be pulled out to 
constants or a util method



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2538

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Justine Olshan]

Hey Rowland,

Not sure why this message showed up in a different thread from the other
KIP-939 discussion (is it just me?)

In KIP-939, we do away with having any transactional timeout on the Kafka
side. The external coordinator is fully responsible for controlling whether
the transaction completes.

While I think there is some use in having a prepare stage, I just wanted to
clarify what the current KIP is proposing.

Thanks,
Justine

On Wed, Jan 3, 2024 at 7:49 PM Rowland Smith  wrote:

> Hi Artem,
>
> I saw your response in the thread I started discussing Kafka distributed
> transaction support and the XA interface. I would like to work with you to
> add XA support to Kafka on top of the excellent foundational work that you
> have started with KIP-939. I agree that explicit XA support should not be
> included in the Kafka codebase as long as the right set of basic operations
> are provided. I will begin pulling together a KIP to follow KIP-939.
>
> I did have one comment on KIP-939 itself. I see that you considered an
> explicit "prepare" RPC, but decided not to add it. If I understand your
> design correctly, that would mean that a 2PC transaction would have a
> single timeout that would need to be long enough to ensure that prepared
> transactions are not aborted when an external coordinator fails. However,
> this also means that an unprepared transaction would not be aborted without
> waiting for the same timeout. Since long running transactions block
> transactional consumers, having a long timeout for all transactions could
> be disruptive. An explicit "prepare " RPC would allow the server to abort
> unprepared transactions after a relatively short timeout, and apply a much
> longer timeout only to prepared transactions. The explicit "prepare" RPC
> would make Kafka server more resilient to client failure at the cost of an
> extra synchronous RPC call. I think its worth reconsidering this.
>
> With an XA implementation this might become a more significant issue since
> the transaction coordinator has no memory of unprepared transactions across
> restarts. Such transactions would need to be cleared by hand through the
> admin client even when the transaction coordinator restarts successfully.
>
> - Rowland
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #132

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16083) Exclude throttle time when expiring inflight requests on a connection

[Adithya Chandra (Jira)]

Adithya Chandra created KAFKA-16083:
---

 Summary: Exclude throttle time when expiring inflight requests on 
a connection
 Key: KAFKA-16083
 URL: https://issues.apache.org/jira/browse/KAFKA-16083
 Project: Kafka
  Issue Type: Bug
  Components: clients
Reporter: Adithya Chandra


When expiring inflight requests, the network client does not take throttle time 
into account. If a connection has multiple inflight requests (default of 5) and 
each request is throttled then some of the requests can incorrectly marked as 
expired. Subsequently the connection is closed and the client establishes a new 
connection to the broker. This behavior leads to unnecessary connections to the 
broker, leads to connection storms and increases latencies. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Apoorv Mittal]

Hi Stan,
I have opened the minor PR: https://github.com/apache/kafka/pull/15127 to
fix publishing the dependency. Once discussed and merged in trunk, I'll
update the 3.7 branch as well.

Regards,
Apoorv Mittal
+44 7721681581


On Thu, Jan 4, 2024 at 12:49 PM Matthias J. Sax  wrote:

> We found a blocker for 3.7:
> https://issues.apache.org/jira/browse/KAFKA-16077
>
> Already having a PR under review to fix it.
>
>
> -Matthias
>
> On 1/3/24 10:43 AM, Stanislav Kozlovski wrote:
> > Hey all, happy new year.
> >
> > Thanks for the heads up Almog. Makes sense.
> >
> > To give an update - I haven't been able to resolve the gradlewAll publish
> > failure, and as such haven't been able to release an RC.
> > As a minor barrier, I have to also update the year in the NOTICE file,
> > otherwise the release script won't let me continue -
> > https://github.com/apache/kafka/pull/15111
> >
> > Me and Apoorv synced offline and ran a few tests to debug the issue
> > regarding the clients build. I successfully executed `publish` when
> > pointing toward a custom jfrog repo with both JDK 8 and 17. Inspecting
> the
> > debug logs, the task that previously failed
> > `:clients:publishMavenJavaPublicationToMavenRepository'` passed
> > successfully. Here's a sample of the logs -
> >
> https://gist.github.com/stanislavkozlovski/841060cb467ec1d179cc9f293c8702e7
> >
> > Having read the release.py script a few times, I am not able to see what
> is
> > different in the setup there. It simply clones the repo anew, gets the
> 3.7
> > branch and runs the same command.
> >
> > At this point, I am contemplating pushing a commit to 3.7 that modifies
> the
> > release.py file that enables debug on the command:
> > diff --git a/release.py b/release.py
> > index 43c5809861..e299e10e74 100755
> > --- a/release.py
> > +++ b/release.py
> > @@ -675,7 +675,7 @@ with
> > open(os.path.expanduser("~/.gradle/gradle.properties")) as f:
> >   contents = f.read()
> >   if not user_ok("Going to build and upload mvn artifacts based on these
> > settings:\n" + contents + '\nOK (y/n)?: '):
> >   fail("Retry again later")
> > -cmd("Building and uploading archives", "./gradlewAll publish",
> > cwd=kafka_dir, env=jdk8_env, shell=True)
> > +cmd("Building and uploading archives", "./gradlewAll publish --debug",
> > cwd=kafka_dir, env=jdk8_env, shell=True)
> >   cmd("Building and uploading archives", "mvn deploy -Pgpg-signing",
> > cwd=streams_quickstart_dir, env=jdk8_env, shell=True)
> >
> >   release_notification_props = { 'release_version': release_version,
> > (END)
> >
> > and continuing to debug through that.
> >
> > Since the release.py script grabs a new copy of origin, we have to modify
> > upstream. An alternative is for me to use my local github Kafka repo, but
> > that may result in the script pushing a build of that into the remote
> > servers.
> >
> > On Tue, Jan 2, 2024 at 8:17 PM Almog Gavra 
> wrote:
> >
> >> Hello Stan,
> >>
> >> I wanted to give you a heads up that
> >> https://github.com/apache/kafka/pull/15073 (
> >> https://issues.apache.org/jira/browse/KAFKA-16046) was identified as a
> >> blocker regression and should be merged to trunk by EOD.
> >>
> >> Cheers,
> >> Almog
> >>
> >> On Tue, Jan 2, 2024 at 4:20 AM Stanislav Kozlovski
> >>  wrote:
> >>
> >>> Hi Apoorv,
> >>>
> >>> Thanks for taking ownership and looking into this! One more caveat is
> >> that
> >>> I believe this first publish is ran with JDK 8, as the release.py runs
> >> with
> >>> both JDK 8 and (if I recall correctly) 17 versions. This seems to fail
> on
> >>> the first one - so JDK 8.
> >>> Not sure if that is related in any way. And I'm also not sure if it
> >> should
> >>> be kafka-clients or just clients.
> >>>
> >>> On Sat, Dec 30, 2023 at 10:48 AM Apoorv Mittal <
> apoorvmitta...@gmail.com
> >>>
> >>> wrote:
> >>>
>  Hi Stan,
>  Thanks for looking into the release. I worked with `./gradlewAll
>  publishToMavenLocal` which generates the respective
> `kafka-clients.jar`
>  and deploys to maven local, I believed that `./gradlewAll publish`
> >> should
>  just publish the artifacts to remote repository and hence should
> always
>  work as jars successfully gets deployed to local maven.
> 
>  Though now I set up the remote private maven repository for myself (on
>  jfrog) and tried `./gradlewAll publish` on the 3.7 branch and
>  successfully completed the build with all artifacts uploaded to the
> >>> remote
>  repository. What seems strange to me is the error you mentioned in the
>  previous email regarding the reference of the clients jar. I suppose
> >> the
>  reference should be to `kafka-clients.jar` rather than `clients.jar`,
> I
>  might be missing if something else gets triggered in the release
> >>> pipeline.
>  Do you think I should set up the remote repository as per the
> >>> instructions
>  in `release.py` and try running `./release.py` as that might do
> >> something
>  different, 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2537

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #47

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 458207 lines...]
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :server:compileTestJava
> Task :server:testClasses
> Task :core:compileScala
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :streams:generateMetadataFileForMavenJavaPublication
> Task :raft:compileTestJava
> Task :raft:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.7/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."

> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.7/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
2 warnings

> Task :clients:javadocJar
> Task :metadata:compileTestJava
> Task :metadata:testClasses
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :connect:api:generateMetadataFileForMavenJavaPublication
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :streams:javadoc
> Task :streams:javadocJar
> Task :streams:srcJar
> Task :streams:processTestResources UP-TO-DATE
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 7m 29s
95 actionable tasks: 41 executed, 54 up-to-date

Publishing build scan...
https://ge.apache.org/s/bfyyesmjk4bcw

[Pipeline] sh
+ + grepcut ^version= -d= gradle.properties -f
 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_3.7/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.7.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.7.0-SNAPSHOT/streams-quickstart-3.7.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.7.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources 

[jira] [Created] (KAFKA-16082) JBOD: Possible dataloss when moving leader partition

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-16082:
-

 Summary: JBOD: Possible dataloss when moving leader partition
 Key: KAFKA-16082
 URL: https://issues.apache.org/jira/browse/KAFKA-16082
 Project: Kafka
  Issue Type: Bug
  Components: jbod
Affects Versions: 3.7.0
Reporter: Proven Provenzano
Assignee: Gaurav Narula
 Fix For: 3.7.0


There is a possible dataloss scenario

when using JBOD,

when moving the partition leader log from one directory to another on the same 
broker,

when after the destination log has caught up to the source log and after the 
broker has sent an update to the partition assignment

if the broker accepts and commits a new record for the partition and then the 
broker restarts and the original partition leader log is lost

then the destination log would not contain the new record.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16081) Limit number of ssl connections in brokers

[Jimmy Wang (Jira)]

Jimmy Wang created KAFKA-16081:
--

 Summary: Limit number of ssl connections in brokers
 Key: KAFKA-16081
 URL: https://issues.apache.org/jira/browse/KAFKA-16081
 Project: Kafka
  Issue Type: New Feature
Reporter: Jimmy Wang
Assignee: Jimmy Wang


In Kafka, an SSL connection occupies approximately 100KB of memory, while a 
plaintext connection occupies around 250 bytes, resulting in a memory footprint 
ratio of approximately 400:1. Therefore, there should be a limitation for SSL 
connections.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.7.0 Release

[Matthias J. Sax]

We found a blocker for 3.7: 
https://issues.apache.org/jira/browse/KAFKA-16077


Already having a PR under review to fix it.


-Matthias

On 1/3/24 10:43 AM, Stanislav Kozlovski wrote:

Hey all, happy new year.

Thanks for the heads up Almog. Makes sense.

To give an update - I haven't been able to resolve the gradlewAll publish
failure, and as such haven't been able to release an RC.
As a minor barrier, I have to also update the year in the NOTICE file,
otherwise the release script won't let me continue -
https://github.com/apache/kafka/pull/15111

Me and Apoorv synced offline and ran a few tests to debug the issue
regarding the clients build. I successfully executed `publish` when
pointing toward a custom jfrog repo with both JDK 8 and 17. Inspecting the
debug logs, the task that previously failed
`:clients:publishMavenJavaPublicationToMavenRepository'` passed
successfully. Here's a sample of the logs -
https://gist.github.com/stanislavkozlovski/841060cb467ec1d179cc9f293c8702e7

Having read the release.py script a few times, I am not able to see what is
different in the setup there. It simply clones the repo anew, gets the 3.7
branch and runs the same command.

At this point, I am contemplating pushing a commit to 3.7 that modifies the
release.py file that enables debug on the command:
diff --git a/release.py b/release.py
index 43c5809861..e299e10e74 100755
--- a/release.py
+++ b/release.py
@@ -675,7 +675,7 @@ with
open(os.path.expanduser("~/.gradle/gradle.properties")) as f:
  contents = f.read()
  if not user_ok("Going to build and upload mvn artifacts based on these
settings:\n" + contents + '\nOK (y/n)?: '):
  fail("Retry again later")
-cmd("Building and uploading archives", "./gradlewAll publish",
cwd=kafka_dir, env=jdk8_env, shell=True)
+cmd("Building and uploading archives", "./gradlewAll publish --debug",
cwd=kafka_dir, env=jdk8_env, shell=True)
  cmd("Building and uploading archives", "mvn deploy -Pgpg-signing",
cwd=streams_quickstart_dir, env=jdk8_env, shell=True)

  release_notification_props = { 'release_version': release_version,
(END)

and continuing to debug through that.

Since the release.py script grabs a new copy of origin, we have to modify
upstream. An alternative is for me to use my local github Kafka repo, but
that may result in the script pushing a build of that into the remote
servers.

On Tue, Jan 2, 2024 at 8:17 PM Almog Gavra  wrote:


Hello Stan,

I wanted to give you a heads up that
https://github.com/apache/kafka/pull/15073 (
https://issues.apache.org/jira/browse/KAFKA-16046) was identified as a
blocker regression and should be merged to trunk by EOD.

Cheers,
Almog

On Tue, Jan 2, 2024 at 4:20 AM Stanislav Kozlovski
 wrote:


Hi Apoorv,

Thanks for taking ownership and looking into this! One more caveat is

that

I believe this first publish is ran with JDK 8, as the release.py runs

with

both JDK 8 and (if I recall correctly) 17 versions. This seems to fail on
the first one - so JDK 8.
Not sure if that is related in any way. And I'm also not sure if it

should

be kafka-clients or just clients.

On Sat, Dec 30, 2023 at 10:48 AM Apoorv Mittal 
Hi Stan,
Thanks for looking into the release. I worked with `./gradlewAll
publishToMavenLocal` which generates the respective `kafka-clients.jar`
and deploys to maven local, I believed that `./gradlewAll publish`

should

just publish the artifacts to remote repository and hence should always
work as jars successfully gets deployed to local maven.

Though now I set up the remote private maven repository for myself (on
jfrog) and tried `./gradlewAll publish` on the 3.7 branch and
successfully completed the build with all artifacts uploaded to the

remote

repository. What seems strange to me is the error you mentioned in the
previous email regarding the reference of the clients jar. I suppose

the

reference should be to `kafka-clients.jar` rather than `clients.jar`, I
might be missing if something else gets triggered in the release

pipeline.

Do you think I should set up the remote repository as per the

instructions

in `release.py` and try running `./release.py` as that might do

something

different, though I suspect that it should?

[image: Screenshot 2023-12-30 at 9.33.42 AM.png]


Regards,
Apoorv Mittal


On Fri, Dec 29, 2023 at 2:13 AM Colin McCabe 

wrote:



Just to update this thread, everything in KAFKA-14127 is done now. A

few

tasks got moved to a separate umbrella JIRA.

Some folks are going to do more testing, both manual and automated, in
the next week or two. I think this will give us a good indicator of
stability and what we need to fix.

Right now I'm leaning towards just making it GA since that's how most
features work. It's kind of rare for us to do a multi-step rollout for

new

features.

best,
Colin


On Wed, Dec 20, 2023, at 03:43, Mickael Maison wrote:

Hi,

With the current timeline for 3.7, I tend to agree with Viktor that
JBOD support in KRaft is unlikely to receive the extensive testing
this 

[jira] [Created] (KAFKA-16080) partiton not retention after execute ALTER_REPLICA_LOG_DIRS and LEADER_AND_ISR request at the same time

[wangliucheng (Jira)]

wangliucheng created KAFKA-16080:


 Summary: partiton not retention after execute 
ALTER_REPLICA_LOG_DIRS and LEADER_AND_ISR request at the same time
 Key: KAFKA-16080
 URL: https://issues.apache.org/jira/browse/KAFKA-16080
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.3.2
Reporter: wangliucheng


Hi,
I found a reproducible problem,

when server running a task which is ALTER_REPLICA_LOG_DIRS 
e.g. test-1 /data01/kafka/log/test01-1 -> /data02/kafka/log/test01-1.xxx-future
then 
1) thread task kafka-log-retention not work both /data01/kafka/log/test01-1 and 
/data02/kafka/log/test01-1.xxx-future,
   The result is that the data will not  retention while task is running
   analysis: The kafka-log-retention thread not work on test01-1  after 
invoking logManager.abortAndPauseCleaning(topicPartition)
   
2) If LEADER_AND_ISR is request while ALTER_REPLICA_LOG_DIRS task is running，
   After the task is end,  the data which in /data02/kafka/log/test01-1  will 
not be deleted all the time 
   analysis: logManager.abortAndPauseCleaning(topicPartition) invoked twice, 
but resumed only once

How to optimize this problem
Thanks 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Road to Kafka 4.0

[Mickael Maison]

Hi Ismael,

Yes both KIPs have been voted.
My point, which admittedly wasn't clear, was to gauge the interest in
getting them done and if so identifying people to drive these tasks.

KIP-719 shouldn't require too much more work to complete. There's a PR
[0] which is relatively straightforward. I pinged Lee Dongjin.
KIP-653 is more involved and depends on KIP-719. There's also a PR [1]
which is pretty large.

Yes log4j3 was on my mind as it's expected to be compatible with
log4j2 and bring significant improvements.

0: https://github.com/apache/kafka/pull/10244
1: https://github.com/apache/kafka/pull/7898

Thanks,
Mickael

On Thu, Jan 4, 2024 at 11:34 AM Ismael Juma  wrote:
>
> Hi Mickael,
>
> Given that KIP-653 was accepted, the current position is that we would move
> to log4j2 - provided that someone is available to drive that. It's also
> worth noting that log4j3 is now a thing (but not yet final):
>
> https://logging.apache.org/log4j/3.x/
>
> Ismael
>
> On Thu, Jan 4, 2024 at 2:15 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I've not seen replies about log4j2.
> > The plan was to deprecated the appender (KIP-719) and switch to log4j2
> > (KIP-653).
> >
> > While reload4j works well, I'd still be in favor of switching to
> > log4j2 in Kafka 4.0.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, Dec 29, 2023 at 2:19 AM Colin McCabe  wrote:
> > >
> > > Hi all,
> > >
> > > Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for
> > a Kafka 3.8.x release" email thread.
> > >
> > > Colin
> > >
> > >
> > > On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote:
> > > > Hi Divij,
> > > >
> > > > Thanks for the feedback. I agree that having a 3.8 release is
> > > > beneficial but some of the comments in this message are inaccurate and
> > > > could mislead the community and users.
> > > >
> > > > On Thu, Dec 21, 2023 at 7:00 AM Divij Vaidya 
> > wrote:
> > > >> 1\ Durability/availability bugs in kraft - Even though kraft has been
> > > >> around for a while, we keep finding bugs that impact availability and
> > data
> > > >> durability in it almost with every release [1] [2]. It's a complex
> > feature
> > > >> and such bugs are expected during the stabilization phase. But we
> > can't
> > > >> remove the alternative until we see stabilization in kraft i.e. no new
> > > >> stability/durability bugs for at least 2 releases.
> > > >
> > > > I took a look at both of these issues and neither of them are bugs
> > > > that affect KRaft's durability and availability.
> > > >
> > > >> [1] https://issues.apache.org/jira/browse/KAFKA-15495
> > > >
> > > > This issue is not specific to KRaft and has been an issue in Apache
> > > > Kafka since the ISR leader election and replication algorithm was
> > > > added to Apache Kafka. I acknowledge that this misunderstanding is
> > > > partially due to the Jira description which insinuates that this only
> > > > applies to KRaft which is not true.
> > > >
> > > >> [2] https://issues.apache.org/jira/browse/KAFKA-15489
> > > >
> > > > First, technically this issue was not first discovered in some recent
> > > > release. This issue was identified by me back in January of 2022:
> > > > https://issues.apache.org/jira/browse/KAFKA-13621. I decided to lower
> > > > the priority as it requires a very specific network partition where
> > > > the controllers are partitioned from the current leader but the
> > > > brokers are not.
> > > >
> > > > This is not a durability bug as the KRaft cluster metadata partition
> > > > leader will not be able to advance the HWM and hence commit records.
> > > >
> > > > Regarding availability, The KRaft's cluster metadata partition favors
> > > > consistency and partition tolerance versus availability from CAP. This
> > > > is by design and not a bug in the protocol or implementation.
> > > >
> > > >> 2\ Parity with Zk - There are also pending bugs [3] which are in the
> > > >> category of Zk parity. Removing Zk from Kafka without having full
> > feature
> > > >> parity with Zk will leave some Kafka users with no upgrade path.
> > > >> 3\ Test coverage - We also don't have sufficient test coverage for
> > kraft
> > > >> since quite a few tests are Zk only at this stage.
> > > >>
> > > >> Given these concerns, I believe we need to reach 100% Zk parity and
> > allow
> > > >> new feature stabilisation (such as scram, JBOD) for at least 1 version
> > > >> (maybe more if we find bugs in that feature) before we remove Zk. I
> > also
> > > >> agree with the point of view that we can't delay 4.0 indefinitely and
> > we
> > > >> need a clear cut line.
> > > >
> > > > There seems to be some misunderstanding regarding Apache Kafka
> > > > versioning scheme. Minor versions (e.g. 3.x) are needed for feature
> > > > releases like new RPCs and configurations. They are not needed for bug
> > > > fixes. Bug fixes can and should be done in patch releases (e.g.
> > > > 3.7.x).
> > > >
> > > > This means that you don't need a 3.8 or 3.9 release to fix a 

Re: [DISCUSS] Road to Kafka 4.0

[Ismael Juma]

Hi Mickael,

Given that KIP-653 was accepted, the current position is that we would move
to log4j2 - provided that someone is available to drive that. It's also
worth noting that log4j3 is now a thing (but not yet final):

https://logging.apache.org/log4j/3.x/

Ismael

On Thu, Jan 4, 2024 at 2:15 AM Mickael Maison 
wrote:

> Hi,
>
> I've not seen replies about log4j2.
> The plan was to deprecated the appender (KIP-719) and switch to log4j2
> (KIP-653).
>
> While reload4j works well, I'd still be in favor of switching to
> log4j2 in Kafka 4.0.
>
> Thanks,
> Mickael
>
> On Fri, Dec 29, 2023 at 2:19 AM Colin McCabe  wrote:
> >
> > Hi all,
> >
> > Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for
> a Kafka 3.8.x release" email thread.
> >
> > Colin
> >
> >
> > On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote:
> > > Hi Divij,
> > >
> > > Thanks for the feedback. I agree that having a 3.8 release is
> > > beneficial but some of the comments in this message are inaccurate and
> > > could mislead the community and users.
> > >
> > > On Thu, Dec 21, 2023 at 7:00 AM Divij Vaidya 
> wrote:
> > >> 1\ Durability/availability bugs in kraft - Even though kraft has been
> > >> around for a while, we keep finding bugs that impact availability and
> data
> > >> durability in it almost with every release [1] [2]. It's a complex
> feature
> > >> and such bugs are expected during the stabilization phase. But we
> can't
> > >> remove the alternative until we see stabilization in kraft i.e. no new
> > >> stability/durability bugs for at least 2 releases.
> > >
> > > I took a look at both of these issues and neither of them are bugs
> > > that affect KRaft's durability and availability.
> > >
> > >> [1] https://issues.apache.org/jira/browse/KAFKA-15495
> > >
> > > This issue is not specific to KRaft and has been an issue in Apache
> > > Kafka since the ISR leader election and replication algorithm was
> > > added to Apache Kafka. I acknowledge that this misunderstanding is
> > > partially due to the Jira description which insinuates that this only
> > > applies to KRaft which is not true.
> > >
> > >> [2] https://issues.apache.org/jira/browse/KAFKA-15489
> > >
> > > First, technically this issue was not first discovered in some recent
> > > release. This issue was identified by me back in January of 2022:
> > > https://issues.apache.org/jira/browse/KAFKA-13621. I decided to lower
> > > the priority as it requires a very specific network partition where
> > > the controllers are partitioned from the current leader but the
> > > brokers are not.
> > >
> > > This is not a durability bug as the KRaft cluster metadata partition
> > > leader will not be able to advance the HWM and hence commit records.
> > >
> > > Regarding availability, The KRaft's cluster metadata partition favors
> > > consistency and partition tolerance versus availability from CAP. This
> > > is by design and not a bug in the protocol or implementation.
> > >
> > >> 2\ Parity with Zk - There are also pending bugs [3] which are in the
> > >> category of Zk parity. Removing Zk from Kafka without having full
> feature
> > >> parity with Zk will leave some Kafka users with no upgrade path.
> > >> 3\ Test coverage - We also don't have sufficient test coverage for
> kraft
> > >> since quite a few tests are Zk only at this stage.
> > >>
> > >> Given these concerns, I believe we need to reach 100% Zk parity and
> allow
> > >> new feature stabilisation (such as scram, JBOD) for at least 1 version
> > >> (maybe more if we find bugs in that feature) before we remove Zk. I
> also
> > >> agree with the point of view that we can't delay 4.0 indefinitely and
> we
> > >> need a clear cut line.
> > >
> > > There seems to be some misunderstanding regarding Apache Kafka
> > > versioning scheme. Minor versions (e.g. 3.x) are needed for feature
> > > releases like new RPCs and configurations. They are not needed for bug
> > > fixes. Bug fixes can and should be done in patch releases (e.g.
> > > 3.7.x).
> > >
> > > This means that you don't need a 3.8 or 3.9 release to fix a bug in
> Kafka.
> > >
> > > Thanks!
> > > --
> > > -José
>

Re: [DISCUSS] Road to Kafka 4.0

[Mickael Maison]

Hi,

I've not seen replies about log4j2.
The plan was to deprecated the appender (KIP-719) and switch to log4j2
(KIP-653).

While reload4j works well, I'd still be in favor of switching to
log4j2 in Kafka 4.0.

Thanks,
Mickael

On Fri, Dec 29, 2023 at 2:19 AM Colin McCabe  wrote:
>
> Hi all,
>
> Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for a 
> Kafka 3.8.x release" email thread.
>
> Colin
>
>
> On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote:
> > Hi Divij,
> >
> > Thanks for the feedback. I agree that having a 3.8 release is
> > beneficial but some of the comments in this message are inaccurate and
> > could mislead the community and users.
> >
> > On Thu, Dec 21, 2023 at 7:00 AM Divij Vaidya  
> > wrote:
> >> 1\ Durability/availability bugs in kraft - Even though kraft has been
> >> around for a while, we keep finding bugs that impact availability and data
> >> durability in it almost with every release [1] [2]. It's a complex feature
> >> and such bugs are expected during the stabilization phase. But we can't
> >> remove the alternative until we see stabilization in kraft i.e. no new
> >> stability/durability bugs for at least 2 releases.
> >
> > I took a look at both of these issues and neither of them are bugs
> > that affect KRaft's durability and availability.
> >
> >> [1] https://issues.apache.org/jira/browse/KAFKA-15495
> >
> > This issue is not specific to KRaft and has been an issue in Apache
> > Kafka since the ISR leader election and replication algorithm was
> > added to Apache Kafka. I acknowledge that this misunderstanding is
> > partially due to the Jira description which insinuates that this only
> > applies to KRaft which is not true.
> >
> >> [2] https://issues.apache.org/jira/browse/KAFKA-15489
> >
> > First, technically this issue was not first discovered in some recent
> > release. This issue was identified by me back in January of 2022:
> > https://issues.apache.org/jira/browse/KAFKA-13621. I decided to lower
> > the priority as it requires a very specific network partition where
> > the controllers are partitioned from the current leader but the
> > brokers are not.
> >
> > This is not a durability bug as the KRaft cluster metadata partition
> > leader will not be able to advance the HWM and hence commit records.
> >
> > Regarding availability, The KRaft's cluster metadata partition favors
> > consistency and partition tolerance versus availability from CAP. This
> > is by design and not a bug in the protocol or implementation.
> >
> >> 2\ Parity with Zk - There are also pending bugs [3] which are in the
> >> category of Zk parity. Removing Zk from Kafka without having full feature
> >> parity with Zk will leave some Kafka users with no upgrade path.
> >> 3\ Test coverage - We also don't have sufficient test coverage for kraft
> >> since quite a few tests are Zk only at this stage.
> >>
> >> Given these concerns, I believe we need to reach 100% Zk parity and allow
> >> new feature stabilisation (such as scram, JBOD) for at least 1 version
> >> (maybe more if we find bugs in that feature) before we remove Zk. I also
> >> agree with the point of view that we can't delay 4.0 indefinitely and we
> >> need a clear cut line.
> >
> > There seems to be some misunderstanding regarding Apache Kafka
> > versioning scheme. Minor versions (e.g. 3.x) are needed for feature
> > releases like new RPCs and configurations. They are not needed for bug
> > fixes. Bug fixes can and should be done in patch releases (e.g.
> > 3.7.x).
> >
> > This means that you don't need a 3.8 or 3.9 release to fix a bug in Kafka.
> >
> > Thanks!
> > --
> > -José

Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

[Federico Valeri]

Thanks! This will finally reconcile Javadoc and implementation.

+1 (non binding)

On Thu, Jan 4, 2024 at 6:49 AM Yash Mayya  wrote:
>
> Hi Chris,
>
> +1 (binding), thanks for the KIP.
>
> Based on discussion in other threads, it looks like the community is
> aligned with having a 3.8 release before the 4.0 release so we should be
> able to remove the 'tasks.max.enforce' connector property in 4.0 (we'd
> discussed potentially having to live with this property until 5.0 in this
> KIP's discussion thread). Once we have confirmation of a 3.8 release, will
> this KIP be updated to reflect the exact AK versions where the deprecated
> property will be introduced and removed?
>
> Thanks,
> Yash
>
> On Wed, Jan 3, 2024 at 11:37 PM Greg Harris 
> wrote:
>
> > Hey Chris,
> >
> > Thanks for the KIP! I think the aggressive default and deprecation
> > schedule is the right choice for this change.
> >
> > +1 (binding)
> >
> > On Wed, Jan 3, 2024 at 9:01 AM Mickael Maison 
> > wrote:
> > >
> > > Hi Chris,
> > >
> > > +1 (binding), thanks for the KIP
> > >
> > > Mickael
> > >
> > > On Tue, Jan 2, 2024 at 8:55 PM Hector Geraldino (BLOOMBERG/ 919 3RD A)
> > >  wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks Chris!
> > > >
> > > > From: dev@kafka.apache.org At: 01/02/24 11:49:18 UTC-5:00To:
> > dev@kafka.apache.org
> > > > Subject: Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka
> > Connect
> > > >
> > > > Hi all,
> > > >
> > > > Happy New Year! Wanted to give this a bump now that the holidays are
> > over
> > > > for a lot of us. Looking forward to people's thoughts!
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Mon, Dec 4, 2023 at 10:36 AM Chris Egerton  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-1004, which adds enforcement for
> > the
> > > > > tasks.max connector property in Kafka Connect.
> > > > >
> > > > > The KIP:
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+
> > > > property+in+Kafka+Connect
> > > > >
> > > > > The discussion thread:
> > > > > https://lists.apache.org/thread/scx75cjwm19jyt19wxky41q9smf5nx6d
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > >
> > > >
> >

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[Ismael Juma]

Hi Ziming,

Why is the flag required at all? Can we use incremental and fallback
automatically if it's not supported by the broker? At this point, the vast
majority of clusters should support it.

Ismael

On Mon, Dec 18, 2023 at 7:58 PM ziming deng 
wrote:

>
> Hello, I want to start a discussion on KIP-1011, to make the broker config
> change path unified with that of user/topic/client-metrics and avoid some
> bugs.
>
> Here is the link:
>
> KIP-1011: Use incrementalAlterConfigs when updating broker configs by
> kafka-configs.sh - Apache Kafka - Apache Software Foundation
> 
> cwiki.apache.org
> 
> [image: favicon.ico]
> 
> 
>
> Best,
> Ziming.
>

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[Federico Valeri]

On Fri, Dec 29, 2023 at 8:49 AM ziming deng  wrote:
>
> Hello all,
>
> We have 2 people prefer using "--enable-incremental" and it makes sense to 
> move the incompatible process to 4.X, I changed the default way to still use 
> `alterConfigs` and use `incrementalAlterConfigs` only when 
> "--enable-incremental" is specified.
>

I agree with this change, much better. Thanks.

> I will initiate a vote process if no further opinions coming.
>
> Best,
> Ziming
>
>
> > On Dec 26, 2023, at 13:21, Kamal Chandraprakash 
> >  wrote:
> >
> > Hi Ziming,
> >
> > Thanks for the KIP! The proposal LGTM.
> >
> > I'm also inclined towards option 2 (i.e. add an explicit
> > --enable-incremental flag in 3.X version)
> > to avoid any incompatible change in v3.X. As mentioned in this thread, many
> > users might be
> > using external tools to do the topic rebalance and apply throttle config on
> > the brokers, it might
> > break for them when the tools are upgraded before the broker binary.
> >
> > cruise-control is currently using kafka-version: 3.1
> > https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_5/gradle.properties#L5
> >
> > On Mon, Dec 25, 2023 at 7:15 PM Divij Vaidya 
> > wrote:
> >
> >> Thank you for the summary, Ziming.
> >>
> >> Personally, I would prefer the latter i.e. having the incompatible change
> >> in 4.x instead of 3.x. This is because a major version upgrade goes through
> >> additional scrutiny by the users and usually comes with inevitable code
> >> changes required on the client. Hence, this incompatibility will be part of
> >> one amongst many changes that users will perform to upgrade to 4.x. This is
> >> unlike a major version change from 3.7 to 3.8 where users expect a simple
> >> upgrade without any code changes.
> >>
> >> Let's wait and hear what others think about this.
> >>
> >> --
> >> Divij Vaidya
> >>
> >>
> >>
> >> On Mon, Dec 25, 2023 at 1:18 PM ziming deng 
> >> wrote:
> >>
> >>> Hello Divij Vaidya,
> >>>
> >>> You are right that users should update existing scripts to add
> >>> ‘—disable-incremental’, and you mentioned another upgrade path which is
> >>> similar, the summary of the 2 schemes are:
> >>> we change existing scripts to use `incrementalAlterConfigs` and add
> >>> "--disable-incremental" flag for old servers in Kafka 3.X, and remove it
> >> in
> >>> Kafka 4.X.
> >>> we keep existing scripts unchanged and add an "--enable-incremental" flag
> >>> for new servers in Kafka 3.X, and remove it in Kafka 4.X.
> >>>
> >>> I think there will always be an incompatible upgrade process to move
> >>> default behavior from `alterConfigs` to `incrementalConfigs`. In the
> >> first
> >>> scheme we are doing this incompatible upgrade in Kafka 3.X, and in the
> >>> second scheme we are moving it to 4.X, I think we should do it as soon as
> >>> possible if it's inevitable.
> >>> However, I will add this to , and I'm open to this
> >>> if more people think it's more suitable.
> >>>
> >>>
> >>> --,
> >>> Ziming
> >>>
>  On Dec 22, 2023, at 18:13, Divij Vaidya 
> >> wrote:
> 
>  Divij Vaidya
> >>>
> >>>
> >>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2536

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16079) Fix leak in LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest

[Luke Chen (Jira)]

Luke Chen created KAFKA-16079:
-

 Summary: Fix leak in 
LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest
 Key: KAFKA-16079
 URL: https://issues.apache.org/jira/browse/KAFKA-16079
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


Fix leak in 
LocalLeaderEndPointTest/FinalizedFeatureChangeListenerTest/KafkaApisTest/ReplicaManagerConcurrencyTest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)