Re: [ANNOUNCE] Apache Kafka 3.9.0

[Luke Chen]

Hi Colin,

Thanks for all the hard work for the release!

Luke

On Fri, Nov 8, 2024 at 7:12 AM Colin McCabe  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 3.9.0
>
> - This is a major release, the final one in the 3.x line. (There may of
> course be other minor releases in this line, such as 3.9.1.)
> - Tiered storage will be considered production-ready in this release.
> - This will be the final major release to feature the deprecated ZooKeeper
> mode.
>
> This release includes the following KIPs:
> - KIP-853: Support dynamically changing KRaft controller membership
> - KIP-1057: Add remote log metadata flag to the dump log tool
> - KIP-1049: Add config log.summary.interval.ms to Kafka Streams
> - KIP-1040: Improve handling of nullable values in InsertField,
> ExtractField, and other transformations
> - KIP-1031: Control offset translation in MirrorSourceConnector
> - KIP-1033: Add Kafka Streams exception handler for exceptions occurring
> during processing
> - KIP-1017: Health check endpoint for Kafka Connect
> - KIP-1025: Optionally URL-encode clientID and clientSecret in
> authorization header
> - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> - KIP-950: Tiered Storage Disablement
> - KIP-956: Tiered Storage Quotas
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.9.0/RELEASE_NOTES.html
>
>
> An overview of the release can be found in our announcement blog post:
> https://kafka.apache.org/blog#apache_kafka_390_release_announcement
>
> You can download the source and binary release from:
> https://kafka.apache.org/downloads#3.9.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 133 contributors to this release!
> (Please report an unintended omission)
>
> Abhijeet Kumar, abhi-ksolves, Abhinav Dixit, Adrian Preston, Alieh Saeedi,
> Alyssa Huang, Anatoly Popov, Andras Katona, Andrew Schofield, Andy
> Wilkinson,
> Anna Sophie Blee-Goldman, Antoine Pourchet, Apoorv Mittal, Arnav Dadarya,
> Arnout Engelen, Arpit Goyal, Arun Mathew, A. Sophie Blee-Goldman, Ayoub
> Omari,
> bachmanity1, Bill Bejeck, brenden20, Bruno Cadonna, Chia Chuan Yu,
> Chia-Ping
> Tsai, ChickenchickenLove, Chirag Wadhwa, Chris Egerton, Christo Lolov,
> Ming-Yen Chung, Colin P. McCabe, Cy, David Arthur, David Jacot,
> demo...@csie.io, dengziming, Dimitar Dimitrov, Dmitry Werner, Dongnuo Lyu,
> dujian0068, Edoardo Comar, Farbod Ahmadian, Federico Valeri, Fiore Mario
> Vitale, Florin Ak ermann, Francois Visconte, GANESH SADANALA, Gantigmaa
> Selenge, Gaurav Narula, gongxuanzhang, Greg Harris, Do Gyeongwon, Harry
> Fallows, Hongten, Ian McDonald, Igor Soarez, Ismael Juma, Ivan Yurchenko,
> Jakub Scholz, Jason Gustafson, Jeff Kim, Jim Galasyn, Jin yong Choi, Johnny
> Hsu, José Armando García Sancio, Josep Prat, Jun Rao, Justine Olshan, Kamal
> Chandraprakash, Ken Huang, Kevin Wu, Kirk True, Kondrat Bertalan, Krishna
> Agarwal, KrishVora01, Kuan-Po (Cooper) Tseng, Kuan-Po Tseng, Lee Dongjin,
> Lianet Magrans , Logan Zhu, Loïc GREFFIER, Lucas Brutschy, Luke Chen,
> Maciej
> Moscicki, Manikumar Reddy, Mason Chen, Matthias J. Sax, Max Riedel, Mickael
> Maison, Murali Basani, 120571969+nancy-ksol...@users.noreply.github.com,
> NICOLAS GUYO

Re: [DISCUSS] Apache Kafka 4.0.0 release

[Luke Chen]

Hi Kamal,

Just added them in to the release plan.

Thanks.
Luke

On Thu, Nov 7, 2024 at 11:11 AM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Luke,
>
> Please add the KIP-1058 and KIP-1105 to the release plan.
> KIP-1058 is accepted and PRs are ready.
> KIP-1105 is a minor one, and will open a voting thread soon.
>
> Thanks,
> Kamal
>
> On Thu, Nov 7, 2024 at 7:28 AM Luke Chen  wrote:
>
> > Hi Alyssa,
> >
> > I've added KIP-966 into the release plan.
> >
> > Thanks.
> > Luke
> >
> > On Thu, Nov 7, 2024 at 3:54 AM Alyssa Huang  >
> > wrote:
> >
> > > Thanks David for being the release manager for 4.0!
> > > I noticed the release plan
> > > <https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+4.0.0>
> > > does
> > > not yet have KIP-996 (Pre-Vote) listed, can I edit the page to add it?
> > >
> > > Best,
> > > Alyssa
> > >
> > > On Sun, Oct 20, 2024 at 11:36 AM Colin McCabe 
> > wrote:
> > >
> > > > Hi TengYao Chi,
> > > >
> > > > The KIP freeze for Apache Kafka 4.0 is. November 20th. So if KIP-1092
> > has
> > > > been accepted already, it should make it into 4.0 without any further
> > > > discussion (assuming the code is ready by feature freeze.)
> > > >
> > > > Cheers,
> > > > Colin
> > > >
> > > > On Fri, Oct 18, 2024, at 20:32, TengYao Chi wrote:
> > > > > Hi David,
> > > > >
> > > > > I would like to request the inclusion of KIP-1092 in the 4.0
> release
> > > > plan.
> > > > > This KIP is newly accepted, and I believe it will bring valuable
> > > > > enhancements for both the Consumer and Kafka Streams components.
> > > > >
> > > > > Thank you for considering this addition.
> > > > >
> > > > > Best,
> > > > > TengYao
> > > > >
> > > > > Colin McCabe  於 2024年10月11日 週五 上午12:22寫道：
> > > > >
> > > > >> Yes (although I don't believe there's a JIRA for that specifically
> > > yet.)
> > > > >>
> > > > >> best,
> > > > >> Colin
> > > > >>
> > > > >>
> > > > >> On Thu, Oct 10, 2024, at 09:11, David Jacot wrote:
> > > > >> > Hi Colin,
> > > > >> >
> > > > >> > Thanks. That seems perfect. Does it include removing --zookeeper
> > > from
> > > > the
> > > > >> > command line tools (e.g. kafka-configs)?
> > > > >> >
> > > > >> > Best,
> > > > >> > David
> > > > >> >
> > > > >> > On Thu, Oct 10, 2024 at 6:06 PM Colin McCabe <
> cmcc...@apache.org>
> > > > wrote:
> > > > >> >
> > > > >> >> Hi David & Luke,
> > > > >> >>
> > > > >> >> We have been using
> > > https://issues.apache.org/jira/browse/KAFKA-17611
> > > > as
> > > > >> >> the umbrella JIRA for ZK removal tasks.  Progress has been
> pretty
> > > > >> rapid, I
> > > > >> >> do think we will get there by January. (Well hopefully even
> > earlier
> > > > :)
> > > > >> >>
> > > > >> >> best,
> > > > >> >> Colin
> > > > >> >>
> > > > >> >>
> > > > >> >> On Thu, Oct 10, 2024, at 05:55, David Jacot wrote:
> > > > >> >> > Hi Luke,
> > > > >> >> >
> > > > >> >> > That's a good point. I think that we should try to stick to
> the
> > > > dates
> > > > >> >> > though. In my opinion, we should ensure that ZK and all the
> > > related
> > > > >> >> public
> > > > >> >> > facing apis are gone in 4.0 by code freeze. The simplest way
> > > would
> > > > be
> > > > >> to
> > > > >> >> > have an epic for the removal with all the related tasks. We
> can
> > > > then
> > > > >> mark
> > > > >> >> > it as a blocker for 4.0. We may alrea

Re: [DISCUSS] Apache Kafka 4.0.0 release

[Luke Chen]

Hi Alyssa,

I've added KIP-966 into the release plan.

Thanks.
Luke

On Thu, Nov 7, 2024 at 3:54 AM Alyssa Huang 
wrote:

> Thanks David for being the release manager for 4.0!
> I noticed the release plan
> <https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+4.0.0>
> does
> not yet have KIP-996 (Pre-Vote) listed, can I edit the page to add it?
>
> Best,
> Alyssa
>
> On Sun, Oct 20, 2024 at 11:36 AM Colin McCabe  wrote:
>
> > Hi TengYao Chi,
> >
> > The KIP freeze for Apache Kafka 4.0 is. November 20th. So if KIP-1092 has
> > been accepted already, it should make it into 4.0 without any further
> > discussion (assuming the code is ready by feature freeze.)
> >
> > Cheers,
> > Colin
> >
> > On Fri, Oct 18, 2024, at 20:32, TengYao Chi wrote:
> > > Hi David,
> > >
> > > I would like to request the inclusion of KIP-1092 in the 4.0 release
> > plan.
> > > This KIP is newly accepted, and I believe it will bring valuable
> > > enhancements for both the Consumer and Kafka Streams components.
> > >
> > > Thank you for considering this addition.
> > >
> > > Best,
> > > TengYao
> > >
> > > Colin McCabe  於 2024年10月11日 週五 上午12:22寫道：
> > >
> > >> Yes (although I don't believe there's a JIRA for that specifically
> yet.)
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Thu, Oct 10, 2024, at 09:11, David Jacot wrote:
> > >> > Hi Colin,
> > >> >
> > >> > Thanks. That seems perfect. Does it include removing --zookeeper
> from
> > the
> > >> > command line tools (e.g. kafka-configs)?
> > >> >
> > >> > Best,
> > >> > David
> > >> >
> > >> > On Thu, Oct 10, 2024 at 6:06 PM Colin McCabe 
> > wrote:
> > >> >
> > >> >> Hi David & Luke,
> > >> >>
> > >> >> We have been using
> https://issues.apache.org/jira/browse/KAFKA-17611
> > as
> > >> >> the umbrella JIRA for ZK removal tasks.  Progress has been pretty
> > >> rapid, I
> > >> >> do think we will get there by January. (Well hopefully even earlier
> > :)
> > >> >>
> > >> >> best,
> > >> >> Colin
> > >> >>
> > >> >>
> > >> >> On Thu, Oct 10, 2024, at 05:55, David Jacot wrote:
> > >> >> > Hi Luke,
> > >> >> >
> > >> >> > That's a good point. I think that we should try to stick to the
> > dates
> > >> >> > though. In my opinion, we should ensure that ZK and all the
> related
> > >> >> public
> > >> >> > facing apis are gone in 4.0 by code freeze. The simplest way
> would
> > be
> > >> to
> > >> >> > have an epic for the removal with all the related tasks. We can
> > then
> > >> mark
> > >> >> > it as a blocker for 4.0. We may already have one. Let me search
> > >> around.
> > >> >> >
> > >> >> > Best,
> > >> >> > David
> > >> >> >
> > >> >> > On Thu, Oct 10, 2024 at 2:43 PM Luke Chen 
> > wrote:
> > >> >> >
> > >> >> >> Hi David,
> > >> >> >>
> > >> >> >> The release plan looks good to me.
> > >> >> >>
> > >> >> >> But since the 4.0 release is a release without ZK, I'm wondering
> > if
> > >> we
> > >> >> >> should list some release criteria for it?
> > >> >> >> The Problem I can imagine is like, when the code freeze date is
> > >> reached,
> > >> >> >> but there are still many ZK removal tasks open, what should we
> do
> > >> with
> > >> >> >> them? We can remove the rest of codes in later releases, but if
> > there
> > >> >> are
> > >> >> >> still some public APIs related to ZK open for users to use, is
> > that
> > >> >> still
> > >> >> >> OK to release 4.0? Ex: The `--zookeeper` option in
> > kafka-configs.sh?
> > >> >> >>
> > >> >> >> Instead of only using time-based management in v

Re: [DISCUSS] New label for breaking changes in 4.0

[Luke Chen]

Hi Divij,

Thanks for raising this!
I agree we should have some ways to identify the breaking changes in
v4.0.0. As I proposed in another thread, in v4.0.0, instead of tracking
with time as before, we should also consider the status of some features
because there are some changes only allowed in the major release. So not
just for documentation, it is also good for release managers to identify
the breaking changes tasks.

Thank you.
Luke

On Wed, Nov 6, 2024 at 12:37 AM Chia-Ping Tsai  wrote:

> I'd like to propose one more rule:
>
> 3. The PR should update `upgrade.html` (
> https://github.com/apache/kafka/blob/trunk/docs/upgrade.html#L25) if it
> brings a breaking change.
>
> Best,
> Chia-Ping
>
>
> Divij Vaidya  於 2024年11月6日 週三 上午12:04寫道：
>
> > Hi folks
> >
> > We would be making many breaking changes in 4.0 such as removal of
> > deprecated metrics, removal of configs etc.
> >
> > It would be nice if we can document all such backward incompatible
> > behaviour in our documentation. The alternative is to leave the user with
> > trying to figure out which changes are breaking from the release notes,
> > which is quite challenging since not all users may be familiar with
> > details.
> >
> > Hence, I propose the following:
> > 1. Add a label "breaking" to the JIRAs scheduled for release in 4.0 which
> > contain backward incompatible changes to public interfaces.
> > 2. During the release process, we will document a "guidance for upgrade"
> > which will call out breaking changes per component.
> >
> > For this proposal to be successful, all committers should be mindful of
> > adding the "breaking" label when closing a JIRA as resolved.
> >
> > Thoughts?
> >
> > --
> > Divij Vaidya
> >
>

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Luke Chen]

Had another read of the KIP, LGTM!
+1 from me.
Thanks for the KIP!

Luke

On Wed, Nov 6, 2024 at 1:58 PM Satish Duggana 
wrote:

> +1
>
> Thanks Kamal for the KIP. Discussed offline with Kamal on the proposal and
> the changes.
> It improves the performance of consumers' remote reads in read_committed
> mode. It brings minor changes in a backward compatible manner.
>
> ~Satish.
>
> On Mon, 4 Nov 2024 at 19:33, Divij Vaidya  wrote:
>
> > +1
> >
> > (I have participated in the review and I think we have a good solution in
> > the KIP)
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Nov 4, 2024 at 5:41 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > Bumping up the voting thread.
> > >
> > > On Mon, Sep 30, 2024 at 3:27 PM Francois Visconte
> > >  wrote:
> > >
> > > > Could we vote on this? This is causing a bunch of tiered storage read
> > > > issues as many consumers default to READ_COMMITTED (eg. librdkafka)
> > > >
> > > > Thanks,
> > > > F.
> > > >
> > > > On Mon, Sep 16, 2024 at 7:20 AM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Bumping this thread for vote. PTAL.
> > > > >
> > > > > On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash <
> > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to open voting for KIP-1058. This KIP improves the
> > consumer
> > > > > > reading from remote storage when READ_COMMITTED isolation level
> is
> > > > > enabled.
> > > > > > PTAL.
> > > > > >
> > > > > > KIP-1058
> > > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1058%3A+Txn+consumer+exerts+pressure+on+remote+storage+when+collecting+aborted+transactions
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Kamal
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] 3.9.0 RC5

[Luke Chen]

Hi Colin,

Thanks for the update.
I've ran a sanity check as Mickael suggested for the maven artifacts with
the one in https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/.
I used jarcomp <https://activityworkshop.net/software/jarcomp/index.html>
to verify:
1. kafka_2.13-3.9.0.jar
2. kafka-clients-3.9.0.jar
3. kafka-metadata-3.9.0.jar

I confirmed all the class files in the jars have the same content.

So +1 (binding) from me.

Thank you.
Luke

On Fri, Nov 1, 2024 at 1:08 AM Mickael Maison 
wrote:

> Hi,
>
> Can we add sanity checks to ensure all artifacts are present when
> running the release?
> Also should we use a different folder for each RC? This would
> eliminate the risk of artifacts from previous RCs hiding an issue.
> Luke spotted the old artifacts this time but this could have been
> missed.
>
> Thanks,
> Mickael
>
> On Thu, Oct 31, 2024 at 5:55 PM Colin McCabe  wrote:
> >
> > Hi David,
> >
> > Thanks for checking. I took a look and yes, I did forget to upload the
> Scala 2.12 artifacts previously. The script runs gradlewAll, but when I was
> trying to do the steps manually, I missed that and just ran gradlew, which
> only did Scala 2.13.
> >
> > This should be fixed now!
> >
> > The scala 2.12 artifacts are here:
> >
> > https://repository.apache.org/content/repositories/orgapachekafka-1827/
> >
> > and also here:
> >
> >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.12/3.9.0/
> >
> > I wonder if we should change the release.py script to just run the
> gradlew steps manually. The gradlewAll script seems a bit confusing (or
> maybe that's just me?)
> >
> > best,
> > Colin
> >
> >
> > On Wed, Oct 30, 2024, at 11:02, David Arthur wrote:
> > > Should there be Scala 2.12 artifacts in the Maven staging?
> > >
> > > I see these for 2.13:
> > > *
> > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka_2.13/3.9.0/
> > > *
> > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-streams-scala_2.13/3.9.0/
> > >
> > > but I don't see 2.12 for 3.9.0 release.
> > >
> > > I can confirm that the other artifacts (such as kafka-clients) do have
> > > today's timestamp for the 3.9.0 artifact.
> > >
> > > -David
> > >
> > > On Wed, Oct 30, 2024 at 12:36 PM Colin McCabe 
> wrote:
> > >
> > >> Thanks for checking, Luke. I did actually visit
> > >> https://repository.apache.org/#stagingRepositories and hit close on
> the
> > >> artifacts as the release.py script says to do. I don't know why they
> ended
> > >> up not showing up.
> > >>
> > >> Today I dropped all the maven artifacts for previous 3.9 RCs,
> switched to
> > >> java 8, and re-ran "gradlew publish" and "mvn deploy -Pgpg-signing".
> Then I
> > >> closed the two artifacts that were uploaded. You can see them here:
> > >>
> > >>
> https://repository.apache.org/content/repositories/orgapachekafka-1825/
> > >>
> https://repository.apache.org/content/repositories/orgapachekafka-1826/
> > >>
> > >> Hopefully these should now show up in Maven as expected.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Tue, Oct 29, 2024, at 23:51, Luke Chen wrote:
> > >> > Hi Colin,
> > >> >
> > >> > I was trying to test the RC, but I found the artifacts in Maven
> artifacts
> > >> > to be voted upon are not up-to-date.
> > >> > Not only the "Last Modified" time in the 3.9.0 artifacts are on
> Oct. 10,
> > >> > but also the source code in the artifacts didn't include the latest
> > >> commits
> > >> > in RC5 here <https://github.com/apache/kafka/commits/3.9.0-rc5/>.
> > >> >
> > >> > Could you help verify it?
> > >> >
> > >> > But I confirmed the release artifacts in "
> > >> > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/"; is
> up-to-date.
> > >> > I did:
> > >> > - checked the checksums and signatures
> > >> > - ran upgrade from v3.8.1 to v3.9.0
> > >> > - ran quickstart with the 2.13 binaries
> > >> > - verify tiered storage and KIP-950 (dynamically disablement)
> > >> >
> > >> > I'll be +1 a

Re: [VOTE] 3.9.0 RC5

[Luke Chen]

Hi Justine,

> As a side note, I was looking into how to configure kip-853 and it took me
a moment to find the instructions in the documentation, I wonder if it
could be included in the notable changes section of the upgrade notes. Not
a blocker as I believe we can still update documentation.

Good point! I've added a note into v3.9.0 notable changes section in my doc
improvement PR <https://github.com/apache/kafka/pull/17598>.

Thanks.
Luke


On Wed, Oct 30, 2024 at 2:51 PM Luke Chen  wrote:

> Hi Colin,
>
> I was trying to test the RC, but I found the artifacts in Maven artifacts
> to be voted upon are not up-to-date.
> Not only the "Last Modified" time in the 3.9.0 artifacts are on Oct. 10,
> but also the source code in the artifacts didn't include the latest commits
> in RC5 here <https://github.com/apache/kafka/commits/3.9.0-rc5/>.
>
> Could you help verify it?
>
> But I confirmed the release artifacts in "
> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/"; is up-to-date.
> I did:
> - checked the checksums and signatures
> - ran upgrade from v3.8.1 to v3.9.0
> - ran quickstart with the 2.13 binaries
> - verify tiered storage and KIP-950 (dynamically disablement)
>
> I'll be +1 after the maven artifacts are updated.
>
> Thank you for running the release.
> Luke
>
>
> On Wed, Oct 30, 2024 at 5:40 AM Justine Olshan
>  wrote:
>
>> Hey Colin,
>>
>> Thanks for the new RC. I've tested the --feature functionality that was
>> missing before. Seems to be working as expected now.
>> As a side note, I was looking into how to configure kip-853 and it took me
>> a moment to find the instructions in the documentation, I wonder if it
>> could be included in the notable changes section of the upgrade notes. Not
>> a blocker as I believe we can still update documentation.
>>
>> I did some other spot checks on the rest of the release. +1 (binding) from
>> me
>>
>> Justine
>>
>> On Tue, Oct 29, 2024 at 12:45 PM Colin McCabe  wrote:
>>
>> > Thanks, Anton. And thanks Chia-Ping Tsai for taking a look at how we can
>> > improve the docs here...
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Tue, Oct 29, 2024, at 02:39, Anton Agestam wrote:
>> > > Hi Chia-Ping,
>> > >
>> > > Thanks for pointing those two fields out. I retract my -1.
>> > >
>> > > Cheers,
>> > > Anton
>> > >
>> > > Den sön 27 okt. 2024 kl 17:40 skrev Chia-Ping Tsai <
>> chia7...@apache.org
>> > >:
>> > >
>> > >> hi Anton
>> > >>
>> > >> Thanks for sharing your insights on Kafka serialization—it’s really
>> cool
>> > >> and interesting to me. Additionally, you inspired me to file a JIRA
>> > issue
>> > >> (KAFKA-17882) to improve the documentation.
>> > >>
>> > >> The most important aspect of Kafka is compatibility, and the
>> > undocumented
>> > >> behavior has been in place for some time [0][1]. This means there’s
>> no
>> > need
>> > >> to rush your improvement for 3.9, as we’ll need to explicitly add
>> > default
>> > >> values after applying your patch to ensure we generate the same
>> binary
>> > data.
>> > >>
>> > >> In short, we can improve the documentation first. In the meantime, we
>> > can
>> > >> continue discussing behavior clarification for 4.0, and RM can keep
>> > running
>> > >> the RC for 3.9. Everything is on track.
>> > >>
>> > >> Best,
>> > >> Chia-Ping
>> > >>
>> > >> [0]
>> > >>
>> >
>> https://github.com/apache/kafka/blob/3.8/clients/src/main/resources/common/message/FetchSnapshotResponse.json#L43
>> > >> [1]
>> > >>
>> >
>> https://github.com/apache/kafka/blob/3.8/group-coordinator/src/main/resources/common/message/ConsumerGroupMemberMetadataValue.json#L39
>> > >>
>> > >> On 2024/10/27 15:28:05 Anton Agestam wrote:
>> > >> > -1, refer to comments on the RC 2 thread.
>> > >> >
>> > >> > Den sön 27 okt. 2024 kl 02:51 skrev Colin McCabe <
>> cmcc...@apache.org
>> > >:
>> > >> >
>> > >> > > This is the RC5 candidate for the release of Apache Kafka 3.9.0.
>> > >> > >
>> > >> > > - This is a ma

Re: [ANNOUNCE] Apache Kafka 3.8.1

[Luke Chen]

Hi Josep,

Thanks for the effort to run the release!

Luke

On Wed, Oct 30, 2024 at 2:01 AM 黃竣陽  wrote:

> Hello Josep
>
> Thanks for hard working on running this release!
>
> Best regards,
> Jiunn-Yang
>
> > TengYao Chi  於 2024年10月30日 凌晨12:35 寫道：
> >
> > Hi Josep
> > Thanks for running this release
> >
> > Cheers,
> > TengYao
> >
> > Apoorv Mittal  於 2024年10月30日 週三 上午12:18寫道：
> >
> >> Thanks Josep for all the hard work for the release.
> >>
> >> Regards,
> >> Apoorv Mittal
> >>
> >>
> >>
> >> On Tue, Oct 29, 2024 at 2:51 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >>> Thanks Josep for running this release!
> >>>
> >>> Mickael
> >>>
> >>> On Tue, Oct 29, 2024 at 3:23 PM Josep Prat  wrote:
> >>>>
> >>>> The Apache Kafka community is pleased to announce the release for
> >>>> Apache Kafka 3.8.1
> >>>>
> >>>> This is a bug fix release and it includes fixes and improvements.
> >>>>
> >>>> All of the changes in this release can be found in the release notes:
> >>>> https://www.apache.org/dist/kafka/3.8.1/RELEASE_NOTES.html
> >>>>
> >>>>
> >>>> An overview of the release can be found in our announcement blog post:
> >>>> https://kafka.apache.org/blog#apache_kafka_381_release_announcement
> >>>>
> >>>> You can download the source and binary release (Scala )
> from:
> >>>> https://kafka.apache.org/downloads#3.8.1
> >>>>
> >>>>
> >>>
> >>
> ---
> >>>>
> >>>>
> >>>> Apache Kafka is a distributed streaming platform with four core APIs:
> >>>>
> >>>>
> >>>> ** The Producer API allows an application to publish a stream of
> >> records
> >>> to
> >>>> one or more Kafka topics.
> >>>>
> >>>> ** The Consumer API allows an application to subscribe to one or more
> >>>> topics and process the stream of records produced to them.
> >>>>
> >>>> ** The Streams API allows an application to act as a stream processor,
> >>>> consuming an input stream from one or more topics and producing an
> >>>> output stream to one or more output topics, effectively transforming
> >> the
> >>>> input streams to output streams.
> >>>>
> >>>> ** The Connector API allows building and running reusable producers or
> >>>> consumers that connect Kafka topics to existing applications or data
> >>>> systems. For example, a connector to a relational database might
> >>>> capture every change to a table.
> >>>>
> >>>>
> >>>> With these APIs, Kafka can be used for two broad classes of
> >> application:
> >>>>
> >>>> ** Building real-time streaming data pipelines that reliably get data
> >>>> between systems or applications.
> >>>>
> >>>> ** Building real-time streaming applications that transform or react
> >>>> to the streams of data.
> >>>>
> >>>>
> >>>> Apache Kafka is in use at large and small companies worldwide,
> >> including
> >>>> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> >> Rabobank,
> >>>> Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >>>>
> >>>> A big thank you for the following 24 contributors to this release!
> >>>> (Please report an unintended omission)
> >>>>
> >>>> Andrew Schofield, Apoorv Mittal, Bill Bejeck, Bruno Cadonna, Chia-Ping
> >>>> Tsai, Chris Egerton, Colin P. McCabe, David Arthur, Guillaume Mallet,
> >>>> Igor Soarez, Josep Prat, Justine Olshan, Ken Huang, Kondrat Bertalan,
> >>>> Kuan-Po Tseng, Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael
> >>>> Maison, PoAn Yang, Rohan, TengYao Chi, Vikas Singh
> >>>>
> >>>> We welcome your help and feedback. For more information on how to
> >>>> report problems, and to get involved, visit the project website at
> >>>> https://kafka.apache.org/
> >>>>
> >>>> Thank you!
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Josep Prat
> >>>> Release Manager for Apache Kafka 3.8.1
> >>>
> >>
>
>

Re: [VOTE] 3.9.0 RC5

[Luke Chen]

Hi Colin,

I was trying to test the RC, but I found the artifacts in Maven artifacts
to be voted upon are not up-to-date.
Not only the "Last Modified" time in the 3.9.0 artifacts are on Oct. 10,
but also the source code in the artifacts didn't include the latest commits
in RC5 here .

Could you help verify it?

But I confirmed the release artifacts in "
https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/"; is up-to-date.
I did:
- checked the checksums and signatures
- ran upgrade from v3.8.1 to v3.9.0
- ran quickstart with the 2.13 binaries
- verify tiered storage and KIP-950 (dynamically disablement)

I'll be +1 after the maven artifacts are updated.

Thank you for running the release.
Luke


On Wed, Oct 30, 2024 at 5:40 AM Justine Olshan 
wrote:

> Hey Colin,
>
> Thanks for the new RC. I've tested the --feature functionality that was
> missing before. Seems to be working as expected now.
> As a side note, I was looking into how to configure kip-853 and it took me
> a moment to find the instructions in the documentation, I wonder if it
> could be included in the notable changes section of the upgrade notes. Not
> a blocker as I believe we can still update documentation.
>
> I did some other spot checks on the rest of the release. +1 (binding) from
> me
>
> Justine
>
> On Tue, Oct 29, 2024 at 12:45 PM Colin McCabe  wrote:
>
> > Thanks, Anton. And thanks Chia-Ping Tsai for taking a look at how we can
> > improve the docs here...
> >
> > best,
> > Colin
> >
> >
> > On Tue, Oct 29, 2024, at 02:39, Anton Agestam wrote:
> > > Hi Chia-Ping,
> > >
> > > Thanks for pointing those two fields out. I retract my -1.
> > >
> > > Cheers,
> > > Anton
> > >
> > > Den sön 27 okt. 2024 kl 17:40 skrev Chia-Ping Tsai <
> chia7...@apache.org
> > >:
> > >
> > >> hi Anton
> > >>
> > >> Thanks for sharing your insights on Kafka serialization—it’s really
> cool
> > >> and interesting to me. Additionally, you inspired me to file a JIRA
> > issue
> > >> (KAFKA-17882) to improve the documentation.
> > >>
> > >> The most important aspect of Kafka is compatibility, and the
> > undocumented
> > >> behavior has been in place for some time [0][1]. This means there’s no
> > need
> > >> to rush your improvement for 3.9, as we’ll need to explicitly add
> > default
> > >> values after applying your patch to ensure we generate the same binary
> > data.
> > >>
> > >> In short, we can improve the documentation first. In the meantime, we
> > can
> > >> continue discussing behavior clarification for 4.0, and RM can keep
> > running
> > >> the RC for 3.9. Everything is on track.
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> [0]
> > >>
> >
> https://github.com/apache/kafka/blob/3.8/clients/src/main/resources/common/message/FetchSnapshotResponse.json#L43
> > >> [1]
> > >>
> >
> https://github.com/apache/kafka/blob/3.8/group-coordinator/src/main/resources/common/message/ConsumerGroupMemberMetadataValue.json#L39
> > >>
> > >> On 2024/10/27 15:28:05 Anton Agestam wrote:
> > >> > -1, refer to comments on the RC 2 thread.
> > >> >
> > >> > Den sön 27 okt. 2024 kl 02:51 skrev Colin McCabe <
> cmcc...@apache.org
> > >:
> > >> >
> > >> > > This is the RC5 candidate for the release of Apache Kafka 3.9.0.
> > >> > >
> > >> > > - This is a major release, the final one in the 3.x line. (There
> > may of
> > >> > > course be other minor releases in this line, such as 3.9.1.)
> > >> > > - Tiered storage will be considered production-ready in this
> > release.
> > >> > > - This will be the final major release to feature the deprecated
> > >> ZooKeeper
> > >> > > mode.
> > >> > >
> > >> > > This release includes the following KIPs:
> > >> > > - KIP-853: Support dynamically changing KRaft controller
> membership
> > >> > > - KIP-1057: Add remote log metadata flag to the dump log tool
> > >> > > - KIP-1049: Add config log.summary.interval.ms to Kafka Streams
> > >> > > - KIP-1040: Improve handling of nullable values in InsertField,
> > >> > > ExtractField, and other transformations
> > >> > > - KIP-1031: Control offset translation in MirrorSourceConnector
> > >> > > - KIP-1033: Add Kafka Streams exception handler for exceptions
> > >> occurring
> > >> > > during processing
> > >> > > - KIP-1017: Health check endpoint for Kafka Connect
> > >> > > - KIP-1025: Optionally URL-encode clientID and clientSecret in
> > >> > > authorization header
> > >> > > - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> > >> > > - KIP-950: Tiered Storage Disablement
> > >> > > - KIP-956: Tiered Storage Quotas
> > >> > >
> > >> > > Release notes for the 3.9.0 release:
> > >> > >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc5/RELEASE_NOTES.html
> > >> > >
> > >> > > *** Please download, test and vote by October 30, 2024.
> > >> > >
> > >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > > https://kafka.apache.org/KEYS
> > >> > >
> > >> > > * Release artifacts to be voted up

Re: [VOTE] 3.8.1 RC1

[Luke Chen]

Hi Josep,

There is no such rule to explicitly exclude RM or not.
But it'd be better if there is the 3rd +1 binding other than RM to have one
more validation.
So, maybe you can give people 1 last day to validate it, then close the
vote with your +1.

Thanks.
Luke

On Mon, Oct 28, 2024 at 7:36 PM Josep Prat  wrote:

> Hi all,
>
> Thanks for all the ones testing and voting!
>
> @Luke Chen  and @Mickael Maison
> , would my implicit +1 count towards the 3 PMCs
> needed to approve the PR? Or as a RM am I excluded?
>
> Best,
>
> --
> Josep Prat
> Open Source Engineering Director, Aiven
> josep.p...@aiven.io   |   +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Mon, Oct 28, 2024, 10:52 Federico Valeri  wrote:
>
>> Hi Josep,
>>
>> - compiled from source
>> - ran all unit and integration tests
>> - ran ZK and KRaft quickstarts
>> - spot checked documentation
>>
>> +1 (non binding)
>>
>> Thanks
>> Fede
>>
>> On Mon, Oct 28, 2024 at 4:31 AM Luke Chen  wrote:
>> >
>> > Hi Josep,
>> >
>> > Thanks for the RC.
>> >
>> > I did the following
>> > - checked the checksums and signatures
>> > - ran upgrade from v3.7.1 to v3.8.1
>> > - ran quickstart with the 2.13 binaries
>> > - reviewed doc update PR.
>> >
>> > +1 (binding) from me.
>> >
>> > Thanks.
>> > Luke
>> >
>> > On Mon, Oct 28, 2024 at 8:16 AM 黃竣陽  wrote:
>> >
>> > > Hi Josep,
>> > >
>> > > +1(non-binding)
>> > >
>> > > I did the following
>> > > - Built from different JDK (8, 11, 17, 21) in Docker image
>> > > - Ran all the unit test and integration test in container
>> > >
>> > > Best regards,
>> > > Jiunn-Yang
>> > >
>> > > > TengYao Chi  於 2024年10月26日 凌晨12:22 寫道：
>> > > >
>> > > > Hi Josep,
>> > > >
>> > > > Thanks for handling the release of the 3.8.1 RC.
>> > > >
>> > > > I did the following:
>> > > > - Built from source using different LTS JDK versions (8 to 21) and
>> > > > distributions in Docker (Zulu and Corretto)
>> > > > - Ran all the unit tests in each container
>> > > >
>> > > > There were a few failures during the first run, but when I tested it
>> > > again,
>> > > > all tests passed successfully. So, I don't think it's an issue.
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > Thanks,
>> > > > TengYao
>> > > >
>> > > > Mickael Maison  於 2024年10月25日 週五
>> 下午8:21寫道：
>> > > >
>> > > >> Hi Josep,
>> > > >>
>> > > >> +1 (binding)
>> > > >>
>> > > >> I did the following
>> > > >> - checked the checksums and signatures
>> > > >> - ran quickstart with the 2.13 binaries, docker and docker-native
>> image
>> > > >> - built and ran tests from source
>> > > >>
>> > > >> Thanks,
>> > > >> Mickael
>> > > >>
>> > > >>
>> > > >> On Thu, Oct 24, 2024 at 8:03 PM Josep Prat
>> <https://www.google.com/maps/search/at+?entry=gmail&source=g>
>> > > > >
>> > > >> wrote:
>> > > >>>
>> > > >>> Hi all!
>> > > >>> Gentle ping on this thread.
>> > > >>>
>> > > >>> Thanks!
>> > > >>>
>> > > >>> --
>> > > >>> Josep Prat
>> > > >>> Open Source Engineering Director, Aiven
>> > > >>> josep.p...@aiven.io   |   +491715557497 | aiven.io
>> > > >>> Aiven Deutschland GmbH
>> > > >>> Alexanderufer 3-7, 10117 Berlin
>> > > >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
>> > > >>> Anna Richardson, Kenneth Chen
>> > > >>> Amtsgericht Charlottenburg, HRB 209739 B
>> > > >>>
>> > > >>> On Thu, Oct 17, 2024, 22:27 Josep Prat 
>> wrote:
>> > &

Re: [VOTE] 3.9.0 RC2

[Luke Chen]

Hi Anton,

Thanks for raising the discussion.
I read through this discussion and I can understand your confusion.

>  The behavior is new.
I think like Colin said, this behavior actually stays consistent as in the
tagged field in ApiVersionsResponse.json named SupportedFeatures. It
doesn't specify an explicit default, so it gets the implicit default of the
empty array if nothing is sent for this field. And you're right, the
SupportedFeatures field is an array field, which is different from what you
described here. However, the main point is the same: Any fields in the
message object that have no explicit default value, it will be reset to
default values, even if it's in the same version. And the default value of
the array field is an empty array, like the host is reset to an empty
string... etc. I think that's what Colin tried to express; this is not a
"new" behavior introduced in v3.9.0.

> The behavior is undocumented.
Yes, I completely agree this is not well documented, and thanks Chia-Ping
for filing KAFKA-17882 
for this improvement. Welcome to provide your comments there.

> The behavior is bad API design.
I think you can always open a KIP or simply start a discussion thread in
the dev mailing list to come up with a better design with the community.

> The behavior does not really save bytes *in practice*.
Again, welcome to open a KIP or simply start a discussion thread to discuss
with the community.


For the "new" behavior, I understand you said this is "new" because there
was no nested tagged entity field with non-explicit defaults in any API
schema version up until 3.9. But no matter how we define the "new"
behavior, the point here is, how we can move it forward. As you proposed:
1. defaults declared in the "UpdateRaftVoterResponse" schema:
Since the KIP had completed discussion and vote process. It requires more
discussion to change the behavior or created a new KIP. If you still think
the default value in CurrentLeader field is necessary, you can always open
a KIP to change it.
2. "nested tagged entity fields are only to be considered as having an
implicit default value if all of fields in the nested entity also have
explicit defaults" like the PR 
did:
I can see it won't break existing behavior, but obviously there are people
object to this change and it needs more discussion.

Given we already delay 3.9.0 release for almost 2 months, and this doesn't
break any functionality in Kafka, just some "not perfect design" issue, do
you agree we improve the documentation in v3.9.0 and come up with a new
proposal for API design enhancement or other enhancement after v3.9.0?

Thank you.
Luke

On Sun, Oct 27, 2024 at 4:45 PM Anton Agestam
 wrote:

> Colin
>
> I have presented four reasons, I'll list them again below. Please let me
> know which ones there isn't already enough information on the thread
> already.
>
> - The behavior is new.
> - The behavior is undocumented.
> - The behavior is bad API design.
> - The behavior does not really save bytes *in practice*.
>
> I don't see why *fixing* the release candidate to not break documented
> behavior should require a KIP, I would actually expect the opposite -- the
> new behavior that is being introduced should really have required one.
>
> > These two behaviors, taken together, save space on the wire
>
> Then you are implicitly arguing that the combination of host="" port=0
> are common enough that this will practically save bytes on the wire, I find
> that hard to believe.
>
> For any future schema that we want to save bytes, there is just as much
> opportunity to save bytes on the wire with my proposal, they just have to
> explicitly define default nested values in order to do so.
>
> BR,
> Anton
>
>
> Den sön 27 okt. 2024 kl 02:56 skrev Colin McCabe :
>
> > Hi Anton,
> >
> > The behavior where we leave out tagged fields when the default value is
> > present is intentional. As is the behavior where default values are
> treated
> > as 0, the empty array, etc. when defaults are not explicitly specified.
> > These two behaviors, taken together, save space on the wire, and are
> > simpler for us to implement than what you have proposed. You haven't
> > presented a single reason why this should change.
> >
> > There simply isn't any reason to change the current tagged fields
> > behavior. And even if we wanted to, it would require a KIP, whereas 3.9
> is
> > past KIP freeze (along with feature freeze, code freeze, and every other
> > freeze.)
> >
> > best,
> > Colin
> >
> >
> > On Sat, Oct 26, 2024, at 09:45, Anton Agestam wrote:
> > > Hi Colin,
> > >
> > >> The current behavior is older than 3.9
> > >
> > > No it is not. The *Java implementation* is older than 3.9. But the
> > behavior
> > > is a sum of Java implementation + JSON schema. It's a new construct of
> > JSON
> > > schema that exposes the new behavior. It is fully possible to adjust
> > > behavior only 

Re: [VOTE] 3.8.1 RC1

[Luke Chen]

Hi Josep,

Thanks for the RC.

I did the following
- checked the checksums and signatures
- ran upgrade from v3.7.1 to v3.8.1
- ran quickstart with the 2.13 binaries
- reviewed doc update PR.

+1 (binding) from me.

Thanks.
Luke

On Mon, Oct 28, 2024 at 8:16 AM 黃竣陽  wrote:

> Hi Josep,
>
> +1(non-binding)
>
> I did the following
> - Built from different JDK (8, 11, 17, 21) in Docker image
> - Ran all the unit test and integration test in container
>
> Best regards,
> Jiunn-Yang
>
> > TengYao Chi  於 2024年10月26日 凌晨12:22 寫道：
> >
> > Hi Josep,
> >
> > Thanks for handling the release of the 3.8.1 RC.
> >
> > I did the following:
> > - Built from source using different LTS JDK versions (8 to 21) and
> > distributions in Docker (Zulu and Corretto)
> > - Ran all the unit tests in each container
> >
> > There were a few failures during the first run, but when I tested it
> again,
> > all tests passed successfully. So, I don't think it's an issue.
> >
> > +1 (non-binding)
> >
> > Thanks,
> > TengYao
> >
> > Mickael Maison  於 2024年10月25日 週五 下午8:21寫道：
> >
> >> Hi Josep,
> >>
> >> +1 (binding)
> >>
> >> I did the following
> >> - checked the checksums and signatures
> >> - ran quickstart with the 2.13 binaries, docker and docker-native image
> >> - built and ran tests from source
> >>
> >> Thanks,
> >> Mickael
> >>
> >>
> >> On Thu, Oct 24, 2024 at 8:03 PM Josep Prat  >
> >> wrote:
> >>>
> >>> Hi all!
> >>> Gentle ping on this thread.
> >>>
> >>> Thanks!
> >>>
> >>> --
> >>> Josep Prat
> >>> Open Source Engineering Director, Aiven
> >>> josep.p...@aiven.io   |   +491715557497 | aiven.io
> >>> Aiven Deutschland GmbH
> >>> Alexanderufer 3-7, 10117 Berlin
> >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >>> Anna Richardson, Kenneth Chen
> >>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>
> >>> On Thu, Oct 17, 2024, 22:27 Josep Prat  wrote:
> >>>
>  Hello Kafka users, developers and client-developers,
> 
>  This is the second release candidate of Apache Kafka 3.8.1.
> 
>  This is a bugfix release with several fixes.
> 
>  Release notes for the 3.8.1 release:
> 
> >>
> https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/RELEASE_NOTES.html
> 
>   Please download, test and vote by Tuesday, October 22, 9am ET*
> 
>  Kafka's KEYS file containing PGP keys we use to sign the release:
>  https://kafka.apache.org/KEYS
> 
>  * Release artifacts to be voted upon (source and binary):
>  https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/
> 
>  * Docker release artifacts to be voted upon:
>  apache/kafka:3.8.1-rc1
>  apache/kafka-native:3.8.1-rc1
> 
>  * Maven artifacts to be voted upon:
> 
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> 
>  * Javadoc:
>  https://dist.apache.org/repos/dist/dev/kafka/3.8.1-rc1/javadoc/
> 
>  * Tag to be voted upon (off 3.8 branch) is the 3.8.1 tag:
>  https://github.com/apache/kafka/releases/tag/3.8.1-rc1
> 
>  * Documentation:
>  Mind that the home.apache.org server is retired now.
>  https://kafka.apache.org/38/documentation.html
>  And https://github.com/apache/kafka-site/pull/635
> 
>  * Protocol:
>  https://kafka.apache.org/38/protocol.html
>  And https://github.com/apache/kafka-site/pull/635
> 
>  * Jenkins builds for the 3.8 branch:
>  Unit/integration tests: There are some flaky tests, with the
> >> combination
>  of these 4 builds all tests passed at least once:
> 
> >>
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.8/103/testReport/
>  (latest build)
> 
> >>
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.8/101/testReport/,
> 
> >>
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.8/102/testReport/
>  and
> 
> >> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.8/97/testReport/
>  All tests pass locally
> 
>  System tests: Between these 2 runs all tests were successful:
> 
> 
> >>
> https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-10-07--001.af519a09-fdc8-4d46-8478-e0280854e43e--1728373295--confluentinc--3.8--7dbc44143a/report.html
> 
> 
> >>
> https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-10-01--001.e7b0a1be-bac1-4792-96da-ec94116e20ce--1727846843--confluentinc--3.8--99746d683a/report.html
> 
>  * Successful Docker Image Github Actions Pipeline for 3.8 branch:
>  Docker Build Test Pipeline (JVM):
>  https://github.com/apache/kafka/actions/runs/11390962530
>  Docker Build Test Pipeline (Native):
>  https://github.com/apache/kafka/actions/runs/11391548205
> 
>  /**
> 
>  Thanks,
> 
>  --
>  [image: Aiven] 
> 
>  *Josep Prat*
>  Open Source Engineering Director

Re: [VOTE] 3.9.0 RC3

[Luke Chen]

Hi Colin,

I opened a PR <https://github.com/apache/kafka/pull/17598> to add the
limitation section in the "Controller membership changes" chapter.
If there are other things needed to be added, please let me know.

But since it's only doc change, it won't block the RC voting.

Thanks.
Luke

On Fri, Oct 25, 2024 at 2:31 PM Luke Chen  wrote:

> Hi Colin,
>
> Thanks for the explanation. And I agree with Mickael that if there are
> such limitations, we should clearly document it.
> So far, in the v3.9 document
> <https://kafka.apache.org/39/documentation.html>, I can't see anything
> about it.
> Also, it seems metrics are still not added, we should document it, too.
>
>
> Thanks.
> Luke
>
>
> On Fri, Oct 25, 2024 at 3:20 AM Mickael Maison 
> wrote:
>
>> Hi Colin,
>>
>> This is a major caveat. We've advertised KRaft as production ready and
>> anyone who has already migrated is now stuck with a static quorum. The
>> whole point of 3.9.0 was to reach "feature parity" with ZooKeeper,
>> this is now true for only a subset of users (users that have not
>> already migrated, or created new KRaft clusters).
>>
>> I think at this point 3.9.0 has to go through. At the very least we
>> should make sure it's clearly explained, I didn't see anything about
>> this limitation in our docs or release notes.
>>
>> Anyway, thanks for the new RC, I plan to test it tomorrow.
>>
>> Thanks,
>> Mickael
>>
>>
>> On Thu, Oct 24, 2024 at 7:49 PM Colin McCabe  wrote:
>> >
>> > Hi Luke,
>> >
>> > Unfortunately, we don't have a way to upgrade from static quorums to
>> dynamic quorums yet. We are hoping to include this in the next release.
>> We're trakcing it under https://issues.apache.org/jira/browse/KAFKA-16538
>> .
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Thu, Oct 24, 2024, at 05:25, Luke Chen wrote:
>> > > Hi Colin,
>> > >
>> > > I found I cannot upgrade to v3.9.0 and then successfully add a new
>> > > controller or remove a controller. I've opened KAFKA-17866
>> > > <https://issues.apache.org/jira/browse/KAFKA-17866>. Please take a
>> look.
>> > >
>> > > Thanks.
>> > > Luke
>> > >
>> > > On Thu, Oct 24, 2024 at 2:27 PM TengYao Chi 
>> wrote:
>> > >
>> > >> Hi Colin,
>> > >>
>> > >> Thanks for handling the release of the new RC.
>> > >>
>> > >> I built from source and ran all the unit and integration tests on
>> each
>> > >> combination of different LTS JDK versions and distributions in
>> Docker (Zulu
>> > >> and Corretto).
>> > >>
>> > >> +1 (non-binding)
>> > >>
>> > >> Thanks,
>> > >> TengYao
>> > >>
>> > >> Justine Olshan  於 2024年10月24日 週四
>> 下午2:24寫道：
>> > >>
>> > >> > Hey Colin,
>> > >> >
>> > >> > Thanks for running the release.
>> > >> >
>> > >> > I checked the keys and scanned the docs
>> > >> > I built from source, used kraft quickstart, ran a transactions
>> workload,
>> > >> > played around with a few other things including running some of the
>> > >> feature
>> > >> > version commands.
>> > >> >
>> > >> > I noticed that some of the KIP-1022 work that was meant to make it
>> in
>> > >> this
>> > >> > release was removed.
>> > >> > Without any specifications, the cluster is meant to take the latest
>> > >> version
>> > >> > of all available features. The code to do that was removed.
>> Perhaps there
>> > >> > is a reason, but we should either change the KIP or remedy this.
>> > >> > Not only that, but the ability to configure a feature via the
>> --feature
>> > >> > flag in the storage tool also seems to be removed but the text
>> remains
>> > >> when
>> > >> > running with no arguments. I don't think it is a blocker, but this
>> should
>> > >> > be fixed for 4.0.
>> > >> >
>> > >> > I looked at two runs of the system tests and they looked fine to
>> me.
>> > >> >
>> > >> > +1 (bin

Re: [VOTE] 3.9.0 RC3

[Luke Chen]

Hi Colin,

Thanks for the explanation. And I agree with Mickael that if there are such
limitations, we should clearly document it.
So far, in the v3.9 document
<https://kafka.apache.org/39/documentation.html>, I can't see anything
about it.
Also, it seems metrics are still not added, we should document it, too.


Thanks.
Luke


On Fri, Oct 25, 2024 at 3:20 AM Mickael Maison 
wrote:

> Hi Colin,
>
> This is a major caveat. We've advertised KRaft as production ready and
> anyone who has already migrated is now stuck with a static quorum. The
> whole point of 3.9.0 was to reach "feature parity" with ZooKeeper,
> this is now true for only a subset of users (users that have not
> already migrated, or created new KRaft clusters).
>
> I think at this point 3.9.0 has to go through. At the very least we
> should make sure it's clearly explained, I didn't see anything about
> this limitation in our docs or release notes.
>
> Anyway, thanks for the new RC, I plan to test it tomorrow.
>
> Thanks,
> Mickael
>
>
> On Thu, Oct 24, 2024 at 7:49 PM Colin McCabe  wrote:
> >
> > Hi Luke,
> >
> > Unfortunately, we don't have a way to upgrade from static quorums to
> dynamic quorums yet. We are hoping to include this in the next release.
> We're trakcing it under https://issues.apache.org/jira/browse/KAFKA-16538
> .
> >
> > best,
> > Colin
> >
> >
> > On Thu, Oct 24, 2024, at 05:25, Luke Chen wrote:
> > > Hi Colin,
> > >
> > > I found I cannot upgrade to v3.9.0 and then successfully add a new
> > > controller or remove a controller. I've opened KAFKA-17866
> > > <https://issues.apache.org/jira/browse/KAFKA-17866>. Please take a
> look.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Thu, Oct 24, 2024 at 2:27 PM TengYao Chi 
> wrote:
> > >
> > >> Hi Colin,
> > >>
> > >> Thanks for handling the release of the new RC.
> > >>
> > >> I built from source and ran all the unit and integration tests on each
> > >> combination of different LTS JDK versions and distributions in Docker
> (Zulu
> > >> and Corretto).
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Thanks,
> > >> TengYao
> > >>
> > >> Justine Olshan  於 2024年10月24日 週四
> 下午2:24寫道：
> > >>
> > >> > Hey Colin,
> > >> >
> > >> > Thanks for running the release.
> > >> >
> > >> > I checked the keys and scanned the docs
> > >> > I built from source, used kraft quickstart, ran a transactions
> workload,
> > >> > played around with a few other things including running some of the
> > >> feature
> > >> > version commands.
> > >> >
> > >> > I noticed that some of the KIP-1022 work that was meant to make it
> in
> > >> this
> > >> > release was removed.
> > >> > Without any specifications, the cluster is meant to take the latest
> > >> version
> > >> > of all available features. The code to do that was removed. Perhaps
> there
> > >> > is a reason, but we should either change the KIP or remedy this.
> > >> > Not only that, but the ability to configure a feature via the
> --feature
> > >> > flag in the storage tool also seems to be removed but the text
> remains
> > >> when
> > >> > running with no arguments. I don't think it is a blocker, but this
> should
> > >> > be fixed for 4.0.
> > >> >
> > >> > I looked at two runs of the system tests and they looked fine to me.
> > >> >
> > >> > +1 (binding) from me,
> > >> >
> > >> > Justine
> > >> >
> > >> > On Tue, Oct 22, 2024 at 1:21 PM Bill Bejeck 
> wrote:
> > >> >
> > >> > > Hi Colin,
> > >> > >
> > >> > > Thanks for running the release.
> > >> > >
> > >> > > I did the following verification steps:
> > >> > >
> > >> > >1. Built from source
> > >> > >2. Validated all checksums
> > >> > >3. Ran all the unit tests
> > >> > >4. Pulled both docker images and started up Kafka
> > >> > >5. Ran the KRaft quick start
> > >> > >6. Ran the ZK quick start
> > >> > > 

Re: [VOTE] 3.9.0 RC3

[Luke Chen]

Hi Colin,

I found I cannot upgrade to v3.9.0 and then successfully add a new
controller or remove a controller. I've opened KAFKA-17866
. Please take a look.

Thanks.
Luke

On Thu, Oct 24, 2024 at 2:27 PM TengYao Chi  wrote:

> Hi Colin,
>
> Thanks for handling the release of the new RC.
>
> I built from source and ran all the unit and integration tests on each
> combination of different LTS JDK versions and distributions in Docker (Zulu
> and Corretto).
>
> +1 (non-binding)
>
> Thanks,
> TengYao
>
> Justine Olshan  於 2024年10月24日 週四 下午2:24寫道：
>
> > Hey Colin,
> >
> > Thanks for running the release.
> >
> > I checked the keys and scanned the docs
> > I built from source, used kraft quickstart, ran a transactions workload,
> > played around with a few other things including running some of the
> feature
> > version commands.
> >
> > I noticed that some of the KIP-1022 work that was meant to make it in
> this
> > release was removed.
> > Without any specifications, the cluster is meant to take the latest
> version
> > of all available features. The code to do that was removed. Perhaps there
> > is a reason, but we should either change the KIP or remedy this.
> > Not only that, but the ability to configure a feature via the --feature
> > flag in the storage tool also seems to be removed but the text remains
> when
> > running with no arguments. I don't think it is a blocker, but this should
> > be fixed for 4.0.
> >
> > I looked at two runs of the system tests and they looked fine to me.
> >
> > +1 (binding) from me,
> >
> > Justine
> >
> > On Tue, Oct 22, 2024 at 1:21 PM Bill Bejeck  wrote:
> >
> > > Hi Colin,
> > >
> > > Thanks for running the release.
> > >
> > > I did the following verification steps:
> > >
> > >1. Built from source
> > >2. Validated all checksums
> > >3. Ran all the unit tests
> > >4. Pulled both docker images and started up Kafka
> > >5. Ran the KRaft quick start
> > >6. Ran the ZK quick start
> > >7. Ran the Kafka Streams quick start
> > >8. Spot checked the documentation
> > >9. Spot checked the javadocs
> > >
> > > +1(binding)
> > >
> > > Thanks,
> > > Bill
> > >
> > >
> > >
> > > On Tue, Oct 22, 2024 at 9:08 AM Federico Valeri 
> > > wrote:
> > >
> > > > Hi, I did the following tests:
> > > >
> > > > - Ran all unit and integration tests
> > > > - Spot checked the documentation
> > > > - Tried out new Maven dependencies
> > > > - Tested KIP-853, KIP-950, KIP-956 and KIP-1057
> > > >
> > > > +1 (non binding)
> > > >
> > > > Thanks
> > > > Fede
> > > >
> > > >
> > > > On Mon, Oct 21, 2024 at 11:12 PM Colin McCabe 
> > > wrote:
> > > > >
> > > > > This is the third candidate for the release of Apache Kafka 3.9.0.
> I
> > > > have titled it rc3 since I had an rc1 which got very far, even to the
> > > point
> > > > of pushing tags and docker images, before I spotted an issue. So
> rather
> > > > than mutate the tags, I decided to skip over rc1.
> > > > >
> > > > > - This is a major release, the final one in the 3.x line. (There
> may
> > of
> > > > course be other minor releases in this line, such as 3.9.1.)
> > > > > - Tiered storage will be considered production-ready in this
> release.
> > > > > - This will be the final major release to feature the deprecated
> > > > ZooKeeper mode.
> > > > >
> > > > > This release includes the following KIPs:
> > > > > - KIP-853: Support dynamically changing KRaft controller membership
> > > > > - KIP-1057: Add remote log metadata flag to the dump log tool
> > > > > - KIP-1049: Add config log.summary.interval.ms to Kafka Streams
> > > > > - KIP-1040: Improve handling of nullable values in InsertField,
> > > > ExtractField, and other transformations
> > > > > - KIP-1031: Control offset translation in MirrorSourceConnector
> > > > > - KIP-1033: Add Kafka Streams exception handler for exceptions
> > > occurring
> > > > during processing
> > > > > - KIP-1017: Health check endpoint for Kafka Connect
> > > > > - KIP-1025: Optionally URL-encode clientID and clientSecret in
> > > > authorization header
> > > > > - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> > > > > - KIP-950: Tiered Storage Disablement
> > > > > - KIP-956: Tiered Storage Quotas
> > > > >
> > > > > Release notes for the 3.9.0 release:
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc3/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by October 24, 2024.
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://dist.apache.org/repos/dist/dev/kafka/3.9.0-rc3/
> > > > >
> > > > > * Docker release artifacts to be voted upon:
> > > > > apache/kafka:3.9.0-rc3
> > > > > apache/kafka-native:3.9.0-rc3
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://reposito

[jira] [Created] (KAFKA-17866) kraft.version is not updated after upgrading to MV 3.9

[Luke Chen (Jira)]

Luke Chen created KAFKA-17866:
-

 Summary: kraft.version is not updated after upgrading to MV 3.9
 Key: KAFKA-17866
 URL: https://issues.apache.org/jira/browse/KAFKA-17866
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.9.0
Reporter: Luke Chen


When upgrading old controllers to v3.9.0, we didn't update the kraft.version to 
1, which causes it blocks the following controller addition/removal.

 

Steps to reproduce:
1. format 1 controller node in MV 3.8:
{code:java}
 bin/kafka-storage.sh format -t r0NrSlvGTqmZCtFtZ8tBIg -c 
config/kraft/controller1.properties --release-version 3.8 
{code}
2. start up the controller
3. update the metadata version to 3.9:
{code:java}
bin/kafka-features.sh --bootstrap-controller localhost:9093 upgrade --metadata 
3.9

or

bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature 
kraft.version=1
{code}
4. format one additional controller to be expanded
{code:java}
bin/kafka-storage.sh format --cluster-id r0NrSlvGTqmZCtFtZ8tBIg --config 
config/kraft/controller2.properties --no-initial-controllers
{code}
5. Try to add this node
{code:java}
bin/kafka-metadata-quorum.sh --command-config 
config/kraft/controller2.properties --bootstrap-controller localhost:9093 
add-controller
{code}
It'll fail with:
{code:java}
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't 
support adding voter because the kraft.version feature is 0
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.UnsupportedVersionException: Cluster doesn't 
support adding voter because the kraft.version feature is 0
at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at 
org.apache.kafka.tools.MetadataQuorumCommand.handleAddController(MetadataQuorumCommand.java:431)
at 
org.apache.kafka.tools.MetadataQuorumCommand.execute(MetadataQuorumCommand.java:147)
at 
org.apache.kafka.tools.MetadataQuorumCommand.mainNoExit(MetadataQuorumCommand.java:81)
at 
org.apache.kafka.tools.MetadataQuorumCommand.main(MetadataQuorumCommand.java:76)
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException: Cluster 
doesn't support adding voter because the kraft.version feature is 0
{code}
 

Under this situation, I can't find any workaround to update kraft.version even 
if rebooting the node.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 4.0.0 release

[Luke Chen]

Hi David,

The release plan looks good to me.

But since the 4.0 release is a release without ZK, I'm wondering if we
should list some release criteria for it?
The Problem I can imagine is like, when the code freeze date is reached,
but there are still many ZK removal tasks open, what should we do with
them? We can remove the rest of codes in later releases, but if there are
still some public APIs related to ZK open for users to use, is that still
OK to release 4.0? Ex: The `--zookeeper` option in kafka-configs.sh?

Instead of only using time-based management in v4.0, I'm suggesting we
should list some "must-do" tasks and track them when milestone date
approaches. And we can discuss with the community if there are some delayed
tasks. Does that make sense?

Thanks.
Luke



On Mon, Oct 7, 2024 at 9:29 PM David Jacot 
wrote:

> Hi all,
>
> I have drafted the release plan for Apache Kafka 4.0.0:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+4.0.0.
> Please take a look and let me know what you think.
>
> I would also appreciate it if you could review the list of KIPs that we
> will ship in this release. I took the list from the approved list in the
> wiki but I am pretty sure that the list is not accurate.
>
> Best,
> Davud
>
> On Wed, Sep 25, 2024 at 5:02 AM Luke Chen  wrote:
>
> > +1 from me.
> > Since v4.0 will be a huge change release, I'm not sure if you need
> another
> > person (vice release manager?) to help on releasing tasks.
> > If so, I'm willing to help.
> >
> > Thanks.
> > Luke
> >
> > On Wed, Sep 25, 2024 at 1:33 AM Colin McCabe  wrote:
> >
> > > +1. Thanks, David.
> > >
> > > best,
> > > Colin
> > >
> > > On Mon, Sep 23, 2024, at 10:11, Chris Egerton wrote:
> > > > Thanks David! +1
> > > >
> > > > On Mon, Sep 23, 2024, 13:07 José Armando García Sancio
> > > >  wrote:
> > > >
> > > >> +1, thanks for volunteering David!
> > > >>
> > > >> On Mon, Sep 23, 2024 at 11:56 AM David Arthur 
> > wrote:
> > > >> >
> > > >> > +1, thanks David!
> > > >> >
> > > >> > On Mon, Sep 23, 2024 at 11:48 AM Satish Duggana <
> > > >> satish.dugg...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > +1
> > > >> > > Thanks David for volunteering!
> > > >> > >
> > > >> > > On Mon, 23 Sept 2024 at 19:27, Lianet M. 
> > > wrote:
> > > >> > > >
> > > >> > > > +1
> > > >> > > >
> > > >> > > > Thanks David!
> > > >> > > >
> > > >> > > > On Mon, Sep 23, 2024, 9:54 a.m. Justine Olshan
> > > >> > > 
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > +1 and thanks!
> > > >> > > > >
> > > >> > > > > On Mon, Sep 23, 2024 at 6:36 AM Chia-Ping Tsai <
> > > chia7...@gmail.com
> > > >> >
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > +1 and thanks David for volunteering!
> > > >> > > > > >
> > > >> > > > > > Best,
> > > >> > > > > > Chia-Ping
> > > >> > > > > >
> > > >> > > > > > Josep Prat  於 2024年9月23日 週一
> > > >> 下午9:13寫道：
> > > >> > > > > >
> > > >> > > > > > > Thanks David for volunteering!
> > > >> > > > > > > +1
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Sep 23, 2024 at 3:11 PM David Jacot
> > > >> > > > >  > > >> > > > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi all,
> > > >> > > > > > > >
> > > >> > > > > > > > I would like to volunteer to be the release manager
> > > driving
> > > >> the
> > > >> > > next
> > > >> > > > > > > > release - Apache Kafka 4.0.0.
> > > >> > > > > > > >
> > > >> > > > > > > > If there are no objections, I'll start building a
> > release
> > > >> plan
> > > >> > > in the
> > > >> > > > > > > wiki
> > > >> > > > > > > > in the next couple of days.
> > > >> > > > > > > >
> > > >> > > > > > > > Best,
> > > >> > > > > > > > David
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > --
> > > >> > > > > > > [image: Aiven] <https://www.aiven.io>
> > > >> > > > > > >
> > > >> > > > > > > *Josep Prat*
> > > >> > > > > > > Open Source Engineering Director, *Aiven*
> > > >> > > > > > > josep.p...@aiven.io   |   +491715557497
> > > >> > > > > > > aiven.io <https://www.aiven.io>   |   <
> > > >> > > > > > https://www.facebook.com/aivencloud
> > > >> > > > > > > >
> > > >> > > > > > >   <https://www.linkedin.com/company/aiven/>   <
> > > >> > > > > > > https://twitter.com/aiven_io>
> > > >> > > > > > > *Aiven Deutschland GmbH*
> > > >> > > > > > > Alexanderufer 3-7, 10117 Berlin
> > > >> > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > >> > > > > > > Anna Richardson, Kenneth Chen
> > > >> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > David Arthur
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> -José
> > > >>
> > >
> >
>

Re: [DISCUSS] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Luke Chen]

Hi Kamal,

Sorry for the late review.
Thanks for the KIP, this will improve the transaction reading for remote
storage.
Overall LGTM, just one minor thought:

Currently, we only store the `TxnIndexEmpty` bool value in the segment
metadata.
Do you think it is helpful if we store the "least abort start offset in the
segment", and -1 means no txnIndex. So that we can have a way to know if we
need to fetch this txn index or not.

Thanks.
Luke

On Mon, Sep 9, 2024 at 3:26 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi all,
>
> If there are no more comments, I'll start a voting thread soon.
>
> Thanks,
> Kamal
>
> On Fri, Sep 6, 2024 at 7:28 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Bumping this thread again for review!
> >
> > Reduced the scope of the proposal to minimum. We will be adding only one
> > field (txnIdxEmpty) to the
> > RemoteLogSegmentMetadata event which is backward compatible. PTAL.
> >
> > Thanks,
> > Kamal
> >
> >
> > On Tue, Aug 13, 2024 at 6:33 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> >> Bumping this thread for KIP review!
> >>
> >> We can go for the simplest solution that is proposed in this KIP and
> >> it can be improved in the subsequent iteration. PTAL.
> >>
> >> Thanks,
> >> Kamal
> >>
> >> On Fri, Aug 2, 2024 at 11:42 AM Kamal Chandraprakash <
> >> kamal.chandraprak...@gmail.com> wrote:
> >>
> >>> Hi Divij,
> >>>
> >>> Thanks for the review! And, sorry for the late reply.
> >>>
> >>> From the UnifiedLog.scala
> >>> <
> https://sourcegraph.com/github.com/apache/kafka@trunk/-/blob/core/src/main/scala/kafka/log/UnifiedLog.scala?L421-427
> >
> >>> doc:
> >>>
> >>> """
> >>> The last stable offset (LSO) is defined as the first offset such that
> >>> all lower offsets have been "decided."
> >>>* Non-transactional messages are considered decided immediately, but
> >>> transactional messages are only decided when
> >>>* the corresponding COMMIT or ABORT marker is written. This implies
> >>> that the last stable offset will be equal
> >>>* to the high watermark if there are no transactional messages in
> the
> >>> log. Note also that the LSO cannot advance
> >>>* beyond the high watermark.
> >>> """
> >>> While rolling the active segment to passive, if LSO equals to HW, then
> >>> all the messages in that segment are
> >>> decided and we can store the `lastStableOffsetLag` as an attribute in
> >>> the rolled segment. We can then propagate
> >>> the `lastStableOffsetLag` information in the RemoteLogMetadata events.
> >>>
> >>> While reading the remote log segment, if the `lastStableOffsetLag` is
> 0,
> >>> then there is no need to traverse to
> >>> the subsequent segments for aborted transactions which covers the case
> >>> for the dominant case where the
> >>> partition had no transactions at all.
> >>>
> >>> With Log compaction, the shrinked segments might get merged. One option
> >>> is to take the max of `lastStableOffsetLag`
> >>> and store it in the new LogSegment. Since, the tiered storage does not
> >>> support compacted topics / historical compacted
> >>> topics, we can omit this case.
> >>>
> >>> If this approach looks good, I can update the KIP with the details.
> >>>
> >>> --
> >>> Kamal
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Jun 25, 2024 at 4:24 PM Divij Vaidya 
> >>> wrote:
> >>>
>  Hi Kamal
> 
>  Thanks for the bump. I have been thinking about this passively for the
>  past
>  few days.
> 
>  The simplest solution is to store a state at segment level metadata.
> The
>  state should specify whether the trx index is empty or not. It would
> be
>  populated during segment archival. We would then iterate over the
>  metadata
>  for future segments without having to make a remote call to download
> the
>  trx index itself.
> 
>  The other solution for storing state at a partition level wouldn't
>  work, as
>  you mentioned, because we will have to change the state on every
>  mutation
>  to the log i.e. at expiration of segments and append.
> 
>  I have been thinking whether we can do something better than the
> simple
>  solution, hence the delay in replying. Let me tell you my half baked
>  train
>  of thoughts, perhaps, you can explore this as well. I have been
> thinking
>  about using LSO (last stable offset) to handle the case when the
>  partition
>  never had any transactions. For a partition which never had any
>  transaction, I would assume that the LSO is never initialized (or is
>  equal
>  to log start offset)? Or is it equal to HW in that case? This is
>  something
>  that I am yet to verify. If this idea works, then we would not have to
>  iterate through the metadata for the dominant case where the partition
>  had
>  no transactions at all.
> 
>  --
>  Divij Vaidya
> 
> 
> 
>  On Tue, Jun 25, 2024 at 11:4

[ANNOUNCE] New committer: Kamal Chandraprakash

[Luke Chen]

Hi all,

The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal
Chandraprakash.

Kamal has been a Kafka contributor since May 2017. He has made significant
contributions to the tiered storage feature (KIP-405). He authored KIP-1018
and KIP-1075 which improved tiered storage operation. He also contributed
to discussing and reviewing many KIPs.

Congratulations, Kamal!

Thanks,
Luke (on behalf of the Apache Kafka PMC)

[jira] [Resolved] (KAFKA-17584) Fix incorrect synonym handling for dynamic log configurations

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17584.
---
Fix Version/s: 3.9.0
   3.8.1
   Resolution: Fixed

> Fix incorrect synonym handling for dynamic log configurations
> -
>
> Key: KAFKA-17584
> URL: https://issues.apache.org/jira/browse/KAFKA-17584
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.9.0
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Blocker
> Fix For: 3.9.0, 3.8.1
>
>
> Updating certain dynamic configurations (for example `message.max.bytes`) 
> causes retention based on time to reset to the default value (source code) 
> for log.retention.ms. This poses a durability issue if users have set their 
> retention by using log.retention.hours or log.retention.minutes. In other 
> words, if a user has set log.retention.hours=-1 (infinite retention) and they 
> dynamically change `message.max.bytes` their retention will immediately 
> change back to the default of 60480 ms (7 days) and data before this will 
> be scheduled for deletion immediately.
> Steps to reproduce:
>  1. Add log.retention.minutes=1,log.retention.check.interval.ms=1000 to 
> server.properties
>  2. Start a single ZK or KRaft instance + a single Kafka instance
>  3. Create a topic using
> {code:java}
> bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic A 
> --replication-factor 1 --partitions 1 --config min.insync.replicas=1 --config 
> segment.bytes=512{code}
>  4. Create a few segments with the console producer
>  5. Observe that they are deleted after 1 minute
>  6. Use the following command
> {code:java}
> bin/kafka-configs.sh --bootstrap-server loclahost:9092 --entity-type brokers 
> --entity-default --alter --add-config message.max.bytes=1048609{code}
> (the value of `message.max.bytes` is irrelevant)
>  7. Create a few more segments with the console producer
>  8. Observe that segments are no longer deleted after 1 minute



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: So long, Jenkins 👋

[Luke Chen]

Cool! Thanks for the effort, David!

Luke


On Thu, Sep 26, 2024 at 1:26 PM Josep Prat 
wrote:

> Awesome David!
>
> Best,
> --
> Josep Prat
> Open Source Engineering Director, Aiven
> josep.p...@aiven.io   |   +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Thu, Sep 26, 2024, 04:36 Kirk True  wrote:
>
> > This is amazing work David! You leveled up every contributor and
> committer
> > on the project.
> >
> > Thanks
> >
> > > On Sep 25, 2024, at 6:27 PM, Chris Egerton 
> > wrote:
> > >
> > > David, you're a legend. Thanks for spearheading this effort!
> > >
> > > On Wed, Sep 25, 2024, 20:51 David Arthur  wrote:
> > >
> > >> Today, we disabled the Jenkins build on trunk. With this change, we
> > should
> > >> now be expecting all green status checks on PRs before merging. Of
> > course,
> > >> flaky tests still exist, but generally speaking we should have green
> > builds
> > >> (see KIP-1090 for some plans on flaky tests).
> > >>
> > >> Any committer or "collaborator" (as defined in .asf.yaml) is able to
> > >> manually re-run a GitHub Action via the UI.
> > >>
> > >> For non-committers, someone must approve the workflow. There is a
> > >> "approve-workflows.py" script in committer-tools to help with this.
> I'm
> > >> still investigating options to improve this.
> > >>
> > >> We will keep the Jenkins build enabled for 3.9 and other release
> > branches.
> > >>
> > >> Cheers,
> > >> David A
> > >>
> >
> >
>

Re: [DISCUSS] bug fix to allow dynamically change some log configs

[Luke Chen]

Hi Colin,

Thanks for the update.
Sounds good to me!

Luke

On Wed, Sep 25, 2024 at 1:47 AM Colin McCabe  wrote:

> Thanks for posting this here, Luke. Thinking about this more, I'll just go
> back to the old behavior where the synonyms are ignored if you set them as
> dynamic configurations. I will fix the problem where static synonyms are
> ignored sometimes, though, since that seems more "surprising" and bad.
>
> Maybe we should have removed those synonyms in 4.0, but I guess that ship
> has sailed. Perhaps 5.0 :)
>
> best,
> Colin
>
>
> On Tue, Sep 24, 2024, at 02:52, Luke Chen wrote:
> > Hi all,
> >
> > KAFKA-17584 <https://issues.apache.org/jira/browse/KAFKA-17584> is
> about a
> > bug related to the incorrect synonym handling for dynamic log
> > configurations. While there is already a PR
> > <https://github.com/apache/kafka/pull/17258> to fix the issue, we found
> > there are some log configs that should be "cluster-wide" configs, but due
> > to the bug in KAFKA-17584
> > <https://issues.apache.org/jira/browse/KAFKA-17584>, we incorrectly
> treat
> > them as "read-only" configs, since v3.0.0 (or even earlier). This issue
> is
> > that we only treat 1 config as "cluster-wide" in the same synonym group.
> > The list is as follows:
> >
> > Configs is now "read-only" but should be "cluster-wide" (i.e. dynamically
> > changeable):
> > - log.retention.hours
> >   log.retention.minutes
> >   (log.retention.ms is already a "cluster-wide" config)
> > - log.roll.hours
> >   (log.roll.ms is already a "cluster-wide" config)
> > - log.roll.jitter.hours
> >   (log.roll.jitter.ms is already a "cluster-wide" config)
> > - log.flush.scheduler.interval.ms
> >   (log.flush.interval.ms is already a "cluster-wide" config)
> >
> > Note that, in all the groups, there is 1 config being treated as a
> > "cluster-wide" config. So, it should not block any operation.
> > After this fix, all the configs above will become "cluster-wide" configs.
> > Although these are public interface changes, I think it is more like a
> bug
> > fix. But if people think we should go through the KIP process to have
> > proper discussion, I can open one for it. But for v3.9.0, I think we
> should
> > include this fix to avoid any possible unexpected data loss as described
> in
> > KAFKA-17584 <https://issues.apache.org/jira/browse/KAFKA-17584>, even if
> > KIP is required and not accepted.
> >
> > Please let me know what you think.
> >
> > Thank you.
> > Luke
>

Re: [VOTE] KIP-1073: Return fenced brokers in DescribeCluster response

[Luke Chen]

Hi Tina,

Thanks for the KIP.
+1 (binding) from me.

Luke


On Mon, Sep 23, 2024 at 9:03 PM Gantigmaa Selenge 
wrote:

> Hi everyone,
>
> I would like to start voting for KIP-1073 that extends DescribeCluster API
> to optionally return fenced brokers in the response.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1073%3A+Return+fenced+brokers+in+DescribeCluster+response
>
> Thanks.
> Gantigmaa
>

Re: [DISCUSS] Apache Kafka 4.0.0 release

[Luke Chen]

+1 from me.
Since v4.0 will be a huge change release, I'm not sure if you need another
person (vice release manager?) to help on releasing tasks.
If so, I'm willing to help.

Thanks.
Luke

On Wed, Sep 25, 2024 at 1:33 AM Colin McCabe  wrote:

> +1. Thanks, David.
>
> best,
> Colin
>
> On Mon, Sep 23, 2024, at 10:11, Chris Egerton wrote:
> > Thanks David! +1
> >
> > On Mon, Sep 23, 2024, 13:07 José Armando García Sancio
> >  wrote:
> >
> >> +1, thanks for volunteering David!
> >>
> >> On Mon, Sep 23, 2024 at 11:56 AM David Arthur  wrote:
> >> >
> >> > +1, thanks David!
> >> >
> >> > On Mon, Sep 23, 2024 at 11:48 AM Satish Duggana <
> >> satish.dugg...@gmail.com>
> >> > wrote:
> >> >
> >> > > +1
> >> > > Thanks David for volunteering!
> >> > >
> >> > > On Mon, 23 Sept 2024 at 19:27, Lianet M. 
> wrote:
> >> > > >
> >> > > > +1
> >> > > >
> >> > > > Thanks David!
> >> > > >
> >> > > > On Mon, Sep 23, 2024, 9:54 a.m. Justine Olshan
> >> > > 
> >> > > > wrote:
> >> > > >
> >> > > > > +1 and thanks!
> >> > > > >
> >> > > > > On Mon, Sep 23, 2024 at 6:36 AM Chia-Ping Tsai <
> chia7...@gmail.com
> >> >
> >> > > wrote:
> >> > > > >
> >> > > > > > +1 and thanks David for volunteering!
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Chia-Ping
> >> > > > > >
> >> > > > > > Josep Prat  於 2024年9月23日 週一
> >> 下午9:13寫道：
> >> > > > > >
> >> > > > > > > Thanks David for volunteering!
> >> > > > > > > +1
> >> > > > > > >
> >> > > > > > > On Mon, Sep 23, 2024 at 3:11 PM David Jacot
> >> > > > >  >> > > > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi all,
> >> > > > > > > >
> >> > > > > > > > I would like to volunteer to be the release manager
> driving
> >> the
> >> > > next
> >> > > > > > > > release - Apache Kafka 4.0.0.
> >> > > > > > > >
> >> > > > > > > > If there are no objections, I'll start building a release
> >> plan
> >> > > in the
> >> > > > > > > wiki
> >> > > > > > > > in the next couple of days.
> >> > > > > > > >
> >> > > > > > > > Best,
> >> > > > > > > > David
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > [image: Aiven] 
> >> > > > > > >
> >> > > > > > > *Josep Prat*
> >> > > > > > > Open Source Engineering Director, *Aiven*
> >> > > > > > > josep.p...@aiven.io   |   +491715557497
> >> > > > > > > aiven.io    |   <
> >> > > > > > https://www.facebook.com/aivencloud
> >> > > > > > > >
> >> > > > > > >      <
> >> > > > > > > https://twitter.com/aiven_io>
> >> > > > > > > *Aiven Deutschland GmbH*
> >> > > > > > > Alexanderufer 3-7, 10117 Berlin
> >> > > > > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> >> > > > > > > Anna Richardson, Kenneth Chen
> >> > > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > David Arthur
> >>
> >>
> >>
> >> --
> >> -José
> >>
>

[DISCUSS] bug fix to allow dynamically change some log configs

[Luke Chen]

Hi all,

KAFKA-17584  is about a
bug related to the incorrect synonym handling for dynamic log
configurations. While there is already a PR
 to fix the issue, we found
there are some log configs that should be "cluster-wide" configs, but due
to the bug in KAFKA-17584
, we incorrectly treat
them as "read-only" configs, since v3.0.0 (or even earlier). This issue is
that we only treat 1 config as "cluster-wide" in the same synonym group.
The list is as follows:

Configs is now "read-only" but should be "cluster-wide" (i.e. dynamically
changeable):
- log.retention.hours
  log.retention.minutes
  (log.retention.ms is already a "cluster-wide" config)
- log.roll.hours
  (log.roll.ms is already a "cluster-wide" config)
- log.roll.jitter.hours
  (log.roll.jitter.ms is already a "cluster-wide" config)
- log.flush.scheduler.interval.ms
  (log.flush.interval.ms is already a "cluster-wide" config)

Note that, in all the groups, there is 1 config being treated as a
"cluster-wide" config. So, it should not block any operation.
After this fix, all the configs above will become "cluster-wide" configs.
Although these are public interface changes, I think it is more like a bug
fix. But if people think we should go through the KIP process to have
proper discussion, I can open one for it. But for v3.9.0, I think we should
include this fix to avoid any possible unexpected data loss as described in
KAFKA-17584 , even if
KIP is required and not accepted.

Please let me know what you think.

Thank you.
Luke

Re: [DISCUSS] Apache Kafka 3.8.1 release

[Luke Chen]

Thanks Josep!
+1

Luke

On Fri, Sep 20, 2024 at 3:36 PM Josep Prat 
wrote:

> Hey folks,
>
> I'd like to volunteer to be the release manager for a bug fix release of
> the 3.8 series. This will be the first bug fix release and will be version
> 3.8.1.
>
> If no one has any objections, I will send out a release plan on 2024/09/23
> that includes a list of all of the fixes we are targeting for 3.8.1 along
> with a timeline (aiming probably for a release happening at the beginning
> of October).
>
> Thanks!
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>

[jira] [Created] (KAFKA-17583) kafka-config script cannot set `cleanup.policy=delete,compact`

[Luke Chen (Jira)]

Luke Chen created KAFKA-17583:
-

 Summary: kafka-config script cannot set 
`cleanup.policy=delete,compact`
 Key: KAFKA-17583
 URL: https://issues.apache.org/jira/browse/KAFKA-17583
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.8.0, 3.5.0
Reporter: Luke Chen


Kafka-config.sh cannot set configs with "," in the value, because we split the 
option value with "," 
[here|https://github.com/apache/kafka/blob/3783385dc1cc27246cf09ec791e4b43f577a26ea/core/src/main/scala/kafka/admin/ConfigCommand.scala#L300].

{code:java}
> bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type topics 
> --entity-name t1 --alter --add-config cleanup.policy=delete,compact

requirement failed: Invalid entity config: all configs to be added must be in 
the format "key=val".
{code}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1021: Allow to get last stable offset (LSO) in kafka-get-offsets.sh

[Luke Chen]

Hi Ahmed,

Are you still interested in this KIP?
Should we take it over if you're busy on other things?

Thanks.
Luke


On Wed, Mar 27, 2024 at 12:26 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Ahmed,
>
> Thanks for the KIP! It will be a useful addition to know the
> last-stable-offset (LSO)
> for a partition. Motivation section is not clear to me. Can you document
> the
> scenarios on how the exposed LSO can be used for txn support/debug?
>
> --
> Kamal
>
> On Thu, Mar 21, 2024 at 11:30 PM Justine Olshan
>  wrote:
>
> > Hey Ahmed,
> >
> > Echoing Andrew's comments to clean up the public interfaces, it was
> unclear
> > to me if this new flag applies to all the options or just the "latest"
> > option.
> > Clarifying that and showing a few examples could help.
> >
> > Thanks,
> > Justine
> >
> > On Thu, Mar 21, 2024 at 9:43 AM Andrew Schofield <
> > andrew_schofield_j...@outlook.com> wrote:
> >
> > > Hi,
> > > Glad to hear you’re better.
> > >
> > > Re-reading the KIP, I think the changes to the public interfaces are
> > > probably only the addition
> > > of the new `--isolation` flag on the `kafka-get-offsets.sh` tool. The
> > > `KafkaAdminClient.listOffsets`
> > > parameters already have what you need I think (OffsetSpec.LATEST and
> > > ListOffsetOptions.isolationLevel).
> > >
> > > If you tidy up the `Public Interfaces` section with the revised syntax
> > for
> > > the tool, I think the
> > > committers will be able to see more easily what the changes are and it
> > > would then be ready to vote
> > > in my opinion.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 20 Mar 2024, at 12:56, Ahmed Sobeh 
> > > wrote:
> > > >
> > > > Hi Andrew,
> > > >
> > > > Apologies for disappearing, I had to undergo knee surgery, all good
> > now!
> > > >
> > > > I adjusted the KIP as you suggested, do you think it's ready for the
> > > voting
> > > > stage?
> > > >
> > > > On Wed, Mar 6, 2024 at 4:02 PM Andrew Schofield <
> > > > andrew_schofield_j...@outlook.com> wrote:
> > > >
> > > >> Hi Ahmed,
> > > >> Looks good to me with one remaining comment.
> > > >>
> > > >> You’ve used:
> > > >>
> > > >> bin/kafka-get-offsets.sh  … --time -1 --isolation -committed
> > > >> bin/kafka-get-offsets.sh  … --time latest --isolation -committed
> > > >> bin/kafka-get-offsets.sh  … --time -1 --isolation -uncommitted
> > > >> bin/kafka-get-offsets.sh  … --time latest --isolation -uncommitted
> > > >>
> > > >> I find the flags starting with a single - to be a bit unusual and
> > > doesn’t
> > > >> match the --time options such as “latest”. I suggest:
> > > >>
> > > >> bin/kafka-get-offsets.sh  … --time -1 --isolation committed
> > > >> bin/kafka-get-offsets.sh  … --time latest --isolation committed
> > > >> bin/kafka-get-offsets.sh  … --time -1 --isolation uncommitted
> > > >> bin/kafka-get-offsets.sh  … --time latest --isolation uncommitted
> > > >>
> > > >> Or even:
> > > >>
> > > >> bin/kafka-get-offsets.sh  … --time -1 --isolation read-committed
> > > >> bin/kafka-get-offsets.sh  … --time latest --isolation read-committed
> > > >> bin/kafka-get-offsets.sh  … --time -1 --isolation read-uncommitted
> > > >> bin/kafka-get-offsets.sh  … --time latest --isolation
> read-uncommitted
> > > >>
> > > >> Apart from that nit, looks good to me.
> > > >>
> > > >> Thanks,
> > > >> Andrew
> > > >>
> > > >>
> > > >>> On 5 Mar 2024, at 16:35, Ahmed Sobeh  >
> > > >> wrote:
> > > >>>
> > > >>> I adjusted the KIP according to what we agreed on, let me know if
> you
> > > >> have
> > > >>> any comments!
> > > >>>
> > > >>> Best,
> > > >>> Ahmed
> > > >>>
> > > >>> On Thu, Feb 29, 2024 at 1:44 AM Luke Chen 
> wrote:
> > > >>>
> > > >>>> Hi Ahmed,
> > > >>>>
> > > >>>&g

Re: [VOTE] 3.9.0 RC0

[Luke Chen]

Same here, no 3.9.0 artifacts in the staging repo.

Thanks.
Luke


On Wed, Sep 18, 2024 at 4:49 PM Jakub Scholz  wrote:

> Hi Colin,
>
> Thanks for the RC.
>
> It looks like the Maven artifacts are missing for 3.9.0 in the staging
> repo. Are we still waiting for some staging job to complete? Or is that
> some CDN issue and I'm the only one not seeing them? Or is that some
> general problem?
>
> Thanks & Regards
> Jakub
>
> On Wed, Sep 18, 2024 at 5:07 AM Colin McCabe  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> >
> >
> > This is the first candidate for release of Apache Kafka 3.9.0.
> >
> >
> > - This is a major release, the final one in the 3.x line. (There may of
> > course be other minor releases in this line, such as 3.9.1.)
> > - Tiered storage will be considered production-ready in this release.
> > - This will be the final major release to feature the deprecated
> ZooKeeper
> > mode.
> >
> > This release includes the following KIPs:
> > - KIP-853: Support dynamically changing KRaft controller membership
> > - KIP-1057: Add remote log metadata flag to the dump log tool
> > - KIP-1049: Add config log.summary.interval.ms to Kafka Streams
> > - KIP-1040: Improve handling of nullable values in InsertField,
> > ExtractField, and other transformations
> > - KIP-1031: Control offset translation in MirrorSourceConnector
> > - KIP-1033: Add Kafka Streams exception handler for exceptions occurring
> > during processing
> > - KIP-1017: Health check endpoint for Kafka Connect
> > - KIP-1025: Optionally URL-encode clientID and clientSecret in
> > authorization header
> > - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> > - KIP-950: Tiered Storage Disablement
> > - KIP-956: Tiered Storage Quotas
> >
> > Release notes for the 3.9.0 release:
> > https://home.apache.org/~cmccabe/kafka-3.9.0-rc0/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by September 23, 2024.
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~cmccabe/kafka-3.9.0-rc0/
> >
> > * Docker release artifacts to be voted upon:
> > apache/kafka:3.9.0-rc0
> > apache/kafka-native:3.9.0-rc0
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~cmccabe/kafka-3.9.0-rc0/javadoc/
> >
> > * Documentation:
> > https://kafka.apache.org/39/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/39/protocol.html
> >
> > * Tag to be voted upon (off 3.9 branch) is the 3.9.0-rc0 tag:
> > https://github.com/apache/kafka/releases/tag/3.9.0-rc0
> >
> > I've run several successful junit and ducktape builds. While there were
> > some flaky tests, when re-running them, they passed.
> >
> > * Successful Docker Image Github Actions Pipeline for 3.8 branch:
> > Docker Build Test Pipeline (JVM):
> > https://github.com/apache/kafka/actions/runs/10912118987
> > Docker Build Test Pipeline (Native):
> > https://github.com/apache/kafka/actions/runs/10914303515
> >
> > Thanks to everyone who helped with this release candidate, either by
> > contributing code, testing, or documentation.
> >
> > Regards,
> > Colin
> >
>

[jira] [Resolved] (KAFKA-17508) Adding some guard for fallback deletion logic

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17508.
---
Resolution: Fixed

> Adding some guard for fallback deletion logic
> -
>
> Key: KAFKA-17508
> URL: https://issues.apache.org/jira/browse/KAFKA-17508
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 3.8.0
>Reporter: Luke Chen
>Assignee: PoAn Yang
>Priority: Major
> Fix For: 4.0.0
>
>
> In KAFKA-16424, we added a fallback logic to delete the logs, but the file 
> has no parent. It'd be better we have some guard from it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Build Updates for week of Sep 9, 2024

[Luke Chen]

Thanks for the improvement, David!
The "De-flaking Integration Tests" looks great!
I'll use it next time when handling flaky tests.

Thanks.
Luke

On Fri, Sep 13, 2024 at 8:22 AM Chia-Ping Tsai  wrote:

> Thanks for David to bring this great improvement to Kafka CI!!!
>
> Best regards,
> Chia-Ping
>
> Josep Prat  於 2024年9月13日 週五 上午2:29寫道：
>
> > Thanks for the great summary David!
> > GH CI looks indeed really good.
> >
> > Best,
> > --
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > Anna Richardson, Kenneth Chen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Thu, Sep 12, 2024, 20:17 David Arthur  wrote:
> >
> > > A lot has been happening with the GitHub Actions build in the past few
> > > weeks. I thought I would share some updates.
> > >
> > > *Build Statistics*
> > > Now that we have all PRs builds running the test suite (see note
> below),
> > we
> > > can do a better comparison between GH and Jenkins
> > >
> > > Github Actions
> > > Successful trunk builds (1):
> > > 1h56m 5%
> > > 1h58m avg
> > > 2h1m 95%
> > >
> > > Github Actions
> > > Successful PR builds:
> > > 1h14m 5%
> > > 1h35m avg
> > > 1h59m 95%
> > >
> > > Jenkins
> > > Successful trunk builds:
> > > 1h27m 5%
> > > 4h7m avg
> > > 5h36m 95%
> > >
> > > Jenkins
> > > Successful PR builds:
> > > 1h22m 5%
> > > 3h48m avg
> > > 5h35m 95%
> > >
> > > It's pretty clear that the GitHub Actions build is significantly more
> > > stable than Jenkins and actually faster on average despite running on
> > > slower hardware.
> > >
> > > 1) We are seeing timeouts occasionally on GH due to a test getting
> stuck.
> > > We have narrowed it down to one test class.
> > >
> > > *Enabling GitHub Actions by default*
> > > In https://github.com/apache/kafka/pull/17105 we turned on the full
> "CI"
> > > workflow by default for PRs. This has been running now for a few days
> and
> > > so far we are well under the quota limit for GH Action Runner usage.
> > >
> > > *Green trunk Builds*
> > > Most of our trunk commits have had green builds on GH Actions and
> > Jenkins.
> > > This has been the result of a lot of focused effort on fixing flaky
> > tests,
> > > which is great to see!
> > >
> > > On Jenkins, we are continuing to see very erratic build times
> presumably
> > > due to resource contention. On Github, our trunk build times are much
> > more
> > > consistent (presumably due to better isolation).
> > >
> > > *Gradle Build Cache*
> > > Pull Requests now can take advantage of the Gradle Build Cache. The way
> > > this works is that trunk will write to a cache managed by GitHub
> Actions
> > > and PRs will read from it. In theory, if a PR only changes some code in
> > > ":streams", none of the ":core" tests will be run (and vica-versa).
> > >
> > > Here is an example PR build that cut its testing time by around 1hr
> > > https://ge.apache.org/s/dj2svkxx2edno/timeline.
> > >
> > > In practice, we are still seeing a lot of cache misses since the cache
> > will
> > > slightly lag behind trunk. Stay tuned for improvements to this...
> > >
> > > *Gradle Build Scans*
> > > We are now able to publish Gradle Build Scans for PRs from public
> forks.
> > > This is very exciting as it will allow contributors (not just
> > committers!)
> > > to gain insights into their builds and have very nice looking test
> > reports.
> > >
> > > Another improvement here is that the build scan links will be included
> in
> > > the PR "Checks". This is much easier to navigate to than finding it in
> > the
> > > workflow run.
> > >
> > > *De-flaking Integration Tests*
> > > A new "deflake" action was added to our GH Actions. It can be used to
> > > repeatedly run a @ClusterTest in the CI environment. I wrote up some
> > > instructions in a doc on our wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606545#FlakyTests-GitHub
> > > "deflake"Action
> > >
> > > *Closing old PRs*
> > > We have finished KAFKA-15073. Our "stale" workflow will now actually
> > close
> > > PRs that are inactive for more than 120 days.
> > >
> > >
> > > Cheers,
> > > David A
> > >
> >
>

[jira] [Created] (KAFKA-17508) Adding some guard for fallback deletion logic

[Luke Chen (Jira)]

Luke Chen created KAFKA-17508:
-

 Summary: Adding some guard for fallback deletion logic
 Key: KAFKA-17508
 URL: https://issues.apache.org/jira/browse/KAFKA-17508
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 3.8.0
Reporter: Luke Chen
Assignee: PoAn Yang


In KAFKA-16424, we added a fallback logic to delete the logs, but the has no 
parent. It'd be better we have some guard from it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: New release branch 3.9

[Luke Chen]

Hi Colin,

I've made KAFKA-17412  (
PR ) as v3.9.0 blocker for
documenting the unclean leader election behavior change in KRaft mode.
This is just a document change, so I think it should be fine.

Let me know if you have any concerns.

Thanks.
Luke

On Sat, Aug 31, 2024 at 2:53 AM Chia-Ping Tsai  wrote:

> hi Colin
>
> I open the https://issues.apache.org/jira/browse/KAFKA-17454 to be a
> blocker for 3.9 since the e2e transactions_mixed_versions_test fail on 3.9
> definitely
>
> We will fix it tomorrow.
>
> Best,
> Chia-Ping
>
> On 2024/08/30 15:38:48 Colin McCabe wrote:
> > No worries. Thanks for checking.
> >
> > best,
> > Colin
> >
> > On Thu, Aug 29, 2024, at 19:00, Kirk True wrote:
> > > Thanks Chia-Ping!
> > >
> > > Sorry for the noise, everyone else :)
> > >
> > >> On Aug 29, 2024, at 6:36 PM, Chia-Ping Tsai 
> wrote:
> > >>
> > >> hi Kirk
> > >>
> > >> I have updated the fix versions of KAFKA-17335 to include 3.9
> > >>
> > >> https://issues.apache.org/jira/browse/KAFKA-17335
> > >>
> > >> Best,
> > >> Chia-Ping
> > >>
> > >> Chia-Ping Tsai  於 2024年8月30日 週五 上午9:32寫道：
> > >>
> > >>> hi Kirk
> > >>>
> > >>> KAFKA-17335 is already in 3.9 (
> > >>>
> https://github.com/apache/kafka/commit/a87b501a47a55a1f9038b6c52e31b71590c3a80a
> > >>> )
> > >>>
> > >>> Best,
> > >>> Chia-Ping
> > >>>
> > >>> Kirk True  於 2024年8月30日 週五 上午4:36寫道：
> > >>>
> >  Hi Colin,
> > 
> >  KAFKA-17335 (Lack of default for URL encoding configuration for
> OAuth
> >  causes NPE) affects the internal use of the OAuth-based
> >  AuthenticateCallbackHandler. Can we back port the fix (
> >  https://github.com/apache/kafka/pull/16990) to the 3.9 branch?
> > 
> >  Thanks,
> >  Kirk
> > 
> > > On Aug 29, 2024, at 11:29 AM, Colin McCabe 
> wrote:
> > >
> > > Hi all,
> > >
> > > Code freeze has officially arrived for Apache Kafka 3.9! If you
> have
> >  additional fixes to get into the branch at this point, please ping
> me first
> >  (unless it's already marked as a blocker in JIRA).
> > >
> > > I did a first pass on the list of open JIRAs for 3.9, and moved
> out any
> >  that were not blockers for 3.9. In a few cases, I asked a question
> about
> >  whether the JIRA was a blocker or not.
> > >
> > > If all goes according to plan, we should now have two more weeks of
> >  stabilization before the release.
> > >
> > > Thanks, everyone.
> > >
> > > Colin
> > >
> > >
> > > On Fri, Aug 23, 2024, at 13:30, Colin McCabe wrote:
> > >> I wanted to give a quick update on 3.9. Code freeze is coming up
> next
> > >> Thursday on the 29th. As usual, after that point, only critical
> bug
> > >> fixes will be accepted.
> > >>
> > >> Thanks again to everyone who has helped with this release.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Fri, Aug 23, 2024, at 13:18, Colin McCabe wrote:
> > >>> Thanks for the heads up, Josep. I saw both emails but I didn't
> make
> >  the
> > >>> connection until now. It does seem likely that we'll have to
> modify
> > >>> this (either for 3.9... or soon.)
> > >>>
> > >>> best,
> > >>> Colin
> > >>>
> > >>> On Tue, Aug 13, 2024, at 05:09, Josep Prat wrote:
> >  Hi Colin,
> > 
> >  Just to make sure you (as a Release Manager) are aware of the
> >  change. ASF
> >  Infra just communicated the removal of the home.apache.org
> machines
> >  (in 4
> >  weeks they will be gone). This means the release.py script
> wouldn't
> >  work if
> >  executed after that date and would need to be modified to store
> RC
> >  artifacts under dist.apache.org/dev instead.
> > 
> >  Best,
> > 
> >  On Fri, Aug 9, 2024 at 11:38 PM Colin McCabe <
> cmcc...@apache.org>
> >  wrote:
> > 
> > > Updates:
> > >
> > > KIP-1007: I have removed this PR from the Apache Kafka 3.9
> release
> >  plan.
> > > Since the KIP was discarded, I did not add it in the
> "postponed to
> > > subsequent release" section.
> > >
> > > KIP-1025: Marked 3.9 status as Done
> > >
> > > KIP-1023: Postponed to subsequent release
> > >
> > > KIP-1005: Marked 3.9 status as Done
> > >
> > > KIP-996: Postponed to subsequent release
> > >
> > > KIP-994: Postponed to subsequent release
> > >
> > > KIP-997: Postponed to subsequent release
> > >
> > > KIP-966: Postponed to subsequent release
> > >
> > > KIP-956: Postponed to subsequent release
> > >
> > > KIP-950: Marked 3.9 status as Done
> > >
> > > best,
> > > Colin
> > >
> > > On Fri, Aug 9, 202

[jira] [Resolved] (KAFKA-17062) RemoteLogManager - RemoteStorageException causes data loss

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17062.
---
Resolution: Fixed

> RemoteLogManager - RemoteStorageException causes data loss
> --
>
> Key: KAFKA-17062
> URL: https://issues.apache.org/jira/browse/KAFKA-17062
> Project: Kafka
>  Issue Type: Bug
>  Components: Tiered-Storage
>Affects Versions: 3.8.0, 3.7.1, 3.9.0
>Reporter: Guillaume Mallet
>Assignee: Luke Chen
>Priority: Critical
>  Labels: tiered-storage
> Fix For: 3.9.0
>
>
> When Tiered Storage is configured, retention.bytes defines the limit for the 
> amount of data stored in the filesystem and in remote storage. However a 
> failure while offloading to remote storage can cause segments to be dropped 
> before the retention limit is met.
> What happens
> Assuming a topic configured with {{retention.bytes=4294967296}} (4GB) and a 
> {{local.retention.bytes=1073741824}} (1GB, equal to segment.bytes) we would 
> expect Kafka to keep up to 3 segments (3GB) in the remote store and 1 segment 
> locally (the local segment) and possibly more if the remote storage is 
> offline. i.e. segments in the following RemoteLogSegmentStates in the 
> RemoteLogMetadataManager (RLMM) :
>  * Segment 3 ({{{}COPY_SEGMENT_FINISHED{}}})
>  * Segment 2 ({{{}COPY_SEGMENT_FINISHED{}}})
>  * Segment 1 ({{{}COPY_SEGMENT_FINISHED{}}})
> Let's assume the RLMM starts failing when segment 4 rolls. At the first 
> iteration of an RLMTask we will have -
>  * 
> [{{copyLogSegmentsToRemote}}|https://github.com/apache/kafka/blob/d0dfefbe6394276eb329b6ca998842a984add506/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L773]
>  : is called first
>  ** RLMM becomes aware of Segment 4 and adds it to the metadata:
>  *** Segment 4 ({{{}COPY_SEGMENT_STARTED{}}}),
>  *** Segment 3 ({{{}COPY_SEGMENT_FINISHED{}}}),
>  *** Segment 2 ({{{}COPY_SEGMENT_FINISHED{}}}),
>  *** Segment 1 ({{{}COPY_SEGMENT_FINISHED{}}})
>  ** An exception is raised during the copy operation 
> ([{{copyLogSegmentData}}|https://github.com/apache/kafka/blob/d0dfefbe6394276eb329b6ca998842a984add506/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteStorageManager.java#L93]
>  in RemoteStorageManager) which is caught with the error message “{{Error 
> occurred while copying log segments of partition}}” and no further copy will 
> be attempted for the duration of this RLMTask.
>  ** At that point the Segment will never move to {{COPY_SEGMENT_FINISHED}} 
> but will transition to {{DELETE_SEGMENT_STARTED}} eventually before being 
> cleaned up when the associated segment is deleted.
>  * 
> [{{cleanupExpiredRemoteLogSegments}}|https://github.com/apache/kafka/blob/d0dfefbe6394276eb329b6ca998842a984add506/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1122]
>  is then called
>  ** Retention size is computed in 
> [{{buildRetentionSizeData}}|https://github.com/apache/kafka/blob/d0dfefbe6394276eb329b6ca998842a984add506/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1296]
>  as the sum of all the segments size regardless of their state so computed 
> size of the topic is 1 (local) + 4 (remote)
>  ** Segment 1 as being the oldest will be dropped.
> At the second iteration after 
> [{{remote.log.manager.task.interval.ms}}|https://github.com/apache/kafka/blob/d0dfefbe6394276eb329b6ca998842a984add506/storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java#L395]
>  (default: 30s), the same will happen. The RLMM will now have 2 x Segment 4 
> in a {{COPY_SEGMENT_STARTED}} state each with a different 
> {{RemoteLogSegmentId}} and Segment 2 will be dropped. The same will happen to 
> Segment 3 after another iteration.
> At that point, we now have the RLMM composed of 4 copies of Segment 4 in 
> {{COPY_SEGMENT_STARTED}} state. Segment 4 is marked for deletion increasing 
> the LSO at the same time and causing the UnifiedLog to delete the local and 
> remote data for Segment 4 including its metadata.
> Under those circumstances Kafka can quickly delete segments that were not 
> meant for deletion causing a data loss.
> Steps to reproduce the problem:
> 1. Enable tiered storage
> {code:bash}
> mkdir -p /tmp/tieredStorage/kafka-tiered-storage/
> cat <> config/kraft/server.properties
> remote.log.storage.system.enable=True
> remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage
> remote.log.manager.task.interval.ms=5000
> remote.log.metadata.

Re: [DISCUSS] KIP-1085: Fix leaking *_DOC variables in StreamsConfig

[Luke Chen]

Hi Jiunn,

Thanks for the KIP.

In the motivation section:
"The overly large scope of these variables can cause development issues."

I can't see why a "doc description variable" will cause any development
issue.
Could you explain more clearly what problem we have now?

Thanks.
Luke

On Thu, Aug 29, 2024 at 9:43 AM 黃竣陽  wrote:

> Hello everyone,
>
> I would like to start a discussion about KIP-1085
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1085%3A+Fix+leaking+*_DOC+variables+in+StreamsConfig
> >
> In this KIP, we plan to deprecate the public doc variables which only used
> in `StreamConfig`
> and make them private in a future release.
>
> Any feedback and suggestions for the KIP are welcome in this email thread.
>
> Thank you!
> Best regards,
> Jiunn Yang

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

[Luke Chen]

Hi Dima,

Before v3.9.0, brokers in KRaft mode didn't honor
`unclean.leader.election.enable` config.
So, the static config and dynamic config didn't work.
The only thing works before v3.9.0 is using he admin API or cli to run
`kafka-leader-election.sh` with `unclean` option.

Thanks.
Luke

On Thu, Aug 29, 2024 at 10:39 AM Dima Brodsky  wrote:

> Thanks Luke,
>
> I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example.
>
> ttyl
> Dima
>
>
> On Wed, Aug 28, 2024 at 7:36 PM Luke Chen  wrote:
>
> > Hi Dima,
> >
> > > I assume that unclean leader election exists as a static config in
> kafka
> > kraft, but is still unavailable dynamically?  Or is it unavailable in
> > general?
> >
> > Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` in
> > KRaft will work, either static config or dynamic config.
> > It's just the behavior changes when dynamically enabling the
> > `unclean.leader.election.enable`, which will not trigger unclean leader
> > election immediately after config change. The detailed behavior change is
> > as described above.
> >
> > Let me know if it's still not clear.
> >
> > Thanks.
> > Luke
> >
> >
> > On Thu, Aug 29, 2024 at 1:58 AM Dima Brodsky 
> wrote:
> >
> > > Hi Luke,
> > >
> > > I assume that unclean leader election exists as a static config in
> kafka
> > > kraft, but is still unavailable dynamically?  Or is it unavailable in
> > > general?
> > >
> > > Thanks!
> > > ttyl
> > > Dima
> > >
> > >
> > > On Wed, Aug 28, 2024 at 3:45 AM Luke Chen  wrote:
> > >
> > > > Hi all,
> > > >
> > > > In v3.9.0, we're planning to support the
> > `unclean.leader.election.enable`
> > > > config in KRaft. Currently, the implementation PR
> > > > <https://github.com/apache/kafka/pull/16866> is under review now.
> But
> > > I'd
> > > > like to raise the discussion to let you know there will be one
> behavior
> > > > change when enabling `unclean.leader.election.enable` config
> > dynamically
> > > in
> > > > KRaft.
> > > >
> > > > scenario:
> > > > 1. broker [0, 1] config are all set
> > > `unclean.leader.election.enable=false`.
> > > > 2. topic A is created with 1 partition, 2 replication factors and
> > > > `unclean.leader.election.enable=false`. Leader of the partition 0 is
> 0.
> > > ISR
> > > > is [0].
> > > > 3. broker 0 is down, which leaves ISR to empty [ ], and leader none.
> > > > 4. dynamically alter config using admin API to set
> > > > `unclean.leader.election.enable=true`, no matter it changes to the
> > topic
> > > A,
> > > > or to broker 1, or default broker.
> > > >
> > > > In ZK mode, with this scenario, it'll trigger unclean leader election
> > > > immediately, and broker 1 will be elected as the leader, which brings
> > the
> > > > availability of the partition back.
> > > > In KRaft mode, with this scenario, the unclean leader election will
> be
> > > > triggered in an unclean election thread, which is scheduled every 5
> > mins
> > > by
> > > > default. That is, this partition might still be unavailable even if
> it
> > > > enables `unclean.leader.election.enable`. If users want to elect a
> > leader
> > > > for this partition, run `kafka-leader-election.sh` with `unclean`
> > option
> > > to
> > > > trigger the unclean leader election immediately.
> > > >
> > > > The main reason we didn't implement what ZK did is because of this
> > > > complication
> > > > <https://github.com/apache/kafka/pull/16284#discussion_r1635410206>
> we
> > > > faced.
> > > >
> > > > Other behaviors keep the same. Ex: When the topic config, or broker
> > > config,
> > > > or default broker config set `unclean.leader.election.enable=true`,
> and
> > > > later, a partition has leader down with empty ISR [ ], and having
> some
> > > > alive non-ISR replicas, in this case, the unclean leader election
> will
> > be
> > > > triggered, as long as the config was set earlier.
> > > >
> > > > But since the PR is still under review, any comments or thoughts are
> > > > welcomed.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > >
> > >
> > > --
> > > ddbrod...@gmail.com
> > >
> > > "The price of reliability is the pursuit of the utmost simplicity.
> > > It is a price which the very rich find the most hard to pay."
> > >(Sir
> > > Antony Hoare, 1980)
> > >
> >
>
>
> --
> ddbrod...@gmail.com
>
> "The price of reliability is the pursuit of the utmost simplicity.
> It is a price which the very rich find the most hard to pay."
>(Sir
> Antony Hoare, 1980)
>

Re: [ANNOUNCE] New committer: Lianet Magrans

[Luke Chen]

Congratulations Lianet!

Luke

On Thu, Aug 29, 2024 at 9:54 AM Lianet M.  wrote:

> Thank you very much everyone! It truly takes the great shared knowledge you
> all put out there with amazing reviews and discussions.
>
> Looking forward to continuing working with you all!
>
> Lianet
>
> On Wed, Aug 28, 2024, 7:12 p.m. Sophie Blee-Goldman  >
> wrote:
>
> > Congrats Lianet! Well deserved
> >
> > On Wed, Aug 28, 2024 at 2:27 PM Jiashen Zhang 
> > wrote:
> >
> > > Congratulations Lianet! Awesome!
> > >
> > > On Wed, Aug 28, 2024 at 2:10 PM Andrew Schofield <
> > > andrew_schofi...@live.com>
> > > wrote:
> > >
> > > > Awesome news. Well done, Lianet.
> > > >
> > > > Andrew
> > > >
> > > > > On 28 Aug 2024, at 17:47, Bruno Cadonna 
> wrote:
> > > > >
> > > > > Well deserved!
> > > > >
> > > > > Best,
> > > > > Bruno
> > > > >
> > > > > On 8/28/24 6:37 PM, Bill Bejeck wrote:
> > > > >> Congrats Lianet!
> > > > >> On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax <
> mj...@apache.org>
> > > > wrote:
> > > > >>> Congrats! Very well deserved!
> > > > >>>
> > > > >>> On 8/28/24 9:18 AM, Justine Olshan wrote:
> > > >  Congratulations Lianet!!
> > > >  Justine
> > > > 
> > > >  On Wed, Aug 28, 2024 at 8:58 AM David Arthur 
> > > > wrote:
> > > > 
> > > > > Congrats, Lianet!
> > > > >
> > > > > On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison <
> > > > >>> mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Congratulations Lianet!
> > > > >>
> > > > >> On Wed, Aug 28, 2024 at 5:40 PM Josep Prat
> > > >  > > > 
> > > > >> wrote:
> > > > >>>
> > > > >>> Congrats Lianet!
> > > > >>>
> > > > >>> On Wed, Aug 28, 2024 at 5:38 PM Chia-Ping Tsai <
> > > > chia7...@apache.org>
> > > > >> wrote:
> > > > >>>
> > > >  Congratulations, Lianet!!!
> > > > 
> > > >  On 2024/08/28 15:35:23 David Jacot wrote:
> > > > > Hi all,
> > > > >
> > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > > committer,
> > > > > Lianet Magrans.
> > > > >
> > > > > Lianet has been a Kafka contributor since June 2023. In
> > > addition
> > > > to
> > > > > being a regular contributor and reviewer, she has made
> > > > significant
> > > > > contributions to the next generation of the consumer
> > rebalance
> > > > > protocol (KIP-848) and to the new consumer. She has also
> > > > > contributed
> > > > > to discussing and reviewing many KIPs.
> > > > >
> > > > > Congratulations, Lianet!
> > > > >
> > > > > Thanks,
> > > > > David (on behalf of the Apache Kafka PMC)
> > > > >
> > > > 
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> [image: Aiven] 
> > > > >>>
> > > > >>> *Josep Prat*
> > > > >>> Open Source Engineering Director, *Aiven*
> > > > >>> josep.p...@aiven.io   |   +491715557497
> > > > >>> aiven.io    |   <
> > > > >> https://www.facebook.com/aivencloud>
> > > > >>>   <
> > > > >> https://twitter.com/aiven_io>
> > > > >>> *Aiven Deutschland GmbH*
> > > > >>> Alexanderufer 3-7, 10117 Berlin
> > > > >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > >>> Anna Richardson, Kenneth Chen
> > > > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > David Arthur
> > > > >
> > > > 
> > > > >>>
> > > >
> > > >
> > >
> > > --
> > > Thanks,
> > > Jiashen
> > >
> >
>

Re: [DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

[Luke Chen]

Hi Dima,

> I assume that unclean leader election exists as a static config in kafka
kraft, but is still unavailable dynamically?  Or is it unavailable in
general?

Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` in
KRaft will work, either static config or dynamic config.
It's just the behavior changes when dynamically enabling the
`unclean.leader.election.enable`, which will not trigger unclean leader
election immediately after config change. The detailed behavior change is
as described above.

Let me know if it's still not clear.

Thanks.
Luke


On Thu, Aug 29, 2024 at 1:58 AM Dima Brodsky  wrote:

> Hi Luke,
>
> I assume that unclean leader election exists as a static config in kafka
> kraft, but is still unavailable dynamically?  Or is it unavailable in
> general?
>
> Thanks!
> ttyl
> Dima
>
>
> On Wed, Aug 28, 2024 at 3:45 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > In v3.9.0, we're planning to support the `unclean.leader.election.enable`
> > config in KRaft. Currently, the implementation PR
> > <https://github.com/apache/kafka/pull/16866> is under review now. But
> I'd
> > like to raise the discussion to let you know there will be one behavior
> > change when enabling `unclean.leader.election.enable` config dynamically
> in
> > KRaft.
> >
> > scenario:
> > 1. broker [0, 1] config are all set
> `unclean.leader.election.enable=false`.
> > 2. topic A is created with 1 partition, 2 replication factors and
> > `unclean.leader.election.enable=false`. Leader of the partition 0 is 0.
> ISR
> > is [0].
> > 3. broker 0 is down, which leaves ISR to empty [ ], and leader none.
> > 4. dynamically alter config using admin API to set
> > `unclean.leader.election.enable=true`, no matter it changes to the topic
> A,
> > or to broker 1, or default broker.
> >
> > In ZK mode, with this scenario, it'll trigger unclean leader election
> > immediately, and broker 1 will be elected as the leader, which brings the
> > availability of the partition back.
> > In KRaft mode, with this scenario, the unclean leader election will be
> > triggered in an unclean election thread, which is scheduled every 5 mins
> by
> > default. That is, this partition might still be unavailable even if it
> > enables `unclean.leader.election.enable`. If users want to elect a leader
> > for this partition, run `kafka-leader-election.sh` with `unclean` option
> to
> > trigger the unclean leader election immediately.
> >
> > The main reason we didn't implement what ZK did is because of this
> > complication
> > <https://github.com/apache/kafka/pull/16284#discussion_r1635410206> we
> > faced.
> >
> > Other behaviors keep the same. Ex: When the topic config, or broker
> config,
> > or default broker config set `unclean.leader.election.enable=true`, and
> > later, a partition has leader down with empty ISR [ ], and having some
> > alive non-ISR replicas, in this case, the unclean leader election will be
> > triggered, as long as the config was set earlier.
> >
> > But since the PR is still under review, any comments or thoughts are
> > welcomed.
> >
> > Thank you.
> > Luke
> >
>
>
> --
> ddbrod...@gmail.com
>
> "The price of reliability is the pursuit of the utmost simplicity.
> It is a price which the very rich find the most hard to pay."
>(Sir
> Antony Hoare, 1980)
>

[DISCUSS] `unclean.leader.election.enable` in KRaft behavior change

[Luke Chen]

Hi all,

In v3.9.0, we're planning to support the `unclean.leader.election.enable`
config in KRaft. Currently, the implementation PR
 is under review now. But I'd
like to raise the discussion to let you know there will be one behavior
change when enabling `unclean.leader.election.enable` config dynamically in
KRaft.

scenario:
1. broker [0, 1] config are all set `unclean.leader.election.enable=false`.
2. topic A is created with 1 partition, 2 replication factors and
`unclean.leader.election.enable=false`. Leader of the partition 0 is 0. ISR
is [0].
3. broker 0 is down, which leaves ISR to empty [ ], and leader none.
4. dynamically alter config using admin API to set
`unclean.leader.election.enable=true`, no matter it changes to the topic A,
or to broker 1, or default broker.

In ZK mode, with this scenario, it'll trigger unclean leader election
immediately, and broker 1 will be elected as the leader, which brings the
availability of the partition back.
In KRaft mode, with this scenario, the unclean leader election will be
triggered in an unclean election thread, which is scheduled every 5 mins by
default. That is, this partition might still be unavailable even if it
enables `unclean.leader.election.enable`. If users want to elect a leader
for this partition, run `kafka-leader-election.sh` with `unclean` option to
trigger the unclean leader election immediately.

The main reason we didn't implement what ZK did is because of this
complication
 we
faced.

Other behaviors keep the same. Ex: When the topic config, or broker config,
or default broker config set `unclean.leader.election.enable=true`, and
later, a partition has leader down with empty ISR [ ], and having some
alive non-ISR replicas, in this case, the unclean leader election will be
triggered, as long as the config was set earlier.

But since the PR is still under review, any comments or thoughts are
welcomed.

Thank you.
Luke

[jira] [Created] (KAFKA-17428) remote segments deleted in RLMCopyTask stays `COPY_SEGMENT_START` state

[Luke Chen (Jira)]

Luke Chen created KAFKA-17428:
-

 Summary: remote segments deleted in RLMCopyTask stays 
`COPY_SEGMENT_START` state
 Key: KAFKA-17428
 URL: https://issues.apache.org/jira/browse/KAFKA-17428
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


Currently, we will delete failed uploaded segment and Custom metadata size 
exceeded segments in copyLogSegment in RLMCopyTask. But after deletion, these 
segment states are still in COPY_SEGMENT_STARTED. That "might" cause unexpected 
issues in the future. We'd better to move the state from 
{{COPY_SEGMENT_STARTED}} -> {{DELETE_SEGMENT_STARTED}} -> 
{{DELETE_SEGMENT_FINISHED}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17422) Improve the test time for testDeletionSkippedForSegmentsBeingCopied

[Luke Chen (Jira)]

Luke Chen created KAFKA-17422:
-

 Summary: Improve the test time for 
testDeletionSkippedForSegmentsBeingCopied
 Key: KAFKA-17422
 URL: https://issues.apache.org/jira/browse/KAFKA-17422
 Project: Kafka
  Issue Type: Test
Reporter: Luke Chen


Currently, it takes more than 5 seconds to run 
testDeletionSkippedForSegmentsBeingCopied in my local env. It is too long for 
running an unit test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17412) Doc the different behavior of `unclean.leader.election.enable` in KRaft

[Luke Chen (Jira)]

Luke Chen created KAFKA-17412:
-

 Summary: Doc the different behavior of 
`unclean.leader.election.enable` in KRaft
 Key: KAFKA-17412
 URL: https://issues.apache.org/jira/browse/KAFKA-17412
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


context: https://github.com/apache/kafka/pull/16866



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Luke Chen]

+1 from me.

Thanks.

On Thu, Aug 22, 2024 at 10:27 PM Satish Duggana 
wrote:

> Thanks Kamal for the KIP, LGTM.
>
> +1 from me.
>
> On Mon, 19 Aug 2024 at 10:41, Kamal Chandraprakash
>  wrote:
> >
> > Hi,
> >
> > I would like to open voting for KIP-1075. I've addressed the review
> > comments in the discussion thread. Please vote if the proposal looks
> good.
> >
> >
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=315494171#content/view/315494171
> >
> >
> > Thanks,
> > Kamal
>

[jira] [Created] (KAFKA-17360) Improve log message when log retention ms/bytes is -2

[Luke Chen (Jira)]

Luke Chen created KAFKA-17360:
-

 Summary: Improve log message when log retention ms/bytes is -2
 Key: KAFKA-17360
 URL: https://issues.apache.org/jira/browse/KAFKA-17360
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


 When setting retention.ms/bytes to -2, we can notice this log message:
{code:java}
Deleting segment LogSegment(baseOffset=10045, size=1037087, 
lastModifiedTime=1724040653922, largestRecordTimestamp=1724040653835) due to 
local log retention size -2 breach. Local log size after deletion will be 
13435280. (kafka.log.UnifiedLog) [kafka-scheduler-6]{code}
This is not helpful for users. We should replace -2 with real retention value 
when logging.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Luke Chen]

Hi Kamal,

Thanks for the update.
LGTM.

Luke

On Wed, Aug 14, 2024 at 7:25 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi all,
>
> > I saw we added some new configs/metrics.
>
> I have removed the recent changes to the public interfaces to limit the
> scope of the KIP to minimum. PTAL.
>
> Thanks,
> Kamal
>
> On Wed, Aug 14, 2024 at 9:58 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Luke,
> >
> > > LC5
> > Agree, this is a rare scenario. Given that we have a common pool of
> > request handler threads to accept
> > all the incoming requests and there are no quotas to handle for each
> > request. I'm OK with reusing the
> > same remote-log-reader threads for LIST_OFFSETS requests. There may be
> > noisy neighbor issues
> > in handling the LIST_OFFSETS and FETCH remote requests when we read from
> > remote storage
> > aggressively and all the remote-log-reader threads are busy.
> >
> > >  If so, maybe the additional config/metrics are not necessary?
> > Do you mean to have a separate thread pool and hardcode the num threads?
> >
> > > LC6 and LC7
> > Updated the KIP.
> >
> > Thanks,
> > Kamal
> >
> > On Wed, Aug 14, 2024 at 8:05 AM Luke Chen  wrote:
> >
> >> Hi Kamal,
> >>
> >> Thanks for the response.
> >>
> >> I saw we added some new configs/metrics. Comments:
> >>
> >> LC5: Do you think this is a commonly happened issue that we need to add
> a
> >> separate `remote.log.offset.reader.threads` for it?
> >> I thought this rarely happened. If so, maybe the additional
> config/metrics
> >> are not necessary? It makes the config more complicated.
> >>
> >> LC6: The config name:
> >> `remote.log.offset.read.max.pending.tasks` , should we be consistent to
> >> use
> >> `reader`, instead of `read`?
> >>
> >> LC7: We should set a default value for the newly introduced configs and
> >> written in KIP.
> >>
> >> Thanks.
> >> Luke
> >>
> >> On Tue, Aug 13, 2024 at 8:47 PM Kamal Chandraprakash <
> >> kamal.chandraprak...@gmail.com> wrote:
> >>
> >> > Hi Luke,
> >> >
> >> > Thanks for the review!
> >> >
> >> > > LC2:
> >> > a. If the time taken to process the request is less than 5 mins, then
> >> the
> >> > Admin client will get a response.
> >> > b. If the time taken to process the request is more than 5 mins, then
> >> the
> >> > Admin client will itself timeout the request due to the
> >> > default-api-timeout.
> >> > c. If the time taken to process the request is more than 6 mins, then
> >> > the server will cancel the request in the DelayedRemoteListOffsets
> >> > purgatory (to be implemented) and
> >> > send TimeoutException back to the client if the client is waiting
> >> for
> >> > the response.
> >> >
> >> > > LC3:
> >> > Updated the KIP.
> >> >
> >> > > LC4:
> >> > The consumer retries the LIST_OFFSETS request incase of
> failures/timeout
> >> > but not the AdminClient. So, I think this is a retry feature in the
> >> > Consumer.
> >> >
> >> > Updated the "Public Interfaces" section in the KIP
> >> > <
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Introduce+delayed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async
> >> > >
> >> > by adding more details. PTAL.
> >> >
> >> > Thanks,
> >> > Kamal
> >> >
> >> >
> >> > On Tue, Aug 13, 2024 at 2:03 PM Luke Chen  wrote:
> >> >
> >> > > Hi Kamal,
> >> > >
> >> > > Thanks for the update.
> >> > > LC1: I see. Thanks.
> >> > > LC2: What I still don't understand is what is the relationship
> between
> >> > > remote.list.offsets.request.timeout.ms V.S.
> >> > > request.timeout/default.api.timeout.
> >> > > Suppose we set request timeout to 30 seconds, default.api.timeout=5
> >> mins
> >> > > and remote.list.offsets.request.timeout.ms = 6 mins.
> >> > > So, when Admin sends a list offset request that needs to query
> remote
> >> > > storage, when will it throw timeout 

[jira] [Resolved] (KAFKA-17300) documentation for disablement feature

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17300.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> documentation for disablement feature
> -
>
> Key: KAFKA-17300
> URL: https://issues.apache.org/jira/browse/KAFKA-17300
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.9.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Luke Chen]

Hi Kamal,

Thanks for the response.

I saw we added some new configs/metrics. Comments:

LC5: Do you think this is a commonly happened issue that we need to add a
separate `remote.log.offset.reader.threads` for it?
I thought this rarely happened. If so, maybe the additional config/metrics
are not necessary? It makes the config more complicated.

LC6: The config name:
`remote.log.offset.read.max.pending.tasks` , should we be consistent to use
`reader`, instead of `read`?

LC7: We should set a default value for the newly introduced configs and
written in KIP.

Thanks.
Luke

On Tue, Aug 13, 2024 at 8:47 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Luke,
>
> Thanks for the review!
>
> > LC2:
> a. If the time taken to process the request is less than 5 mins, then the
> Admin client will get a response.
> b. If the time taken to process the request is more than 5 mins, then the
> Admin client will itself timeout the request due to the
> default-api-timeout.
> c. If the time taken to process the request is more than 6 mins, then
> the server will cancel the request in the DelayedRemoteListOffsets
> purgatory (to be implemented) and
> send TimeoutException back to the client if the client is waiting for
> the response.
>
> > LC3:
> Updated the KIP.
>
> > LC4:
> The consumer retries the LIST_OFFSETS request incase of failures/timeout
> but not the AdminClient. So, I think this is a retry feature in the
> Consumer.
>
> Updated the "Public Interfaces" section in the KIP
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Introduce+delayed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async
> >
> by adding more details. PTAL.
>
> Thanks,
> Kamal
>
>
> On Tue, Aug 13, 2024 at 2:03 PM Luke Chen  wrote:
>
> > Hi Kamal,
> >
> > Thanks for the update.
> > LC1: I see. Thanks.
> > LC2: What I still don't understand is what is the relationship between
> > remote.list.offsets.request.timeout.ms V.S.
> > request.timeout/default.api.timeout.
> > Suppose we set request timeout to 30 seconds, default.api.timeout=5 mins
> > and remote.list.offsets.request.timeout.ms = 6 mins.
> > So, when Admin sends a list offset request that needs to query remote
> > storage, when will it throw timeout exception? 30 secs or 5 mins or 6
> mins?
> > We might need to make it clear in the KIP.
> >
> > LC3:
> > "Admin sends only one request and wait for upto default-api-timeout. (eg)
> > If the admin is configured with default-api-timeout as 5 mins and
> > request-timeout as 30 seconds. And, the server takes 50 seconds to
> process
> > the LIST_OFFSETS request, then the admin sends only one LIST_OFFSETS
> > request, then receives the request from server after 50 seconds."
> >
> > In the end of the section, it should be receives the "response" from
> > server?
> >
> > LC4: I found the different consumer and admin behavior when setting
> > "request.timeout" and "default.api.timeout" is confusing. Are they
> expected
> > or a bug?
> >
> > Thank you.
> > Luke
> >
> >
> > On Thu, Aug 8, 2024 at 10:06 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi Luke,
> > >
> > > Thanks for the review!
> > >
> > > LC1: When the consumer starts to read data, then it might need the
> below
> > > offsets:
> > > earliest, latest, and last-committed-offset based on the
> > > "auto.offset.reset" config.
> > >
> > > The earliest and latest offsets have special timestamps -2 and -1,
> those
> > > timestamp corresponding offsets are cached in the
> > > broker memory and get served immediately. The last-committed-offset is
> > also
> > > cached in the GroupMetadata and
> > > gets served in the "OFFSET_FETCH" request. Unless the consumer
> > > explicitly uses the KafkaConsumer#offsetForTimes API,
> > > there won't be any delay in serving the data from the local log.
> > >
> > > In this KIP, we are trying to address the case in which multiple
> > consumers
> > > start at the same time and use the 'offsetForTimes` LIST_OFFSETS API,
> > > assuming the remote requests are slow, then it should not block other
> > > PRODUCE/FETCH requests.
> > >
> > > LC2: Sorry for the confusion. We were planning to introduce only the
> > broker
> > > config "remote.list.offsets.timeout.ms".
> > > If we add the timeout in the ListOffsetsRequest.json, then when old
> > clients
> > > talk with the new broker, we don't have
> > > a timeout to set on the server. Moved adding the timeout to the
> > > ListOffsetsRequest to the rejected alternatives section.
> > >
> > > --
> > > Kamal
> > >
> >
>

Re: [VOTE]: KIP-1006: Remove SecurityManager Support

[Luke Chen]

Hi Greg,

Thanks for the KIP!
This is important when we move to the new jdk version.

Just a minor comment, could we make it much clearer in the compatibility
section?
Like in the 1st line, we can directly say, "this change is backward
compatible", then explain if SecurityManager is unsupported, we'll fall
back to newer implementation.

Also, what does this line mean here?
"Users upgrading to Java 17+ (in which SecurityManager was deprecated)
already receive a warning message on startup."
Maybe we can remove it or add more words at the end?


Anyway, +1 (binding) from me.

Thank you.
Luke

On Tue, Jul 9, 2024 at 2:26 AM Greg Harris 
wrote:

> Hi all,
>
> I'd like to call a vote on
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support
>
>
> Voting for this KIP would signal our intention to remove SecurityManager
> support in the future when the project's minimum supported version is Java
> 21.
>
> Thanks,
> Greg Harris
>

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Luke Chen]

Hi Kamal,

Thanks for the update.
LC1: I see. Thanks.
LC2: What I still don't understand is what is the relationship between
remote.list.offsets.request.timeout.ms V.S.
request.timeout/default.api.timeout.
Suppose we set request timeout to 30 seconds, default.api.timeout=5 mins
and remote.list.offsets.request.timeout.ms = 6 mins.
So, when Admin sends a list offset request that needs to query remote
storage, when will it throw timeout exception? 30 secs or 5 mins or 6 mins?
We might need to make it clear in the KIP.

LC3:
"Admin sends only one request and wait for upto default-api-timeout. (eg)
If the admin is configured with default-api-timeout as 5 mins and
request-timeout as 30 seconds. And, the server takes 50 seconds to process
the LIST_OFFSETS request, then the admin sends only one LIST_OFFSETS
request, then receives the request from server after 50 seconds."

In the end of the section, it should be receives the "response" from server?

LC4: I found the different consumer and admin behavior when setting
"request.timeout" and "default.api.timeout" is confusing. Are they expected
or a bug?

Thank you.
Luke


On Thu, Aug 8, 2024 at 10:06 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Luke,
>
> Thanks for the review!
>
> LC1: When the consumer starts to read data, then it might need the below
> offsets:
> earliest, latest, and last-committed-offset based on the
> "auto.offset.reset" config.
>
> The earliest and latest offsets have special timestamps -2 and -1, those
> timestamp corresponding offsets are cached in the
> broker memory and get served immediately. The last-committed-offset is also
> cached in the GroupMetadata and
> gets served in the "OFFSET_FETCH" request. Unless the consumer
> explicitly uses the KafkaConsumer#offsetForTimes API,
> there won't be any delay in serving the data from the local log.
>
> In this KIP, we are trying to address the case in which multiple consumers
> start at the same time and use the 'offsetForTimes` LIST_OFFSETS API,
> assuming the remote requests are slow, then it should not block other
> PRODUCE/FETCH requests.
>
> LC2: Sorry for the confusion. We were planning to introduce only the broker
> config "remote.list.offsets.timeout.ms".
> If we add the timeout in the ListOffsetsRequest.json, then when old clients
> talk with the new broker, we don't have
> a timeout to set on the server. Moved adding the timeout to the
> ListOffsetsRequest to the rejected alternatives section.
>
> --
> Kamal
>

[jira] [Created] (KAFKA-17300) documentation for disablement feature

[Luke Chen (Jira)]

Luke Chen created KAFKA-17300:
-

 Summary: documentation for disablement feature
 Key: KAFKA-17300
 URL: https://issues.apache.org/jira/browse/KAFKA-17300
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16857) Zookeeper - Add new ZNodes

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16857.
---
Resolution: Not A Problem

We are pausing ZK development so resolving the ticket.

> Zookeeper - Add new ZNodes
> --
>
> Key: KAFKA-16857
> URL: https://issues.apache.org/jira/browse/KAFKA-16857
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Christo Lolov
>Priority: Major
>
> *Summary*
> Additional information needs to be stored in new ZNodes as part of 
> disablement. Ensure that said information makes it into Zookeeper.
> {code:java}
> /brokers/topics/{topic-name}/partitions
>/tieredstorage
>  /tiered_epoch
>  
>  /state {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17031) Make configurations public

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17031.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

This is done in another ticket. Close it.

> Make configurations public
> --
>
> Key: KAFKA-17031
> URL: https://issues.apache.org/jira/browse/KAFKA-17031
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Christo Lolov
>Priority: Major
> Fix For: 3.9.0
>
>
> *Summary*
> We introduce 3 new configurations as part of this KIP. Up to now they have 
> been defined as internal only. We need to change said configurations to 
> public prior to the release.
>  
> Also need to deprecate "remote.log.manager.thread.pool.size" config.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17236) local log should be deleted when remote copy disabled and no remote logs

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17236.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> local log should be deleted when remote copy disabled and no remote logs
> 
>
> Key: KAFKA-17236
> URL: https://issues.apache.org/jira/browse/KAFKA-17236
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.9.0
>
>
> local log should be deleted when remote copy disabled and no remote logs. It 
> doesn't work right now because the remote storage is enabled but no more data 
> will be copied into remote storage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17080) bump metadata version for topic Record

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17080.
---
Resolution: Not A Problem

We don't need to bump MV because we don't use topicRecord anymore.

> bump metadata version for topic Record
> --
>
> Key: KAFKA-17080
> URL: https://issues.apache.org/jira/browse/KAFKA-17080
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Assignee: Muralidhar Basani
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Luke Chen]

Hi Kamal,

Thanks for the KIP!
I think it is a good improvement to avoid users making many list_offset
requests to starve other high priority requests.

Questions:
1. When consumer poll the partitions first time, it'll also try to call
list_offset to get the offset to fetch.
If this offset is located in remote storage, the consumer will need to wait
for more time to start fetch real data.
Doesn't that conflict with what we are trying to achieve?

2. Since we will add a new timeout attribute to `ListOffsetsRequest
`,
please write it out clearly, like other KIP did (ex: KIP-1073

)
3. We added a new timeout attribute for the ListOffsetsRequest
,
as well as a remote.list.offsets.request.timeout.ms broker config.
Could you explain why we need 2 timeout configurations? I guess the broker
one is for the purgatory?
What will happen if the client timeout > broker timeout? And client timeout
< broker timeout?

Thank you.
Luke



On Mon, Aug 5, 2024 at 8:59 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Bumping this thread. Please take a look.
>
> On Fri, Aug 2, 2024 at 12:32 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi all,
> >
> > I would like to start a discussion thread on KIP-1075
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Introduce+delayed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async>
> to
> > make the remote LIST_OFFSETS an async operation.
> >
> > The KIP is here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Introduce+delayed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async
> >
> > Draft PR: https://github.com/apache/kafka/pull/16602
> >
> > Please take a look. Feedbacks and suggestions are welcome.
> >
> > Thanks,
> > Kamal
> >
>

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

[Luke Chen]

Hi Mickael,

Thanks for the KIP.
+1 (binding) from me.

Thanks.
Luke

On Fri, Aug 2, 2024 at 4:57 AM Tom Bentley  wrote:

> +1 (binding).
>
> Thanks Mickael!
>
> On Thu, 1 Aug 2024 at 05:12, Mickael Maison 
> wrote:
>
> > Hi,
> >
> > Bumping this thread to get some more votes and/or feedback.
> >
> > As I restarted the vote on June 10 after major changes, I'm only
> > counting votes since then.
> > So we have 1 binding (Chris) and 1 non-binding (Hector) votes.
> >
> > Thanks,
> > Mickael
> >
> > On Mon, Jul 8, 2024 at 8:40 PM Hector Geraldino (BLOOMBERG/ 919 3RD A)
> >  wrote:
> > >
> > > This will help eliminate some boilerplate code we have for our
> > connectors.
> > >
> > > +1 (non-binding)
> > >
> > > From: dev@kafka.apache.org At: 06/25/24 04:30:27 UTC-4:00To:
> > dev@kafka.apache.org
> > > Subject: Re: [VOTE] KIP-877: Mechanism for plugins and connectors to
> > register metrics
> > >
> > > Bumping this thread.
> > >
> > > Let me know if you have any feedback.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, Jun 10, 2024 at 1:44 PM Chris Egerton  >
> > wrote:
> > > >
> > > > +1 (binding), thanks Mickael!
> > > >
> > > > On Mon, Jun 10, 2024, 04:24 Mickael Maison  >
> > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Following the feedback in the DISCUSS thread, I made significant
> > > > > changes to the proposal. So I'd like to restart a vote for KIP-877:
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi
> > > ns+and+connectors+to+register+metrics
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Thu, Jan 25, 2024 at 2:59 AM Tom Bentley 
> > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > You'll have seen that I left some comments on the discussion
> > thread, but
> > > > > > they're minor enough that I'm happy to vote +1 here.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Tom
> > > > > >
> > > > > > On Thu, 11 Jan 2024 at 06:14, Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Bumping this thread since I've not seen any feedback.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > > On Tue, Dec 19, 2023 at 10:03 AM Mickael Maison
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'd like to start a vote on KIP-877:
> > > > > > > >
> > > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi
> > > ns+and+connectors+to+register+metrics
> > > > > > > >
> > > > > > > > Let me know if you have any feedback.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > >
> > >
> > >
> >
> >
>

Re: [DISCUSS] KIP-1073 Return inactive observer nodes in DescribeQuorum response

[Luke Chen]

Hi Gantigmaa,

Thanks for the KIP!
The motivation and change looks good to me.

Some comments:
1. typo: When a KRaft broker node shuts down, it is in "fenced" state, not
"unfenced" state
2. Will the "–include-inactive-observers" option apply to "
kafka-metadata-quorum.sh describe --replication"?
I don't think we must have it, but maybe it's useful to let users know the
offset lag for the inactive observers?

Thank you.
Luke

On Thu, Jul 25, 2024 at 9:21 PM Gantigmaa Selenge 
wrote:

> Hi everyone,
>
> I would like to start a discussion on KIP-1073 that includes inactive
> observer nodes in the response for describeQuorum request.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1073%3A+Return+inactive+observer+nodes+in+DescribeQuorum+response
>
> The initial discussion on this issue is here, if you would like to see what
> was previously discussed:
> https://lists.apache.org/list.html?dev@kafka.apache.org
>
> Any feedback and suggestions for the KIP are welcome in this email thread.
>
> Thank you.
> Regards,
> Gantigmaa Selenge
>

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Hi Satish,

Thanks for the review.
I've updated the KIP to move ZK based solution to the appendix section.

> One minor comment I have is to change the config name from
"remote.log.copy.disable" to "remote.log.copy.enable" with the default
value being true.

I also think the original name "remote.log.copy.disable" makes more sense
because users are "disabling" the remote copy feature.

Thank you.
Luke

On Thu, Aug 8, 2024 at 12:21 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi Satish,
>
> Thanks for the review!
>
> > One minor comment I have is to change the config name from
> "remote.log.copy.disable" to "remote.log.copy.enable" with the default
> value being true.
>
> I'm inclined towards maintaining the config as "remote.log.copy.disable"
> to keep the default value as "false".
>
> --
> Kamal
>
> On Thu, Aug 8, 2024 at 9:23 AM Satish Duggana 
> wrote:
>
> > Thanks Kamal, and Luke for improving the earlier solution for KRaft.
> >
> > One minor comment I have is to change the config name from
> > "remote.log.copy.disable" to "remote.log.copy.enable" with the default
> > value being true.
> >
> > The solution summary to disable tiered storage on a topic:
> >
> > - When a user wants to disable tiered storage on a topic, we should
> > make sure that local.log and log.retention are same. This is to make
> > sure the user understands the implications of the local storage
> > requirements while disabling tiered storage and set them
> > appropriately.
> >
> > - Stop copying the log segments to remote storage as broker needs to
> > accumulate the required data locally to serve the required data from
> > local storage before we disable in remote storage. This will be done
> > by updating the config "remote.log.copy.enable" as false.
> >
> > - We added a guardrail to make sure user understands that disabling
> > tiered storage will delete the remote storage data. This is by setting
> > "remote.log.delete.on.disable" should be true before setting
> > "remote.storage.enable" as false.
> >
> >
> > I think it is better to refactor the KIP to have only the updated
> > KRaft based solution and move the ZK based solution to the appendix
> > for reference. wdyt?
> >
> > ~Satish.
> >
> >
> >
> > On Wed, 7 Aug 2024 at 17:38, Luke Chen  wrote:
> > >
> > > Hi all,
> > >
> > > Based on the original design:
> > > When tiered storage is disabled or becomes read-only on a topic, the
> > local
> > > retention configuration becomes irrelevant, and all data expiration
> > follows
> > > the topic-wide retention configuration exclusively.
> > >
> > > That works well. But we are afraid users will not check the document
> and
> > > "thought" the local log is bound to the local.retention.ms/bytes after
> > > `remote.log.copy.disable=true` (i.e. read-only remote storage). The
> > > confusion might cause the local disk to be full and bring down the
> > broker.
> > > To avoid this "surprise" to users, we'd like to add one more validation
> > > when `remote.log.copy.disable` is set to true:
> > >   - validation: when `remote.log.copy.disable=true`,
> > > -- `local.retention.ms` must equal to `retention.ms` or -2 (which
> > > means `retention.ms` value to be used)
> > > -- `local.retention.bytes` must equal to `retention.byes` or -2
> > (which
> > > means `retention.ms` value to be used)
> > >
> > > So, basically, we don't change the original design, just want to make
> > sure
> > > users are aware of the retention policy change after disabling remote
> log
> > > copy.
> > >
> > > Let me know if you have any comments.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Fri, Jul 26, 2024 at 8:17 PM Luke Chen  wrote:
> > >
> > > > Thanks Kamal for the comments.
> > > > KIP updated.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Fri, Jul 26, 2024 at 6:56 PM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > >> Luke,
> > > >>
> > > >> Thanks for confirming the topic config change validation on the
> > controller
> > > >> and updating the KIP.
> >

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Hi all,

Based on the original design:
When tiered storage is disabled or becomes read-only on a topic, the local
retention configuration becomes irrelevant, and all data expiration follows
the topic-wide retention configuration exclusively.

That works well. But we are afraid users will not check the document and
"thought" the local log is bound to the local.retention.ms/bytes after
`remote.log.copy.disable=true` (i.e. read-only remote storage). The
confusion might cause the local disk to be full and bring down the broker.
To avoid this "surprise" to users, we'd like to add one more validation
when `remote.log.copy.disable` is set to true:
  - validation: when `remote.log.copy.disable=true`,
-- `local.retention.ms` must equal to `retention.ms` or -2 (which
means `retention.ms` value to be used)
-- `local.retention.bytes` must equal to `retention.byes` or -2 (which
means `retention.ms` value to be used)

So, basically, we don't change the original design, just want to make sure
users are aware of the retention policy change after disabling remote log
copy.

Let me know if you have any comments.

Thank you.
Luke

On Fri, Jul 26, 2024 at 8:17 PM Luke Chen  wrote:

> Thanks Kamal for the comments.
> KIP updated.
>
> Thanks.
> Luke
>
> On Fri, Jul 26, 2024 at 6:56 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
>> Luke,
>>
>> Thanks for confirming the topic config change validation on the controller
>> and updating the KIP.
>> The updated KIP LGTM.
>>
>> 1. Can we update the below sentence in the KIP to clarify that
>> remote.storage.enable should be true during graceful disablement?
>>
>> > Users set the configuration
>> "remote.storage.enable=false,remote.log.delete.on.disable=true", or
>> "remote.copy.disabled=true" for the desired topic, indicating the
>> disablement of tiered storage.
>> to
>> > Users set the configuration
>> "remote.storage.enable=false,remote.log.delete.on.disable=true", or
>> "remote.storage.enable=true,remote.copy.disabled=true" for the desired
>> topic, indicating the disablement of tiered storage.
>>
>> 2. Can we clarify in the public interface that the StopReplica v5,
>> tiered_epoch, and tiered_state changes are required only for ZK mode and
>> won't be implemented?
>>
>> Thanks,
>> Kamal
>>
>> On Fri, Jul 26, 2024 at 1:40 PM Luke Chen  wrote:
>>
>> > Hi Kamal,
>> >
>> > Thanks for the comments.
>> >
>> > For this:
>> > > If we throw an exception from the server for invalid config, then
>> there
>> > will be inconsistency between the CLI tools and the actual state of the
>> > topic in the cluster. This can cause some confusion to the users whether
>> > tiered storage is disabled or not. I don't know how the Kraft topic
>> config
>> > propagation/validation works.
>> >
>> > I've confirmed we can validate the topic configuration change on the
>> > controller level, by comparing existing configuration and new changed
>> > configuration.
>> > In my local POC, we can fail the configuration change if it's invalid
>> like
>> > this:
>> >
>> > # Disable with remote.log.delete.on.disable=false (default)
>> > bin/kafka-configs.sh --bootstrap-server {bootstrap-string} \
>> >--alter --entity-type topics --entity-name {topic-name} \
>> >--add-config 'remote.storage.enable=false'
>> >
>> > Error while executing config command with args '--bootstrap-server
>> > {bootstrap-string} --entity-type topics --entity-name {topic-name}
>> --alter
>> > --add-config remote.storage.enable=false'
>> > java.util.concurrent.ExecutionException:
>> > org.apache.kafka.common.errors.InvalidConfigurationException: It is
>> invalid
>> > to disable remote storage without deleting remote data. If you want to
>> keep
>> > the remote data, but turn to read only, please set
>> `remote.copy.disabled=
>> > true`. If you want to disable remote storage and delete all remote data,
>> > please set
>> `remote.storage.enable=false,remote.log.delete.on.disable=true`.
>> >
>> > I've updated the KIP. Please take a look when available.
>> >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
>> >
>> > Thank you.
>> > Luke
>> >
>> >
>> > On Fri, Jul 26, 2024 at 2:05 AM Kamal Chandraprakash <
>> > kamal.c

[jira] [Resolved] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-15857.
---
Resolution: Fixed

> Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.
> -
>
> Key: KAFKA-15857
> URL: https://issues.apache.org/jira/browse/KAFKA-15857
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Satish Duggana
>Assignee: Christo Lolov
>Priority: Major
>  Labels: need-kip, tiered-storage
> Fix For: 3.9.0
>
>
> Introduce  EarliestLocalOffset and TieredOffset in OffsetSpec which will help 
> in finding respective offsets while using AdminClient#listOffsets().
> EarliestLocalOffset - local log start offset of a topic partition.
> TieredOffset - Highest offset up to which the segments were copied to remote 
> storage.
> We can discuss further on naming and semantics of these offset specs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16154) Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16154.
---
Resolution: Fixed

> Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP
> ---
>
> Key: KAFKA-16154
> URL: https://issues.apache.org/jira/browse/KAFKA-16154
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.9.0
>
>
> A broker should start returning offsets when given a timestamp of -5, which 
> signifies a LATEST_TIERED_TIMESTAMP.
> There are 3 cases.
> Tiered Storage is not enabled. In such a situation asking for 
> LATEST_TIERED_TIMESTAMP should always return no offset.
> Tiered Storage is enabled and there is nothing in remote storage. In such a 
> situation the offset returned should be 0.
> Tiered Storage is enabled and there is something in remote storage. In such a 
> situation the offset returned should be the highest offset the broker is 
> aware of.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16855) KRaft - Wire replaying a TopicRecord

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16855.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> KRaft - Wire replaying a TopicRecord
> 
>
> Key: KAFKA-16855
> URL: https://issues.apache.org/jira/browse/KAFKA-16855
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.9.0
>
>
> *Summary*
> Replaying a TopicRecord containing a new TieredEpoch and TieredState needs to 
> interact with the two thread pools in the RemoteLogManager to add/remove the 
> correct tasks from each



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17245) revert topicRecord changes

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17245.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> revert topicRecord changes
> --
>
> Key: KAFKA-17245
> URL: https://issues.apache.org/jira/browse/KAFKA-17245
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Luke Chen
>Assignee: TengYao Chi
>Priority: Major
> Fix For: 3.9.0
>
>
> We should revert [https://github.com/apache/kafka/pull/16257] because we've 
> changed the design in KRaft to not use TopicRecord anymore.
>  
> See 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement#KIP950:TieredStorageDisablement-Disablement-KRaftBackedCluster]
>  for more detail.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17245) revert topicRecord changes

[Luke Chen (Jira)]

Luke Chen created KAFKA-17245:
-

 Summary: revert topicRecord changes
 Key: KAFKA-17245
 URL: https://issues.apache.org/jira/browse/KAFKA-17245
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


We should revert [https://github.com/apache/kafka/pull/16257] because we've 
changed the design in KRaft to not use TopicRecord anymore.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17236) local log should be deleted when remote copy disabled and no remote logs

[Luke Chen (Jira)]

Luke Chen created KAFKA-17236:
-

 Summary: local log should be deleted when remote copy disabled and 
no remote logs
 Key: KAFKA-17236
 URL: https://issues.apache.org/jira/browse/KAFKA-17236
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


local log should be deleted when remote copy disabled and no remote logs. It 
doesn't work right now because the remote storage is enabled but no more data 
will be copied into remote storage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-17205) Allow topic config validation in controller level in KRaft mode

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17205.
---
Resolution: Fixed

> Allow topic config validation in controller level in KRaft mode
> ---
>
> Key: KAFKA-17205
> URL: https://issues.apache.org/jira/browse/KAFKA-17205
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.9.0
>
>
> Allow topic config validation in controller level. This is required because 
> we need to fail the invalid config change before it is written into metadata 
> log, especially for tiered storage feature.
>  
> Note: This ticket only changes for KRaft mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

[Luke Chen]

; > > >> (Please report an unintended omission)
> > > > >>
> > > > >> Aadithya Chandra, Abhijeet Kumar, Abhinav Dixit, Adrian Preston,
> > > Afshin
> > > > >> Moazami, Ahmed Najiub, Ahmed Sobeh, Akhilesh Chaganti, Almog
> Gavra,
> > > > Alok
> > > > >> Thatikunta, Alyssa Huang, Anatoly Popov, Andras Katona, Andrew
> > > > >> Schofield, Anna Sophie Blee-Goldman, Antoine Pourchet, Anton
> > Agestam,
> > > > >> Anton Liauchuk, Anuj Sharma, Apoorv Mittal, Arnout Engelen, Arpit
> > > > Goyal,
> > > > >> Artem Livshits, Ashwin Pankaj, Ayoub Omari, Bruno Cadonna, Calvin
> > Liu,
> > > > >> Cameron Redpath, charliecheng630, Cheng-Kai, Zhang, Cheryl
> Simmons,
> > > > Chia
> > > > >> Chuan Yu, Chia-Ping Tsai, ChickenchickenLove, Chris Egerton, Chris
> > > > >> Holland, Christo Lolov, Christopher Webb, Colin P. McCabe, Colt
> > > > McNealy,
> > > > >> cooper.ts...@suse.com, Vedarth Sharma, Crispin Bernier, Daan
> > Gerits,
> > > > >> David Arthur, David Jacot, David Mao, dengziming, Divij Vaidya,
> > > DL1231,
> > > > >> Dmitry Werner, Dongnuo Lyu, Drawxy, Dung Ha, Edoardo Comar, Eduwer
> > > > >> Camacaro, Emanuele Sabellico, Erik van Oosten, Eugene Mitskevich,
> > Fan
> > > > >> Yang, Federico Valeri, Fiore Mario Vitale, flashmouse, Florin
> > > Akermann,
> > > > >> Frederik Rouleau, Gantigmaa Selenge, Gaurav Narula, ghostspiders,
> > > > >> gongxuanzhang, Greg Harris, Gyeongwon Do, Hailey Ni, Hao Li,
> Hector
> > > > >> Geraldino, highluck, hudeqi, Hy (하이), IBeyondy, Iblis Lin, Igor
> > > Soarez,
> > > > >> ilyazr, Ismael Juma, Ivan Vaskevych, Ivan Yurchenko, James
> Faulkner,
> > > > >> Jamie Holmes, Jason Gustafson, Jeff Kim, jiangyuan, Jim Galasyn,
> > > > Jinyong
> > > > >> Choi, Joel Hamill, John Doe zh2725284...@gmail.com, John Roesler,
> > > John
> > > > >> Yu, Johnny Hsu, Jorge Esteban Quilcate Otoya, Josep Prat, José
> > Armando
> > > > >> García Sancio, Jun Rao, Justine Olshan, Kalpesh Patel, Kamal
> > > > >> Chandraprakash, Ken Huang, Kirk True, Kohei Nozaki, Krishna
> Agarwal,
> > > > >> KrishVora01, Kuan-Po (Cooper) Tseng, Kvicii, Lee Dongjin, Leonardo
> > > > >> Silva, Lianet Magrans, LiangliangSui, Linu Shibu, lixinyang,
> Lokesh
> > > > >> Kumar, Loïc GREFFIER, Lucas Brutschy, Lucia Cerchie, Luke Chen,
> > > > >> Manikumar Reddy, mannoopj, Manyanda Chitimbo, Mario Pareja,
> Matthew
> > de
> > > > >> Detrich, Matthias Berndt, Matthias J. Sax, Matthias Sax, Max
> Riedel,
> > > > >> Mayank Shekhar Narula, Michael Edgar, Michael Westerby, Mickael
> > > Maison,
> > > > >> Mike Lloyd, Minha, Jeong, Murali Basani, n.izhikov, Nick Telford,
> > > > Nikhil
> > > > >> Ramakrishnan, Nikolay, Octavian Ciubotaru, Okada Haruki, Omnia G.H
> > > > >> Ibrahim, Ori Hoch, Owen Leung, Paolo Patierno, Philip Nee,
> > > > >> Phuc-Hong-Tran, PoAn Yang, Proven Provenzano, Qichao Chu, Ramin
> > > Gharib,
> > > > >> Ritika Reddy, Rittika Adhikari, Rohan, Ron Dagostino, runom,
> > rykovsi,
> > > > >> Sagar Rao, Said Boudjelda, sanepal, Sanskar Jhajharia, Satish
> > Duggana,
> > > > >> Sean Quah, Sebastian Marsching, Sebastien Viale, Sergio Troiano,
> Sid
> > > > >> Yagnik, Stanislav Kozlovski, Stig Døssing, Sudesh Wasnik, TaiJuWu,
> > > > >> TapDang, testn, TingIāu "Ting" Kì, vamossagar12, Vedarth
> > > > >> Sharma, Victor van den Hoven, Vikas Balani, Viktor Somogyi-Vass,
> > > > Vincent
> > > > >> Rose, Walker Carlson, wernerdv, Yang Yu, Yash Mayya, yicheny,
> > Yu-Chen
> > > > >> Lai, yuz10, Zhifeng Chen, Zihao Lin, Ziming Deng, 谭九鼎
> > > > >>
> > > > >> We welcome your help and feedback. For more information on how to
> > > > >> report problems, and to get involved, visit the project website at
> > > > >> https://kafka.apache.org/
> > > > >>
> > > > >> Thank you!
> > > > >>
> > > > >>
> > > > >> Regards,
> > > > >>
> > > > >> Josep Prat
> > > > >> Release Manager for Apache Kafka 3.8.0
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> You received this message because you are subscribed to the Google
> > > > Groups "kafka-clients" group.
> > > > >> To unsubscribe from this group and stop receiving emails from it,
> > send
> > > > an email to kafka-clients+unsubscr...@googlegroups.com.
> > > > >> To view this discussion on the web visit
> > > >
> > >
> >
> https://groups.google.com/d/msgid/kafka-clients/CAOJ18G5D-jOLuyPjR6Qq0msoC8wFHG_1XQPvbn-34_u%2BYHYnhw%40mail.gmail.com
> > > > <
> > > >
> > >
> >
> https://groups.google.com/d/msgid/kafka-clients/CAOJ18G5D-jOLuyPjR6Qq0msoC8wFHG_1XQPvbn-34_u%2BYHYnhw%40mail.gmail.com?utm_medium=email&utm_source=footer
> > > > >.
> > > > >
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > Groups "kafka-clients" group.
> > > > > To unsubscribe from this group and stop receiving emails from it,
> > send
> > > > an email to kafka-clients+unsubscr...@googlegroups.com.
> > > > > To view this discussion on the web visit
> > > >
> > >
> >
> https://groups.google.com/d/msgid/kafka-clients/CAFbGOyxeAoXNhzswSqEVCNMnLndzEe-BJZr-pJBKKbvE03chLw%40mail.gmail.com
> > > > <
> > > >
> > >
> >
> https://groups.google.com/d/msgid/kafka-clients/CAFbGOyxeAoXNhzswSqEVCNMnLndzEe-BJZr-pJBKKbvE03chLw%40mail.gmail.com?utm_medium=email&utm_source=footer
> > > > >.
> > > >
> > >
> >
>

Re: New release branch 3.9

[Luke Chen]

Hi Colin and all,

If KIP-853 can complete in v3.9.0 in time (or a little delay), I agree we
should try to keep v3.9.0 as the last release before v4.0.
This way, it will let all Kafka ecosystem projects have a clear (and
certain) picture about what will happen in Apache Kafka.

Hi Colin,
For KIP-950 (KAFKA-15132 )
to allow to disable tiered storage on topic level, the PR is under review
and we should be able to merge within this week.
For KIP-1005 (KAFKA-15857
) to expose remote
storage related offset in kafka-get-offsets.sh, this KIP was reverted in
v3.8.0 because of MV issue. We'd like to add it back and can be completed
within this week.

These 2 KIPs are important feature for tiered storage, we hope they can be
added into v3.9.0.

Thank you.
Luke



On Wed, Jul 31, 2024 at 7:31 AM Colin McCabe  wrote:

> Yeah, please go ahead. I know a lot of people are waiting for 4.0.
>
> best,
> Colin
>
>
> On Tue, Jul 30, 2024, at 16:05, Matthias J. Sax wrote:
> > Thanks for clarifying Colin. So my assumptions were actually correct.
> >
> > We have a lot of contributors waiting to pick-up 4.0 tickets, and I'll
> > go ahead a tell them that we are ready and they can start to pick them
> up.
> >
> > Thanks.
> >
> >
> > -Matthias
> >
> > On 7/30/24 3:51 PM, Colin McCabe wrote:
> >> Hi Chia-Ping Tsai,
> >>
> >> If you can get them done this week then I think we can merge them in to
> 3.9. If not, then let's wait until 4.0, please.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Tue, Jul 30, 2024, at 09:07, Chia-Ping Tsai wrote:
> >>> hi Colin,
> >>>
> >>> Could you please consider adding
> >>> https://issues.apache.org/jira/browse/KAFKA-1 to 3.9.0
> >>>
> >>> The issue is used to deprecate the formatters in core module. Also, it
> >>> implements the replacements for them.
> >>>
> >>> In order to follow the deprecation rules, it would be nice to have
> >>> KAFKA-1 in 3.9.0
> >>>
> >>> If you agree to have them in 3.9.0, I will cherry-pick them into 3.9.0
> when
> >>> they get merged to trunk.
> >>>
> >>> Best,
> >>> Chia-Ping
> >>>
> >>>
> >>> José Armando García Sancio  於
> 2024年7月30日 週二
> >>> 下午11:59寫道：
> >>>
>  Thanks Colin.
> 
>  For KIP-853 (KRaft Controller Membership Changes), we still have the
>  following features that are in progress.
> 
>  1. UpdateVoter RPC and request handling
>  
>  2. Storage tool changes for KIP-853
>  
>  3. kafka-metadata-quorum describe changes for KIP-853
>  
>  4. kafka-metadata-quorum add voter and remove voter changes
>  
>  5. Sending UpdateVoter request and response handling
>  
> 
>  Can we cherry pick them to the release branch 3.9.0 when they get
> merged to
>  trunk? They have a small impact as they shouldn't affect the rest of
> Kafka
>  and only affect the kraft controller membership change feature. I
> expected
>  them to get merged to the trunk branch in the coming days.
> 
>  Thanks,
> 
>  On Mon, Jul 29, 2024 at 7:02 PM Colin McCabe 
> wrote:
> 
> > Hi Kafka developers and friends,
> >
> > As promised, we now have a release branch for the upcoming 3.9.0
> release.
> > Trunk has been bumped to 4.0.0-SNAPSHOT.
> >
> > I'll be going over the JIRAs to move every non-blocker from this
> release
>  to
> > the next release.
> >
> >  From this point, most changes should go to trunk.
> > *Blockers (existing and new that we discover while testing the
> release)
> > will be double-committed. *Please discuss with your reviewer whether
> your
> > PR should go to trunk or to trunk+release so they can merge
> accordingly.
> >
> > *Please help us test the release! *
> >
> > best,
> > Colin
> >
> 
> 
>  --
>  -José
> 
>

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Thanks Kamal for the comments.
KIP updated.

Thanks.
Luke

On Fri, Jul 26, 2024 at 6:56 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Luke,
>
> Thanks for confirming the topic config change validation on the controller
> and updating the KIP.
> The updated KIP LGTM.
>
> 1. Can we update the below sentence in the KIP to clarify that
> remote.storage.enable should be true during graceful disablement?
>
> > Users set the configuration
> "remote.storage.enable=false,remote.log.delete.on.disable=true", or
> "remote.copy.disabled=true" for the desired topic, indicating the
> disablement of tiered storage.
> to
> > Users set the configuration
> "remote.storage.enable=false,remote.log.delete.on.disable=true", or
> "remote.storage.enable=true,remote.copy.disabled=true" for the desired
> topic, indicating the disablement of tiered storage.
>
> 2. Can we clarify in the public interface that the StopReplica v5,
> tiered_epoch, and tiered_state changes are required only for ZK mode and
> won't be implemented?
>
> Thanks,
> Kamal
>
> On Fri, Jul 26, 2024 at 1:40 PM Luke Chen  wrote:
>
> > Hi Kamal,
> >
> > Thanks for the comments.
> >
> > For this:
> > > If we throw an exception from the server for invalid config, then there
> > will be inconsistency between the CLI tools and the actual state of the
> > topic in the cluster. This can cause some confusion to the users whether
> > tiered storage is disabled or not. I don't know how the Kraft topic
> config
> > propagation/validation works.
> >
> > I've confirmed we can validate the topic configuration change on the
> > controller level, by comparing existing configuration and new changed
> > configuration.
> > In my local POC, we can fail the configuration change if it's invalid
> like
> > this:
> >
> > # Disable with remote.log.delete.on.disable=false (default)
> > bin/kafka-configs.sh --bootstrap-server {bootstrap-string} \
> >--alter --entity-type topics --entity-name {topic-name} \
> >--add-config 'remote.storage.enable=false'
> >
> > Error while executing config command with args '--bootstrap-server
> > {bootstrap-string} --entity-type topics --entity-name {topic-name}
> --alter
> > --add-config remote.storage.enable=false'
> > java.util.concurrent.ExecutionException:
> > org.apache.kafka.common.errors.InvalidConfigurationException: It is
> invalid
> > to disable remote storage without deleting remote data. If you want to
> keep
> > the remote data, but turn to read only, please set `remote.copy.disabled=
> > true`. If you want to disable remote storage and delete all remote data,
> > please set
> `remote.storage.enable=false,remote.log.delete.on.disable=true`.
> >
> > I've updated the KIP. Please take a look when available.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> >
> > Thank you.
> > Luke
> >
> >
> > On Fri, Jul 26, 2024 at 2:05 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Correction:
> > >
> > > (2): Wait for all the remote segments to be deleted async due to breach
> > by
> > > retention time (or) size,
> > >then set the `remote.storage.enable = false` and
> > > `remote.log.delete.on.disable = true`. This step is optional.
> > >
> > > On Thu, Jul 25, 2024 at 11:13 PM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi Chia-Ping,
> > > >
> > > > Thanks for the review!
> > > >
> > > > >  If so, what is the purpose of
> `remote.log.delete.on.disable=false`?
> > > >
> > > > IIUC, the purpose of `remote.log.delete.on.disable` is to get
> explicit
> > > > confirmation from the user
> > > > before deleting the remote log segments. The concern raised in the
> > thread
> > > > is that if the user
> > > > accidentally changes the value of `remote.storage.enable` from true
> to
> > > > false, then remote segments
> > > > get lost.
> > > >
> > > > For ungraceful disablement, (ie) disabling the remote storage for the
> > > > topic and deleting all the
> > > > remote segments, the user should set both the configs at once:
> > > >
> > > > (1) remote.storage.enable = false and remote.log.delete.on.disable =
> 

[jira] [Created] (KAFKA-17205) Allow topic config validation in controller level

[Luke Chen (Jira)]

Luke Chen created KAFKA-17205:
-

 Summary: Allow topic config validation in controller level 
 Key: KAFKA-17205
 URL: https://issues.apache.org/jira/browse/KAFKA-17205
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


Allow topic config validation in controller level. This is required because we 
need to fail the invalid config change before it is written into metadata log, 
especially for tiered storage feature.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Hi Kamal,

Thanks for the comments.

For this:
> If we throw an exception from the server for invalid config, then there
will be inconsistency between the CLI tools and the actual state of the
topic in the cluster. This can cause some confusion to the users whether
tiered storage is disabled or not. I don't know how the Kraft topic config
propagation/validation works.

I've confirmed we can validate the topic configuration change on the
controller level, by comparing existing configuration and new changed
configuration.
In my local POC, we can fail the configuration change if it's invalid like
this:

# Disable with remote.log.delete.on.disable=false (default)
bin/kafka-configs.sh --bootstrap-server {bootstrap-string} \
   --alter --entity-type topics --entity-name {topic-name} \
   --add-config 'remote.storage.enable=false'

Error while executing config command with args '--bootstrap-server
{bootstrap-string} --entity-type topics --entity-name {topic-name} --alter
--add-config remote.storage.enable=false'
java.util.concurrent.ExecutionException:
org.apache.kafka.common.errors.InvalidConfigurationException: It is invalid
to disable remote storage without deleting remote data. If you want to keep
the remote data, but turn to read only, please set `remote.copy.disabled=
true`. If you want to disable remote storage and delete all remote data,
please set `remote.storage.enable=false,remote.log.delete.on.disable=true`.

I've updated the KIP. Please take a look when available.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement

Thank you.
Luke


On Fri, Jul 26, 2024 at 2:05 AM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Correction:
>
> (2): Wait for all the remote segments to be deleted async due to breach by
> retention time (or) size,
>then set the `remote.storage.enable = false` and
> `remote.log.delete.on.disable = true`. This step is optional.
>
> On Thu, Jul 25, 2024 at 11:13 PM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Chia-Ping,
> >
> > Thanks for the review!
> >
> > >  If so, what is the purpose of `remote.log.delete.on.disable=false`?
> >
> > IIUC, the purpose of `remote.log.delete.on.disable` is to get explicit
> > confirmation from the user
> > before deleting the remote log segments. The concern raised in the thread
> > is that if the user
> > accidentally changes the value of `remote.storage.enable` from true to
> > false, then remote segments
> > get lost.
> >
> > For ungraceful disablement, (ie) disabling the remote storage for the
> > topic and deleting all the
> > remote segments, the user should set both the configs at once:
> >
> > (1) remote.storage.enable = false and remote.log.delete.on.disable = true
> >
> > If the user accidentally sets only the remote.storage.enable = true and
> > leaves the `remote.log.delete.on.disable`
> > with default value of `false`, then we will throw ConfigException to
> > prevent the deletion of remote logs.
> >
> > For graceful disablement, the user should set:
> >
> > (1): remote.copy.disabled = true.
> > (2): Wait for all the remote segments to be deleted async due to breach
> by
> > retention time (or) size,
> >then set the `remote.storage.enable = false`. This step is
> > optional.
> >
> > Luke,
> >
> > In ZK mode, once the topic config value gets updated, then it gets saved
> > in the /configs/topics/ znode.
> > If we throw an exception from the server for invalid config, then there
> > will be inconsistency between the CLI tools
> > and the actual state of the topic in the cluster. This can cause some
> > confusion to the users whether tiered storage
> > is disabled or not. I don't know how the Kraft topic config
> > propagation/validation works.
> >
> > --
> > Kamal
> >
> > On Thu, Jul 25, 2024 at 7:10 PM Chia-Ping Tsai 
> wrote:
> >
> >> remote.storage.enable=false
> >> remote.log.delete.on.disable=false (default)
> >> If the topic config is set to this, or changed to this, we'll return
> >> ConfigException during validation.
> >>
> >> Pardon me, I'm a bit confused.
> >>
> >> when `remote.storage.enable=true`, `remote.log.delete.on.disable=false`
> is
> >> no-op
> >> when `remote.storage.enable=false`, `remote.log.delete.on.disable=false`
> >> is
> >> error
> >>
> >> If `remote.log.delete.on.disable` must be true when setting
> >> `remote.storage.enable`
> >> to false, does it mean changing `remote.storage.en

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Hi Christo,

Thanks for your reply.

> keep the remote.log.disable.policy, but only allow it to take a value of
"delete".

I agree, or maybe make it a boolean value, and rename it to
`remote.log.delete.on.disable`, which is clearer.
And because of this new config, there will be a case that the config is
like this:

remote.storage.enable=false
remote.log.delete.on.disable=false (default)

That means, in this case, we'll keep all remote storage data, but close all
remote log tasks, and make "log start offset = local log start offset".
This will make the remote storage metadata in an unknown state because the
data in the remote storage is inaccessible anymore (since log start moved
to LLSO). And once this topic re-enables the `remote.storage.enable`, the
old remote log metadata will be included, but log start offset is not
expected anymore

So, I'd like to propose that we don't allow this configuration:

remote.storage.enable=false
remote.log.delete.on.disable=false (default)

If the topic config is set to this, or changed to this, we'll return
ConfigException during validation.

To make it clear, this is the new proposed solution:
https://docs.google.com/document/d/1Y_cSkXr-qQiFFlFoGqfzGHE9m9MnIvZSgGpFP5l5o4I/edit

Let me know what you think.

Thanks.
Luke



On Thu, Jul 25, 2024 at 8:07 PM Christo Lolov 
wrote:

> Hello!
>
> Thank you for raising this!
>
> Up to now KIP-950 took the stance that you can disable tiering whenever you
> wish as long as you specify what you would like to do with the data in
> remote. Amongst other things it also made the promise that it will not
> delete data without a user explicitly saying that they want their data
> deleted. In other words there is a 2-step verification that the user truly
> wants their data deleted.
>
> From the table of the new proposal I am left with the impression that the
> moment a user tries to disable tiering their data will by deleted. In other
> words, there is no 2-step verification that they want their data deleted.
>
> On a first read, I wouldn't be opposed to this proposal since it provides a
> neat alternative to the tiered epoch as long as there is still a 2-step
> verification that the user is aware their data will be deleted. I think
> that a reasonable way to achieve this is to keep the
> remote.log.disable.policy, but only allow it to take a value of "delete".
>
> What are your thoughts?
>
> Best,
> Christo
>
>
> On Thu, 25 Jul 2024 at 12:10, Luke Chen  wrote:
>
> > Hi all,
> >
> > I just found the table is not able to be displayed correctly in the
> email.
> > I've put the table content in google doc here
> > <
> >
> https://docs.google.com/document/d/1Y_cSkXr-qQiFFlFoGqfzGHE9m9MnIvZSgGpFP5l5o4I/edit?usp=sharing
> > >
> > .
> >
> > Thanks.
> > Luke
> >
> > On Thu, Jul 25, 2024 at 6:30 PM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > While implementing the feature in KRaft mode, I found something we need
> > to
> > > change the original proposal:
> > >
> > > (1) In the KIP of "Disablement - KRaft backed Cluster
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement#KIP950:TieredStorageDisablement-Disablement-KRaftBackedCluster
> > >",
> > > we said:
> > > Controller persists configuration change and completes disablement:
> > >
> > >1. The controller creates a ConfigRecord and persists it in the
> > >metadata topic.
> > >2. The controller creates a TopicRecord to increment the
> tiered_epoch
> > >and update the tiered_state to DISABLED state.
> > >3. This update marks the completion of the disablement process,
> > >indicating that tiered storage has been successfully disabled for
> the
> > >KRaft-backed clusters. Similar to topic deletion all replicas will
> > >eventually pick up the changes from the cluster metadata topic and
> > apply
> > >them to their own state. Any deletion failures will be picked up by
> > the
> > >expiration threads which should be deleting data before the log
> start
> > >offset. If the retention policy is delete, a new expiration thread
> > will be
> > >started on leadership change on any historical tiered topic to
> > confirm that
> > >there aren't any leftover segments in remote which need deletion.
> > After a
> > >cycle in which it didn't delete anything, it will die.
> > >
> > > For the (b) step, I don't think the controller needs to cr

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Hi all,

I just found the table is not able to be displayed correctly in the email.
I've put the table content in google doc here
<https://docs.google.com/document/d/1Y_cSkXr-qQiFFlFoGqfzGHE9m9MnIvZSgGpFP5l5o4I/edit?usp=sharing>
.

Thanks.
Luke

On Thu, Jul 25, 2024 at 6:30 PM Luke Chen  wrote:

> Hi all,
>
> While implementing the feature in KRaft mode, I found something we need to
> change the original proposal:
>
> (1) In the KIP of "Disablement - KRaft backed Cluster
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement#KIP950:TieredStorageDisablement-Disablement-KRaftBackedCluster>",
> we said:
> Controller persists configuration change and completes disablement:
>
>1. The controller creates a ConfigRecord and persists it in the
>metadata topic.
>2. The controller creates a TopicRecord to increment the tiered_epoch
>and update the tiered_state to DISABLED state.
>3. This update marks the completion of the disablement process,
>indicating that tiered storage has been successfully disabled for the
>KRaft-backed clusters. Similar to topic deletion all replicas will
>eventually pick up the changes from the cluster metadata topic and apply
>them to their own state. Any deletion failures will be picked up by the
>expiration threads which should be deleting data before the log start
>offset. If the retention policy is delete, a new expiration thread will be
>started on leadership change on any historical tiered topic to confirm that
>there aren't any leftover segments in remote which need deletion. After a
>cycle in which it didn't delete anything, it will die.
>
> For the (b) step, I don't think the controller needs to create TopicRecord
> because:
> 1. The broker can fetch the "tiered_state" from the ConfigRecord
> 2. The "tiered_epoch" is not necessary because raft protocol will keep the
> order for us. The broker can rely on the raft protocol and apply them in
> order, to get the expected results.
> 3. Marking the completion of the disablement process. In KRaft, it's not
> necessary because once the ConfigRecord is accepted by the controller, it
> must be applied by all the observers "in order".
>
> So, I'd like to propose to remove the (b) step in KRaft mode.
>
> (2) Current configuration make users and implementation confusion.
> This is what originally we proposed in KIP-950:
>
> remote.storage.enable
>
> remote.log.disable.policy(new)
>
> remote storage data
>
> true
>
> null/retain/delete
>
> uploadable + readable
>
> false
>
> retain (default)
>
> readable, but remote storage is disabled? For users, they are also
> surprised if this topic is reading data from remote storage.
>
> Note: This also makes development difficult because it’s unable to
> distinguish between:
>
> (1) a topic never enables remote storage
>
> (2) a topic enabled and then disabled remote storage
>
> A problem we have is when broker startup and trying to set log start
> offset. Since the remote storage is disabled, we originally should set to
> “local log start offset”, but in case (2), we expect it to treat it as
> “remote storage enabled”, which is confusing.
>
> false
>
> delete
>
> All remote data are deleted
>
>
> Therefore, Kamal and I would like to propose a new version of the
> configuration:
>
> remote.storage.enable
>
> remote.copy.disabled (new)
>
> remote storage data
>
> true
>
> false (default)
>
> uploadable + readable
>
> true
>
> true
>
> readable
>
> false
>
> true/false
>
> All remote data are deleted
>
> The advantage is this config makes users clear what it is configuring, and
> the result is expected.
> Also, on the implementation side, we can still rely on
> "remote.storage.enable" to identify is this feature is on/off.
>
> Any thoughts about it?
>
> Thank you.
> Luke
>
>
>
> On Thu, May 30, 2024 at 6:50 PM David Jacot 
> wrote:
>
>> Hi all,
>>
>> Thanks for the KIP. This is definitely a worthwhile feature. However, I am
>> a bit sceptical on the ZK part of the story. The 3.8 release is supposed
>> to
>> be the last one supporting ZK so I don't really see how we could bring it
>> to ZK, knowing that we don't plan to do a 3.9 release (current plan). I
>> strongly suggest clarifying this before implementing the ZK part in order
>> to avoid having new code [1] being deleted right after 3.8 is released
>> :). Personally, I agree with Chia-Ping and Mickael. We could drop the ZK
>> part.
>>

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Luke Chen]

Hi all,

While implementing the feature in KRaft mode, I found something we need to
change the original proposal:

(1) In the KIP of "Disablement - KRaft backed Cluster
",
we said:
Controller persists configuration change and completes disablement:

   1. The controller creates a ConfigRecord and persists it in the metadata
   topic.
   2. The controller creates a TopicRecord to increment the tiered_epoch
   and update the tiered_state to DISABLED state.
   3. This update marks the completion of the disablement process,
   indicating that tiered storage has been successfully disabled for the
   KRaft-backed clusters. Similar to topic deletion all replicas will
   eventually pick up the changes from the cluster metadata topic and apply
   them to their own state. Any deletion failures will be picked up by the
   expiration threads which should be deleting data before the log start
   offset. If the retention policy is delete, a new expiration thread will be
   started on leadership change on any historical tiered topic to confirm that
   there aren't any leftover segments in remote which need deletion. After a
   cycle in which it didn't delete anything, it will die.

For the (b) step, I don't think the controller needs to create TopicRecord
because:
1. The broker can fetch the "tiered_state" from the ConfigRecord
2. The "tiered_epoch" is not necessary because raft protocol will keep the
order for us. The broker can rely on the raft protocol and apply them in
order, to get the expected results.
3. Marking the completion of the disablement process. In KRaft, it's not
necessary because once the ConfigRecord is accepted by the controller, it
must be applied by all the observers "in order".

So, I'd like to propose to remove the (b) step in KRaft mode.

(2) Current configuration make users and implementation confusion.
This is what originally we proposed in KIP-950:

remote.storage.enable

remote.log.disable.policy(new)

remote storage data

true

null/retain/delete

uploadable + readable

false

retain (default)

readable, but remote storage is disabled? For users, they are also
surprised if this topic is reading data from remote storage.

Note: This also makes development difficult because it’s unable to
distinguish between:

(1) a topic never enables remote storage

(2) a topic enabled and then disabled remote storage

A problem we have is when broker startup and trying to set log start
offset. Since the remote storage is disabled, we originally should set to
“local log start offset”, but in case (2), we expect it to treat it as
“remote storage enabled”, which is confusing.

false

delete

All remote data are deleted


Therefore, Kamal and I would like to propose a new version of the
configuration:

remote.storage.enable

remote.copy.disabled (new)

remote storage data

true

false (default)

uploadable + readable

true

true

readable

false

true/false

All remote data are deleted

The advantage is this config makes users clear what it is configuring, and
the result is expected.
Also, on the implementation side, we can still rely on
"remote.storage.enable" to identify is this feature is on/off.

Any thoughts about it?

Thank you.
Luke



On Thu, May 30, 2024 at 6:50 PM David Jacot 
wrote:

> Hi all,
>
> Thanks for the KIP. This is definitely a worthwhile feature. However, I am
> a bit sceptical on the ZK part of the story. The 3.8 release is supposed to
> be the last one supporting ZK so I don't really see how we could bring it
> to ZK, knowing that we don't plan to do a 3.9 release (current plan). I
> strongly suggest clarifying this before implementing the ZK part in order
> to avoid having new code [1] being deleted right after 3.8 is released
> :). Personally, I agree with Chia-Ping and Mickael. We could drop the ZK
> part.
>
> [1] https://github.com/apache/kafka/pull/16131
>
> Best,
> David
>
> On Tue, May 28, 2024 at 1:31 PM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I agree with Chia-Ping, I think we could drop the ZK variant
> > altogether, especially if this is not going to make it in 3.8.0.
> > Even if we end up needing a 3.9.0 release, I wouldn't write a bunch of
> > new ZooKeeper-related code in that release to delete it all right
> > after in 4.0.
> >
> > Thanks,
> > Mickael
> >
> > On Fri, May 24, 2024 at 5:03 PM Christo Lolov 
> > wrote:
> > >
> > > Hello!
> > >
> > > I am closing this vote as ACCEPTED with 3 binding +1 (Luke, Chia-Ping
> and
> > > Satish) and 1 non-binding +1 (Kamal) - thank you for the reviews!
> > >
> > > Realistically, I don't think I have the bandwidth to get this in 3.8.0.
> > > Due to this, I will mark tentatively the Zookeeper part for 3.9 if the
> > > community decides that they do in fact want one more 3.x release.
> > > I will mark the KRaft part as ready to be started and aiming for either
> > 4.0
> > > or 3.9.
>

[jira] [Created] (KAFKA-17191) close RLMM and expire task after empty remote storage in "retain" policy

[Luke Chen (Jira)]

Luke Chen created KAFKA-17191:
-

 Summary: close RLMM and expire task after empty remote storage in 
"retain" policy
 Key: KAFKA-17191
 URL: https://issues.apache.org/jira/browse/KAFKA-17191
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


context: https://github.com/apache/kafka/pull/16653#pullrequestreview-2194120819



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Wiki Kafka Kip access

[Luke Chen]

Hi Mason,

Your accounts are all set.

Thanks.
Luke

On Sun, Jul 21, 2024 at 6:25 PM Mason Chen  wrote:

> Hi there,
>
> I would like to have the right access permission to create Kip. My wiki id
> and jira id are both masonc. Thank you for the help!
>
> Best regards,
> Mason
>

Re: Getting Started: requesting permissions to contribute to Apache Kafka

[Luke Chen]

Hi Rich,

Your accounts are all set.

Thanks.
Luke

On Mon, Jul 22, 2024 at 12:37 AM Rich C.  wrote:

> Hi team,
>
> Following Getting Started #4
> <
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >,
> I would like to request permission to contribute to Apache Kafka. I guess
> the edit permission for wiki (KIP).
>
> Jira: jychen7
> wiki: juchen7 (note: wiki id is different from Jira id, due to a typo
> during signup)
>
> Thanks,
> Rich
>

[jira] [Reopened] (KAFKA-16154) Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen reopened KAFKA-16154:
---

> Make broker changes to return an offset for LATEST_TIERED_TIMESTAMP
> ---
>
> Key: KAFKA-16154
> URL: https://issues.apache.org/jira/browse/KAFKA-16154
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Christo Lolov
>Assignee: Christo Lolov
>Priority: Major
> Fix For: 3.8.0
>
>
> A broker should start returning offsets when given a timestamp of -5, which 
> signifies a LATEST_TIERED_TIMESTAMP.
> There are 3 cases.
> Tiered Storage is not enabled. In such a situation asking for 
> LATEST_TIERED_TIMESTAMP should always return no offset.
> Tiered Storage is enabled and there is nothing in remote storage. In such a 
> situation the offset returned should be 0.
> Tiered Storage is enabled and there is something in remote storage. In such a 
> situation the offset returned should be the highest offset the broker is 
> aware of.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17141) "DescribeTopicPartitions API is not supported" warning message confuses users

[Luke Chen (Jira)]

Luke Chen created KAFKA-17141:
-

 Summary: "DescribeTopicPartitions API is not supported" warning 
message confuses users
 Key: KAFKA-17141
 URL: https://issues.apache.org/jira/browse/KAFKA-17141
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


When running describeTopics admin API, we'll try to invoke 
DescribeTopicPartitions request, and fallback to metadata request if not 
supported, and then log a warning message:
{code:java}
2024-07-15 19:03:34 WARN  KafkaAdminClient:2293 - [AdminClient 
clientId=adminclient-17] The DescribeTopicPartitions API is not supported, 
using Metadata API to describe topics.{code}
 

When seeing this warning message, users might think something is wrong with 
this call. We should try to improve it. Maybe:
1. log it with an INFO level or DEBUG level

2. Make the message clear to not to confuse users

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.8.0 RC0

[Luke Chen]

Sounds good!
Thank you.

Luke

On Mon, Jul 15, 2024 at 5:11 PM Mickael Maison 
wrote:

> Yeah I've not marked it as a blocker for 3.8.0. It's just something we
> need to do in the background.
>
> On Mon, Jul 15, 2024 at 11:05 AM Luke Chen  wrote:
> >
> > Thanks Mickael!
> > +1 for increasing the test coverage for admin clients.
> > But I don't think this should be the blocker for v3.8.0, given the delay
> of
> > v3.8.0 and we already have many releases with this state.
> > What do you think?
> >
> > Thanks.
> > Luke
> >
> > On Mon, Jul 15, 2024 at 4:57 PM Mickael Maison  >
> > wrote:
> >
> > > Hi,
> > >
> > > I'm concerned we did not have tests to catch that issue earlier. Such
> > > an essential API like describeTopics() should be properly tested.
> > > Taking a quick look, it seems a bunch of other Admin APIs also don't
> > > have integration tests. I created
> > > https://issues.apache.org/jira/browse/KAFKA-17137 to address that.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, Jul 15, 2024 at 9:53 AM Josep Prat  >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I'm cancelling the VOTE thread for 3.8.0-RC0. I submitted a PR with
> the
> > > > backport https://github.com/apache/kafka/pull/16593 and I'll
> generate a
> > > new
> > > > RC as soon as it's merged.
> > > >
> > > > Best,
> > > >
> > > > On Sat, Jul 13, 2024 at 7:09 PM Josep Prat 
> wrote:
> > > >
> > > > > Thanks for reviewing the RC Jakub,
> > > > >
> > > > > If you can open a PR with this fix pointing to the 3.8 branch I
> could
> > > cut
> > > > > another RC.
> > > > >
> > > > > Best!
> > > > >
> > > > > --
> > > > > Josep Prat
> > > > > Open Source Engineering Director, Aiven
> > > > > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > > > > Aiven Deutschland GmbH
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > On Sat, Jul 13, 2024, 16:13 Jakub Scholz  wrote:
> > > > >
> > > > >> Hi Josep,
> > > > >>
> > > > >> Thanks for the RC.
> > > > >>
> > > > >> I gave it a quick try and ran into issues with an application
> using
> > > the
> > > > >> Kafka Admin API that looks like this issue:
> > > > >> https://issues.apache.org/jira/browse/KAFKA-16905 ... given that
> this
> > > > >> breaks what was working fine with Kafka 3.7, can the fix be
> > > backported to
> > > > >> 3.8.0 as well? If needed, I tried to create a simple reproducer
> for
> > > the
> > > > >> issue:
> > > https://github.com/scholzj/kafka-admin-api-async-issue-reproducer.
> > > > >>
> > > > >> Thanks & Regards
> > > > >> Jakub
> > > > >>
> > > > >> On Fri, Jul 12, 2024 at 11:46 AM Josep Prat
> > > 
> > > > >> wrote:
> > > > >>
> > > > >> > Hello Kafka users, developers and client-developers,
> > > > >> >
> > > > >> > This is the first candidate for release of Apache Kafka 3.8.0.
> > > > >> > Some of the major features included in this release are:
> > > > >> > * KIP-1028: Docker Official Image for Apache Kafka
> > > > >> > * KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > >> > * KIP-1036: Extend RecordDeserializationException exception
> > > > >> > * KIP-1019: Expose method to determine Metric Measurability
> > > > >> > * KIP-1004: Enforce tasks.max property in Kafka Connect
> > > > >> > * KIP-989: Improved StateStore Iterator metrics for detecting
> leaks
> > > > >> > * KIP-993: Allow restricting files accessed by File and
> Directory
> > > > >> > ConfigProviders
> > > > >> > * KIP-924: customizable task assignment for Streams
> > > > >> > * KIP-813: Shareable State St

[jira] [Resolved] (KAFKA-16661) add a lower `log.initial.task.delay.ms` value to integration test framework

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16661.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> add a lower `log.initial.task.delay.ms` value to integration test framework
> ---
>
> Key: KAFKA-16661
> URL: https://issues.apache.org/jira/browse/KAFKA-16661
> Project: Kafka
>  Issue Type: Test
>    Reporter: Luke Chen
>Assignee: Vinay Agarwal
>Priority: Major
>  Labels: newbie, newbie++
> Fix For: 3.9.0
>
>
> After KAFKA-16552, we created an internal config `log.initial.task.delay.ms` 
> to control the initial task delay in log manager. This ticket follows it up, 
> to set a default low value (100ms, 500ms maybe?) for it, to speed up the 
> tests.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.8.0 RC0

[Luke Chen]

Thanks Mickael!
+1 for increasing the test coverage for admin clients.
But I don't think this should be the blocker for v3.8.0, given the delay of
v3.8.0 and we already have many releases with this state.
What do you think?

Thanks.
Luke

On Mon, Jul 15, 2024 at 4:57 PM Mickael Maison 
wrote:

> Hi,
>
> I'm concerned we did not have tests to catch that issue earlier. Such
> an essential API like describeTopics() should be properly tested.
> Taking a quick look, it seems a bunch of other Admin APIs also don't
> have integration tests. I created
> https://issues.apache.org/jira/browse/KAFKA-17137 to address that.
>
> Thanks,
> Mickael
>
> On Mon, Jul 15, 2024 at 9:53 AM Josep Prat 
> wrote:
> >
> > Hi all,
> >
> > I'm cancelling the VOTE thread for 3.8.0-RC0. I submitted a PR with the
> > backport https://github.com/apache/kafka/pull/16593 and I'll generate a
> new
> > RC as soon as it's merged.
> >
> > Best,
> >
> > On Sat, Jul 13, 2024 at 7:09 PM Josep Prat  wrote:
> >
> > > Thanks for reviewing the RC Jakub,
> > >
> > > If you can open a PR with this fix pointing to the 3.8 branch I could
> cut
> > > another RC.
> > >
> > > Best!
> > >
> > > --
> > > Josep Prat
> > > Open Source Engineering Director, Aiven
> > > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > On Sat, Jul 13, 2024, 16:13 Jakub Scholz  wrote:
> > >
> > >> Hi Josep,
> > >>
> > >> Thanks for the RC.
> > >>
> > >> I gave it a quick try and ran into issues with an application using
> the
> > >> Kafka Admin API that looks like this issue:
> > >> https://issues.apache.org/jira/browse/KAFKA-16905 ... given that this
> > >> breaks what was working fine with Kafka 3.7, can the fix be
> backported to
> > >> 3.8.0 as well? If needed, I tried to create a simple reproducer for
> the
> > >> issue:
> https://github.com/scholzj/kafka-admin-api-async-issue-reproducer.
> > >>
> > >> Thanks & Regards
> > >> Jakub
> > >>
> > >> On Fri, Jul 12, 2024 at 11:46 AM Josep Prat
> 
> > >> wrote:
> > >>
> > >> > Hello Kafka users, developers and client-developers,
> > >> >
> > >> > This is the first candidate for release of Apache Kafka 3.8.0.
> > >> > Some of the major features included in this release are:
> > >> > * KIP-1028: Docker Official Image for Apache Kafka
> > >> > * KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > >> > * KIP-1036: Extend RecordDeserializationException exception
> > >> > * KIP-1019: Expose method to determine Metric Measurability
> > >> > * KIP-1004: Enforce tasks.max property in Kafka Connect
> > >> > * KIP-989: Improved StateStore Iterator metrics for detecting leaks
> > >> > * KIP-993: Allow restricting files accessed by File and Directory
> > >> > ConfigProviders
> > >> > * KIP-924: customizable task assignment for Streams
> > >> > * KIP-813: Shareable State Stores
> > >> > * KIP-719: Deprecate Log4J Appender
> > >> > * KIP-390: Support Compression Level
> > >> > * KIP-1018: Introduce max remote fetch timeout config for
> > >> > DelayedRemoteFetch requests
> > >> > * KIP-1037: Allow WriteTxnMarkers API with Alter Cluster Permission
> > >> > * KIP-1047 Introduce new org.apache.kafka.tools.api.Decoder to
> replace
> > >> > kafka.serializer.Decoder
> > >> > * KIP-899: Allow producer and consumer clients to rebootstrap
> > >> >
> > >> > Release notes for the 3.8.0 release:
> > >> > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/RELEASE_NOTES.html
> > >> >
> > >> > *** Please download, test and vote by Monday, July 15, 12pm PT
> > >> >
> > >> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > https://kafka.apache.org/KEYS
> > >> >
> > >> > * Release artifacts to be voted upon (source and binary):
> > >> > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/
> > >> >
> > >> > * Docker release artifacts to be voted upon:
> > >> > apache/kafka:3.8.0-rc0
> > >> > apache/kafka-native:3.8.0-rc0
> > >> >
> > >> > * Maven artifacts to be voted upon:
> > >> >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> >
> > >> > * Javadoc:
> > >> > https://home.apache.org/~jlprat/kafka-3.8.0-rc0/javadoc/
> > >> >
> > >> > * Tag to be voted upon (off 3.8 branch) is the 3.8.0 tag:
> > >> > https://github.com/apache/kafka/releases/tag/3.8.0-rc0
> > >> >
> > >> >
> > >> > Once https://github.com/apache/kafka-site/pull/608 is merged. You
> will
> > >> be
> > >> > able to find the proper documentation under kafka.apache.org.
> > >> > * Documentation:
> > >> > https://kafka.apache.org/38/documentation.html
> > >> >
> > >> > * Protocol:
> > >> > https://kafka.apache.org/38/protocol.html
> > >> >
> > >> >
> > >> > * Successful Jenkins builds for the 3.8 branch:
> > >> > Unit/integration tests:
> > >> >
> > >> >
> > >>
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%252Fkafka/detail/3.8/67/

Re: [VOTE] KIP-1067: Remove ReplicaVerificationTool in 4.0 (deprecate in 3.9)

[Luke Chen]

+1 (binding) from me.

Thanks Dongjin.
Luke

On Thu, Jul 11, 2024 at 12:23 AM Justine Olshan
 wrote:

> +1 (binding)
>
> Thanks,
> Justine
>
> On Mon, Jul 8, 2024 at 1:59 AM Chia-Ping Tsai  wrote:
>
> > >
> > > Note that we already have this tracker for tools deprecations, but I'm
> > > fine to have a dedicated one for this tool (maybe we can link them).
> > > https://issues.apache.org/jira/browse/KAFKA-14705.
> >
> >
> > happy to know it. I have added the link to
> > https://issues.apache.org/jira/browse/KAFKA-17073
> >
> > Federico Valeri  於 2024年7月8日 週一 下午3:45寫道：
> >
> > > +1
> > >
> > > Note that we already have this tracker for tools deprecations, but I'm
> > > fine to have a dedicated one for this tool (maybe we can link them).
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-14705.
> > >
> > > On Sun, Jul 7, 2024 at 3:58 PM Chia-Ping Tsai 
> > wrote:
> > > >
> > > > +1
> > > >
> > > > Dongjin Lee  於 2024年7月7日 週日 下午9:22寫道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to call for a vote on KIP-1067: Remove
> > > ReplicaVerificationTool in
> > > > > 4.0 (deprecate in 3.9):
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311627623
> > > > >
> > > > > Thanks,
> > > > > Dongjin
> > > > >
> > > > > --
> > > > > *Dongjin Lee*
> > > > >
> > > > > *A hitchhiker in the mathematical world.*
> > > > >
> > > > >
> > > > >
> > > > > *github:  github.com/dongjinleekr
> > > > > keybase:
> > > https://keybase.io/dongjinleekr
> > > > > linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > > speakerdeck:
> > > > > speakerdeck.com/dongjin
> > > > > *
> > > > >
> > >
> >
>

Re: [DISCUSS] KAFKA-17094: How should unregistered broker nodes be handled in KRaft quorum state?

[Luke Chen]

Hi Tina,

Thanks for starting the discussion thread. This is indeed a problem if
users don't know which node ID need to be unregistered.
I'm +1 for adding another field in the response to include registered
observer nodes that are inactive. That will make it clear and won't confuse
existing users.

Thanks.
Luke

On Wed, Jul 10, 2024 at 11:43 PM Gantigmaa Selenge 
wrote:

> Hi all,
>
> As reported in KAFKA-17094 [1], to scale down KRaft-based broker nodes,
> they must first be unregistered via the Kafka Admin API. If a node is
> removed before being unregistered, it can't be listed for unregistration
> because the describeQuorum won't show inactive observer nodes. This happens
> because the quorum state excludes nodes that haven't heartbeated within the
> observer session timeout [2].
>
> To address this issue, we could stop clearing the observers list, changing
> its meaning from "active observer nodes" to "all registered observer
> nodes". While the current code implies the list should only include active
> nodes, there's no documentation explicitly stating this. Moreover, the
> voters list already includes all registered/configured voter nodes,
> inactive or not. Making this change would align the behavior of the
> observers and voters lists.
>
> Alternatively, we could add another field in the response (requiring a KIP)
> to include registered observer nodes that are offline. This would result in
> two separate lists: one for active observer nodes and one for inactive
> observer nodes.
>
> What are your thoughts on this issue?
> [1] https://issues.apache.org/jira/browse/KAFKA-17094
> [2]
>
> https://github.com/apache/kafka/blob/trunk/raft/src/main/java/org/apache/kafka/raft/LeaderState.java#L469
>
>
> Thanks!
> Regards,
> Gantigmaa Selenge
>

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

[Luke Chen]

Hi Mickael,

Thanks for the response.

> 4. Cordoned log directories are persisted to the metadata log via the
RegisterBrokerRecord, BrokerRegistrationChangeRecord records. If a
broker is offline, the controller will use the latest known state of
the broker to determine the broker's cordoned log directories. I've
added a sentence clarifying this point.

OK, so if the broker A goes offline, and the controller is in "fenced"
state, without any cordoned log dirs, then some topic created and assigned
to broker A. Later, broker A starts up with all its log dirs cordoned
configured. At this situation, will the broker A create the partitions?

> 6. I'm leaning towards considering that scenario a configuration
error. If all log directories are cordoned before the internal topics
are created, then the broker will not be able to create them. This
seems like a pretty strange scenario, where it's the first time you
start a broker, you've cordoned all its log directory, and the
internal topics (offsets and transactions) have not yet been created
in the rest of the cluster.

Yes, I agree that this should be a configuration error.
So the follow-up question is: Suppose users encounter this issue, and how
could they resolve it?
Uncordon the log dir dynamically using kafka-configs.sh? Will the
uncordoning config change recreate the partitions we didn't create earlier
because of log dir cordoned?

> The metadata log is different (not managed by LogManager), so I think
it should always be created regardless if its log directory is
cordoned or not.

I agree we should treat "__cluster_metadata" differently.


Thanks.
Luke


On Wed, Jul 10, 2024 at 12:42 AM Mickael Maison 
wrote:

> Hi Luke,
>
> 2. isCordoned() is a new method on LogDirDescription. It does not take
> any arguments. It just returns true if this log directory the
> LogDirDescription represents is cordoned.
>
> 3. Sorry that was a typo. This method will only return a log directory
> that is not cordoned. Fixed
>
> 4. Cordoned log directories are persisted to the metadata log via the
> RegisterBrokerRecord, BrokerRegistrationChangeRecord records. If a
> broker is offline, the controller will use the latest known state of
> the broker to determine the broker's cordoned log directories. I've
> added a sentence clarifying this point.
>
> 5. Yes a log directory can be uncordoned. You can either update the
> properties file and restart the broker or dynamically change the value
> at runtime using kafka-configs. I've added a paragraph about it in the
> KIP.
>
> 6. I'm leaning towards considering that scenario a configuration
> error. If all log directories are cordoned before the internal topics
> are created, then the broker will not be able to create them. This
> seems like a pretty strange scenario, where it's the first time you
> start a broker, you've cordoned all its log directory, and the
> internal topics (offsets and transactions) have not yet been created
> in the rest of the cluster.
> The metadata log is different (not managed by LogManager), so I think
> it should always be created regardless if its log directory is
> cordoned or not.
>
> Thanks,
> Mickael
>
> On Tue, Jul 9, 2024 at 3:48 PM Chia-Ping Tsai  wrote:
> >
> > hi Mickael
> >
> > That is totally a good idea, but I have a question about the
> implementation
> >
> > Do we consider making pluggable ReplicaPlacer (KIP-660) first and then
> add
> > another impl of ReplicaPlacer to offer cordon mechanism? Noted that
> > `ReplicaPlacer` can implement Reconfigurable to get updated at runtime.
> > That is similar to KIP-1066 - change cordoned.log.dirs through configs
> > update.
> >
> > The benefit is to let users have their optimized policy for specific
> > scenario. Also, it can avoid that we add more and more mechanism to our
> > code base. Of course we can merge the mechanism which can be used by 99%
> > users :smile
> >
> > Best,
> > Chia-Ping
> >
> >
> > Luke Chen  於 2024年7月9日 週二 下午9:07寫道：
> >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > > This is a long waiting feature for many users!
> > >
> > > Questions:
> > > 1. I think piggyback the "BrokerHeartbeatRequest" to forward the
> corden log
> > > dir to controller makes sense to me.
> > > We already did similar things for fence, controller shutdown, failed
> log
> > > dir...etc.
> > >
> > > 2. In the admin API, what parameters will the new added isCordoned()
> method
> > > take?
> > >
> > > 3. In the KIP, we said:
> > > "defaultDir(): This method w

Re: [DISCUSS] KIP-1066: Mechanism to cordon brokers and log directories

[Luke Chen]

Hi Mickael,

Thanks for the KIP!
This is a long waiting feature for many users!

Questions:
1. I think piggyback the "BrokerHeartbeatRequest" to forward the corden log
dir to controller makes sense to me.
We already did similar things for fence, controller shutdown, failed log
dir...etc.

2. In the admin API, what parameters will the new added isCordoned() method
take?

3. In the KIP, we said:
"defaultDir(): This method will not return the Uuid of a log directory that
is not cordoned."
--> It's hard to understand. Does that mean we will only return cordoned
log dir?
>From the current java doc of the interface, it doesn't look right:
"Get the default directory for new partitions placed in a given broker."

4. Currently, if a broker is registered and then go offline. In this state,
the controller will still distribute partitions to this broker.
So, if now, the broker get startup with "cordoned.log.dirs" set, what will
happen?
Will the newly assigned partitions be created successfully or not?

5. I think after a log dir get cordoned, we can always uncordon it, right?
I think we should mention it in the KIP.

6. If a broker is startup with "cordoned.log.dirs" set, and does that mean
the internal topics partitions (ex: __consumer_offsets) cannot be created,
either?
Also, if this log dir is happen to be the metadata log dir, what will
happen to the metadata topic creation?

Thanks.
Luke


On Tue, Jul 9, 2024 at 12:12 AM Mickael Maison 
wrote:

> Hi,
>
> Thanks for taking a look.
>
> - Yes you're right, I meant AlterPartitionReassignments. Fixed.
> - That's a good idea. I was expecting users to discover cordoned log
> directories by describing broker configurations. But being able to
> also get this information when describing log directories makes sense.
> I've added that to the KIP.
>
> Thanks,
> Mickael
>
>
> On Fri, Jul 5, 2024 at 8:05 AM Haruki Okada  wrote:
> >
> > Hi,
> >
> > Thank you for the KIP.
> > The motivation sounds make sense to me.
> >
> > I have a few questions:
> >
> > - [nits] "AlterPartitions request" in Error handling section is
> > "AlterPartitionReassignments request" actually, right?
> > - Don't we need to include cordoned information in DescribeLogDirs
> response
> > too? Some tools (e.g. CruiseControl) need to have a way to know which
> > broker/log-dirs are cordoned to generate partition reassignment proposal.
> >
> > Thanks,
> >
> > 2024年7月4日(木) 22:57 Mickael Maison :
> >
> > > Hi,
> > >
> > > I'd like to start a discussion on KIP-1066 that introduces a mechanism
> > > to cordon log directories and brokers.
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1066%3A+Mechanism+to+cordon+brokers+and+log+directories
> > >
> > > Thanks,
> > > Mickael
> > >
> >
> >
> > --
> > 
> > Okada Haruki
> > ocadar...@gmail.com
> > 
>

[jira] [Resolved] (KAFKA-17084) Network Degrade Test fails in System Tests

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17084.
---
Fix Version/s: 3.8.0
   Resolution: Not A Problem

> Network Degrade Test fails in System Tests
> --
>
> Key: KAFKA-17084
> URL: https://issues.apache.org/jira/browse/KAFKA-17084
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Affects Versions: 3.8.0
>Reporter: Josep Prat
>Priority: Critical
> Fix For: 3.8.0
>
> Attachments: TEST-kafka.xml
>
>
> Tests for NetworkDegradeTest fail consistently on the 3.8 branch.
>  
> Tests failing are:
>  
> {noformat}
> Module: kafkatest.tests.core.network_degrade_test
> Class:  NetworkDegradeTest
> Method: test_latency
> Arguments:
> {
>   "device_name": "eth0",
>   "latency_ms": 50,
>   "rate_limit_kbit": 1000,
>   "task_name": "latency-100-rate-1000"
> }
> {noformat}
>  
> and 
>  
> {noformat}
> Module: kafkatest.tests.core.network_degrade_test
> Class:  NetworkDegradeTest
> Method: test_latency
> Arguments:
> {
>   "device_name": "eth0",
>   "latency_ms": 50,
>   "rate_limit_kbit": 0,
>   "task_name": "latency-100"
> }
> {noformat}
>  
> Failure for the first one is:
> {noformat}
> RemoteCommandError({'ssh_config': {'host': 'worker30', 'hostname': 
> '10.140.34.105', 'user': 'ubuntu', 'port': 22, 'password': None, 
> 'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 
> 'hostname': 'worker30', 'ssh_hostname': '10.140.34.105', 'user': 'ubuntu', 
> 'externally_routable_ip': '10.140.34.105', '_logger':  kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_latency.task_name=latency-100-rate-1000.device_name=eth0.latency_ms=50.rate_limit_kbit=1000-1790
>  (DEBUG)>, 'os': 'linux', '_ssh_client':  0x7f17a237dc10>, '_sftp_client':  0x7f17a2393910>, '_custom_ssh_exception_checks': None}, 'ping -i 1 -c 20 
> worker21', 1, b'')
> Traceback (most recent call last):
>   File 
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
>  line 184, in _do_run
> data = self.run_test()
>   File 
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/tests/runner_client.py",
>  line 262, in run_test
> return self.test_context.function(self.test)
>   File 
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/mark/_mark.py",
>  line 433, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/home/semaphore/kafka-overlay/kafka/tests/kafkatest/tests/core/network_degrade_test.py",
>  line 66, in test_latency
> for line in zk0.account.ssh_capture("ping -i 1 -c 20 %s" % 
> zk1.account.hostname):
>   File 
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
>  line 680, in next
> return next(self.iter_obj)
>   File 
> "/home/semaphore/kafka-overlay/kafka/venv/lib/python3.8/site-packages/ducktape/cluster/remoteaccount.py",
>  line 347, in output_generator
> raise RemoteCommandError(self, cmd, exit_status, stderr.read())
> ducktape.cluster.remoteaccount.RemoteCommandError: ubuntu@worker30: Command 
> 'ping -i 1 -c 20 worker21' returned non-zero exit status 1.{noformat}
> And for the second one is:
> {noformat}
> RemoteCommandError({'ssh_config': {'host': 'worker28', 'hostname': 
> '10.140.41.79', 'user': 'ubuntu', 'port': 22, 'password': None, 
> 'identityfile': '/home/semaphore/kafka-overlay/semaphore-muckrake.pem'}, 
> 'hostname': 'worker28', 'ssh_hostname': '10.140.41.79', 'user': 'ubuntu', 
> 'externally_routable_ip': '10.140.41.79', '_logger':  kafkatest.tests.core.network_degrade_test.NetworkDegradeTest.test_latency.task_name=latency-100.device_name=eth0.latency_ms=50.rate_limit_kbit=0-1791
>  (DEBUG)>, 'os': 'linux&#

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Luke Chen]

ite-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/service.py",
> line 345, in run
> self.wait()
>   File
> "/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/background_thread.py",
> line 72, in wait
> super(BackgroundThreadService, self).wait(timeout_sec)
>   File
> "/home/jlprat/projects/kafka/tests/venv39/lib64/python3.9/site-packages/ducktape-0.8.14-py3.9.egg/ducktape/services/service.py",
> line 293, in wait
> raise TimeoutError("Timed out waiting %s seconds for service nodes
> to finish. " % str(timeout_sec)
> ducktape.errors.TimeoutError: Timed out waiting 600 seconds for
> service nodes to finish. These nodes are still alive:
> ['ProducerPerformanceService-0-140496695824336 node 1 on worker3']
>
>
> On Thu, Jul 4, 2024 at 11:57 AM Luke Chen  wrote:
>
> > Hi Josep,
> >
> > For this
> > - QuotaTest --> speaking with Bruno we suspect there is a problem with
> the
> > test setup, failed with "ValueError: max() arg is an empty sequence"
> >
> > It's a known issue: KAFKA-16138
> > <https://issues.apache.org/jira/browse/KAFKA-16138> .
> > It should be passed with local specific tests run.
> > Do you want me help verify it by running it in my environment?
> >
> > Thanks.
> > Luke
> >
> >
> >
> > On Thu, Jul 4, 2024 at 4:03 PM Josep Prat 
> > wrote:
> >
> > > Hi all,
> > >
> > > We have had 2[1][2] runs of the system tests since the last blocker was
> > > merged on 3.8. So far we have 19 tests that failed on both runs. I've
> > > compiled them in this list[3].
> > >
> > > There seems to these different categories of failing tests:
> > > - QuotaTest --> speaking with Bruno we suspect there is a problem with
> > the
> > > test setup, failed with "ValueError: max() arg is an empty sequence"
> > > - Streams cooperative rebalance upgrade --> It fails on versions 2.3.1
> or
> > > older, failed with Timeout
> > > - KRaft Upgrade --> from dev with Isolated and combined KRaft, failed
> > with
> > > RemoteCommandError
> > > - Network degrade test -> failed with RemoteCommandError
> > > - Replica verification tool test --> Timeout for KRaft, but ZK failed
> on
> > > the first run but worked on the second
> > >
> > > If someone has further ideas on what could be causing these failures,
> > > please let me know. Given holidays in the US, the possible test setup
> > > problem might not be able to be fixed today.
> > >
> > > [1]:
> > >
> > >
> >
> https://confluent-open-source-kafka-system-test-results.s3-us-west-2.amazonaws.com/3.8/2024-07-02--001.05d6b151-356a-47e5-b724-6fcd79493422--1719991984--confluentinc--3.8--49d2ee3db9/report.html
> > > [2]:
> > >
> > >
> >
> https://confluent-open-source-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/trunk/2024-07-03--001.4803d99b-52df-4f6d-82c2-3f050a6207fa--1720038529--apache--3.8--2fbe32ecb9/report.html
> > > [3]:
> > >
> > >
> >
> https://docs.google.com/document/d/1wbcyzO6GM2SYQaqTMITBTBjHgZgM7mmiAt7TUfh1xt8/edit
> > >
> > > Best,
> > >
> > > On Tue, Jul 2, 2024 at 7:29 PM Josep Prat  wrote:
> > >
> > > > Hi all,
> > > > Thanks for reviewing and merging the latest blockers for 3.8.0.
> > Tomorrow,
> > > > I will start with the process to get the first RC out.
> > > >
> > > > Best!
> > > >
> > > > On Sat, Jun 29, 2024 at 9:04 PM Josep Prat 
> > wrote:
> > > >
> > > >> Hi Justine,
> > > >>
> > > >> Marking MV 3.8-IV0 as latest
> > > >> production MV is done in this PR (I did both together)
> > > >> https://github.com/apache/kafka/pull/16400
> > > >>
> > > >> Best,
> > > >>
> > > >> --
> > > >> Josep Prat
> > > >> Open Source Engineering Director, Aiven
> > > >> josep.p...@aiven.io   |   +491715557497 | aiven.io
> > > >> Aiven Deutschland GmbH
> > > >> Alexanderufer 3-7, 10117 Berlin
> > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>
> > > >> On Sat, Jun 29, 2024, 00:52 Justine Olshan
> >  > > &g

[jira] [Created] (KAFKA-17080) bump metadata version for topic Record

[Luke Chen (Jira)]

Luke Chen created KAFKA-17080:
-

 Summary: bump metadata version for topic Record
 Key: KAFKA-17080
 URL: https://issues.apache.org/jira/browse/KAFKA-17080
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Luke Chen]

t; >>> >>> >>> > > >>>>
> >>> >>> >>> > > >>>> Best,
> >>> >>> >>> > > >>>> David
> >>> >>> >>> > > >>>>
> >>> >>> >>> > > >>>> Le ven. 14 juin 2024 à 21:57, José Armando García
> Sancio
> >>> >>> >>> > > >>>>  a écrit :
> >>> >>> >>> > > >>>>
> >>> >>> >>> > > >>>>> +1 on the proposed release plan for 3.8.
> >>> >>> >>> > > >>>>>
> >>> >>> >>> > > >>>>> Thanks!
> >>> >>> >>> > > >>>>>
> >>> >>> >>> > > >>>>> On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma <
> >>> >>> m...@ismaeljuma.com
> >>> >>> >>> >
> >>> >>> >>> > > wrote:
> >>> >>> >>> > > >>>>>>
> >>> >>> >>> > > >>>>>> +1 to the plan we converged on in this thread.
> >>> >>> >>> > > >>>>>>
> >>> >>> >>> > > >>>>>> Ismael
> >>> >>> >>> > > >>>>>>
> >>> >>> >>> > > >>>>>> On Fri, Jun 14, 2024 at 10:46 AM Josep Prat
> >>> >>> >>> > > >>  >>> >>> >>> > > >>>>>>
> >>> >>> >>> > > >>>>>> wrote:
> >>> >>> >>> > > >>>>>>
> >>> >>> >>> > > >>>>>>> Hi all,
> >>> >>> >>> > > >>>>>>>
> >>> >>> >>> > > >>>>>>> Thanks Colin, yes go ahead.
> >>> >>> >>> > > >>>>>>>
> >>> >>> >>> > > >>>>>>> As we are now past code freeze I would like to ask
> >>> >>> everyone
> >>> >>> >>> > > involved
> >>> >>> >>> > > >>>>> in a
> >>> >>> >>> > > >>>>>>> KIP that is not yet complete, to verify if what
> >>> landed on
> >>> >>> >>> the 3.8
> >>> >>> >>> > > >>>>> branch
> >>> >>> >>> > > >>>>>>> needs to be reverted or if it can stay.
> Additionally,
> >>> >>> I'll be
> >>> >>> >>> > > pinging
> >>> >>> >>> > > >>>>> KIPs
> >>> >>> >>> > > >>>>>>> and Jira reporters asking for their status as some
> >>> Jiras
> >>> >>> >>> seem to
> >>> >>> >>> > > have
> >>> >>> >>> > > >>>>> all
> >>> >>> >>> > > >>>>>>> related GitHub PRs merged but their status is still
> >>> Open
> >>> >>> or
> >>> >>> >>> In
> >>> >>> >>> > > >>>>> Progress.
> >>> >>> >>> > > >>>>>>> I'll be checking all the open blockers and check if
> >>> they
> >>> >>> are
> >>> >>> >>> > > really a
> >>> >>> >>> > > >>>>>>> blocker or can be pushed.
> >>> >>> >>> > > >>>>>>>
> >>> >>> >>> > > >>>>>>>
> >>> >>> >>> > > >>>>>>> Regarding timeline, I'll attempt to generate the
> >>> first
> >>> >>> RC on
> >>> >>> >>> > > >> Wednesday
> >>> >>> >>> > > >>>>> or
> >>> >>

[jira] [Resolved] (KAFKA-17069) Remote copy throttle metrics

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-17069.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> Remote copy throttle metrics 
> -
>
> Key: KAFKA-17069
> URL: https://issues.apache.org/jira/browse/KAFKA-17069
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Abhijeet Kumar
>Priority: Major
> Fix For: 3.9.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17070) perf: consider to use ByteBufferOutputstream to append records

[Luke Chen (Jira)]

Luke Chen created KAFKA-17070:
-

 Summary: perf: consider to use ByteBufferOutputstream to append 
records
 Key: KAFKA-17070
 URL: https://issues.apache.org/jira/browse/KAFKA-17070
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


Consider to use ByteBufferOutputstream to append records, instead of a 
DataOutputStream. We should add JMH test to confirm this indeed improve the 
performance before merging it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Apache Kafka 3.7.1

[Luke Chen]

Thanks Igor for running this release!

Luke

On Mon, Jul 1, 2024 at 5:45 PM Mickael Maison 
wrote:

> Congratulations!
>
> Thanks Igor for running the release.
>
> Mickael
>
> On Mon, Jul 1, 2024 at 11:26 AM Igor Soarez  wrote:
> >
> > The Apache Kafka community is pleased to announce the release for Apache
> Kafka 3.7.1
> >
> > This is a bug fix release and it includes fixes and improvements.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.7.1/RELEASE_NOTES.html
> >
> >
> > You can download the source and binary release (Scala 2.12 and 2.13)
> from:
> > https://kafka.apache.org/downloads#3.7.1
> >
> >
> ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records
> to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 1086 contributors to this release!
> (Please report an unintended omission)
> >
> > Adrian Preston, Anatoly Popov, Andras Katona, Andrew Schofield, Anna
> Sophie Blee-Goldman, Anton Liauchuk, Apoorv Mittal, Ayoub Omari, Bill
> Bejeck, Bruno Cadonna, Calvin Liu, Cameron Redpath, Cheng-Kai Zhang,
> Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe, David Arthur, David
> Jacot, Divij Vaidya, Dmitry Werner, Edoardo Comar, flashmouse, Florin
> Akermann, Gantigmaa Selenge, Gaurav Narula, Greg Harris, Igor Soarez,
> ilyazr, Ismael Juma, Jason Gustafson, Jeff Kim, jiangyuan, Joel Hamill,
> John Yu, Johnny Hsu, José Armando García Sancio, Josep Prat, Jun Rao,
> Justine Olshan, Kamal Chandraprakash, Ken Huang, Kuan-Po (Cooper) Tseng,
> Lokesh Kumar, Luke Chen, Manikumar Reddy, Mario Pareja, Matthias J. Sax,
> Mayank Shekhar Narula, Mickael Maison, Murali Basani, Omnia Ibrahim, Paolo
> Patierno, PoAn Yang, Sagar Rao, sanepal, Sean Quah, Sebastian Marsching,
> Stanislav Kozlovski, Vedarth Sharma, Walker Carlson, Yash Mayya
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > https://kafka.apache.org/
> >
> > Thank you!
> >
> >
> > Regards,
> >
> > --
> > Igor Soarez
> > Release Manager for Apache Kafka 3.7.1
>

Re: [VOTE] 3.7.1 RC2

[Luke Chen]

Hi Igor,

I think we've passed the vote for the release.
Are we waiting for anything to release v3.7.1?

Please let me know if you need help.

Thanks.
Luke

On Wed, Jun 26, 2024 at 5:17 AM Justine Olshan 
wrote:

> Hey folks,
>
> Sorry for the delay. I have done the following:
>
>
> 1. Checked signatures
> 2. Built from source
> 3. Inspected unit/integration test failures
> 4. Scanned documentation and other artifacts
> 5. Ran ZK and Kraft quick starts and simple workloads on staged binary for
> 2.13
>
> +1 (binding) from me.
> Thanks,
>
> Justine
>
> On Fri, Jun 21, 2024 at 11:03 AM Jakub Scholz  wrote:
>
> > +1 (non-binding). I used the staged binaries (based on Scala 2.13) and
> > Maven artifacts to run my tests. All seems to work fine.
> >
> > Thanks & Regards
> > Jakub
> >
> > On Wed, Jun 19, 2024 at 10:55 AM Igor Soarez  wrote:
> >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 3.7.1.
> > >
> > > This is a bugfix release with several fixes.
> > >
> > > Release notes for the 3.7.1 release:
> > > https://home.apache.org/~soarez/kafka-3.7.1-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Friday June 28, 11am UTC.
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~soarez/kafka-3.7.1-rc2/
> > >
> > > * Docker release artifact to be voted upon:
> > > apache/kafka:3.7.1-rc2
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~soarez/kafka-3.7.1-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 3.7 branch) is the 3.7.1 tag:
> > > https://github.com/apache/kafka/releases/tag/3.7.1-rc2
> > >
> > > * Documentation:
> > > https://kafka.apache.org/37/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/37/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.7 branch:
> > > Unit/integration tests:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/184/
> > > The latest test run includes some flaky tests, all of which were
> > confirmed
> > > to pass locally.
> > >
> > > System tests:
> > > I don't have access to the Jenkins instance used for system tests in
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7
> > > Luke has kindly shared results in the previous RC (thank you Luke!),
> > > and all issues have been addressed.
> > > If anyone with access is able to confirm the latest test results,
> please
> > > reply with details.
> > >
> > > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > > Docker Build Test Pipeline:
> > > https://github.com/apache/kafka/actions/runs/9572915509
> > >
> > > /**
> > >
> > > Thanks,
> > >
> > > --
> > > Igor Soarez
> > >
> >
>

[jira] [Resolved] (KAFKA-16781) Expose advertised.listeners in controller node

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16781.
---
Fix Version/s: 3.9.0
   Resolution: Fixed

> Expose advertised.listeners in controller node
> --
>
> Key: KAFKA-16781
> URL: https://issues.apache.org/jira/browse/KAFKA-16781
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Luke Chen
>Assignee: Gantigmaa Selenge
>Priority: Major
>  Labels: need-kip, newbie, newbie++
> Fix For: 3.9.0
>
>
> After 
> [KIP-919|https://cwiki.apache.org/confluence/display/KAFKA/KIP-919%3A+Allow+AdminClient+to+Talk+Directly+with+the+KRaft+Controller+Quorum+and+add+Controller+Registration],
>  we allow clients to talk to the KRaft controller node directly. But unlike 
> broker node, we don't allow users to config advertised.listeners for clients 
> to connect to. Without this config, the client cannot connect to the 
> controller node if the controller is sitting behind NAT network while the 
> client is in the external network.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-17035) Add debug log to retention cleanupLogs method to help troubleshoot issues

[Luke Chen (Jira)]

Luke Chen created KAFKA-17035:
-

 Summary: Add debug log to retention cleanupLogs method to help 
troubleshoot issues
 Key: KAFKA-17035
 URL: https://issues.apache.org/jira/browse/KAFKA-17035
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


Sometimes, there will be some log segments leftover without getting deleted as 
configured. One example is, the config is set to retention.ms = 7 days, but 
there are some log segments stay in storage for more than 7 days. In this case, 
it's difficult to troubleshoot if there is a bug in kafka or something wrong in 
the log segment (ex: max time is wrongly set).

To help troubleshooting, we could add "trace level" or "debug level" logs when 
doing retentionSize/retentionMs/logStartOffset check for each log segment.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

[Luke Chen]

Hi Fede,

Thanks for the KIP!
+1 from me.

Luke

On Fri, Jun 21, 2024 at 6:44 PM Federico Valeri 
wrote:

> Hi all, I'd like to kick off a vote on KIP-1057.
>
> Design doc:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool
>
> Discussion thread:
> https://lists.apache.org/thread/kxx1h4qwshgcjh4d5xzqltkx5mx9qopm
>
> Thanks,
> Fede
>

Re: [DISCUSS] KIP-1052: Enable warmup in producer performance test

[Luke Chen]

Hi Matt,

Thanks for the KIP!
I agree having the warm-up records could help correctly analyze the
performance.

Some questions:
1. It looks like we will add 2 more options to producer perf tool:
 - --warmup-records
 - --combined-summary

Is this correct?
In the "public interface" section, only 1 is mentioned. Could you update it?
Also, in the KIP, you use the word: "An option such as "--warmup-records"
should be added...", it sounds like it is not decided, yet.
I suggest you update to say, we will add "--warmup-records" option for
" to make it clear.

2. What will be the output containing both warm-up and steady-state results?
Could you give an example directly?

For better understanding, I'd suggest you refer to KIP-1057

to add some examples using `kafka-producer-perf-test.sh` with the new
option, to show what it will output.

Thank you.
Luke

On Fri, Jun 21, 2024 at 10:39 AM Welch, Matt  wrote:

> Hi Divij,
>
> Thanks for your response.  You raise some very important points.
> I've updated the KIP to clarify the changes discussed here.
>
> 1. I agree that warmup stats should be printed separately.  I see two
> cases here, both of which would have two summary lines printed at the end
> of the producer perf test.  In the first case, warmup-separate, the warmup
> stats are printed first as warmup-only, followed by a second print of the
> steady state performance. In the second case, warmup-combined, the first
> print would look identical to the summary line that's currently used and
> would reflect the "whole test", with a second summary print of the
> steady-state performance.  This second case would allow for users to
> compare what the test would have looked like without a warmup to results of
> the test with a warmup. Although I've been looking at the second case
> lately, I can see merits of both cases and would be happy to support the
> warmup-separate case if that's the preference of the community.  Regarding
> the JMX metrics accumulated by Kafka, we need to decide if we should reset
> the JMX metrics between the warmup and steady state. While I like the idea
> of having separate JMX buckets for warmup and steady state, these
> statistics are usually heavily windowed, so should naturally settle toward
> steady-state values after a warmup.
>
> 2. The total number of records sent by the producer and defined by
> '--num-records' MUST be strictly greater than the '--warmup-records' or an
> error will be thrown. Negative warmup records should similarly throw an
> error.  Specifying warmup-records of "0" should have behavior identical to
> the current implementation.
>
> 3.  You're correct that choosing the warmup duration can have a
> significant impact on the test output if care is not taken.  I've updated
> the proposed change to describe a simplistic process to choose how many
> warmup records to use.  Without understanding all the factors that go into
> a warmup, a user could run a test and watch the time series output of the
> producer test to determine when steady state has been reached and warmup
> has completed.  The number of records at which the producer hits steady
> state could then be used in subsequent tests. In practice, we find that 1
> minute is a good warmup for most cases, since aside from networking and
> storage initialization, even the JVM should be warmed up by then and using
> compiled code rather than interpreted byte code. This is more a heuristic,
> however, and measured latency and throughput of the system should be used
> to determine steady state.
>
> 4.  The current design has the user specifying the warmup records like
> they would specify the number of records for the test. While this is
> related to the throughput, it seemed a better option to have the user
> specify the number of records in the warmup, rather than some kind of
> duration which would be more complex to track. I completely agree with your
> concern of warmup affecting steady state, however, especially in short
> tests. With a warmup "removing" some of the high latency from steady state
> results, it could be tempting for users to run very short tests since they
> no longer need to wait long to achieve a repeatable steady-state result. I
> would consider this a case of insufficient warmup since Kafka could still
> be processing the warmup records as you mention. Best practice for warmup
> duration would be to hit steady state during the warmup and only then
> consider it a successful warmup. Our preferred process is to monitor
> producer latency until it hits steady state in a first test, then double
> that duration for the warmup in subsequent testing. One minute is usually
> sufficient. A problem does occur when using unlimited throughput since the
> user does not yet know how fast the producers will send so can't estimate
> warmup records. If the iterative testing described above is not 

[jira] [Resolved] (KAFKA-16531) Fix check-quorum calculation to not assume that the leader is in the voter set

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-16531.
---
Resolution: Fixed

> Fix check-quorum calculation to not assume that the leader is in the voter set
> --
>
> Key: KAFKA-16531
> URL: https://issues.apache.org/jira/browse/KAFKA-16531
> Project: Kafka
>  Issue Type: Sub-task
>  Components: kraft
>Reporter: José Armando García Sancio
>    Assignee: Luke Chen
>Priority: Major
> Fix For: 3.9.0
>
>
> In the check-quorum calculation, the leader should not assume that it is part 
> of the voter set. This may happen when the leader is removing itself from the 
> voter set.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.1 RC2

[Luke Chen]

Hi Igor,

I had run system tests against this RC2, and all passed except:
kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
KAFKA-16962

--> This test tested the upgrade from version [0.9 ~ 3.6] to v3.7, and only
v0.9 failed, others all passed. Given v0.9 version is a very old version,
this issue should not be the blocker for this release.

So, the system tests look good.

I also did:
1. ran the quickstart with the scala 2.12 binaries
2. check checksums.
3. browser javadoc

+1 from me.

Thank you.
Luke



On Wed, Jun 19, 2024 at 11:39 PM Mickael Maison 
wrote:

> Hi Krishna,
>
> Thanks for the clarification!
> In that case +1 (binding)
>
> Thanks,
> Mickael
>
> On Wed, Jun 19, 2024 at 5:30 PM Krishna Agarwal
>  wrote:
> >
> > Hi Mickael,
> >
> > The Docker Image CVE scan report can be found in the Docker Build Test
> > Pipeline[0] shared(in the initial vote email). I see there are no High or
> > Critical CVEs.
> >
> > [0]: https://github.com/apache/kafka/actions/runs/9572915509
> >
> > Regards,
> > Krishna
> >
> > On Wed, Jun 19, 2024 at 8:14 PM Mickael Maison  >
> > wrote:
> >
> > > Hi Igor,
> > >
> > > I did the following:
> > > - Checked signatures and checksums
> > > - Ran the quickstart with the 2.13 binaries and the Docker image
> > > - Built and run the tests from source
> > > - Quickly browsed the javadoc
> > >
> > > It all looks good, but before voting, could you run the Docker Image
> > > CVE Scanner GitHub action [0] on the new image to check it's CVE free?
> > > It looks like we don't have this action in the 3.7 branch so it'll
> > > probably involve some manual steps.
> > >
> > > 0: https://github.com/apache/kafka/actions/workflows/docker_scan.yml
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > > Mickael
> > >
> > > On Wed, Jun 19, 2024 at 10:55 AM Igor Soarez 
> wrote:
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the second candidate for release of Apache Kafka 3.7.1.
> > > >
> > > > This is a bugfix release with several fixes.
> > > >
> > > > Release notes for the 3.7.1 release:
> > > > https://home.apache.org/~soarez/kafka-3.7.1-rc2/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Friday June 28, 11am UTC.
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~soarez/kafka-3.7.1-rc2/
> > > >
> > > > * Docker release artifact to be voted upon:
> > > > apache/kafka:3.7.1-rc2
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~soarez/kafka-3.7.1-rc2/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.7 branch) is the 3.7.1 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.7.1-rc2
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/37/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/37/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.7 branch:
> > > > Unit/integration tests:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.7/184/
> > > > The latest test run includes some flaky tests, all of which were
> > > confirmed to pass locally.
> > > >
> > > > System tests:
> > > > I don't have access to the Jenkins instance used for system tests in
> > > > https://jenkins.confluent.io/job/system-test-kafka/job/3.7
> > > > Luke has kindly shared results in the previous RC (thank you Luke!),
> > > > and all issues have been addressed.
> > > > If anyone with access is able to confirm the latest test results,
> please
> > > > reply with details.
> > > >
> > > > * Successful Docker Image Github Actions Pipeline for 3.7 branch:
> > > > Docker Build Test Pipeline:
> > > https://github.com/apache/kafka/actions/runs/9572915509
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Igor Soarez
> > >
>