Re: [ANNOUNCE] Apache Kafka 3.9.0

[Satish Duggana]

Thanks Colin for all your hard work on running the 3.9.0 release.
Thanks to all the contributors to this release.

~Satish.


~Satish.

On Fri, 8 Nov 2024 at 04:42, Colin McCabe  wrote:
>
> The Apache Kafka community is pleased to announce the release for Apache 
> Kafka 3.9.0
>
> - This is a major release, the final one in the 3.x line. (There may of 
> course be other minor releases in this line, such as 3.9.1.)
> - Tiered storage will be considered production-ready in this release.
> - This will be the final major release to feature the deprecated ZooKeeper 
> mode.
>
> This release includes the following KIPs:
> - KIP-853: Support dynamically changing KRaft controller membership
> - KIP-1057: Add remote log metadata flag to the dump log tool
> - KIP-1049: Add config log.summary.interval.ms to Kafka Streams
> - KIP-1040: Improve handling of nullable values in InsertField, ExtractField, 
> and other transformations
> - KIP-1031: Control offset translation in MirrorSourceConnector
> - KIP-1033: Add Kafka Streams exception handler for exceptions occurring 
> during processing
> - KIP-1017: Health check endpoint for Kafka Connect
> - KIP-1025: Optionally URL-encode clientID and clientSecret in authorization 
> header
> - KIP-1005: Expose EarliestLocalOffset and TieredOffset
> - KIP-950: Tiered Storage Disablement
> - KIP-956: Tiered Storage Quotas
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.9.0/RELEASE_NOTES.html
>
> An overview of the release can be found in our announcement blog post:
> https://kafka.apache.org/blog#apache_kafka_390_release_announcement
>
> You can download the source and binary release from:
> https://kafka.apache.org/downloads#3.9.0
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 133 contributors to this release!
> (Please report an unintended omission)
>
> Abhijeet Kumar, abhi-ksolves, Abhinav Dixit, Adrian Preston, Alieh Saeedi,
> Alyssa Huang, Anatoly Popov, Andras Katona, Andrew Schofield, Andy Wilkinson,
> Anna Sophie Blee-Goldman, Antoine Pourchet, Apoorv Mittal, Arnav Dadarya,
> Arnout Engelen, Arpit Goyal, Arun Mathew, A. Sophie Blee-Goldman, Ayoub Omari,
> bachmanity1, Bill Bejeck, brenden20, Bruno Cadonna, Chia Chuan Yu, Chia-Ping
> Tsai, ChickenchickenLove, Chirag Wadhwa, Chris Egerton, Christo Lolov,
> Ming-Yen Chung, Colin P. McCabe, Cy, David Arthur, David Jacot,
> demo...@csie.io, dengziming, Dimitar Dimitrov, Dmitry Werner, Dongnuo Lyu,
> dujian0068, Edoardo Comar, Farbod Ahmadian, Federico Valeri, Fiore Mario
> Vitale, Florin Ak ermann, Francois Visconte, GANESH SADANALA, Gantigmaa
> Selenge, Gaurav Narula, gongxuanzhang, Greg Harris, Do Gyeongwon, Harry
> Fallows, Hongten, Ian McDonald, Igor Soarez, Ismael Juma, Ivan Yurchenko,
> Jakub Scholz, Jason Gustafson, Jeff Kim, Jim Galasyn, Jin yong Choi, Johnny
> Hsu, José Armando García Sancio, Josep Prat, Jun Rao, Justine Olshan, Kamal
> Chandraprakash, Ken Huang, Kevin Wu, Kirk True, Kondrat Bertalan, Krishna
> Agarwal, KrishVora01, Kuan-Po (Cooper) Tseng, Kuan-Po Tseng, Lee Dongjin,
> Lianet Magrans , Logan Zhu, Loïc GREFFIER, Lucas Brutschy, Luke Chen, Maciej
> Moscicki, Manikumar Reddy, Mason Chen, Matthias J. Sax, Max Riedel, Mickael
> Maison, Murali Basani, 120571969+nancy-k

Re: [VOTE] KIP-1105: Make remote log manager thread-pool configs dynamic

[Satish Duggana]

+1 (binding)

On Thu, 7 Nov 2024 at 13:55, Federico Valeri  wrote:
>
> +1 (non binding)
>
> Thanks
>
> On Thu, Nov 7, 2024 at 5:46 AM Chu Cheng Li  wrote:
> >
> > +1
> >
> > On Thu, Nov 7, 2024 at 12:39 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call for a vote on KIP-1105
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic
> > >
> > > Thanks,
> > > Kamal
> > >

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

[Satish Duggana]

Thanks for updating the KIP by addressing with the review comments.

On Thu, 7 Nov 2024 at 21:38, Kamal Chandraprakash
 wrote:
>
> Hi Satish,
>
> Thanks for the review! Yes, we won't be enabling the deprecated config as
> dynamic.
> Removed the remote.log.manager.thread.pool.size config from the KIP.
>
> Currently, the config is not marked as deprecated which was
> already proposed in
> KIP-950, we can mark that config as deprecated in the source code.
>
> Thanks,
> Kamal
>
> On Thu, Nov 7, 2024 at 9:23 PM Satish Duggana 
> wrote:
>
> > Thanks Kamal for the KIP. This is useful for dynamically changing the
> > thread pool configurations, especially in production environments. We
> > can skip remote.log.manager.thread.pool.size as it is already
> > deprecated, and remote.log.manager.copier.thread.pool.size,
> > remote.log.manager.expiration.thread.pool.size  are derived from that.
> >
> > ~Satish.
> >
> >
> > On Thu, 7 Nov 2024 at 10:07, Kamal Chandraprakash
> >  wrote:
> > >
> > > Hi all,
> > >
> > > If there are no more comments, then I'll start a voting thread as the
> > > change is minor.
> > >
> > > --
> > > Kamal
> > >
> > > On Thu, Nov 7, 2024 at 9:00 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi Federico,
> > > >
> > > > Updated the KIP by replacing the `isInitialized` to `isReady` in the
> > KIP.
> > > >
> > > >
> > > >
> > > > On Wed, Nov 6, 2024 at 12:47 PM Federico Valeri 
> > > > wrote:
> > > >
> > > >> Thanks Kamal, LGTM, but you should replace all instances of
> > > >> isInitialized to isReady in the rest of the KIP.
> > > >>
> > > >> On Wed, Nov 6, 2024 at 5:22 AM Kamal Chandraprakash
> > > >>  wrote:
> > > >> >
> > > >> > Hi Federico,
> > > >> >
> > > >> > Thanks for the review!
> > > >> >
> > > >> > 1. Changed the API name to `isReady`
> > > >> > 2. Added an example of stacktrace in the KIP.
> > > >> >
> > > >> > PTAL.
> > > >> >
> > > >> > Thanks,
> > > >> > Kamal
> > > >> >
> > > >> > On Mon, Nov 4, 2024 at 2:37 PM Federico Valeri <
> > fedeval...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > Hi Kamal, these changes make sense to me. Thanks.
> > > >> > >
> > > >> > > In this case, I wonder if "isReady" could be a better name,
> > instead of
> > > >> > > "isInitialized". Wdyt?
> > > >> > >
> > > >> > > Could you please add an example of the stack trace that the RLMM
> > can
> > > >> > > raise during the initialization phase?
> > > >> > >
> > > >> > > On Sun, Nov 3, 2024 at 4:50 PM Kamal Chandraprakash
> > > >> > >  wrote:
> > > >> > > >
> > > >> > > > Hi all,
> > > >> > > >
> > > >> > > > I would like to start a discussion thread on KIP-1105
> > > >> > > > <
> > > >> > >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic
> > > >> > > >.
> > > >> > > > This KIP is about
> > > >> > > >
> > > >> > > > 1. Configuring the thread-pool used by the remote-log manager
> > > >> dynamically
> > > >> > > > and
> > > >> > > > 2. Graceful handling of remote-log components during server
> > startup.
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic
> > > >> > > >
> > > >> > > > Please take a look and suggest your thoughts.
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Kamal
> > > >> > >
> > > >>
> > > >
> >

Re: [DISCUSS] KIP-1105: Make remote log manager thread-pool configs dynamic

[Satish Duggana]

Thanks Kamal for the KIP. This is useful for dynamically changing the
thread pool configurations, especially in production environments. We
can skip remote.log.manager.thread.pool.size as it is already
deprecated, and remote.log.manager.copier.thread.pool.size,
remote.log.manager.expiration.thread.pool.size  are derived from that.

~Satish.


On Thu, 7 Nov 2024 at 10:07, Kamal Chandraprakash
 wrote:
>
> Hi all,
>
> If there are no more comments, then I'll start a voting thread as the
> change is minor.
>
> --
> Kamal
>
> On Thu, Nov 7, 2024 at 9:00 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi Federico,
> >
> > Updated the KIP by replacing the `isInitialized` to `isReady` in the KIP.
> >
> >
> >
> > On Wed, Nov 6, 2024 at 12:47 PM Federico Valeri 
> > wrote:
> >
> >> Thanks Kamal, LGTM, but you should replace all instances of
> >> isInitialized to isReady in the rest of the KIP.
> >>
> >> On Wed, Nov 6, 2024 at 5:22 AM Kamal Chandraprakash
> >>  wrote:
> >> >
> >> > Hi Federico,
> >> >
> >> > Thanks for the review!
> >> >
> >> > 1. Changed the API name to `isReady`
> >> > 2. Added an example of stacktrace in the KIP.
> >> >
> >> > PTAL.
> >> >
> >> > Thanks,
> >> > Kamal
> >> >
> >> > On Mon, Nov 4, 2024 at 2:37 PM Federico Valeri 
> >> wrote:
> >> >
> >> > > Hi Kamal, these changes make sense to me. Thanks.
> >> > >
> >> > > In this case, I wonder if "isReady" could be a better name, instead of
> >> > > "isInitialized". Wdyt?
> >> > >
> >> > > Could you please add an example of the stack trace that the RLMM can
> >> > > raise during the initialization phase?
> >> > >
> >> > > On Sun, Nov 3, 2024 at 4:50 PM Kamal Chandraprakash
> >> > >  wrote:
> >> > > >
> >> > > > Hi all,
> >> > > >
> >> > > > I would like to start a discussion thread on KIP-1105
> >> > > > <
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic
> >> > > >.
> >> > > > This KIP is about
> >> > > >
> >> > > > 1. Configuring the thread-pool used by the remote-log manager
> >> dynamically
> >> > > > and
> >> > > > 2. Graceful handling of remote-log components during server startup.
> >> > > >
> >> > > >
> >> > >
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1105%3A+Make+remote+log+manager+thread-pool+configs+dynamic
> >> > > >
> >> > > > Please take a look and suggest your thoughts.
> >> > > >
> >> > > > Thanks,
> >> > > > Kamal
> >> > >
> >>
> >

Re: [VOTE] KIP-1058: Txn consumer exerts pressure on remote storage when reading non-txn topic

[Satish Duggana]

+1

Thanks Kamal for the KIP. Discussed offline with Kamal on the proposal and
the changes.
It improves the performance of consumers' remote reads in read_committed
mode. It brings minor changes in a backward compatible manner.

~Satish.

On Mon, 4 Nov 2024 at 19:33, Divij Vaidya  wrote:

> +1
>
> (I have participated in the review and I think we have a good solution in
> the KIP)
>
> --
> Divij Vaidya
>
>
>
> On Mon, Nov 4, 2024 at 5:41 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Hi all,
> >
> > Bumping up the voting thread.
> >
> > On Mon, Sep 30, 2024 at 3:27 PM Francois Visconte
> >  wrote:
> >
> > > Could we vote on this? This is causing a bunch of tiered storage read
> > > issues as many consumers default to READ_COMMITTED (eg. librdkafka)
> > >
> > > Thanks,
> > > F.
> > >
> > > On Mon, Sep 16, 2024 at 7:20 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Bumping this thread for vote. PTAL.
> > > >
> > > > On Mon, Sep 9, 2024 at 2:01 PM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to open voting for KIP-1058. This KIP improves the
> consumer
> > > > > reading from remote storage when READ_COMMITTED isolation level is
> > > > enabled.
> > > > > PTAL.
> > > > >
> > > > > KIP-1058
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1058%3A+Txn+consumer+exerts+pressure+on+remote+storage+when+collecting+aborted+transactions
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Kamal
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Satish Duggana]

Congratulations Kamal!

On Mon, 30 Sept 2024 at 18:18, Josep Prat 
wrote:

> Congrats Kamal!
>
> On Mon, Sep 30, 2024 at 2:46 PM  wrote:
>
> > Congrats Kamal!
> >
> > -
> > Gaurav
> >
> > > On 30 Sep 2024, at 18:10, Mickael Maison 
> > wrote:
> > >
> > > Congratulations Kamal!
> > >
> > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> The PMC of Apache Kafka is pleased to announce a new Kafka committer,
> > Kamal
> > >> Chandraprakash.
> > >>
> > >> Kamal has been a Kafka contributor since May 2017. He has made
> > significant
> > >> contributions to the tiered storage feature (KIP-405). He authored
> > KIP-1018
> > >> and KIP-1075 which improved tiered storage operation. He also
> > contributed
> > >> to discussing and reviewing many KIPs.
> > >>
> > >> Congratulations, Kamal!
> > >>
> > >> Thanks,
> > >> Luke (on behalf of the Apache Kafka PMC)
> >
> >
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> Anna Richardson, Kenneth Chen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] Apache Kafka 4.0.0 release

[Satish Duggana]

+1
Thanks David for volunteering!

On Mon, 23 Sept 2024 at 19:27, Lianet M.  wrote:
>
> +1
>
> Thanks David!
>
> On Mon, Sep 23, 2024, 9:54 a.m. Justine Olshan 
> wrote:
>
> > +1 and thanks!
> >
> > On Mon, Sep 23, 2024 at 6:36 AM Chia-Ping Tsai  wrote:
> >
> > > +1 and thanks David for volunteering!
> > >
> > > Best,
> > > Chia-Ping
> > >
> > > Josep Prat  於 2024年9月23日 週一 下午9:13寫道：
> > >
> > > > Thanks David for volunteering!
> > > > +1
> > > >
> > > > On Mon, Sep 23, 2024 at 3:11 PM David Jacot
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I would like to volunteer to be the release manager driving the next
> > > > > release - Apache Kafka 4.0.0.
> > > > >
> > > > > If there are no objections, I'll start building a release plan in the
> > > > wiki
> > > > > in the next couple of days.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > >
> > > >
> > > > --
> > > > [image: Aiven] 
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.p...@aiven.io   |   +491715557497
> > > > aiven.io    |   <
> > > https://www.facebook.com/aivencloud
> > > > >
> > > >      <
> > > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > Anna Richardson, Kenneth Chen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
> >

[jira] [Resolved] (KAFKA-15434) Tiered Storage Quotas

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15434.

Resolution: Duplicate

Duplicate of https://issues.apache.org/jira/browse/KAFKA-15265

> Tiered Storage Quotas 
> --
>
> Key: KAFKA-15434
> URL: https://issues.apache.org/jira/browse/KAFKA-15434
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Satish Duggana
>Assignee: Abhijeet Kumar
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Jeff Kim

[Satish Duggana]

Congratulations Jeff!

On Mon, 9 Sept 2024 at 18:37, Bruno Cadonna  wrote:
>
> Congrats! Well deserved!
>
> Best,
> Bruno
>
>
>
> On 9/9/24 2:28 PM, Bill Bejeck wrote:
> > Congrats Jeff!!
> >
> > On Mon, Sep 9, 2024 at 7:50 AM Lianet M.  wrote:
> >
> >> Congrats Jeff!!!
> >>
> >> On Mon, Sep 9, 2024, 7:05 a.m. Chris Egerton 
> >> wrote:
> >>
> >>> Congrats!
> >>>
> >>> On Mon, Sep 9, 2024, 06:36 Rajini Sivaram 
> >> wrote:
> >>>
>  Congratulations, Jeff!
> 
>  Regards,
> 
>  Rajini
> 
>  On Mon, Sep 9, 2024 at 10:49 AM Luke Chen  wrote:
> 
> > Congrats, Jeff!
> >
> > On Mon, Sep 9, 2024 at 5:19 PM Viktor Somogyi-Vass
> >  wrote:
> >
> >> Congrats Jeff!
> >>
> >> On Mon, Sep 9, 2024, 11:02 Yash Mayya 
> >> wrote:
> >>
> >>> Congratulations Jeff!
> >>>
> >>> On Mon, 9 Sept, 2024, 12:13 David Jacot, 
> >> wrote:
> >>>
>  Hi all,
> 
>  The PMC of Apache Kafka is pleased to announce a new Kafka
>  committer,
> >>> Jeff
>  Kim.
> 
>  Jeff has been a Kafka contributor since May 2020. In addition
> >> to
> > being
>  a regular contributor and reviewer, he has made significant
>  contributions to the next generation of the consumer rebalance
>  protocol (KIP-848) and to the new group coordinator. He
> >> authored
>  KIP-915 which improved how coordinators can be downgraded. He
> >>> also
>  contributed multiple fixes/improvements to the fetch from
> >>> follower
>  feature.
> 
>  Congratulations, Jeff!
> 
>  Thanks,
>  David (on behalf of the Apache Kafka PMC)
> 
> >>>
> >>
> >
> 
> >>>
> >>
> >

Re: [ANNOUNCE] New committer: Lianet Magrans

[Satish Duggana]

Congratulations Lianet!!

On Thu, 29 Aug 2024 at 10:08, Muralidhar Basani
 wrote:
>
> Congratulations Lianet !!
>
> Murali
>
> On Thu, 29 Aug 2024 at 05:48, Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > Congrats Lianet!
> >
> > On Thu, Aug 29, 2024 at 8:07 AM Luke Chen  wrote:
> >
> > > Congratulations Lianet!
> > >
> > > Luke
> > >
> > > On Thu, Aug 29, 2024 at 9:54 AM Lianet M.  wrote:
> > >
> > > > Thank you very much everyone! It truly takes the great shared knowledge
> > > you
> > > > all put out there with amazing reviews and discussions.
> > > >
> > > > Looking forward to continuing working with you all!
> > > >
> > > > Lianet
> > > >
> > > > On Wed, Aug 28, 2024, 7:12 p.m. Sophie Blee-Goldman <
> > > sop...@responsive.dev
> > > > >
> > > > wrote:
> > > >
> > > > > Congrats Lianet! Well deserved
> > > > >
> > > > > On Wed, Aug 28, 2024 at 2:27 PM Jiashen Zhang <
> > jiashenzz...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congratulations Lianet! Awesome!
> > > > > >
> > > > > > On Wed, Aug 28, 2024 at 2:10 PM Andrew Schofield <
> > > > > > andrew_schofi...@live.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Awesome news. Well done, Lianet.
> > > > > > >
> > > > > > > Andrew
> > > > > > >
> > > > > > > > On 28 Aug 2024, at 17:47, Bruno Cadonna 
> > > > wrote:
> > > > > > > >
> > > > > > > > Well deserved!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Bruno
> > > > > > > >
> > > > > > > > On 8/28/24 6:37 PM, Bill Bejeck wrote:
> > > > > > > >> Congrats Lianet!
> > > > > > > >> On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax <
> > > > mj...@apache.org>
> > > > > > > wrote:
> > > > > > > >>> Congrats! Very well deserved!
> > > > > > > >>>
> > > > > > > >>> On 8/28/24 9:18 AM, Justine Olshan wrote:
> > > > > > >  Congratulations Lianet!!
> > > > > > >  Justine
> > > > > > > 
> > > > > > >  On Wed, Aug 28, 2024 at 8:58 AM David Arthur <
> > > mum...@gmail.com>
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > Congrats, Lianet!
> > > > > > > >
> > > > > > > > On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison <
> > > > > > > >>> mickael.mai...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Congratulations Lianet!
> > > > > > > >>
> > > > > > > >> On Wed, Aug 28, 2024 at 5:40 PM Josep Prat
> > > > > > >  > > > > > > 
> > > > > > > >> wrote:
> > > > > > > >>>
> > > > > > > >>> Congrats Lianet!
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 28, 2024 at 5:38 PM Chia-Ping Tsai <
> > > > > > > chia7...@apache.org>
> > > > > > > >> wrote:
> > > > > > > >>>
> > > > > > >  Congratulations, Lianet!!!
> > > > > > > 
> > > > > > >  On 2024/08/28 15:35:23 David Jacot wrote:
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > The PMC of Apache Kafka is pleased to announce a new
> > > Kafka
> > > > > > > > committer,
> > > > > > > > Lianet Magrans.
> > > > > > > >
> > > > > > > > Lianet has been a Kafka contributor since June 2023. In
> > > > > > addition
> > > > > > > to
> > > > > > > > being a regular contributor and reviewer, she has made
> > > > > > > significant
> > > > > > > > contributions to the next generation of the consumer
> > > > > rebalance
> > > > > > > > protocol (KIP-848) and to the new consumer. She has
> > also
> > > > > > > > contributed
> > > > > > > > to discussing and reviewing many KIPs.
> > > > > > > >
> > > > > > > > Congratulations, Lianet!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > David (on behalf of the Apache Kafka PMC)
> > > > > > > >
> > > > > > > 
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> [image: Aiven] 
> > > > > > > >>>
> > > > > > > >>> *Josep Prat*
> > > > > > > >>> Open Source Engineering Director, *Aiven*
> > > > > > > >>> josep.p...@aiven.io   |   +491715557497
> > > > > > > >>> aiven.io    |   <
> > > > > > > >> https://www.facebook.com/aivencloud>
> > > > > > > >>>   <
> > > > > > > >> https://twitter.com/aiven_io>
> > > > > > > >>> *Aiven Deutschland GmbH*
> > > > > > > >>> Alexanderufer 3-7, 10117 Berlin
> > > > > > > >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > > > > > >>> Anna Richardson, Kenneth Chen
> > > > > > > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > David Arthur
> > > > > > > >
> > > > > > > 
> > > > > > > >>>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Thanks,
> > > > > > Jiashen
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [VOTE] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Satish Duggana]

Thanks Kamal for the KIP, LGTM.

+1 from me.

On Mon, 19 Aug 2024 at 10:41, Kamal Chandraprakash
 wrote:
>
> Hi,
>
> I would like to open voting for KIP-1075. I've addressed the review
> comments in the discussion thread. Please vote if the proposal looks good.
>
> https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=315494171#content/view/315494171
>
>
> Thanks,
> Kamal

Re: [DISCUSS] KIP-1075: Introduce delayed remote list offsets purgatory to make LIST_OFFSETS async

[Satish Duggana]

Thanks Kamal for the KIP.

This is an important enhancement to unblock the io threads as
ListOffsetsAPI may need to fetch data from remote storage and it may
take longer to process them. This feature helped in not starving other
broker api requests in our environments when there are many
ListOffsetsAPi requests on a broker.

Thanks,
Satish.

On Sat, 17 Aug 2024 at 13:55, Kamal Chandraprakash
 wrote:
>
> Hi all,
>
> If there are no more comments, I'll start a vote soon.
>
> Thanks,
> Kamal
>
> On Wed, Aug 14, 2024 at 5:05 PM Luke Chen  wrote:
>
> > Hi Kamal,
> >
> > Thanks for the update.
> > LGTM.
> >
> > Luke
> >
> > On Wed, Aug 14, 2024 at 7:25 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > > I saw we added some new configs/metrics.
> > >
> > > I have removed the recent changes to the public interfaces to limit the
> > > scope of the KIP to minimum. PTAL.
> > >
> > > Thanks,
> > > Kamal
> > >
> > > On Wed, Aug 14, 2024 at 9:58 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > > LC5
> > > > Agree, this is a rare scenario. Given that we have a common pool of
> > > > request handler threads to accept
> > > > all the incoming requests and there are no quotas to handle for each
> > > > request. I'm OK with reusing the
> > > > same remote-log-reader threads for LIST_OFFSETS requests. There may be
> > > > noisy neighbor issues
> > > > in handling the LIST_OFFSETS and FETCH remote requests when we read
> > from
> > > > remote storage
> > > > aggressively and all the remote-log-reader threads are busy.
> > > >
> > > > >  If so, maybe the additional config/metrics are not necessary?
> > > > Do you mean to have a separate thread pool and hardcode the num
> > threads?
> > > >
> > > > > LC6 and LC7
> > > > Updated the KIP.
> > > >
> > > > Thanks,
> > > > Kamal
> > > >
> > > > On Wed, Aug 14, 2024 at 8:05 AM Luke Chen  wrote:
> > > >
> > > >> Hi Kamal,
> > > >>
> > > >> Thanks for the response.
> > > >>
> > > >> I saw we added some new configs/metrics. Comments:
> > > >>
> > > >> LC5: Do you think this is a commonly happened issue that we need to
> > add
> > > a
> > > >> separate `remote.log.offset.reader.threads` for it?
> > > >> I thought this rarely happened. If so, maybe the additional
> > > config/metrics
> > > >> are not necessary? It makes the config more complicated.
> > > >>
> > > >> LC6: The config name:
> > > >> `remote.log.offset.read.max.pending.tasks` , should we be consistent
> > to
> > > >> use
> > > >> `reader`, instead of `read`?
> > > >>
> > > >> LC7: We should set a default value for the newly introduced configs
> > and
> > > >> written in KIP.
> > > >>
> > > >> Thanks.
> > > >> Luke
> > > >>
> > > >> On Tue, Aug 13, 2024 at 8:47 PM Kamal Chandraprakash <
> > > >> kamal.chandraprak...@gmail.com> wrote:
> > > >>
> > > >> > Hi Luke,
> > > >> >
> > > >> > Thanks for the review!
> > > >> >
> > > >> > > LC2:
> > > >> > a. If the time taken to process the request is less than 5 mins,
> > then
> > > >> the
> > > >> > Admin client will get a response.
> > > >> > b. If the time taken to process the request is more than 5 mins,
> > then
> > > >> the
> > > >> > Admin client will itself timeout the request due to the
> > > >> > default-api-timeout.
> > > >> > c. If the time taken to process the request is more than 6 mins,
> > then
> > > >> > the server will cancel the request in the DelayedRemoteListOffsets
> > > >> > purgatory (to be implemented) and
> > > >> > send TimeoutException back to the client if the client is
> > waiting
> > > >> for
> > > >> > the response.
> > > >> >
> > > >> > > LC3:
> > > >> > Updated the KIP.
> > > >> >
> > > >> > > LC4:
> > > >> > The consumer retries the LIST_OFFSETS request incase of
> > > failures/timeout
> > > >> > but not the AdminClient. So, I think this is a retry feature in the
> > > >> > Consumer.
> > > >> >
> > > >> > Updated the "Public Interfaces" section in the KIP
> > > >> > <
> > > >> >
> > > >>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1075%3A+Introduce+delayed+remote+list+offsets+purgatory+to+make+LIST_OFFSETS+async
> > > >> > >
> > > >> > by adding more details. PTAL.
> > > >> >
> > > >> > Thanks,
> > > >> > Kamal
> > > >> >
> > > >> >
> > > >> > On Tue, Aug 13, 2024 at 2:03 PM Luke Chen 
> > wrote:
> > > >> >
> > > >> > > Hi Kamal,
> > > >> > >
> > > >> > > Thanks for the update.
> > > >> > > LC1: I see. Thanks.
> > > >> > > LC2: What I still don't understand is what is the relationship
> > > between
> > > >> > > remote.list.offsets.request.timeout.ms V.S.
> > > >> > > request.timeout/default.api.timeout.
> > > >> > > Suppose we set request timeout to 30 seconds,
> > default.api.timeout=5
> > > >> mins
> > > >> > > and remote.list.offsets.request.timeout.ms = 6 mins.
> > > >> > > So, when Admin sends a list offset request that needs to query
> > > remote
> > > >> > > storage, when will it throw timeout exception? 30 secs o

[jira] [Resolved] (KAFKA-16887) document remote copy/fetch quotas metrics

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-16887.

Resolution: Fixed

> document remote copy/fetch quotas metrics
> -
>
> Key: KAFKA-16887
> URL: https://issues.apache.org/jira/browse/KAFKA-16887
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Assignee: Ksolves India Limited
>Priority: Major
>
> Context: https://github.com/apache/kafka/pull/15820#discussion_r1625304008



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Satish Duggana]

Thanks Kamal, and Luke for improving the earlier solution for KRaft.

One minor comment I have is to change the config name from
"remote.log.copy.disable" to "remote.log.copy.enable" with the default
value being true.

The solution summary to disable tiered storage on a topic:

- When a user wants to disable tiered storage on a topic, we should
make sure that local.log and log.retention are same. This is to make
sure the user understands the implications of the local storage
requirements while disabling tiered storage and set them
appropriately.

- Stop copying the log segments to remote storage as broker needs to
accumulate the required data locally to serve the required data from
local storage before we disable in remote storage. This will be done
by updating the config "remote.log.copy.enable" as false.

- We added a guardrail to make sure user understands that disabling
tiered storage will delete the remote storage data. This is by setting
"remote.log.delete.on.disable" should be true before setting
"remote.storage.enable" as false.


I think it is better to refactor the KIP to have only the updated
KRaft based solution and move the ZK based solution to the appendix
for reference. wdyt?

~Satish.



On Wed, 7 Aug 2024 at 17:38, Luke Chen  wrote:
>
> Hi all,
>
> Based on the original design:
> When tiered storage is disabled or becomes read-only on a topic, the local
> retention configuration becomes irrelevant, and all data expiration follows
> the topic-wide retention configuration exclusively.
>
> That works well. But we are afraid users will not check the document and
> "thought" the local log is bound to the local.retention.ms/bytes after
> `remote.log.copy.disable=true` (i.e. read-only remote storage). The
> confusion might cause the local disk to be full and bring down the broker.
> To avoid this "surprise" to users, we'd like to add one more validation
> when `remote.log.copy.disable` is set to true:
>   - validation: when `remote.log.copy.disable=true`,
> -- `local.retention.ms` must equal to `retention.ms` or -2 (which
> means `retention.ms` value to be used)
> -- `local.retention.bytes` must equal to `retention.byes` or -2 (which
> means `retention.ms` value to be used)
>
> So, basically, we don't change the original design, just want to make sure
> users are aware of the retention policy change after disabling remote log
> copy.
>
> Let me know if you have any comments.
>
> Thank you.
> Luke
>
> On Fri, Jul 26, 2024 at 8:17 PM Luke Chen  wrote:
>
> > Thanks Kamal for the comments.
> > KIP updated.
> >
> > Thanks.
> > Luke
> >
> > On Fri, Jul 26, 2024 at 6:56 PM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> >> Luke,
> >>
> >> Thanks for confirming the topic config change validation on the controller
> >> and updating the KIP.
> >> The updated KIP LGTM.
> >>
> >> 1. Can we update the below sentence in the KIP to clarify that
> >> remote.storage.enable should be true during graceful disablement?
> >>
> >> > Users set the configuration
> >> "remote.storage.enable=false,remote.log.delete.on.disable=true", or
> >> "remote.copy.disabled=true" for the desired topic, indicating the
> >> disablement of tiered storage.
> >> to
> >> > Users set the configuration
> >> "remote.storage.enable=false,remote.log.delete.on.disable=true", or
> >> "remote.storage.enable=true,remote.copy.disabled=true" for the desired
> >> topic, indicating the disablement of tiered storage.
> >>
> >> 2. Can we clarify in the public interface that the StopReplica v5,
> >> tiered_epoch, and tiered_state changes are required only for ZK mode and
> >> won't be implemented?
> >>
> >> Thanks,
> >> Kamal
> >>
> >> On Fri, Jul 26, 2024 at 1:40 PM Luke Chen  wrote:
> >>
> >> > Hi Kamal,
> >> >
> >> > Thanks for the comments.
> >> >
> >> > For this:
> >> > > If we throw an exception from the server for invalid config, then
> >> there
> >> > will be inconsistency between the CLI tools and the actual state of the
> >> > topic in the cluster. This can cause some confusion to the users whether
> >> > tiered storage is disabled or not. I don't know how the Kraft topic
> >> config
> >> > propagation/validation works.
> >> >
> >> > I've confirmed we can validate the topic configuration change on the
> >> > controller level, by comparing existing configuration and new changed
> >> > configuration.
> >> > In my local POC, we can fail the configuration change if it's invalid
> >> like
> >> > this:
> >> >
> >> > # Disable with remote.log.delete.on.disable=false (default)
> >> > bin/kafka-configs.sh --bootstrap-server {bootstrap-string} \
> >> >--alter --entity-type topics --entity-name {topic-name} \
> >> >--add-config 'remote.storage.enable=false'
> >> >
> >> > Error while executing config command with args '--bootstrap-server
> >> > {bootstrap-string} --entity-type topics --entity-name {topic-name}
> >> --alter
> >> > --add-config remote.storage.enable=false'
> >> > java.util.concurrent.ExecutionException:

Re: [VOTE] KIP-1057: Add remote log metadata flag to the dump log tool

[Satish Duggana]

Thanks Federico for the KIP.

+1

~Satish.

On Thu, 27 Jun 2024 at 13:44, Federico Valeri  wrote:
>
> Thanks for the votes so far, bumping this thread to get more.
>
> On Fri, Jun 21, 2024 at 4:23 PM Kamal Chandraprakash
>  wrote:
> >
> > Hi Federico,
> >
> > Thanks for the KIP! +1 from me.
> >
> > On Fri, Jun 21, 2024 at 5:47 PM Luke Chen  wrote:
> >
> > > Hi Fede,
> > >
> > > Thanks for the KIP!
> > > +1 from me.
> > >
> > > Luke
> > >
> > > On Fri, Jun 21, 2024 at 6:44 PM Federico Valeri 
> > > wrote:
> > >
> > > > Hi all, I'd like to kick off a vote on KIP-1057.
> > > >
> > > > Design doc:
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool
> > > >
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/kxx1h4qwshgcjh4d5xzqltkx5mx9qopm
> > > >
> > > > Thanks,
> > > > Fede
> > > >
> > >

Re: [DISCUSS] KIP-1057: Add remote log metadata flag to the dump log tool

[Satish Duggana]

Thanks Federico for the KIP.

This feature is helpful for developers while debugging tiered storage
related issues.

Even though RLMM is a pluggable interface, it is still useful to have
a utility that is meant for the default/inbuilt implementation based
on the internal topic. We can clarify that in the help notes and user
docs.

Users can still use alternatives like others suggested if they need to
dump in a different format
- Running the dump-logs tool with custom decoder
- Running kafka-consumer.sh on the topic.

~Satish.


~Satish.



On Mon, 17 Jun 2024 at 15:55, Federico Valeri  wrote:
>
> Hi Kamal,
>
> On Mon, Jun 17, 2024 at 11:44 AM Kamal Chandraprakash
>  wrote:
> >
> > We can use the console-consumer to read the contents of the
> > `__remote_log_metadata` topic. Why are we proposing a new tool?
> >
> > sh kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
> > __remote_log_metadata  --consumer-property exclude.internal.topics=false
> > --formatter
> > org.apache.kafka.server.log.remote.metadata.storage.serialization.RemoteLogMetadataSerde\$RemoteLogMetadataFormatter
> > --from-beginning
> >
>
> Thanks from bringing this up. It works fine but a running broker is
> required, so it would make it inconvenient for a remote support
> engineer. Also you may have to deal with client security
> configuration, and it would be complicated to only dump specific
> segments. I'm adding to the rejected alternative for now, but I'm open
> to changes.
>
> >
> >
> > On Mon, Jun 17, 2024 at 12:53 PM Federico Valeri 
> > wrote:
> >
> > > Hi Divij,
> > >
> > > On Sun, Jun 16, 2024 at 7:38 PM Divij Vaidya 
> > > wrote:
> > > >
> > > > Hello Federico
> > > >
> > > > Please note that the topic-based RLMM is one of the possible
> > > > implementations of RLMM. Hence, whatever solution we design here should:
> > > 1\
> > > > be explicit that this tooling only works for topic based RLMM 2\ specify
> > > > the handling of the failure mode when topic based RLMM is not being 
> > > > used.
> > > >
> > >
> > > That's true, thanks for pointing out.
> > >
> > > > I would argue that Topic based RLMM cannot be treated the same as other
> > > > internal topics. Topic based RLMM topic is an optional topic which can
> > > have
> > > > any possible schema (depending on plugin implementation) whereas
> > > > other internal topics are always guaranteed to be present with a fixed
> > > > schema.
> > > >
> > >
> > > Right, I updated the KIP with an improved option description.
> > >
> > > > In light of the above statements, the rejected alternative sounds better
> > > to
> > > > me because:
> > > > 1\ it provides the ability to dump logs for "any" RLMM implementation 
> > > > and
> > > > not just topic based RLMM.
> > > > 2\ we don't have to deal with schema evolution of topic based RLMM in
> > > this
> > > > tool. That responsibility will be delegated to the decoder class which
> > > the
> > > > operator can define using the flag "--value-decoder-class".
> > > >
> > > > Is there a reason that you are unable to use the rejected solution 
> > > > (which
> > > > requires no changes) for debugging purposes?
> > > >
> > >
> > > The rejected alternative will still be available, but I thought that
> > > having a dedicated flag would make debugging easier, as I guess most
> > > people will use the default RLMM implementation. I would be happy to
> > > hear other opinions on this.
> > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Sat, Jun 15, 2024 at 4:43 PM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'd like to kick off a discussion for KIP-1057, that proposes to add
> > > > > remote log metadata flag to the dump log tool, which is useful when
> > > > > debugging.
> > > > >
> > > > >
> > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1057%3A+Add+remote+log+metadata+flag+to+the+dump+log+tool
> > > > >
> > > > > Thanks,
> > > > > Fede
> > > > >
> > >

[jira] [Resolved] (KAFKA-15420) Kafka Tiered Storage V1

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15420.

Resolution: Fixed

> Kafka Tiered Storage V1
> ---
>
> Key: KAFKA-15420
> URL: https://issues.apache.org/jira/browse/KAFKA-15420
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 3.6.0
>        Reporter: Satish Duggana
>    Assignee: Satish Duggana
>Priority: Major
>  Labels: KIP-405
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16977) remote log manager dynamic configs are not available after broker restart.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-16977:
--

 Summary: remote log manager dynamic configs are not available 
after broker restart.
 Key: KAFKA-16977
 URL: https://issues.apache.org/jira/browse/KAFKA-16977
 Project: Kafka
  Issue Type: Bug
Reporter: Satish Duggana


The below remote log configs can be configured dynamically:

remote.log.manager.copy.max.bytes.per.second
remote.log.manager.fetch.max.bytes.per.second and
remote.log.index.file.cache.total.size.bytes

If those values are dynamically configured, after the broker restart, it loads 
the static value from the config file instead of the dynamic value. Note that 
the issue happens only when running the server with ZooKeeper.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16976) Improve the dynamic config handling for RemoteLogManagerConfig when a broker is restarted.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-16976:
--

 Summary: Improve the dynamic config handling for 
RemoteLogManagerConfig when a broker is restarted.
 Key: KAFKA-16976
 URL: https://issues.apache.org/jira/browse/KAFKA-16976
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana
Assignee: Kamal Chandraprakash
 Fix For: 3.9.0


This is a followup on the discussion: 
https://github.com/apache/kafka/pull/16353#pullrequestreview-2121953295



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Satish Duggana]

+1 on going with 3.8 release with the existing plan and targeting the
required features in 3.9 timelines. 4.0 will be targeted in the usual
cycle(4 months) after 3.9 is released.


On Fri, 14 Jun 2024 at 15:19, Edoardo Comar  wrote:
>
> Josep,
> past the deadline sorry but I can't see reasons not to cherry-pick this
> https://github.com/apache/kafka/pull/16326
>
> On Wed, 12 Jun 2024 at 17:14, Josep Prat  wrote:
> >
> > Hi Edoardo,
> >
> > Correct, you can still cherry-pick this one.
> >
> > I'll send an email tomorrow morning (CEST) asking maintainers to stop
> > cherry picking commits unless we discuss it beforehand.
> >
> > Best,
> >
> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar  wrote:
> >
> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> > > https://github.com/apache/kafka/pull/16229
> > >
> > > right?
> > > thanks
> > >
> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko  wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner today,
> > > but please proceed with the release if I don't manage.
> > > >
> > > > Ivan
> > > >
> > > > [1] https://github.com/apache/kafka/pull/13277
> > > >
> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > > > > Hi Luke,
> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0 (but he
> > > can
> > > > > confirm this). My question now would be, given that it seems we would
> > > need
> > > > > a v3.9.0, do you think it's important to include
> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > > > >
> > > > > Best,
> > > > >
> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> > > > >
> > > > > > Hi Josep
> > > > > >
> > > > > > For KIP-966, I think Calvin had mentioned he won't complete in
> > > v3.8.0.
> > > > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > > > > >
> > > > > > For unclean leader election, all we need is this PR:
> > > > > > https://github.com/apache/kafka/pull/16284
> > > > > > For this PR, I think it needs one more week to be completed.
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
> > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We are now really close to the planned code freeze deadline (today
> > > EOD).
> > > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch
> > > until we
> > > > > > > achieve feature parity regarding Zookeeper and KRaft. The two main
> > > KIPs
> > > > > > > identified that would achieve this are: KIP-853 [2] and KIP-966
> > > [3].
> > > > > > > At the moment of writing this email both KIPs are not completed. 
> > > > > > > My
> > > > > > > question to the people driving both KIPs would be, how much more
> > > time do
> > > > > > > you think it's needed to bring these KIPs to completion?
> > > > > > >
> > > > > > > - If the time needed would be short, we could still include these
> > > 2 KIPs
> > > > > > in
> > > > > > > the release.
> > > > > > > - However, if the time needed would be on the scale of weeks, we
> > > should
> > > > > > > continue with the release plan for 3.8 and after start working on
> > > the 3.9
> > > > > > > release.
> > > > > > >
> > > > > > > What are your thoughts?
> > > > > > >
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > > > > > [2]:
> > > > > > >
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > > > > > [3]:
> > > > > > >
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > > > > > >
> > > > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat 
> > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini,
> > > > > > > > Yes, we could backport this one to the 3.8 branch. Would you be
> > > able to
> > > > > > > do
> > > > > > > > this once you merge this PR?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > > > > > rajinisiva...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Josep,
> > > > > > > >>
> > > > > > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899
> > > looks
> > > > > > > ready
> > > > > > > >> to be merged (waiting for the PR build).The PR changes several
> > > files,
> > > > > > > but
> > > > > > > >> is relatively straightforward and not risky. Also the changes
> > > are
> > > > > > under
> > > > > > > a
> > > > > > > >> config that is not enabled by default. Since the KIP was
> > > approved
> > > > > > before
> > > > > > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > > > > > >>
> > > > > > > >> Thank you,
> > > > > > > >>
> > > > > > > >> Rajini
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>

[jira] [Resolved] (KAFKA-14877) refactor InMemoryLeaderEpochCheckpoint

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-14877.

Resolution: Invalid

> refactor InMemoryLeaderEpochCheckpoint
> --
>
> Key: KAFKA-14877
> URL: https://issues.apache.org/jira/browse/KAFKA-14877
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Luke Chen
>Priority: Minor
> Fix For: 3.8.0
>
>
> follow up with this comment: 
> https://github.com/apache/kafka/pull/13456#discussion_r1154306477



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16947) Kafka Tiered Storage V2

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-16947:
--

 Summary: Kafka Tiered Storage V2
 Key: KAFKA-16947
 URL: https://issues.apache.org/jira/browse/KAFKA-16947
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 3.6.0
Reporter: Satish Duggana
Assignee: Satish Duggana
 Fix For: 3.8.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16890) Failing to build aux state on broker failover

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-16890.

Resolution: Fixed

> Failing to build aux state on broker failover
> -
>
> Key: KAFKA-16890
> URL: https://issues.apache.org/jira/browse/KAFKA-16890
> Project: Kafka
>  Issue Type: Bug
>  Components: Tiered-Storage
>Affects Versions: 3.7.0, 3.7.1
>Reporter: Francois Visconte
>Assignee: Kamal Chandraprakash
>Priority: Major
> Fix For: 3.8.0
>
>
> We have clusters where we replace machines often falling into a state where 
> we keep having "Error building remote log auxiliary state for 
> loadtest_topic-22" and the partition being under-replicated until the leader 
> is manually restarted. 
> Looking into a specific case, here is what we observed in 
> __remote_log_metadata topic:
> {code:java}
>  
> partition: 29, offset: 183593, value: 
> RemoteLogSegmentMetadata{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, startOffset=10823, endOffset=11536, 
> brokerId=10013, maxTimestampMs=1715774588597, eventTimestampMs=1715781657604, 
> segmentLeaderEpochs={125=10823, 126=10968, 128=11047, 130=11048, 131=11324, 
> 133=11442, 134=11443, 135=11445, 136=11521, 137=11533, 139=11535}, 
> segmentSizeInBytes=704895, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_STARTED}
> partition: 29, offset: 183594, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_FINISHED, eventTimestampMs=1715781658183, brokerId=10013}
> partition: 29, offset: 183669, value: 
> RemoteLogSegmentMetadata{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, startOffset=10823, endOffset=11544, 
> brokerId=10008, maxTimestampMs=1715781445270, eventTimestampMs=1715782717593, 
> segmentLeaderEpochs={125=10823, 126=10968, 128=11047, 130=11048, 131=11324, 
> 133=11442, 134=11443, 135=11445, 136=11521, 137=11533, 139=11535, 140=11537, 
> 142=11543}, segmentSizeInBytes=713088, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_STARTED}
> partition: 29, offset: 183670, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_FINISHED, eventTimestampMs=1715782718370, brokerId=10008}
> partition: 29, offset: 186215, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_STARTED, eventTimestampMs=1715867874617, brokerId=10008}
> partition: 29, offset: 186216, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_FINISHED, eventTimestampMs=1715867874725, brokerId=10008}
> partition: 29, offset: 186217, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_STARTED, eventTimestampMs=1715867874729, brokerId=10008}
> partition: 29, offset: 186218, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_FINISHED, eventTimestampMs=1715867874817, brokerId=10008}
> {code}
>  
> It seems that at the time the leader is restarted (10013), a second copy of 
> the same segment is tiered by the new leader (10008). Interestingly the 
> segment doesn't have the same end offset, which is concerning. 
> Then the follower sees the following error repeatedly until the leader is 
> restarted: 
>  
> {code:java}
> [2024-05-17 20:46:42,133] DEBUG [ReplicaFetcher replicaId=10013, 
> leaderId=10008, fetcherId=0] Handling errors in processFetchRequest for 
> partitions HashSet(loadtest_topic-22) (kafka.server.ReplicaFetcherThread)
> [2024-05-17 20:46:43,174] DEBUG [ReplicaFetcher replicaId=10013, 
> leaderId=10008, fetcherId=0] Receiv

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Satish Duggana]

+1
Thanks Christo for addressing the review comments. We can update the
KIP for any minor comments/clarifications.


On Thu, 16 May 2024 at 15:21, Luke Chen  wrote:
>
> Thanks Chia-Ping!
> Since ZK is going to be removed, I agree the KRaft part has higher priority.
> But if Christo or the community contributor has spare time, it's good to
> have ZK part, too!
>
> Thanks.
> Luke
>
> On Thu, May 16, 2024 at 5:45 PM Chia-Ping Tsai  wrote:
>
> > +1 but I prefer to ship it to KRaft only.
> >
> > I do concern that community have enough time to accept more feature in 3.8
> > :(
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/05/14 15:20:50 Christo Lolov wrote:
> > > Heya!
> > >
> > > I would like to start a vote on KIP-950: Tiered Storage Disablement in
> > > order to catch the last Kafka release targeting Zookeeper -
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> > >
> > > Best,
> > > Christo
> > >
> >

Re: [VOTE] KIP-1018: Introduce max remote fetch timeout config

[Satish Duggana]

Thanks Kamal for the KIP.

+1 from me.

~Satish.

On Thu, 9 May 2024 at 17:52, Christo Lolov  wrote:
>
> Heya Kamal,
>
> Thanks for the KIP and the answers in the discussion!
>
> +1 from me :)
>
> Best,
> Christo
>
> On Thu, 9 May 2024 at 11:11, Federico Valeri  wrote:
>
> > +1 non binding
> >
> > Thanks
> >
> > On Thu, May 9, 2024 at 12:05 PM Luke Chen  wrote:
> > >
> > > Hi Kamal,
> > >
> > > Thanks for the KIP!
> > > +1 from me.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Mon, May 6, 2024 at 5:03 PM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We would like to start a voting thread for KIP-1018: Introduce
> > > > max remote fetch timeout config for DelayedRemoteFetch requests.
> > > >
> > > > The KIP is available on
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests
> > > >
> > > > If you have any suggestions, feel free to participate in the discussion
> > > > thread:
> > > > https://lists.apache.org/thread/9x21hzpxzmrt7xo4vozl17d70fkg3chk
> > > >
> > > > --
> > > > Kamal
> > > >
> >

Re: [VOTE] KIP-932: Queues for Kafka

[Satish Duggana]

Hi Andrew,
Thanks for the nice KIP, it will allow other messaging use cases to be
onboarded to Kafka.

+1 from me.

Satish.

On Tue, 7 May 2024 at 03:41, Jun Rao  wrote:
>
> Hi, Andrew,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar 
> wrote:
>
> > Thanks Andrew,
> >
> > +1 (binding)
> >
> > Edo
> >
> > On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >  wrote:
> > >
> > > Hi Andrew
> > >
> > > + 1 (Non-Binding)
> > >
> > > This will be great addition to Kafka
> > >
> > > On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal 
> > > wrote:
> > >
> > > > Hi Andrew,
> > > > Thanks for writing the KIP. This is indeed going to be a valuable
> > addition
> > > > to the Kafka, excited to see the KIP.
> > > >
> > > > + 1 (Non-Binding)
> > > >
> > > > Regards,
> > > > Apoorv Mittal
> > > > +44 7721681581
> > > >
> > > >
> > > > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > > > andrew_schofield_j...@outlook.com> wrote:
> > > >
> > > > > Hi,
> > > > > I’ve been working to complete KIP-932 over the past few months and
> > > > > discussions have quietened down.
> > > > >
> > > > > I’d like to open the voting for KIP-932:
> > > > >
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > >
> >

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

[Satish Duggana]

Thanks Abhijeet for the KIP.
+1 from me.

~Satish

On Fri, 26 Apr 2024 at 8:35 PM, Jun Rao  wrote:

> Hi, Abhijeet,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Thu, Apr 25, 2024 at 10:30 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I would like to start the vote for KIP-1023 - Follower fetch from tiered
> > offset
> >
> > The KIP is here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+tiered+offset
> >
> > Regards.
> > Abhijeet.
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

[Satish Duggana]

Congratulations Greg!

On Mon, 15 Apr 2024 at 11:04, Claude Warren  wrote:
>
> Congrats Greg!  All the hard work paid off.
>
> On Mon, Apr 15, 2024 at 6:58 AM Ivan Yurchenko  wrote:
>
> > Congrats Greg!
> >
> > On Sun, Apr 14, 2024, at 22:51, Sophie Blee-Goldman wrote:
> > > Congrats Greg! Happy to have you
> > >
> > > On Sun, Apr 14, 2024 at 9:26 AM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > > > Congrats, Greg!!
> > > >
> > > > On Sun 14. Apr 2024 at 15.05, Josep Prat 
> > > > wrote:
> > > >
> > > > > Congrats Greg!!!
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Josep Prat
> > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > +491715557497 | aiven.io
> > > > > Aiven Deutschland GmbH
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > On Sun, Apr 14, 2024, 12:30 Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > > Congratulations Greg!
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Apr 14, 2024 at 6:39 AM Kamal Chandraprakash <
> > > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > >
> > > > > > > Congratulations, Greg!
> > > > > > >
> > > > > > > On Sun, Apr 14, 2024 at 8:57 AM Yash Mayya  > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Congrats Greg!
> > > > > > > >
> > > > > > > > On Sun, 14 Apr, 2024, 05:56 Randall Hauch, 
> > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Greg!
> > > > > > > > >
> > > > > > > > > On Sat, Apr 13, 2024 at 6:36 PM Luke Chen  > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats, Greg!
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 14, 2024 at 7:05 AM Viktor Somogyi-Vass
> > > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > > Congrats Greg! :)
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 14, 2024, 00:35 Bill Bejeck <
> > bbej...@gmail.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congrats Greg!
> > > > > > > > > > > >
> > > > > > > > > > > > -Bill
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Apr 13, 2024 at 4:25 PM Boudjelda Mohamed Said
> > <
> > > > > > > > > > > bmsc...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat 13 Apr 2024 at 20:42, Chris Egerton <
> > > > > > > ceger...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Greg Harris has been a Kafka committer since July
> > 2023.
> > > > > He
> > > > > > > has
> > > > > > > > > > > remained
> > > > > > > > > > > > > > very active and instructive in the community since
> > > > > > becoming a
> > > > > > > > > > > > committer.
> > > > > > > > > > > > > > It's my pleasure to announce that Greg is now a
> > member
> > > > of
> > > > > > > Kafka
> > > > > > > > > > PMC.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Congratulations, Greg!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris, on behalf of the Apache Kafka PMC
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> LinkedIn: http://www.linkedin.com/in/claudewarren

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

ined behaviour and where we
> > retain
> > > half of the remote log and delete the other half.*
> > >
> > > * 103. Do we plan to add any metrics related to this feature?*
> > >
> > >
> > >
> > > *Any recommendations?*
> > > *We may emit a gauge showing the enablement state of a topic but we could
> > > gather that info from the logs as well.*
> > > *The total duration for remote topic deletion could be added as well but
> > > this is more of a metric for the RemotePartitionRemover itself.*
> > >
> > >
> > >
> > > *104. Please add configuration details about copier thread pool,
> > expiration
> > > thread pool and the migration of the existing
> > > RemoteLogManagerScheduledThreadPool.*
> > >
> > > *Will add the details.*
> > >
> > > 105. How is the behaviour with topic or partition deletion request
> > > handled when tiered storage disablement request is still being
> > > processed on a topic?
> > >
> > > *If the disablement policy is Delete then a successive topic deletion
> > > request is going to be a NOOP because RemoteLogs is already being
> > deleted.*
> > > *If the disablement policy is Retain, then we only moved the
> > LogStartOffset
> > > and didn't touch RemoteLogs anyway, so the delete topic request will
> > > result*
> > >
> > > *in the initiation of RemoteLog deletion.*
> > >
> > >
> > > On Tue, 26 Mar 2024 at 18:21, Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for the KIP! Overall the KIP looks good and covered most of the
> > > > items.
> > > >
> > > > 1. Could you explain how the brokers will handle the DisableRemoteTopic
> > > API
> > > > request?
> > > >
> > > > 2. Who will initiate the controller interaction sequence? Does the
> > > > controller listens for
> > > > topic config updates and initiate the disablement?
> > > >
> > > > --
> > > > Kamal
> > > >
> > > > On Tue, Mar 26, 2024 at 4:40 PM Satish Duggana <
> > satish.dugg...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks Mehari, Divij, Christo etal for the KIP.
> > > > >
> > > > > I had an initial review of the KIP and left the below comments.
> > > > >
> > > > > 101. For remote.log.disable.policy=delete:
> > > > > Does it delete the remote log data immediately and the data in remote
> > > > > storage will not be taken into account by any replica? That means
> > > > > log-start-offset is moved to the earlier local-log-start-offset.
> > > > >
> > > > > 102. Can we update the remote.log.disable.policy after tiered storage
> > > > > is disabled on a topic?
> > > > >
> > > > > 103. Do we plan to add any metrics related to this feature?
> > > > >
> > > > > 104. Please add configuration details about copier thread pool,
> > > > > expiration thread pool and the migration of the existing
> > > > > RemoteLogManagerScheduledThreadPool.
> > > > >
> > > > > 105. How is the behaviour with topic or partition deletion request
> > > > > handled when tiered storage disablement request is still being
> > > > > processed on a topic?
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Mon, 25 Mar 2024 at 13:34, Doğuşcan Namal <
> > namal.dogus...@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Christo and Luke,
> > > > > >
> > > > > > I think the KRaft section of the KIP requires slight improvement.
> > The
> > > > > metadata propagation in KRaft is handled by the RAFT layer instead of
> > > > > sending Controller -> Broker RPCs. In fact, KIP-631 deprecated these
> > > > RPCs.
> > > > > >
> > > > > > I will come up with some recommendations on how we could improve
> > that
> > > > > one but until then, @Luke please feel free to review the KIP.
> > > > > >
> > > > > > @Satish, if we want this to make it to Kafka 3.8 I believe we need
> > to
> > > > > aim to get th

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

[Satish Duggana]

+1 to Jun for adding the consumer fetching from a follower scenario
also to the existing section that talked about the drawback when a
node built with last-tiered-offset has become a leader. As Abhijeet
mentioned, we plan to have a follow-up KIP that will address these by
having a deprioritzation of these brokers. The deprioritization of
those brokers can be removed once they catchup until the local log
retention.

Thanks,
Satish.

On Tue, 9 Apr 2024 at 14:08, Luke Chen  wrote:
>
> Hi Abhijeet,
>
> Thanks for the KIP to improve the tiered storage feature!
>
> Questions:
> 1. We could also get the "pending-upload-offset" and epoch via remote log
> metadata, instead of adding a new API to fetch from the leader. Could you
> explain why you choose the later approach, instead of the former?
> 2.
> > We plan to have a follow-up KIP that will address both the
> deprioritization
> of these brokers from leadership, as well as
> for consumption (when fetching from followers is allowed).
>
> I agree with Jun that we might need to make it clear all possible drawbacks
> that could have. So, could we add the drawbacks that Jun mentioned about
> the performance issue when consumer fetch from follower?
>
> 3. Could we add "Rejected Alternatives" section to the end of the KIP to
> add some of them?
> Like the "ListOffsetRequest" approach VS "Earliest-Pending-Upload-Offset"
> approach, or getting the "Earliest-Pending-Upload-Offset" from remote log
> metadata... etc.
>
> Thanks.
> Luke
>
>
> On Tue, Apr 9, 2024 at 2:25 PM Abhijeet Kumar 
> wrote:
>
> > Hi Christo,
> >
> > Please find my comments inline.
> >
> > On Fri, Apr 5, 2024 at 12:36 PM Christo Lolov 
> > wrote:
> >
> > > Hello Abhijeet and Jun,
> > >
> > > I have been mulling this KIP over a bit more in recent days!
> > >
> > > re: Jun
> > >
> > > I wasn't aware we apply 2.1 and 2.2 for reserving new timestamps - in
> > > retrospect it should have been fairly obvious. I would need to go an
> > update
> > > KIP-1005 myself then, thank you for giving the useful reference!
> > >
> > > 4. I think Abhijeet wants to rebuild state from latest-tiered-offset and
> > > fetch from latest-tiered-offset + 1 only for new replicas (or replicas
> > > which experienced a disk failure) to decrease the time a partition spends
> > > in under-replicated state. In other words, a follower which has just
> > fallen
> > > out of ISR, but has local data will continue using today's Tiered Storage
> > > replication protocol (i.e. fetching from earliest-local). I further
> > believe
> > > he has taken this approach so that local state of replicas which have
> > just
> > > fallen out of ISR isn't forcefully wiped thus leading to situation 1.
> > > Abhijeet, have I understood (and summarised) what you are proposing
> > > correctly?
> > >
> > > Yes, your understanding is correct. We want to limit the behavior changes
> > only to new replicas.
> >
> >
> > > 5. I think in today's Tiered Storage we know the leader's
> > log-start-offset
> > > from the FetchResponse and we can learn its local-log-start-offset from
> > the
> > > ListOffsets by asking for earliest-local timestamp (-4). But granted,
> > this
> > > ought to be added as an additional API call in the KIP.
> > >
> > >
> > Yes, I clarified this in my reply to Jun. I will add this missing detail in
> > the KIP.
> >
> >
> > > re: Abhijeet
> > >
> > > 101. I am still a bit confused as to why you want to include a new offset
> > > (i.e. pending-upload-offset) when you yourself mention that you could use
> > > an already existing offset (i.e. last-tiered-offset + 1). In essence, you
> > > end your Motivation with "In this KIP, we will focus only on the follower
> > > fetch protocol using the *last-tiered-offset*" and then in the following
> > > sections you talk about pending-upload-offset. I understand this might be
> > > classified as an implementation detail, but if you introduce a new offset
> > > (i.e. pending-upload-offset) you have to make a change to the ListOffsets
> > > API (i.e. introduce -6) and thus document it in this KIP as such.
> > However,
> > > the last-tiered-offset ought to already be exposed as part of KIP-1005
> > > (under implementation). Am I misunderstanding something here?
> > >
> >
> > I have tried to clarify this in my reply to Jun.
> >
> > > The follower needs to build the local data starting from the offset
> > > Earliest-Pending-Upload-Offset. Hence it needs the offset and the
> > > corresponding leader-epoch.
> > > There are two ways to do this:
> > >1. We add support in ListOffsetRequest to be able to fetch this offset
> > > (and leader epoch) from the leader.
> > >2. Or, fetch the tiered-offset (which is already supported). From this
> > > offset, we can get the Earliest-Pending-Upload-Offset. We can just add 1
> > to
> > > the tiered-offset.
> > >   However, we still need the leader epoch for offset, since there is
> > > no guarantee that the leader epoch for Earliest-Pending-Upload-Offset
> > will
> 

Re: [ANNOUNCE] New committer: Christo Lolov

[Satish Duggana]

Congratulations Christo!

On Tue, 26 Mar 2024 at 19:20, Ivan Yurchenko  wrote:
>
> Congrats!
>
> On Tue, Mar 26, 2024, at 14:48, Lucas Brutschy wrote:
> > Congrats!
> >
> > On Tue, Mar 26, 2024 at 2:44 PM Federico Valeri  
> > wrote:
> > >
> > > Congrats!
> > >
> > > On Tue, Mar 26, 2024 at 2:27 PM Mickael Maison  
> > > wrote:
> > > >
> > > > Congratulations Christo!
> > > >
> > > > On Tue, Mar 26, 2024 at 2:26 PM Chia-Ping Tsai  
> > > > wrote:
> > > > >
> > > > > Congrats Christo!
> > > > >
> > > > > Chia-Ping
> >

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

Thanks Mehari, Divij, Christo etal for the KIP.

I had an initial review of the KIP and left the below comments.

101. For remote.log.disable.policy=delete:
Does it delete the remote log data immediately and the data in remote
storage will not be taken into account by any replica? That means
log-start-offset is moved to the earlier local-log-start-offset.

102. Can we update the remote.log.disable.policy after tiered storage
is disabled on a topic?

103. Do we plan to add any metrics related to this feature?

104. Please add configuration details about copier thread pool,
expiration thread pool and the migration of the existing
RemoteLogManagerScheduledThreadPool.

105. How is the behaviour with topic or partition deletion request
handled when tiered storage disablement request is still being
processed on a topic?

~Satish.

On Mon, 25 Mar 2024 at 13:34, Doğuşcan Namal  wrote:
>
> Hi Christo and Luke,
>
> I think the KRaft section of the KIP requires slight improvement. The 
> metadata propagation in KRaft is handled by the RAFT layer instead of sending 
> Controller -> Broker RPCs. In fact, KIP-631 deprecated these RPCs.
>
> I will come up with some recommendations on how we could improve that one but 
> until then, @Luke please feel free to review the KIP.
>
> @Satish, if we want this to make it to Kafka 3.8 I believe we need to aim to 
> get the KIP approved in the following weeks otherwise it will slip and we can 
> not support it in Zookeeper mode.
>
> I also would like to better understand what is the community's stand for 
> adding a new feature for Zookeeper since it is marked as deprecated already.
>
> Thanks.
>
>
>
> On Mon, 18 Mar 2024 at 13:42, Christo Lolov  wrote:
>>
>> Heya,
>>
>> I do have some time to put into this, but to be honest I am still after
>> reviews of the KIP itself :)
>>
>> After the latest changes it ought to be detailing both a Zookeeper approach
>> and a KRaft approach.
>>
>> Do you have any thoughts on how it could be improved or should I start a
>> voting thread?
>>
>> Best,
>> Christo
>>
>> On Thu, 14 Mar 2024 at 06:12, Luke Chen  wrote:
>>
>> > Hi Christo,
>> >
>> > Any update with this KIP?
>> > If you don't have time to complete it, I can collaborate with you to work
>> > on it.
>> >
>> > Thanks.
>> > Luke
>> >
>> > On Wed, Jan 17, 2024 at 11:38 PM Satish Duggana 
>> > wrote:
>> >
>> > > Hi Christo,
>> > > Thanks for volunteering to contribute to the KIP discussion. I suggest
>> > > considering this KIP for both ZK and KRaft as it will be helpful for
>> > > this feature to be available in 3.8.0 running with ZK clusters.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > > On Wed, 17 Jan 2024 at 19:04, Christo Lolov 
>> > > wrote:
>> > > >
>> > > > Hello!
>> > > >
>> > > > I volunteer to get this KIP moving forward and implemented in Apache
>> > > Kafka
>> > > > 3.8.
>> > > >
>> > > > I have caught up with Mehari offline and we have agreed that given
>> > Apache
>> > > > Kafka 4.0 being around the corner we would like to propose this feature
>> > > > only for KRaft clusters.
>> > > >
>> > > > Any and all reviews and comments are welcome!
>> > > >
>> > > > Best,
>> > > > Christo
>> > > >
>> > > > On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
>> > > > wrote:
>> > > >
>> > > > > Hi everyone, any progress on the status of this KIP? Overall looks
>> > > good to
>> > > > > me but I wonder whether we still need to support it for Zookeeper
>> > mode
>> > > > > given that it will be deprecated in the next 3 months.
>> > > > >
>> > > > > On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
>> > > > > > Hi everyone,
>> > > > > > I would like to start a discussion on KIP-950: Tiered Storage
>> > > Disablement
>> > > > > (
>> > > > >
>> > > > >
>> > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
>> > > > > ).
>> > > > > >
>> > > > > > This KIP proposes adding the ability to disable and re-enable
>> > tiered
>> > > > > storage on a topic.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mehari
>> > > > > >
>> > > > >
>> > >
>> >

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Satish Duggana]

+1, Thanks Mani for volunteering.

On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
>
> +1, Thanks Manikumar!
>
> On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna  wrote:
>
> > Thanks Manikumar!
> >
> > +1
> >
> > Best,
> > Bruno
> >
> > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > +1 thanks for volunteering!
> > >
> > > Best
> > > ---
> > >
> > > Josep Prat
> > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya 
> > wrote:
> > >
> > >> +1
> > >>
> > >> Thank you for volunteering.
> > >>
> > >> --
> > >> Divij Vaidya
> > >>
> > >>
> > >>
> > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > >> 
> > >> wrote:
> > >>
> > >>> Thanks Manikumar!
> > >>> +1 from me
> > >>>
> > >>> Justine
> > >>>
> > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar 
> > >>> wrote:
> > >>>
> >  Hi,
> > 
> >  I'd like to volunteer to be the release manager for a bug fix release
> > >> of
> >  the 3.6 line.
> >  If there are no objections, I'll send out the release plan soon.
> > 
> >  Thanks,
> >  Manikumar
> > 
> > >>>
> > >>
> > >
> >

Re: [VOTE] KIP-956: Tiered Storage Quotas

[Satish Duggana]

Thanks Abhijeet for the KIP, +1 from me.


On Sat, 9 Mar 2024 at 1:51 AM, Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> +1 (non-binding), Thanks for the KIP, Abhijeet!
>
> --
> Kamal
>
> On Fri, Mar 8, 2024 at 11:02 PM Jun Rao  wrote:
>
> > Hi, Abhijeet,
> >
> > Thanks for the KIP. +1
> >
> > Jun
> >
> > On Fri, Mar 8, 2024 at 3:44 AM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to start the vote for KIP-956 - Tiered Storage Quotas
> > >
> > > The KIP is here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas
> > >
> > > Regards.
> > > Abhijeet.
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Satish Duggana]

Thanks Josep, +1.

On Tue, 27 Feb 2024 at 17:29, Divij Vaidya  wrote:
>
> Thank you for volunteering Josep. +1 from me.
>
> --
> Divij Vaidya
>
>
>
> On Tue, Feb 27, 2024 at 9:35 AM Bruno Cadonna  wrote:
>
> > Thanks Josep!
> >
> > +1
> >
> > Best,
> > Bruno
> >
> > On 2/26/24 9:53 PM, Chris Egerton wrote:
> > > Thanks Josep, I'm +1 as well.
> > >
> > > On Mon, Feb 26, 2024 at 12:32 PM Justine Olshan
> > >  wrote:
> > >
> > >> Thanks Joesp. +1 from me.
> > >>
> > >> On Mon, Feb 26, 2024 at 3:37 AM Josep Prat  > >
> > >> wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> I'd like to volunteer as release manager for the Apache Kafka 3.8.0
> > >>> release.
> > >>> If there are no objections, I'll start building a release plan (or
> > >> adapting
> > >>> the one Colin made some weeks ago) in the wiki in the next days.
> > >>>
> > >>> Thank you.
> > >>>
> > >>> --
> > >>> [image: Aiven] 
> > >>>
> > >>> *Josep Prat*
> > >>> Open Source Engineering Director, *Aiven*
> > >>> josep.p...@aiven.io   |   +491715557497
> > >>> aiven.io    |   <
> > >> https://www.facebook.com/aivencloud
> > 
> > >>>   <
> > >>> https://twitter.com/aiven_io>
> > >>> *Aiven Deutschland GmbH*
> > >>> Alexanderufer 3-7, 10117 Berlin
> > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>
> > >>
> > >
> >

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 3.7.0

[Satish Duggana]

Thanks Stanislav for all your hard work on running the release. Thanks
to all the contributors to this release.


On Wed, 28 Feb 2024 at 13:43, Bruno Cadonna  wrote:
>
> Thanks Stan and all contributors for the release!
>
> Best,
> Bruno
>
> On 2/27/24 7:01 PM, Stanislav Kozlovski wrote:
> > The Apache Kafka community is pleased to announce the release of
> > Apache Kafka 3.7.0
> >
> > This is a minor release that includes new features, fixes, and
> > improvements from 296 JIRAs
> >
> > An overview of the release and its notable changes can be found in the
> > release blog post:
> > https://kafka.apache.org/blog#apache_kafka_370_release_announcement
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html
> >
> > You can download the source and binary release (Scala 2.12, 2.13) from:
> > https://kafka.apache.org/downloads#3.7.0
> >
> > ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you to the following 146 contributors to this release!
> > (Please report an unintended omission)
> >
> > Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra,
> > Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew
> > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal,
> > Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj,
> > ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna,
> > Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin
> > Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David
> > Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, Dongnuo
> > Lyu, Eaugene Thomas, Eduwer Camacaro, Eike Thaden, Federico Valeri,
> > Florin Akermann, Gantigmaa Selenge, Gaurav Narula, gongzhongqiang,
> > Greg Harris, Guozhang Wang, Gyeongwon, Do, Hailey Ni, Hanyu Zheng, Hao
> > Li, Hector Geraldino, hudeqi, Ian McDonald, Iblis Lin, Igor Soarez,
> > iit2009060, Ismael Juma, Jakub Scholz, James Cheng, Jason Gustafson,
> > Jay Wang, Jeff Kim, Jim Galasyn, John Roesler, Jorge Esteban Quilcate
> > Otoya, Josep Prat, José Armando García Sancio, Jotaniya Jeel, Jouni
> > Tenhunen, Jun Rao, Justine Olshan, Kamal Chandraprakash, Kirk True,
> > kpatelatwork, kumarpritam863, Laglangyue, Levani Kokhreidze, Lianet
> > Magrans, Liu Zeyu, Lucas Brutschy, Lucia Cerchie, Luke Chen, maniekes,
> > Manikumar Reddy, mannoopj, Maros Orsak, Matthew de Detrich, Matthias
> > J. Sax, Max Riedel, Mayank Shekhar Narula, Mehari Beyene, Michael
> > Westerby, Mickael Maison, Nick Telford, Nikhil Ramakrishnan, Nikolay,
> > Okada Haruki, olalamichelle, Omnia G.H Ibrahim, Owen Leung, Paolo
> > Patierno, Philip Nee, Phuc-Hong-Tran, Proven Provenzano, Purshotam
> > Chauhan, Qichao Chu, Matthias J. Sax, Rajini Sivaram, Renaldo Baur
> > Filho, Ritika Reddy, Robert Wagner, Rohan, Ron Dagostino, Roon, runom,
> > Ruslan Krivoshein, rykovsi, Sagar Rao, Said Boudjelda, Satish Duggana,
> > shuoer86, Stanislav Kozlovski, Taher Ghaleb, Tang Yunzi, TapDa

[jira] [Created] (KAFKA-16161) Avoid creating remote log metadata snapshot file in partition data directory.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-16161:
--

 Summary: Avoid creating remote log metadata snapshot file in 
partition data directory.
 Key: KAFKA-16161
 URL: https://issues.apache.org/jira/browse/KAFKA-16161
 Project: Kafka
  Issue Type: Improvement
Reporter: Satish Duggana


Avoid creating remote log metadata snapshot file in a partition data directory. 
This can be added when the snapshots implementation related functionality is 
enabled end to end. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

Hi Christo,
Thanks for volunteering to contribute to the KIP discussion. I suggest
considering this KIP for both ZK and KRaft as it will be helpful for
this feature to be available in 3.8.0 running with ZK clusters.

Thanks,
Satish.

On Wed, 17 Jan 2024 at 19:04, Christo Lolov  wrote:
>
> Hello!
>
> I volunteer to get this KIP moving forward and implemented in Apache Kafka
> 3.8.
>
> I have caught up with Mehari offline and we have agreed that given Apache
> Kafka 4.0 being around the corner we would like to propose this feature
> only for KRaft clusters.
>
> Any and all reviews and comments are welcome!
>
> Best,
> Christo
>
> On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
> wrote:
>
> > Hi everyone, any progress on the status of this KIP? Overall looks good to
> > me but I wonder whether we still need to support it for Zookeeper mode
> > given that it will be deprecated in the next 3 months.
> >
> > On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> > > Hi everyone,
> > > I would like to start a discussion on KIP-950: Tiered Storage Disablement
> > (
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> > ).
> > >
> > > This KIP proposes adding the ability to disable and re-enable tiered
> > storage on a topic.
> > >
> > > Thanks,
> > > Mehari
> > >
> >

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

[Satish Duggana]

+1 (binding)

Thanks,
Satish.

On Thu, 11 Jan 2024 at 17:52, Divij Vaidya  wrote:
>
> +1 (binding)
>
> Divij Vaidya
>
>
>
> On Tue, Dec 26, 2023 at 7:05 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > +1 (non-binding). Thanks for the KIP!
> >
> > --
> > Kamal
> >
> > On Thu, Dec 21, 2023 at 2:23 PM Christo Lolov 
> > wrote:
> >
> > > Heya all!
> > >
> > > KIP-1005 (
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Expose+EarliestLocalOffset+and+TieredOffset
> > > )
> > > has been open for around a month with no further comments - I would like
> > to
> > > start a voting round on it!
> > >
> > > Best,
> > > Christo
> > >
> >

Re: [VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

[Satish Duggana]

Thanks Kamal for the KIP.

+1 (binding)

On Fri, 5 Jan 2024 at 17:04, Divij Vaidya  wrote:
>
> +1 (binding)
>
> --
> Divij Vaidya
>
>
>
> On Thu, Dec 21, 2023 at 10:30 AM Luke Chen  wrote:
>
> > Hi Kamal,
> >
> > Thanks for the KIP.
> > +1 (binding) from me.
> >
> > Luke
> >
> > On Thu, Dec 21, 2023 at 4:51 PM Christo Lolov 
> > wrote:
> >
> > > Heya Kamal,
> > >
> > > The proposed change makes sense to me as it will be a more explicit
> > > behaviour than what Kafka does today - I am happy with it!
> > >
> > > +1 (non-binding) from me
> > >
> > > Best,
> > > Christo
> > >
> > > On Tue, 12 Dec 2023 at 09:01, Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to call a vote for KIP-1007
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1007%3A+Introduce+Remote+Storage+Not+Ready+Exception
> > > > >.
> > > > This KIP aims to introduce a new error code for retriable remote
> > storage
> > > > errors. Thanks to everyone who reviewed the KIP!
> > > >
> > > > --
> > > > Kamal
> > > >
> > >
> >

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

[Satish Duggana]

Thanks for the KIP Kamal, LGTM.

On Tue, 26 Dec 2023 at 10:23, Kamal Chandraprakash
 wrote:
>
> Hi Divij,
>
> Thanks for reviewing the KIP! I've updated the KIP with the below
> documentation. Let me know if it needs to be changed:
>
> The consumer can read the local data as long as it knows the offset from
> where to fetch the data from.
> When there is no initial offset, the consumer decides the offset based on
> the below config:
>
> ```
> auto.offset.reset = earliest / latest / none
> ```
>
>- For `earliest` offset policy and any offset that lies in the remote
>storage, the consumer (FETCH request)
>cannot be able to make progress until the remote log metadata gets
>synced.
>- In a FETCH request, when there are multiple partitions where a subset
>of them are consuming from local
>and others from remote, then only the partitions which are consuming
>from the remote cannot make
>progress and the partitions that fetch data from local storage should be
>able to make progress.
>- In a FETCH request, when the fetch-offset for a partition is within
>the local-storage, then it should be able
>to consume the messages.
>- All the calls to LIST_OFFETS will fail until the remote log metadata
>gets synced.
>
>
> The code link that is mentioned is referring to the `LIST_OFFSETS` handler.
> Usually, consumers don't use this API
> unless it's explicitly called by the user.
>
>
> On Fri, Dec 22, 2023 at 4:10 PM Divij Vaidya 
> wrote:
>
> > Thanks for the KIP, Kamal.
> >
> > The change looks good to me, though, I think we can do a better job at
> > documenting what the error means for the clients and users.
> >
> > Correct me if I'm wrong, when remote metadata is being synced on a new
> > leader, we cannot fetch even the local data (as per [1]), hence, partition
> > is considered "unreadable" but writes (and all other operations such as
> > admin operations) can continue to work on that partition. If my
> > understanding is correct, perhaps, please clarify this in the error
> > description. In absence of it, it is difficult to determine what this error
> > means for operations that can be performed on a partition.
> >
> > [1]
> >
> > https://github.com/apache/kafka/blob/82808873cbf6a95611243c2e7984c4aa6ff2cfff/core/src/main/scala/kafka/log/UnifiedLog.scala#L1336
> >
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Tue, Dec 12, 2023 at 9:58 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Thanks Luke for reviewing this KIP!
> > >
> > > If there are no more comments from others, I'll start the VOTE since this
> > > is a minor KIP.
> > >
> > > On Mon, Dec 11, 2023 at 1:01 PM Luke Chen  wrote:
> > >
> > > > Hi Kamal,
> > > >
> > > > Thanks for the KIP!
> > > > LGTM.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Wed, Nov 22, 2023 at 7:28 PM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would like to start a discussion to introduce a new error code for
> > > > > retriable remote storage errors. Please take a look at the proposal:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1007%3A+Introduce+Remote+Storage+Not+Ready+Exception
> > > > >
> > > >
> > >
> >

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

[Satish Duggana]

Thanks Ismael for the proposal.

Adopting JDK 17 enhances developer productivity and has reached a
level of maturity that has led to its adoption by several other major
projects, signifying its reliability and effectiveness.

+1 (binding)


~Satish.

On Wed, 3 Jan 2024 at 06:59, Justine Olshan
 wrote:
>
> Thanks for driving this.
>
> +1 (binding) from me.
>
> Justine
>
> On Tue, Jan 2, 2024 at 4:30 PM Ismael Juma  wrote:
>
> > Hi all,
> >
> > I would like to start a vote on KIP-1013.
> >
> > As stated in the discussion thread, this KIP was proposed after the KIP
> > freeze for Apache Kafka 3.7, but it is purely a documentation update (if we
> > decide to adopt it) and I believe it would serve our users best if we
> > communicate the deprecation for removal sooner (i.e. 3.7) rather than later
> > (i.e. 3.8).
> >
> > Please take a look and cast your vote.
> >
> > Link:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> >
> > Ismael
> >

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

[Satish Duggana]

Congratulations Divij!

On Thu, 28 Dec 2023 at 07:21, Kamal Chandraprakash
 wrote:
>
> Congrats Divij!
>
> On Thu, Dec 28, 2023, 07:09 Kirk True  wrote:
>
> > Congrats Divij!!!
> >
> > On Wed, Dec 27, 2023, at 1:44 PM, Jorge Esteban Quilcate Otoya wrote:
> > > Congratulations Divij!!
> > >
> > > On Wed 27. Dec 2023 at 14.56, Tom Bentley  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > On Thu, 28 Dec 2023 at 06:17, Philip Nee  wrote:
> > > >
> > > > > congrats divij!
> > > > >
> > > > > On Wed, Dec 27, 2023 at 8:55 AM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Congratulations Divij!
> > > > > >
> > > > > > On Wed, Dec 27, 2023 at 4:20 AM Gaurav Narula 
> > > > wrote:
> > > > > >
> > > > > > > Congratulations Divij!
> > > > > > >
> > > > > > > Regards,
> > > > > > > Gaurav
> > > > > > >
> > > > > > > > On 27-Dec-2023, at 17:44, Mickael Maison <
> > mickael.mai...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Congratulations Divij!
> > > > > > > >
> > > > > > > >> On Wed, Dec 27, 2023 at 1:05 PM Sagar <
> > sagarmeansoc...@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Congrats Divij! Absolutely well deserved !
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >> Sagar.
> > > > > > > >>
> > > > > > > >>> On Wed, Dec 27, 2023 at 5:15 PM Luke Chen  > >
> > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> Hi, Everyone,
> > > > > > > >>>
> > > > > > > >>> Divij has been a Kafka committer since June, 2023. He has
> > > > remained
> > > > > > very
> > > > > > > >>> active and instructive in the community since becoming a
> > > > committer.
> > > > > > > It's my
> > > > > > > >>> pleasure to announce that Divij is now a member of Kafka PMC.
> > > > > > > >>>
> > > > > > > >>> Congratulations Divij!
> > > > > > > >>>
> > > > > > > >>> Luke
> > > > > > > >>> on behalf of Apache Kafka PMC
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: New Release Branch 3.7

[Satish Duggana]

Hi Stanislav,
+1 to what Luke mentioned in the earlier email. I have also replied to your
comment on https://issues.apache.org/jira/browse/KAFKA-15147

These are minor changes and we plan to target the remaining changes to be
merged in the next few days(before 20th Dec).

Thanks,
Satish.

On Thu, 14 Dec 2023 at 17:54, Luke Chen  wrote:

> Hi Stanislav,
>
> For KIP-963, there are only 1 remaining task and 2 PRs which should be
> close to get merged. (ref:
> https://issues.apache.org/jira/browse/KAFKA-15147)
> Since this is an improvement for tiered storage metrics, not a huge
> change, I'd like to backport them into 3.7 branch if no objection from
> you.
>
> Thanks.
> Luke
>
> On Thu, Dec 14, 2023 at 5:15 PM Stanislav Kozlovski
>  wrote:
> >
> > Hey all,
> >
> > (thanks to Josep for reviewing the 3.8 bump PR)
> >
> > I have two more PRs to get reviewed regarding the release:
> > - targeting trunk: MINOR: Update documentation.html with the 3.7 release
> > #15010 
> > - targeting 3.7: MINOR: Update documentation.html with the 3.7 release
> > #15011 
> >
> > Additionally, I have one ask:
> > - if you are reading this message, can you double-check the list of KIPs
> > being released in the Release Page
> > 
> and
> > if you recognize any KIP you are involved it - can you ensure that the
> data
> > in the page & the associated JIRA/KIP (target release, merge status, vote
> > status) is up to date?
> >
> > I have myself went over the list of KIPs and a bit of the commit
> history. I
> > think KIPs:
> > - KIP-998: Give ProducerConfig(props, doLog) constructor protected access
> > - KIP-938: Add more metrics for measuring KRaft performance
> > are slipping this release.
> >
> > By far our biggest feature this release, KIP-848 The Next Generation of
> the
> > Consumer Rebalance Protocol, is a bit on the border. The main PR that KIP
> > is dependent on is this KAFKA-15456: Commit/Fetch error handling
> > improvements and V9 support #14557
> > .
> > While it's a gray area, I am weighing this change more on the
> stabilization
> > side as well as allowing it to come in the middle of feature freeze for
> two
> > chief reasons: a) it fixes issues that were recently discovered, hence
> can
> > be marked as stabilization work and b) it only touches the code path used
> > for Early Access, and not the existing production code path (hence
> doesn't
> > substantially risk the release)
> >
> > Additionally, KIP-858 has one minor but important change pending
> > .
> >
> > With that, I remind you that there are only 6 days to code freeze! It's
> not
> > long until we will have our very first Apache Kafka 3.7 RC! Let's get
> this
> > shipped.
> >
> > Best,
> > Stanislav
> >
> > On Tue, Dec 12, 2023 at 2:56 PM Stanislav Kozlovski <
> stanis...@confluent.io>
> > wrote:
> >
> > > Hello Kafka developers and friends,
> > >
> > > As promised, we now have a release branch for 3.7 release.
> > > Trunk is being bumped to 3.8.0-SNAPSHOT (please help review the PR <
> https://github.com/apache/kafka/pull/14993>).
> > >
> > > I'll be going over the JIRAs to move every non-blocker from this
> release to the next release.
> > >
> > > From this point, most changes should go to trunk.
> > > *- Blockers (existing and new that we discover while testing the
> release) will be double-committed.*
> > > *- Please discuss with your reviewer whether your PR should go to
> trunk or to trunk+release so they can merge accordingly.*
> > > *- Please help us test the release!*
> > >
> > > Thanks!
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
> >
> > --
> > Best,
> > Stanislav
>

[jira] [Created] (KAFKA-15864) Add more tests asserting the log-start-offset, local-log-start-offset, and HW/LSO/LEO in rolling over segments with tiered storage.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15864:
--

 Summary: Add more tests asserting the log-start-offset, 
local-log-start-offset, and HW/LSO/LEO in rolling over segments with tiered 
storage.
 Key: KAFKA-15864
 URL: https://issues.apache.org/jira/browse/KAFKA-15864
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Satish Duggana
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

[Satish Duggana]

Thanks Christo for starting the discussion on the KIP.

As mentioned in KAFKA-15857[1], the goal is to add new entries for
local-log-start-offset and tierd-offset in OffsetSpec. This will be
used in AdminClient APIs and also to be added as part of
GetOffsetShell. This was also raised by Kamal in the earlier email.

OffsetSpec related changes for these entries also need to be mentioned
as part of the PublicInterfaces section because these are exposed to
users as public APIs through Admin#listOffsets() APIs[2, 3].

Please update the KIP with the above details.

1. https://issues.apache.org/jira/browse/KAFKA-15857
2.  
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/Admin.java#L1238
3. 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/Admin.java#L1226

~Satish.

On Mon, 20 Nov 2023 at 18:35, Kamal Chandraprakash
 wrote:
>
> Hi Christo,
>
> Thanks for the KIP!
>
> Similar to the earliest-local-log offset, can we also expose the
> highest-copied-remote-offset via
> GetOffsetShell tool? This will be useful during the debugging session.
>
>
> On Mon, Nov 20, 2023 at 5:38 PM Christo Lolov 
> wrote:
>
> > Hello all!
> >
> > I would like to start a discussion for
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Add+EarliestLocalOffset+to+GetOffsetShell
> > .
> >
> > A new offset called local log start offset was introduced as part of
> > KIP-405: Kafka Tiered Storage. KIP-1005 aims to expose this offset by
> > changing the AdminClient and in particular the GetOffsetShell tool.
> >
> > I am looking forward to your suggestions for improvement!
> >
> > Best,
> > Christo
> >

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

[Satish Duggana]

+1 (binding)
Thanks for the KIP and the discussion.

Discussion mail thread for the KIP:
https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45



On Tue, 21 Nov 2023 at 05:21, Kamal Chandraprakash
 wrote:
>
> +1 (non-binding). Thanks for the KIP!
>
> On Tue, Nov 21, 2023, 03:04 Divij Vaidya  wrote:
>
> > + 1 (binding)
> >
> > This Kip will greatly improve Tiered Storage troubleshooting. Thank you
> > Christo.
> >
> > On Mon 20. Nov 2023 at 17:21, Christo Lolov 
> > wrote:
> >
> > > Hello all!
> > >
> > > Now that the discussion for KIP-963 has winded down, I would like to open
> > > it for a vote targeting 3.7.0 as the release. You can find the current
> > > version of the KIP at
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Additional+metrics+in+Tiered+Storage
> > >
> > > Best,
> > > Christo
> > >
> >

[jira] [Created] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15857:
--

 Summary: Introduce LocalLogStartOffset and TieredOffset in 
OffsetSpec.
 Key: KAFKA-15857
 URL: https://issues.apache.org/jira/browse/KAFKA-15857
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Satish Duggana


Introduce  EarliestLocalOffset and TieredOffset in OffsetSpec which will help 
in finding respective offsets while using AdminClient#listOffsets().

EarliestLocalOffset - local log start offset of a topic partition.
TieredOffset - Highest offset up to which the segments were copied to remote 
storage.

We can discuss further on naming and semantics of these offset specs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Satish Duggana]

Hi Christo,
I think we can start the vote thread on the KIP which is updated with the
finalized metrics. We can have followup KIPs with other metrics if needed
in future.

Thanks,
Satish.


On Fri, 17 Nov 2023 at 22:16, Christo Lolov  wrote:

> Heya all!
>
> I have updated the KIP so please have another read through when you have
> the time. I know we are cutting it a bit close, but I would be grateful if
> I could start a vote early next week in order to get this in 3.7.
>
> re: Satish
>
> 104. I envision that ideally we will compare this metric with
> *RemoteCopyLagSegments*, yes.
>
> re: Jorge
>
> I have been thinking about your suggestion for RemoteDeleteBytesPerSec. The
> *BytesPerSec make sense on the Copy and Read paths because there we
> actually write a whole segment or read it, but on the delete path we tend
> to just mark it for deletion, as such we don't really have deleted bytes
> per sec. Or am I misunderstanding why you want this metric added?
>
> I have also thought a bit more about the LocalDeleteLag and your
> description. If I understand correctly you propose this metric to monitor
> the segments expired due to local retention, have the .deleted suffix, but
> haven't yet been actually deleted by the LogCleaner. This will serve as a
> proxy for how much data which we should be serving from remote but we are
> serving from local? However, I believe that the moment we add the .delete
> suffix we stop serving traffic from those segments, hence we will be
> serving requests for those from remote. Am I wrong?
>
> Best,
> Christo
>
> On Thu, 16 Nov 2023 at 08:45, Satish Duggana 
> wrote:
>
> > Thanks Christo for your reply.
> >
> > 101 and 102 We have conclusion on them.
> >
> > 103. I am not strongly opinionated on this. I am fine if it is helpful
> > for your scenarios.
> >
> > 104. It seems you want to compare this metric with the number of
> > segments that are copied. Do you have such a metric now?
> >
> > Kamal and Luke,
> > I agree some of the metrics are needed outside of RSM layer in remote
> > fetch path. Can we take those fine grained remote fetch flow sequence
> > metrics separately later?
> >
> > Thanks,
> > Satish.
> >
> > On Tue, 14 Nov 2023 at 22:07, Christo Lolov 
> > wrote:
> > >
> > > Heya everyone,
> > >
> > > Apologies for the delay in my response and thank you very much for all
> > your
> > > comments! I will start answering in reverse:
> > >
> > > *re: Satish*
> > >
> > > 101. I am happy to scope down this KIP and start off by emitting those
> > > metrics on a topic level. I had a preference to emit them on a
> partition
> > > level because I have ran into situations where data wasn't evenly
> spread
> > > across partitions and not having that granularity made it harder to
> > > discover.
> > >
> > > 102. Fair enough, others have expressed the same preference. I will
> scope
> > > down the KIP to only bytes-based and segment-based metrics.
> > >
> > > 103. I agree that we could do this, but I personally prefer this to be
> a
> > > metric. At the very least a newcomer might not know to look for the log
> > > line, while most metric-collection systems allow you to explore the
> whole
> > > namespace. For example, I really dislike that while log loading happens
> > > Kafka emits log lines of "X/Y logs loaded" rather than just show me the
> > > progress via a metric. If you are strongly against this, however, I am
> > > happy to scope down on this as well.
> > >
> > > 104. Ideally we have only one metadata in remote storage for every
> > segment
> > > of the correct lineage. Due to leadership changes, however, this is not
> > > always the case. I envisioned that exposing such a metric will showcase
> > if
> > > there are problems with too many metadata records not part of the
> correct
> > > lineage of a log.
> > >
> > > *re: Luke*
> > >
> > > 1. I am a bit conflicted on this one. As discussed earlier with Jorge,
> in
> > > my head such metrics are better left to plugin implementations. If you
> > and
> > > Kamal feel strongly about this being included I will add it to the KIP.
> > >
> > > 2. After running tiered storage in production for a while I ran into
> > > problems where a partition-level metric would have allowed me to zone
> in
> > on
> > > the problem sooner. I tried balancing this with not exposing everything
&

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Satish Duggana]

Thanks Christo for your reply.

101 and 102 We have conclusion on them.

103. I am not strongly opinionated on this. I am fine if it is helpful
for your scenarios.

104. It seems you want to compare this metric with the number of
segments that are copied. Do you have such a metric now?

Kamal and Luke,
I agree some of the metrics are needed outside of RSM layer in remote
fetch path. Can we take those fine grained remote fetch flow sequence
metrics separately later?

Thanks,
Satish.

On Tue, 14 Nov 2023 at 22:07, Christo Lolov  wrote:
>
> Heya everyone,
>
> Apologies for the delay in my response and thank you very much for all your
> comments! I will start answering in reverse:
>
> *re: Satish*
>
> 101. I am happy to scope down this KIP and start off by emitting those
> metrics on a topic level. I had a preference to emit them on a partition
> level because I have ran into situations where data wasn't evenly spread
> across partitions and not having that granularity made it harder to
> discover.
>
> 102. Fair enough, others have expressed the same preference. I will scope
> down the KIP to only bytes-based and segment-based metrics.
>
> 103. I agree that we could do this, but I personally prefer this to be a
> metric. At the very least a newcomer might not know to look for the log
> line, while most metric-collection systems allow you to explore the whole
> namespace. For example, I really dislike that while log loading happens
> Kafka emits log lines of "X/Y logs loaded" rather than just show me the
> progress via a metric. If you are strongly against this, however, I am
> happy to scope down on this as well.
>
> 104. Ideally we have only one metadata in remote storage for every segment
> of the correct lineage. Due to leadership changes, however, this is not
> always the case. I envisioned that exposing such a metric will showcase if
> there are problems with too many metadata records not part of the correct
> lineage of a log.
>
> *re: Luke*
>
> 1. I am a bit conflicted on this one. As discussed earlier with Jorge, in
> my head such metrics are better left to plugin implementations. If you and
> Kamal feel strongly about this being included I will add it to the KIP.
>
> 2. After running tiered storage in production for a while I ran into
> problems where a partition-level metric would have allowed me to zone in on
> the problem sooner. I tried balancing this with not exposing everything on
> a partition level so not to explode the cardinality too much (point 101
> from Satish). I haven't ran into a situation where knowing the
> RemoteLogSizeComputationTime on a partition level helped me, but this
> doesn't mean there isn't one.
>
> 3. I was thinking that the metric can be emitted while reading of those
> records is happening i.e. if it takes a long time then it will just
> gradually increase as we read. What do you think?
>
> *re: Jorge*
>
> 3.5. Sure, I will aim to add my thoughts to the KIP
>
> 4. Let me check and come back to you on this one. I have a vague memory
> this wasn't as easy to calculate, but if it is, I will include
> RemoteDeleteBytesPerSec as well.
>
> 7. Yes, this is I believe a better explanation than the one I have in the
> KIP, so I will aim to update it with your one. Thank you! LocalDeleteLag
> makes sense to me as well, I will aim to include it.
>
> *re: Kamal*
>
> 1. I can add this to the KIP, but similar to what I have mentioned earlier,
> I believe these are better left to plugin implementations, no?
>
> 2. Yeah, this makes sense.
>
> Best,
> Christo
>
> On Fri, 10 Nov 2023 at 09:33, Satish Duggana 
> wrote:
>
> > Thanks Christo for the KIP and the interesting discussion.
> >
> > 101. Adding metrics at partition level may increase the cardinality of
> > these metrics. We should be cautious of that and see whether they are
> > really needed. RLM related operations do not generally affect based on
> > partition(s) but it is mostly because of the remote storage or broker
> > level issues.
> >
> > 102. I am not sure whether the records metric is much useful when we
> > have other bytes and segments related metrics available. If needed,
> > records level information can be derived once we have segments/bytes
> > metrics.
> >
> > 103. Regarding RemoteLogSizeComputationTime, we can add logs for
> > debugging purposes to print the required duration while computing size
> > instead of generating a metric. If there is any degradation in remote
> > log size computation, it will have an effect on RLM task leading to
> > remote log copy and delete lags.
> >
> > RLMM and RSM implementations can always add more metric

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

[Satish Duggana]

Thanks Qichao for the KIP.

+1 (binding)

~Satish.

On Thu, 16 Nov 2023 at 02:20, Jorge Esteban Quilcate Otoya
 wrote:
>
> Qichao, thanks again for leading this proposal!
>
> +1 (non-binding)
>
> Cheers,
> Jorge.
>
> On Wed, 15 Nov 2023 at 19:17, Divij Vaidya  wrote:
>
> > +1 (binding)
> >
> > I was involved in the discussion thread for this KIP and support it in its
> > current form.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics.
> > >
> > > Please take a look here:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> > >
> > > Best,
> > > Qichao Chu
> > >
> >

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Satish Duggana]

Thanks Christo for the KIP and the interesting discussion.

101. Adding metrics at partition level may increase the cardinality of
these metrics. We should be cautious of that and see whether they are
really needed. RLM related operations do not generally affect based on
partition(s) but it is mostly because of the remote storage or broker
level issues.

102. I am not sure whether the records metric is much useful when we
have other bytes and segments related metrics available. If needed,
records level information can be derived once we have segments/bytes
metrics.

103. Regarding RemoteLogSizeComputationTime, we can add logs for
debugging purposes to print the required duration while computing size
instead of generating a metric. If there is any degradation in remote
log size computation, it will have an effect on RLM task leading to
remote log copy and delete lags.

RLMM and RSM implementations can always add more metrics for
observability based on the respective implementations.

104. What is the purpose of RemoteLogMetadataCount as a metric?

Thanks,
Satish.

On Fri, 10 Nov 2023 at 04:10, Jorge Esteban Quilcate Otoya
 wrote:
>
> Hi Christo,
>
> I'd like to add another suggestion:
>
> 7. Adding on TS lag formulas, my understanding is that per pertition:
> - RemoteCopyLag: difference between: latest local segment candidate for
> upload - latest remote segment
>   - Represents how Remote Log Manager task is handling backlog of segments.
>   - Ideally, this lag is zero -- grows when upload is slower than the
> increase on candidate segments to upload
>
> - RemoteDeleteLag: difference between: latest remote candidate segment to
> keep based on retention - oldest remote segment
>   - Represents how many segments Remote Log Manager task is missing to
> delete at a given point in time
>   - Ideally, this lag is zero -- grows when retention condition changes but
> RLM task is not able to schedule deletion yet.
>
> Is my understanding of these lags correct?
>
> I'd like to also consider an additional lag:
> - LocalDeleteLag: difference between: latest local candidate segment to
> keep based on local retention - oldest local segment
>   - Represents how many segments are still available locally when they are
> candidate for deletion. This usually happens when log cleaner has not been
> scheduled yet. It's important because it represents how much data is stored
> locally when it could be removed, and it represents how much data that can
> be sourced from remote tier is still been sourced from local tier.
>   - Ideally, this lag returns to zero when log cleaner runs; but could be
> growing if there are issues uploading data (other lag) or removing data
> internally.
>
> Thanks,
> Jorge.
>
> On Thu, 9 Nov 2023 at 10:51, Luke Chen  wrote:
>
> > Hi Christo,
> >
> > Thanks for the KIP!
> >
> > Some comments:
> > 1. I agree with Kamal that a metric to cover the time taken to read data
> > from remote storage is helpful.
> >
> > 2. I can see there are some metrics are only on topic level, but some are
> > on partition level.
> > Could you explain why some of them are only on topic level?
> > Like RemoteLogSizeComputationTime, it's different from partition to
> > partition, will it be better to be exposed as partition metric?
> >
> > 3. `RemoteLogSizeBytes` metric hanging.
> > To compute the RemoteLogSizeBytes, we need to wait until all records in the
> > metadata topic loaded.
> > What will happen if it takes long to load the data from metadata topic?
> > Should we instead return -1 or something to indicate it's still loading?
> >
> > Thanks.
> > Luke
> >
> > On Fri, Nov 3, 2023 at 1:53 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi Christo,
> > >
> > > Thanks for expanding the scope of the KIP!  We should also cover the time
> > > taken to
> > > read data from remote storage. This will give our users a fair idea about
> > > the P99, P95,
> > > and P50 Fetch latency to read data from remote storage.
> > >
> > > The Fetch API request metrics currently provides a breakdown of the time
> > > spent on each item:
> > >
> > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/RequestChannel.scala#L517
> > > Should we also provide `RemoteStorageTimeMs` item (only for FETCH API) so
> > > that users can
> > > understand the overall and per-step time taken?
> > >
> > > Regarding the Remote deletion metrics, should we also emit a metric to
> > > expose the oldest segment time?
> > > Users can configure the topic retention either by size (or) time. If time
> > > is configured, then emitting
> > > the oldest segment time allows the user to configure an alert on top of
> > it
> > > and act accordingly.
> > >
> > > On Wed, Nov 1, 2023 at 7:07 PM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > > > Thanks, Christo!
> > > >
> > > > 1. Agree. Having a further look into how many latency metrics are
> > > included
> > > > on the broker side there

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Satish Duggana]

Thank you, everyone!

~Satish.

On Mon, 30 Oct 2023 at 16:14, Josep Prat  wrote:
>
> Congrats Satish!
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Oct 30, 2023, 11:37 Bruno Cadonna  wrote:
>
> > Congrats, Satish!
> >
> > Bruno
> >
> > On 10/29/23 2:42 PM, John Roesler wrote:
> > > Congratulations, Satish!
> > > -John
> > >
> > > On Sun, Oct 29, 2023, at 08:09, Randall Hauch wrote:
> > >> Congratulations, Satish!
> > >>
> > >> On Sun, Oct 29, 2023 at 1:47 AM Tom Bentley 
> > wrote:
> > >>
> > >>> Congratulations!
> > >>>
> > >>> On Sun, 29 Oct 2023 at 5:41 PM, Guozhang Wang <
> > guozhang.wang...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Congratulations Satish!
> > >>>>
> > >>>> On Sat, Oct 28, 2023 at 12:59 AM Luke Chen  wrote:
> > >>>>>
> > >>>>> Congrats Satish!
> > >>>>>
> > >>>>> Luke
> > >>>>>
> > >>>>> On Sat, Oct 28, 2023 at 11:16 AM ziming deng <
> > dengziming1...@gmail.com
> > >>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Congratulations Satish!
> > >>>>>>
> > >>>>>>> On Oct 27, 2023, at 23:03, Jun Rao 
> > >>> wrote:
> > >>>>>>>
> > >>>>>>> Hi, Everyone,
> > >>>>>>>
> > >>>>>>> Satish Duggana has been a Kafka committer since 2022. He has been
> > >>>> very
> > >>>>>>> instrumental to the community since becoming a committer. It's my
> > >>>>>> pleasure
> > >>>>>>> to announce that Satish is now a member of Kafka PMC.
> > >>>>>>>
> > >>>>>>> Congratulations Satish!
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>> on behalf of Apache Kafka PMC
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> >

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Satish Duggana]

Thanks Luke for volunteering for 3.5.2 release.

On Tue, 17 Oct 2023 at 11:58, Josep Prat  wrote:
>
> Hi Luke,
>
> Thanks for taking this one!
>
> Best,
>
> On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
> > have an important bug/vulnerability fix release for 3.5.1.
> >
> > If there are no objections, I'll start building a release plan in thewiki
> > in the next couple of weeks.
> >
> > Thanks,
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-15612) Followup on whether the indexes need to be materialized before they are passed to RSM for writing them to tiered storage.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15612:
--

 Summary: Followup on whether the indexes need to be materialized 
before they are passed to RSM for writing them to tiered storage. 
 Key: KAFKA-15612
 URL: https://issues.apache.org/jira/browse/KAFKA-15612
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana


Followup on the [PR 
comment|https://github.com/apache/kafka/pull/14529#discussion_r1355263700]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Apache Kafka 3.6.0

[Satish Duggana]

I'd like to clarify calling out 3.6.0 as a minor release.

We use semantic versioning*(Major.Minor.Patch) in Apache Kafka
releases. Releases like 3.5.0 or 3.6.0 are called minor releases
according to the semantic versioning. The next major release will be
4.0.0.

3.6.0 release is packed with big features and improvements as
mentioned in the blog post[1], and release notes[2], even though it is
called a minor release :)

1. https://kafka.apache.org/blog
2. https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html
* https://semver.org/

Thanks,
Satish.


On Wed, 11 Oct 2023 at 12:00, Luke Chen  wrote:
>
> Thanks for running the release, Satish!
>
> BTW, 3.6.0 should be a major release, not a minor one. :)
>
> Luke
>
> On Wed, Oct 11, 2023 at 1:39 PM Satish Duggana  wrote:
>
> > The Apache Kafka community is pleased to announce the release for
> > Apache Kafka 3.6.0
> >
> > This is a minor release and it includes fixes and improvements from 238
> > JIRAs.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html
> >
> > An overview of the release can be found in our announcement blog post:
> > https://kafka.apache.org/blog
> >
> > You can download the source and binary release (Scala 2.12 and Scala 2.13)
> > from:
> > https://kafka.apache.org/downloads#3.6.0
> >
> >
> > ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 139 contributors to this release!
> > (Please report an unintended omission)
> >
> > This was a community effort, so thank you to everyone who contributed
> > to this release, including all our users and our 139 contributors:
> > A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
> > Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
> > Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
> > Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
> > bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
> > Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
> > Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
> > Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
> > David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
> > Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
> > Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
> > Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
> > Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
> > iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
> > Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
> > Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
> > Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
> > Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas
> > Brutschy, Lucent-Wong, Lucia Cerchie, Luke Ch

[jira] [Created] (KAFKA-15594) Add 3.6.0 to streams upgrade/compatibility tests

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15594:
--

 Summary: Add 3.6.0 to streams upgrade/compatibility tests
 Key: KAFKA-15594
 URL: https://issues.apache.org/jira/browse/KAFKA-15594
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15593) Add 3.6.0 to broker/client upgrade/compatibility tests

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15593:
--

 Summary: Add 3.6.0 to broker/client upgrade/compatibility tests
 Key: KAFKA-15593
 URL: https://issues.apache.org/jira/browse/KAFKA-15593
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15576) Add 3.2.0 to broker/client and streams upgrade/compatibility tests

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15576:
--

 Summary: Add 3.2.0 to broker/client and streams 
upgrade/compatibility tests
 Key: KAFKA-15576
 URL: https://issues.apache.org/jira/browse/KAFKA-15576
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Kafka 3.6.0

[Satish Duggana]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.6.0

This is a minor release and it includes fixes and improvements from 238 JIRAs.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html

An overview of the release can be found in our announcement blog post:
https://kafka.apache.org/blog

You can download the source and binary release (Scala 2.12 and Scala 2.13) from:
https://kafka.apache.org/downloads#3.6.0

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 139 contributors to this release!
(Please report an unintended omission)

This was a community effort, so thank you to everyone who contributed
to this release, including all our users and our 139 contributors:
A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas
Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, Manikumar Reddy,
Manyanda Chitimbo, Maros Orsak, Matthew de Detrich, Matthias J. Sax,
maulin-vasavada, Max Riedel, Mehari Beyene, Michal Cabak (@miccab),
Mickael Maison, Milind Mantri, minjian.cai, mojh7, Nikolay, Okada
Haruki, Omnia G H Ibrahim, Owen Leung, Philip Nee, prasanthV, Proven
Provenzano, Purshotam Chauhan, Qichao Chu, Rajini Sivaram, Randall
Hauch, Renaldo Baur Filho, Ritika Reddy, Rittika Adhikari, Rohan, Ron
Dagostino, Sagar Rao, Said Boudjelda, Sambhav Jain, Satish Duggana,
sciclon2, Shekhar Rajak, Sungyun Hur, Sushant Mahajan, Tanay
Karmarkar, tison, Tom Bentley, vamossagar12, Victoria Xia, Vincent
Jiang, vveicc, Walker Carlson, Yash Mayya, Yi-Sheng Lien, Ziming Deng,
蓝士钦

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Satish Duggana

[jira] [Resolved] (KAFKA-15535) Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property.

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15535.

Resolution: Fixed

> Add documentation of "remote.log.index.file.cache.total.size.bytes" 
> configuration property. 
> 
>
> Key: KAFKA-15535
> URL: https://issues.apache.org/jira/browse/KAFKA-15535
> Project: Kafka
>  Issue Type: Task
>  Components: documentation
>Reporter: Satish Duggana
>Assignee: hudeqi
>Priority: Major
>  Labels: tiered-storage
> Fix For: 3.7.0
>
>
> Add documentation of "remote.log.index.file.cache.total.size.bytes" 
> configuration property. 
> Please double check all the existing public tiered storage configurations. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

+1 to add it as part of upgrade page. Users refer to the upgrade page
to access any information related to upgrades for a release as it is
expected to contain all the relevant text and references regarding the
upgrade in that particular release.

David,

On Thu, 5 Oct 2023 at 17:22, Ismael Juma  wrote:
>
> The upgrade page is the right place to add it, in my opinion.
>
> Ismael
>
> On Thu, Oct 5, 2023 at 5:02 PM David Arthur
>  wrote:
>
> > Hey folks, we found a significant ZK migration bug today
> > https://issues.apache.org/jira/browse/KAFKA-15552. Since the release is
> > already voted in, the fix will wait for 3.6.1. In the meantime, do we have
> > a way to indicate known significant issues in the release notes? Maybe just
> > a note in the blog?
> >
> > Thanks!
> > David
> >
> > On Fri, Sep 29, 2023 at 1:42 PM Ismael Juma  wrote:
> >
> > > Thanks Satish!
> > >
> > > Ismael
> > >
> > > On Thu, Sep 28, 2023 at 6:39 AM Satish Duggana  > >
> > > wrote:
> > >
> > > > We do not have any pending release blockers for now. RC1 release
> > > > blocker KAFKA-15498[1] is merged to 3.6.
> > > >  I will create RC2 by 29th Sep 12:00 pm PT and start a new RC thread.
> > > >
> > > > 1. https://github.com/apache/kafka/pull/14434
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > >
> > > > On Wed, 27 Sept 2023 at 13:36, Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > Ismael,
> > > > > Thank you for checking.
> > > > > Multiple other folks have validated after I left the comment here
> > that
> > > > > it doesn't impact log truncation and hence won't lead to data loss. I
> > > > > agree that it's not a blocker.
> > > > >
> > > > > (ref: https://github.com/apache/kafka/pull/14457)
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > > On Wed, Sep 27, 2023 at 8:50 PM Ismael Juma 
> > wrote:
> > > > > >
> > > > > > Doesn't look like a blocker to me.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Sep 27, 2023 at 2:36 AM Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey team
> > > > > > >
> > > > > > > I need help in determining whether
> > > > > > > https://github.com/apache/kafka/pull/14457 is a release blocker
> > > bug
> > > > or
> > > > > > > not. If someone is familiar with replication protocol (on the log
> > > > > > > diverange and reconciliation process), please add your comments
> > on
> > > > the
> > > > > > > PR.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > > On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > A community member reported another bug in TS feature in 3.6 -
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15511
> > > > > > > >
> > > > > > > > I don't consider it as a blocker for release because the bug
> > > > occurs in
> > > > > > > > rare situations when the index on disk or in a remote store is
> > > > > > > > corrupted and fails a sanity check.
> > > > > > > > Sharing it here as an FYI.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Divij Vaidya
> > > > > > > >
> > > > > > > > On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Found a bug while testing TS feature in 3.6 -
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15481
> > > > > > > > >
> > > > > > > > > I don&

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

Hi,
Thank you all for participating in the 3.6.0 release candidate vote threads.

This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes.

+1 votes
PMC Members:
* Luke Chen
* Bill Bejeck
* Chris Egerton
* Justine Olshan

Community:
* Jakub Scholz
* Federico Valeri
* Divij Vaidya
* Kamal Chandraprakash
* Josep Prat
* Proven Provenzano
* Greg Harris

0 votes
* No votes

-1 votes
* No votes

I will continue with the release process and the release announcement
will follow in the next few days.

Thanks,
Satish.

On Tue, 3 Oct 2023 at 12:35, Satish Duggana  wrote:
>
> Hi,
> Thank you all for participating in the 3.6.0 release candidate vote threads.
>
> This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes.
>
> +1 votes
> PMC Members:
> * Luke Chen
> * Bill Bejeck
> * Chris Egerton
> * Justine Olshan
>
> Community:
> * Jakub Scholz
> * Federico Valeri
> * Divij Vaidya
> * Kamal Chandraprakash
> * Josep Prat
> * Proven Provenzano
> * Greg Harris
>
> 0 votes
> * No votes
>
> -1 votes
> * No votes
>
> I will continue with the release process and the release announcement
> will follow in the next few days.
>
> Thanks,
> Satish.
>
> On Tue, 3 Oct 2023 at 09:54, Justine Olshan
>  wrote:
> >
> > Thanks folks for following up. Given my previous testing and the results
> > you've provided, I'm +1 (binding)
> >
> > I will also follow up with the non-blocking metrics documentation.
> >
> > Thanks!
> > Justine
> >
> > On Tue, Oct 3, 2023 at 8:17 AM Chris Egerton 
> > wrote:
> >
> > > Hi Satish,
> > >
> > > Thanks for running this release!
> > >
> > > To verify, I:
> > > - Built from source using Java 11 with both:
> > > - - the 3.6.0-rc2 tag on GitHub
> > > - - the kafka-3.6.0-src.tgz artifact from
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > - Checked signatures and checksums
> > > - Ran the quickstart using the kafka_2.13-3.6.0.tgz artifact from
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/ with Java 11 and Scala
> > > 13
> > > in KRaft mode
> > > - Ran all unit tests
> > > - Ran all integration tests for Connect and MM2
> > > - Verified that the connect-test-plugins module is present in the staging
> > > Maven artifacts (https://issues.apache.org/jira/browse/KAFKA-15249)
> > >
> > > Everything looks good to me!
> > >
> > > +1 (binding)
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Oct 3, 2023 at 6:43 AM Satish Duggana 
> > > wrote:
> > >
> > > > Thanks Luke for helping on running system tests on RCs and updating
> > > > the status on this email thread.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 3 Oct 2023 at 05:04, Luke Chen  wrote:
> > > > >
> > > > > Hi Justine and all,
> > > > >
> > > > > The system test result for 3.6.0 RC2 can be found below.
> > > > > In short, no failed tests. The flaky tests will pass in the 2nd run.
> > > > >
> > > >
> > > https://drive.google.com/drive/folders/1qwIKg-B4CBrswUeo5fBRv65KWpDsGUiS?usp=sharing
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Tue, Oct 3, 2023 at 7:08 AM Justine Olshan
> > > > 
> > > > > wrote:
> > > > >
> > > > > > I realized Luke shared the results here for RC1
> > > > > >
> > > > https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC
> > > > > > Given we had some runs that looked reasonable, and we made a small
> > > > change,
> > > > > > I'm ok with this. But I wouldn't be upset if we had another set of
> > > > runs :)
> > > > > >
> > > > > > As for the validation:
> > > > > >
> > > > > >- I've compiled from source with java 17, 2.13, run the
> > > > transactional
> > > > > >produce bench
> > > > > >- Run unit tests
> > > > > >- Validated the checksums
> > > > > >- Downloaded and ran the 2.12 version of the release
> > > > > >- Briefly took a look at the documentation
> > > > > >- I was browsing through the site html files and I noticed the
> > > html
> > >

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

Hi,
Thank you all for participating in the 3.6.0 release candidate vote threads.

This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes.

+1 votes
PMC Members:
* Luke Chen
* Bill Bejeck
* Chris Egerton
* Justine Olshan

Community:
* Jakub Scholz
* Federico Valeri
* Divij Vaidya
* Kamal Chandraprakash
* Josep Prat
* Proven Provenzano
* Greg Harris

0 votes
* No votes

-1 votes
* No votes

I will continue with the release process and the release announcement
will follow in the next few days.

Thanks,
Satish.

On Tue, 3 Oct 2023 at 09:54, Justine Olshan
 wrote:
>
> Thanks folks for following up. Given my previous testing and the results
> you've provided, I'm +1 (binding)
>
> I will also follow up with the non-blocking metrics documentation.
>
> Thanks!
> Justine
>
> On Tue, Oct 3, 2023 at 8:17 AM Chris Egerton 
> wrote:
>
> > Hi Satish,
> >
> > Thanks for running this release!
> >
> > To verify, I:
> > - Built from source using Java 11 with both:
> > - - the 3.6.0-rc2 tag on GitHub
> > - - the kafka-3.6.0-src.tgz artifact from
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > - Checked signatures and checksums
> > - Ran the quickstart using the kafka_2.13-3.6.0.tgz artifact from
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/ with Java 11 and Scala
> > 13
> > in KRaft mode
> > - Ran all unit tests
> > - Ran all integration tests for Connect and MM2
> > - Verified that the connect-test-plugins module is present in the staging
> > Maven artifacts (https://issues.apache.org/jira/browse/KAFKA-15249)
> >
> > Everything looks good to me!
> >
> > +1 (binding)
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Oct 3, 2023 at 6:43 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Luke for helping on running system tests on RCs and updating
> > > the status on this email thread.
> > >
> > > ~Satish.
> > >
> > > On Tue, 3 Oct 2023 at 05:04, Luke Chen  wrote:
> > > >
> > > > Hi Justine and all,
> > > >
> > > > The system test result for 3.6.0 RC2 can be found below.
> > > > In short, no failed tests. The flaky tests will pass in the 2nd run.
> > > >
> > >
> > https://drive.google.com/drive/folders/1qwIKg-B4CBrswUeo5fBRv65KWpDsGUiS?usp=sharing
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Tue, Oct 3, 2023 at 7:08 AM Justine Olshan
> > > 
> > > > wrote:
> > > >
> > > > > I realized Luke shared the results here for RC1
> > > > >
> > > https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC
> > > > > Given we had some runs that looked reasonable, and we made a small
> > > change,
> > > > > I'm ok with this. But I wouldn't be upset if we had another set of
> > > runs :)
> > > > >
> > > > > As for the validation:
> > > > >
> > > > >- I've compiled from source with java 17, 2.13, run the
> > > transactional
> > > > >produce bench
> > > > >- Run unit tests
> > > > >- Validated the checksums
> > > > >- Downloaded and ran the 2.12 version of the release
> > > > >- Briefly took a look at the documentation
> > > > >- I was browsing through the site html files and I noticed the
> > html
> > > for
> > > > >documentation.html seemed to be for 3.4. Not sure if this is a
> > > blocker,
> > > > > but
> > > > >wanted to flag it. This seems to be the case for the previous
> > > release
> > > > >candidates as well. (As well as 3.5 release it seems)
> > > > >
> > > > >
> > > > > I will hold off on voting until we figure that part out. I will also
> > > follow
> > > > > up with the documentation Divij mentioned outside this thread.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Mon, Oct 2, 2023 at 3:05 PM Greg Harris
> > > 
> > > > > wrote:
> > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > I verified KIP-898 functionality and the KAFKA-15473 patch.
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks!
> > > >

[jira] [Created] (KAFKA-15535) Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15535:
--

 Summary: Add documentation of 
"remote.log.index.file.cache.total.size.bytes" configuration property. 
 Key: KAFKA-15535
 URL: https://issues.apache.org/jira/browse/KAFKA-15535
 Project: Kafka
  Issue Type: Task
  Components: documentation
Reporter: Satish Duggana
 Fix For: 3.7.0


Add documentation of "remote.log.index.file.cache.total.size.bytes" 
configuration property. 

Please double check all the existing public tiered storage configurations. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

d the release
> > manager
> > > > > > public keys are available at
> > > > > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > > > > >
> > > > > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are
> > also
> > > > > > being emitted in 3.6.0 (with Zk).
> > > > > >
> > > > > > Problems (but not blockers):
> > > > > > 1. Metrics added in
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > > > > aren't available in the documentation (cc: Justine). I don't
> > consider
> > > > > this
> > > > > > as a release blocker but we should add it as a fast follow-up.
> > > > > >
> > > > > > 2. Metric added in
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > > > > isn't available in documentation (cc: David). I don't consider
> > this as
> > > > a
> > > > > > release blocker but we should add it as a fast follow-up.
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri <
> > fedeval...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Satish, I did the following to verify the release:
> > > > > > >
> > > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > - Ran all unit and integration tests
> > > > > > > - Spot checked documentation
> > > > > > > - Ran custom client applications using staging artifacts on a
> > 3-nodes
> > > > > > > cluster
> > > > > > > - Tested tiered storage with one of the available RSM
> > implementations
> > > > > > >
> > > > > > > +1 (non binding)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Fede
> > > > > > >
> > > > > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I verified with:
> > > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > > 2. Making sure the checksum are correct
> > > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > > > 4. Verified the tiered storage feature works well.
> > > > > > > >
> > > > > > > > +1 (binding).
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > Luke
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz 
> > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and the
> > staged
> > > > > Maven
> > > > > > > > > artifacts and run my tests. Everything seems to work fine
> > for me.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Jakub
> > > > > > > > >
> > > > > > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > > > > > satish.dugg...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > > >
> > > > > > > > > > This is the third candidate for the release of Apache Kafka
> > > > > 3.6.0.
> > > > > > > > > > Some of the major features include:
> > > > > > > > > >
> > > > > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > > > > > * KIP-917: Additional custom metadata for remote log
> > segment
> > > > > > > > > >
> > > > > > > > > > Release notes for the 3.6.0 release:
> > > > > > > > > >
> > > > > >
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > > > > > > >
> > > > > > > > > > *** Please download, test and vote by Tuesday, October 3,
> > 12pm
> > > > PT
> > > > > > > > > >
> > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > release:
> > > > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > > > >
> > > > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > > > > > > >
> > > > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > > > >
> > > > > > >
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > >
> > > > > > > > > > * Javadoc:
> > > > > > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > > > > > > > > >
> > > > > > > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2
> > tag:
> > > > > > > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > > > > > > > > >
> > > > > > > > > > * Documentation:
> > > > > > > > > > https://kafka.apache.org/36/documentation.html
> > > > > > > > > >
> > > > > > > > > > * Protocol:
> > > > > > > > > > https://kafka.apache.org/36/protocol.html
> > > > > > > > > >
> > > > > > > > > > * Successful Jenkins builds for the 3.6 branch:
> > > > > > > > > > There are a few runs of unit/integration tests. You can
> > see the
> > > > > > > latest
> > > > > > > > > > at
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/.
> > > > We
> > > > > > > will
> > > > > > > > > > continue running a few more iterations.
> > > > > > > > > > System tests:
> > > > > > > > > > We will send an update once we have the results.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Satish.
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

> > > > > overall lower P99.8 latencies.
> > > > > > >
> > > > > > > 2. I have verified that detached signature is correct using
> > > > > > > https://www.apache.org/info/verification.html and the release
> > > manager
> > > > > > > public keys are available at
> > > > > > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > > > > > >
> > > > > > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk)
> > are
> > > also
> > > > > > > being emitted in 3.6.0 (with Zk).
> > > > > > >
> > > > > > > Problems (but not blockers):
> > > > > > > 1. Metrics added in
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > > > > > aren't available in the documentation (cc: Justine). I don't
> > > consider
> > > > > > this
> > > > > > > as a release blocker but we should add it as a fast follow-up.
> > > > > > >
> > > > > > > 2. Metric added in
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > > > > > isn't available in documentation (cc: David). I don't consider
> > > this as
> > > > > a
> > > > > > > release blocker but we should add it as a fast follow-up.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri <
> > > fedeval...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish, I did the following to verify the release:
> > > > > > > >
> > > > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > > - Ran all unit and integration tests
> > > > > > > > - Spot checked documentation
> > > > > > > > - Ran custom client applications using staging artifacts on a
> > > 3-nodes
> > > > > > > > cluster
> > > > > > > > - Tested tiered storage with one of the available RSM
> > > implementations
> > > > > > > >
> > > > > > > > +1 (non binding)
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Fede
> > > > > > > >
> > > > > > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen 
> > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > >
> > > > > > > > > I verified with:
> > > > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > > > 2. Making sure the checksum are correct
> > > > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > > > > 4. Verified the tiered storage feature works well.
> > > > > > > > >
> > > > > > > > > +1 (binding).
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and the
> > > staged
> > > > > > Maven
> > > > > > > > > > artifacts and run my tests. Everything seems to work fine
> > > for me.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Jakub
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > > > > > > satish.dugg...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > > > >
> > > > > > > > > > > This is the third candidate for the release of Apache
> > Kafka
> > > > > > 3.6.0.
> > > > > > > > > > > Some of the major features include:
> > > > > > > > > > >
> > > > > > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > > > > > > * KIP-938: Add more metrics for measuring KRaft
> > performance
> > > > > > > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > > > > > > * KIP-917: Additional custom metadata for remote log
> > > segment
> > > > > > > > > > >
> > > > > > > > > > > Release notes for the 3.6.0 release:
> > > > > > > > > > >
> > > > > > >
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > > > > > > > >
> > > > > > > > > > > *** Please download, test and vote by Tuesday, October 3,
> > > 12pm
> > > > > PT
> > > > > > > > > > >
> > > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > > release:
> > > > > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > > > > >
> > > > > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > > > > > > > >
> > > > > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > > >
> > > > > > > > > > > * Javadoc:
> > > > > > > > > > >
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > > > > > > > > > >
> > > > > > > > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2
> > > tag:
> > > > > > > > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > > > > > > > > > >
> > > > > > > > > > > * Documentation:
> > > > > > > > > > > https://kafka.apache.org/36/documentation.html
> > > > > > > > > > >
> > > > > > > > > > > * Protocol:
> > > > > > > > > > > https://kafka.apache.org/36/protocol.html
> > > > > > > > > > >
> > > > > > > > > > > * Successful Jenkins builds for the 3.6 branch:
> > > > > > > > > > > There are a few runs of unit/integration tests. You can
> > > see the
> > > > > > > > latest
> > > > > > > > > > > at
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/.
> > > > > We
> > > > > > > > will
> > > > > > > > > > > continue running a few more iterations.
> > > > > > > > > > > System tests:
> > > > > > > > > > > We will send an update once we have the results.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Satish.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

Thanks Divij for the observations.

KAFKA-15483 is added to address the missing metrics related to KRaft
and ZK to KRaft migration. Please leave a comment at
https://github.com/apache/kafka-site/pull/548 in which related metrics
documentation is added.

https://issues.apache.org/jira/browse/KAFKA-15530 is raised to
followup on the missing documentation of metrics introduced in
KIP-890.

~Satish.

On Mon, 2 Oct 2023 at 05:37, Divij Vaidya  wrote:
>
> + 1 (non-binding)
>
> Verifications:
> 1. I ran a produce-consume workload with plaintext auth, JDK17, zstd
> compression using an open messaging benchmark and found 3.6 to be better
> than or equal to 3.5.1 across all dimensions. Notably, 3.6 had consistently
> 6-7% lower CPU utilization, lesser spikes on P99 produce latencies and
> overall lower P99.8 latencies.
>
> 2. I have verified that detached signature is correct using
> https://www.apache.org/info/verification.html and the release manager
> public keys are available at
> https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
>
> 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are also
> being emitted in 3.6.0 (with Zk).
>
> Problems (but not blockers):
> 1. Metrics added in
> https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> aren't available in the documentation (cc: Justine). I don't consider this
> as a release blocker but we should add it as a fast follow-up.
>
> 2. Metric added in
> https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> isn't available in documentation (cc: David). I don't consider this as a
> release blocker but we should add it as a fast follow-up.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri  wrote:
>
> > Hi Satish, I did the following to verify the release:
> >
> > - Built from source with Java 17 and Scala 2.13
> > - Ran all unit and integration tests
> > - Spot checked documentation
> > - Ran custom client applications using staging artifacts on a 3-nodes
> > cluster
> > - Tested tiered storage with one of the available RSM implementations
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I verified with:
> > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > 2. Making sure the checksum are correct
> > > 3. Browsing release notes, documents, javadocs, protocols.
> > > 4. Verified the tiered storage feature works well.
> > >
> > > +1 (binding).
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > >
> > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven
> > > > artifacts and run my tests. Everything seems to work fine for me.
> > > >
> > > > Thanks
> > > > Jakub
> > > >
> > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the third candidate for the release of Apache Kafka 3.6.0.
> > > > > Some of the major features include:
> > > > >
> > > > > * KIP-405 : Kafka Tiered Storage
> > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > >
> > > > > Release notes for the 3.6.0 release:
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Tuesday, October 3, 12pm PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >

[jira] [Created] (KAFKA-15530) Add missing documentation of metrics introduced as part of KAFKA-15196

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15530:
--

 Summary: Add missing documentation of metrics introduced as part 
of KAFKA-15196
 Key: KAFKA-15530
 URL: https://issues.apache.org/jira/browse/KAFKA-15530
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana


This is a followup to the 3.6.0 RC2 verification email 
[thread|https://lists.apache.org/thread/js2nmq3ggn46qg122h4jg5p2fcq5hr2s]. Add 
the missing documentation of a few metrics added as part of the 
[change|https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] 3.6.0 RC2

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the third candidate for the release of Apache Kafka 3.6.0.
Some of the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, October 3, 12pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc2

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest
at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
continue running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

We do not have any pending release blockers for now. RC1 release
blocker KAFKA-15498[1] is merged to 3.6.
 I will create RC2 by 29th Sep 12:00 pm PT and start a new RC thread.

1. https://github.com/apache/kafka/pull/14434

Thanks,
Satish.



On Wed, 27 Sept 2023 at 13:36, Divij Vaidya  wrote:
>
> Ismael,
> Thank you for checking.
> Multiple other folks have validated after I left the comment here that
> it doesn't impact log truncation and hence won't lead to data loss. I
> agree that it's not a blocker.
>
> (ref: https://github.com/apache/kafka/pull/14457)
>
> --
> Divij Vaidya
>
> On Wed, Sep 27, 2023 at 8:50 PM Ismael Juma  wrote:
> >
> > Doesn't look like a blocker to me.
> >
> > Ismael
> >
> > On Wed, Sep 27, 2023 at 2:36 AM Divij Vaidya 
> > wrote:
> >
> > > Hey team
> > >
> > > I need help in determining whether
> > > https://github.com/apache/kafka/pull/14457 is a release blocker bug or
> > > not. If someone is familiar with replication protocol (on the log
> > > diverange and reconciliation process), please add your comments on the
> > > PR.
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya 
> > > wrote:
> > > >
> > > > A community member reported another bug in TS feature in 3.6 -
> > > > https://issues.apache.org/jira/browse/KAFKA-15511
> > > >
> > > > I don't consider it as a blocker for release because the bug occurs in
> > > > rare situations when the index on disk or in a remote store is
> > > > corrupted and fails a sanity check.
> > > > Sharing it here as an FYI.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > > On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya 
> > > wrote:
> > > > >
> > > > > Found a bug while testing TS feature in 3.6 -
> > > > > https://issues.apache.org/jira/browse/KAFKA-15481
> > > > >
> > > > > I don't consider it as a blocker for release since it's a concurrency
> > > > > bug that should occur rarely for a feature which is early access.
> > > > > Sharing it here as FYI in case someone else thinks differently.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > > On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana <
> > > satish.dugg...@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Divij for raising a PR for doc formatting issue.
> > > > > >
> > > > > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya, 
> > > > > > 
> > > wrote:
> > > > > >
> > > > > > > Hey Satish
> > > > > > >
> > > > > > > I filed a PR to fix the website formatting bug in 3.6
> > > documentation -
> > > > > > > https://github.com/apache/kafka/pull/14419
> > > > > > > Please take a look when you get a chance.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton
> > > 
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I think this qualifies as a blocker. This API has been around
> > > for years
> > > > > > > now
> > > > > > > > and, while we don't document it as not exposing duplicates*, it
> > > has come
> > > > > > > > with that implicit contract since its inception. More
> > > importantly, it has
> > > > > > > > also never exposed plugins that cannot be used on the worker.
> > > This change
> > > > > > > > in behavior not only introduces duplicates*, it causes
> > > unreachable
> > > > > > > plugins
> > > > > > > > to be displayed. With this in mind, it seems to qualify pretty
> > > clearly
> > > > > > > as a
> > > > > > > > regression and we should not put out a release that includes it.
> > > > > > > >
> > > > > > > > * - Really, these aren't duplicates; rather, they&

[jira] [Created] (KAFKA-15503) CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15503:
--

 Summary: CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 
10.0.16, 11.0.16, 12.0.1
 Key: KAFKA-15503
 URL: https://issues.apache.org/jira/browse/KAFKA-15503
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.7.0, 2.6.1, 3.4.1, 3.6.0, 3.5.1
Reporter: Rafael Rios Saavedra
Assignee: Divij Vaidya
 Fix For: 2.8.0, 2.7.1, 2.6.2, 3.0.0


CVE-2023-40167 and CVE-2023-36479 vulnerabilities affects Jetty version 
{*}9.4.51{*}. For more information see 
[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40167] 
[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-364749] 

Upgrading to Jetty version *9.4.52, 10.0.16, 11.0.16, 12.0.1* should address 
this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Luke for reporting the High CVE issue. Ismael also confirmed
that High CVEs are generally considered as release blockers.

Agree with all that we should consider this CVE as a release blocker.
We will include other CVE also with the next RC.

~Satish.


On Mon, 25 Sept 2023 at 03:04, Luke Chen  wrote:
>
> Hi Divij,
>
> About the system tests, it's me helping Satish working on it since our team
> has internal jenkins pipeline for it.
> Here's the result:
> https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC?usp=sharing
>
> I'm mainly focusing on the failed tests.
> For kraft_upgrade_test, I've fixed in this PR
> <https://github.com/apache/kafka/pull/14424>. After the fix, the rerun
> results look good.
> For quota_test, it's a known issue in our environment. After rerun, all
> passed.
>
> Thanks.
> Luke
>
>
>
> On Mon, Sep 25, 2023 at 4:09 PM Divij Vaidya 
> wrote:
>
> > Correction: posted the wrong JIRA in my previous email. Instead of
> > https://issues.apache.org/jira/browse/KAFKA-15001, please consider
> > this https://issues.apache.org/jira/browse/KAFKA-15487
> >
> > --
> > Divij Vaidya
> >
> > On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya 
> > wrote:
> > >
> > > Hi Satish
> > >
> > > 1. I agree with Luke. It's a "high" severity vulnerability and we
> > > should create another RC with the upgraded Snappy version. If we
> > > create another RC, we should also fix a different CVE resported in
> > > https://issues.apache.org/jira/browse/KAFKA-15001
> > >
> > > 2. I was hoping you could post the results of system tests before I
> > > vote on this. I am particularly interested in looking at
> > > producer/consumer performance results since we have quite a few
> > > changes in this release. What is the plan on the system tests?
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Mon, Sep 25, 2023 at 9:10 AM Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > Snappy-java published a new vulnerability
> > > > <
> > https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv
> > >
> > > > that will cause OOM error in the server.
> > > > Kafka is also impacted by this vulnerability since it's like
> > CVE-2023-34455
> > > > <https://nvd.nist.gov/vuln/detail/CVE-2023-34455>.
> > > > We'd better bump the snappy-java version to bypass this vulnerability.
> > > > PR <https://github.com/apache/kafka/pull/14434> is created to run the
> > CI
> > > > build.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks to everyone who voted for this release.
> > > > >
> > > > > We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> > > > > deadline. Please try RC1 and send your vote to this email thread.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > >
> > > > > On Sun, 24 Sept 2023 at 13:23, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I've done the following:
> > > > > > - Verified signature
> > > > > > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > > > > > - Run unit + integration tests
> > > > > > - Ran a shorter Trogdor transactional-produce-bench on a single
> > broker
> > > > > > cluster (KRaft and ZK) to verify transactional workloads worked
> > > > > reasonably
> > > > > >
> > > > > > Minor thing (we can discuss elsewhere and is non-blocking for the
> > > > > release)
> > > > > > but if ZK has been deprecated since 3.5 we should move up the
> > Kraft setup
> > > > > > in the quickstart guide  <http://goog_2103708782>here
> > > > > > <https://kafka.apache.org/quickstart>.
> > > > > >
> > > > > > +1 (binding) from me.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On S

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks to everyone who voted for this release.

We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
deadline. Please try RC1 and send your vote to this email thread.

Thanks,
Satish.


On Sun, 24 Sept 2023 at 13:23, Justine Olshan
 wrote:
>
> Hi Satish,
>
> I've done the following:
> - Verified signature
> - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> - Run unit + integration tests
> - Ran a shorter Trogdor transactional-produce-bench on a single broker
> cluster (KRaft and ZK) to verify transactional workloads worked reasonably
>
> Minor thing (we can discuss elsewhere and is non-blocking for the release)
> but if ZK has been deprecated since 3.5 we should move up the Kraft setup
> in the quickstart guide  <http://goog_2103708782>here
> <https://kafka.apache.org/quickstart>.
>
> +1 (binding) from me.
>
> Justine
>
> On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri 
> wrote:
>
> > Hi Satish, I did the following to verify the release:
> >
> > - Verified signature and checksum
> > - Built from source with Java 17 and Scala 2.13
> > - Ran all unit and integration tests
> > - Spot checked release notes and documentation
> > - Ran a custom client using staging artifacts on a 3-nodes cluster
> > - Tested tiered storage with one of the available RSM implementations
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> >
> > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I verified with:
> > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > 2. Making sure the checksum are correct
> > > 3. Browsing release notes, documents, javadocs, protocols.
> > >
> > > I filed KAFKA-15491 <https://issues.apache.org/jira/browse/KAFKA-15491
> > >for
> > > log output improvement while testing stream application.
> > > It won't be blocker in v3.6.0.
> > >
> > > For KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489>, I'm
> > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > >
> > > +1 (binding) from me.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:
> > >
> > > > Given that this is not a regression and there have been no reports for
> > over
> > > > a year, I think it's ok for this to land in 3.6.1.
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Luke for reporting KRaft issue[1].
> > > > >
> > > > > I am not sure whether it is a release blocker for 3.6.0. Need input
> > > > > from other KRaft experts also to finalize the decision. Even if we
> > > > > adopt a fix, do not we need to bake it for some time before it is
> > > > > pushed to production to avoid any regressions as this change is in
> > the
> > > > > critical paths?
> > > > >
> > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I found the current KRaft implementation will have "split brain"
> > issue
> > > > > when
> > > > > > network partition happens, which will cause inconsistent metadata
> > > > > returned
> > > > > > from the controller.
> > > > > > Filed KAFKA-15489 <
> > https://issues.apache.org/jira/browse/KAFKA-15489>
> > > > > for
> > > > > > this issue, and PR <https://github.com/apache/kafka/pull/14428> is
> > > > ready
> > > > > > for review.
> > > > > >
> > > > > > Even though this is not a regression issue (this has already
> > existed
> > > > > since
> > > > > > the 1st release of KRaft feature), I think this is an important
> > issue
> > > > > since
> > > > > > KRaft is announced production ready.
> > > > > > Not sure what other people's thoughts are.
> > > > > >
>

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[Satish Duggana]

Congratulations Justine!!

On Sat, 23 Sept 2023 at 15:46, Bill Bejeck  wrote:
>
> Congrats Justine!
>
> -Bill
>
> On Sat, Sep 23, 2023 at 6:23 PM Greg Harris 
> wrote:
>
> > Congratulations Justine!
> >
> > On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
> >  wrote:
> > >
> > > Congrats Justin !
> > >
> > > On Sat 23 Sep 2023 at 14:44, Randall Hauch  wrote:
> > >
> > > > Congratulations, Justine!
> > > >
> > > > On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Congrats Justine!
> > > > >
> > > > > On Sat, Sep 23, 2023, 13:28 Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > > Congratulations Justine!
> > > > > >
> > > > > > On Sat 23. Sep 2023 at 07:06, Chris Egerton <
> > fearthecel...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats Justine!
> > > > > > > On Fri, Sep 22, 2023, 20:47 Guozhang Wang <
> > > > guozhang.wang...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations!
> > > > > > > >
> > > > > > > > On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <
> > > > > > tzuli...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Congratulations Justine!
> > > > > > > > >
> > > > > > > > > On Fri, Sep 22, 2023, 19:25 Philip Nee 
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats Justine!
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <
> > show...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Everyone,
> > > > > > > > > > >
> > > > > > > > > > > Justine Olshan has been a Kafka committer since Dec.
> > 2022.
> > > > She
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > very active and instrumental to the community since
> > becoming
> > > > a
> > > > > > > > committer.
> > > > > > > > > > > It's my pleasure to announce that Justine is now a
> > member of
> > > > > > Kafka
> > > > > > > > PMC.
> > > > > > > > > > >
> > > > > > > > > > > Congratulations Justine!
> > > > > > > > > > >
> > > > > > > > > > > Luke
> > > > > > > > > > > on behalf of Apache Kafka PMC
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Luke for reporting KRaft issue[1].

I am not sure whether it is a release blocker for 3.6.0. Need input
from other KRaft experts also to finalize the decision. Even if we
adopt a fix, do not we need to bake it for some time before it is
pushed to production to avoid any regressions as this change is in the
critical paths?

1. https://issues.apache.org/jira/browse/KAFKA-15489

Thanks,
Satish.

On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
>
> Hi Satish,
>
> I found the current KRaft implementation will have "split brain" issue when
> network partition happens, which will cause inconsistent metadata returned
> from the controller.
> Filed KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489> for
> this issue, and PR <https://github.com/apache/kafka/pull/14428> is ready
> for review.
>
> Even though this is not a regression issue (this has already existed since
> the 1st release of KRaft feature), I think this is an important issue since
> KRaft is announced production ready.
> Not sure what other people's thoughts are.
>
> Thank you.
> Luke
>
> On Thu, Sep 21, 2023 at 6:33 PM Josep Prat 
> wrote:
>
> > Hi Satish,
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Verified Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> >
> >
> > I +1 this release (non-binding)
> >
> > Thanks for your efforts!
> >
> > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Greg for verifying the release including the earlier
> > > blocker(KAFKA-15473) verification.
> > >
> > > ~Satish.
> > >
> > > On Wed, 20 Sept 2023 at 22:30, Greg Harris  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I verified the functionality of KIP-898 and the recent fix for
> > > > KAFKA-15473 with the following steps:
> > > >
> > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > publicly available plugins installed
> > > > 2. I captured the output of /connector-plugins
> > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > 4. I captured the output of /connector-plugins with various settings
> > > > of plugin.discovery
> > > > 5. I ran the migration script to add manifests to my plugins
> > > > 6. I captured the output of /connector-plugins with various settings
> > > > of plugin.discovery
> > > > 7. I downgraded the worker to 3.5.1
> > > > 8. I diffed the output of /connector-plugins across the different
> > > > cases and observed the expected changes.
> > > > a. When plugins are migrated for 3.6.0, all modes produce identical
> > > > results.
> > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > service_load is missing plugins
> > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > constructors are hidden, AK plugins now have versions, multi-interface
> > > > plugins now show each interface type, and plugins using AppInfoParser
> > > > change versions.
> > > > d. The startup logs now include descriptive errors for invalid
> > > > plugins that otherwise would have been thrown at runtime
> > > > d. The fix for KAFKA-15473 prevents duplicates
> > > > e. The output for 3.5.1 after downgrading is identical to before.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks Satish for running the release!
> > > >
> > > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> > wrote:
> > > > >
> > > > > Hey Satish
> > > > >
> > > > > My comments about documentation misses from RC0 vote thread [1] are
> > > > > still not addressed (such as missing metric documentation, formatting
> > > > > problems etc). Could you please mention why we shouldn't consider
> > them
> > > > > as blockers to make RC1 as the final release?
> > > > >
> > > > > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > > > >
> > > > > On Wed, Sep 20, 2023 at 4:53 PM Sa

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Divij for raising a PR for doc formatting issue.

On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya,  wrote:

> Hey Satish
>
> I filed a PR to fix the website formatting bug in 3.6 documentation -
> https://github.com/apache/kafka/pull/14419
> Please take a look when you get a chance.
>
> --
> Divij Vaidya
>
> On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton 
> wrote:
> >
> > Hi Satish,
> >
> > I think this qualifies as a blocker. This API has been around for years
> now
> > and, while we don't document it as not exposing duplicates*, it has come
> > with that implicit contract since its inception. More importantly, it has
> > also never exposed plugins that cannot be used on the worker. This change
> > in behavior not only introduces duplicates*, it causes unreachable
> plugins
> > to be displayed. With this in mind, it seems to qualify pretty clearly
> as a
> > regression and we should not put out a release that includes it.
> >
> > * - Really, these aren't duplicates; rather, they're multiple copies of
> the
> > same plugin that come from different locations on the worker
> >
> > Best,
> >
> > Chris
> >
> > On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana  >
> > wrote:
> >
> > > Hi Greg,
> > > Is this API documented that it does not return duplicate entries?
> > >
> > > Can we also get an opinion from PMC/Committers who have KafkaConnect
> > > expertise on whether this issue is a release blocker?
> > >
> > > If we agree that it is not a release blocker then we can have a
> > > release note clarifying this behaviour and add a reference to the JIRA
> > > that follows up on the possible solutions.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Tue, 19 Sept 2023 at 03:29, Greg Harris
> 
> > > wrote:
> > > >
> > > > Hey Satish,
> > > >
> > > > After investigating further, I believe that this is a regression, but
> > > > mostly a cosmetic one.
> > > > I don't think there is significant risk of breaking clients with this
> > > > change, but it would be confusing for users, so I'd still like to get
> > > > the fix into the next RC.
> > > > I've opened a PR here: https://github.com/apache/kafka/pull/14398
> and
> > > > I'll work to get it merged promptly.
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris 
> > > wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > While validating 3.6.0-rc0, I noticed this regression as compared
> to
> > > > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> > > > >
> > > > > Impact: The `connector-plugins` endpoint lists duplicates which may
> > > > > cause confusion for users, or poor behavior in clients.
> > > > > Using the other REST API endpoints appears unaffected.
> > > > > I'll open a PR for this later today.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> > > > >  wrote:
> > > > > >
> > > > > > Thanks Justine for the update. I saw in the morning that these
> > > changes
> > > > > > are pushed to trunk and 3.6.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > > We were able to merge
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > > > > and pick to 3.6.
> > > > > > >
> > > > > > > Hopefully nothing more from me on this release.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Luke for the update.
> > > > > > > >
> > > > > > > > ~Satis

Re: [ANNOUNCE] New committer: Lucas Brutschy

[Satish Duggana]

Congratulations Lucas!

On Thu, 21 Sept 2023 at 22:58, Viktor Somogyi-Vass
 wrote:
>
> Congrats Lucas!
>
> On Thu, Sep 21, 2023 at 7:12 PM Alexander Sorokoumov
>  wrote:
>
> > Congratulations, Lucas!
> >
> > On Thu, Sep 21, 2023 at 10:09 AM Walker Carlson
> >  wrote:
> >
> > > Congrats Lucas!
> > >
> > > On Thu, Sep 21, 2023 at 11:42 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Congrats Lucas!
> > > >
> > > > On Thu, Sep 21, 2023, 22:05 Boudjelda Mohamed Said 
> > > > wrote:
> > > >
> > > > > Congratulations, Lucas!!
> > > > >
> > > > > On Thu 21 Sep 2023 at 18:34, Lianet M.  wrote:
> > > > >
> > > > > > Congratulations Lucas!
> > > > > >
> > > > > > On Thu, Sept 21, 2023, 11:45 a.m. Bruno Cadonna <
> > cado...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > committer
> > > > > > > Lucas Brutschy.
> > > > > > >
> > > > > > > Lucas' major contributions are around Kafka Streams.
> > > > > > >
> > > > > > > Lucas' significantly contributed to the state updater
> > > > > > > (https://issues.apache.org/jira/browse/KAFKA-10199) and he
> > drives
> > > > the
> > > > > > > implementation of the new threading model for Kafka Streams
> > > > > > > (https://issues.apache.org/jira/browse/KAFKA-15326).
> > > > > > >
> > > > > > > Lucas' contributions to KIP discussions and PR reviews are very
> > > > > > thoughtful.
> > > > > > >
> > > > > > > Congratulations, Lucas!
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Bruno (on behalf of the Apache Kafka PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [ANNOUNCE] New committer: Yash Mayya

[Satish Duggana]

Congratulations Yash!!

On Thu, 21 Sept 2023 at 22:57, Viktor Somogyi-Vass
 wrote:
>
> Congrats Yash!
>
> On Thu, Sep 21, 2023 at 7:04 PM Josep Prat 
> wrote:
>
> > Congrats Yash!
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Thu, Sep 21, 2023, 18:55 Raymond Ng  wrote:
> >
> > > Congrats Yash! Well-deserved!
> > >
> > > /Ray
> > >
> > > On Thu, Sep 21, 2023 at 9:40 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Congratulations Yash!
> > > >
> > > > On Thu, Sep 21, 2023, 22:03 Bill Bejeck  wrote:
> > > >
> > > > > Congrats Yash!
> > > > >
> > > > > On Thu, Sep 21, 2023 at 12:26 PM Divij Vaidya <
> > divijvaidy...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations Yash!
> > > > > >
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 21, 2023 at 6:18 PM Sagar 
> > > > wrote:
> > > > > > >
> > > > > > > Congrats Yash !
> > > > > > > On Thu, 21 Sep 2023 at 9:38 PM, Ashwin
> > >  > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Awesome ! Congratulations Yash !!
> > > > > > > >
> > > > > > > > On Thu, Sep 21, 2023 at 9:25 PM Edoardo Comar <
> > > > edoardli...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Yash
> > > > > > > > >
> > > > > > > > > On Thu, 21 Sept 2023 at 16:28, Bruno Cadonna <
> > > cado...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > > > committer
> > > > > > > > > > Yash Mayya.
> > > > > > > > > >
> > > > > > > > > > Yash's major contributions are around Connect.
> > > > > > > > > >
> > > > > > > > > > Yash authored the following KIPs:
> > > > > > > > > >
> > > > > > > > > > KIP-793: Allow sink connectors to be used with
> > topic-mutating
> > > > > SMTs
> > > > > > > > > > KIP-882: Kafka Connect REST API configuration validation
> > > > timeout
> > > > > > > > > > improvements
> > > > > > > > > > KIP-970: Deprecate and remove Connect's redundant task
> > > > > > configurations
> > > > > > > > > > endpoint
> > > > > > > > > > KIP-980: Allow creating connectors in a stopped state
> > > > > > > > > >
> > > > > > > > > > Overall, Yash is known for insightful and friendly input to
> > > > > > discussions
> > > > > > > > > > and his high quality contributions.
> > > > > > > > > >
> > > > > > > > > > Congratulations, Yash!
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Bruno (on behalf of the Apache Kafka PMC)
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Greg for verifying the release including the earlier
blocker(KAFKA-15473) verification.

~Satish.

On Wed, 20 Sept 2023 at 22:30, Greg Harris 
wrote:

> Hi all,
>
> I verified the functionality of KIP-898 and the recent fix for
> KAFKA-15473 with the following steps:
>
> 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> publicly available plugins installed
> 2. I captured the output of /connector-plugins
> 3. I upgraded the worker to 3.6.0-rc1
> 4. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 5. I ran the migration script to add manifests to my plugins
> 6. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 7. I downgraded the worker to 3.5.1
> 8. I diffed the output of /connector-plugins across the different
> cases and observed the expected changes.
> a. When plugins are migrated for 3.6.0, all modes produce identical
> results.
> b. When plugins are not migrated for 3.6.0, only_scan and
> hybrid_warn produce identical results, hybrid_fail crashes, and
> service_load is missing plugins
> c. When upgrading from 3.5.1 I see that plugins with invalid
> constructors are hidden, AK plugins now have versions, multi-interface
> plugins now show each interface type, and plugins using AppInfoParser
> change versions.
> d. The startup logs now include descriptive errors for invalid
> plugins that otherwise would have been thrown at runtime
> d. The fix for KAFKA-15473 prevents duplicates
> e. The output for 3.5.1 after downgrading is identical to before.
>
> +1 (non-binding)
>
> Thanks Satish for running the release!
>
> On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
> >
> > Hey Satish
> >
> > My comments about documentation misses from RC0 vote thread [1] are
> > still not addressed (such as missing metric documentation, formatting
> > problems etc). Could you please mention why we shouldn't consider them
> > as blockers to make RC1 as the final release?
> >
> > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> >
> > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for the release of Apache Kafka 3.6.0.
> Some of the major features include:
> > >
> > > * KIP-405 : Kafka Tiered Storage
> > > * KIP-868 : KRaft Metadata Transactions
> > > * KIP-875: First-class offsets support in Kafka Connect
> > > * KIP-898: Modernize Connect plugin discovery
> > > * KIP-938: Add more metrics for measuring KRaft performance
> > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > * KIP-917: Additional custom metadata for remote log segment
> > >
> > > Release notes for the 3.6.0 release:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Saturday, September 23, 8am PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/36/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/36/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.6 branch:
> > > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > > System tests:
> > > We will send an update once we have the results.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Hi Divij,
Thanks for calling out documentation issues. Some of them are being
addressed in the existing documentation PRs. We will keep addressing the
remaining documentation issues by making respective changes in kafka-site
repo for 3.6.0 release.

I think documentation is important but I do not consider documentation
issues as release blockers. I want others to review and test the release
candidate sooner to finalize the RC for 3.6.0 release.

~Satish.

On Wed, 20 Sept 2023 at 21:04, Divij Vaidya  wrote:

> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some
> of the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

[jira] [Created] (KAFKA-15483) Update metrics documentation for the new metrics implemented as part of KIP-938

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15483:
--

 Summary: Update metrics documentation for the new metrics 
implemented as part of KIP-938
 Key: KAFKA-15483
 URL: https://issues.apache.org/jira/browse/KAFKA-15483
 Project: Kafka
  Issue Type: Task
  Components: docs, documentation
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15092) KafkaClusterTestKit in test jar depends on MockFaultHandler

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15092.

Resolution: Invalid

> KafkaClusterTestKit in test jar depends on MockFaultHandler
> ---
>
> Key: KAFKA-15092
> URL: https://issues.apache.org/jira/browse/KAFKA-15092
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Gary Russell
>Priority: Major
>
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/kafka/server/fault/MockFaultHandler
>   at 
> kafka.testkit.KafkaClusterTestKit$SimpleFaultHandlerFactory.(KafkaClusterTestKit.java:119)
>   at 
> kafka.testkit.KafkaClusterTestKit$Builder.(KafkaClusterTestKit.java:143)
> {noformat}
> MockFaultHandler is missing from the test jar.
> This PR https://github.com/apache/kafka/pull/13375/files seems to work around 
> it by adding the {code}server-common sourcesets.test.output{code} to the 
> class path.
> The class needs to be available for third parties to create an embedded KRaft 
> broker.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15482) kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15482.

Resolution: Invalid

> kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar
> -
>
> Key: KAFKA-15482
> URL: https://issues.apache.org/jira/browse/KAFKA-15482
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Gary Russell
>Priority: Major
>
> Commit 
> 7eea2a3908fdcee1627c18827e6dcb5ed0089fdd 
> Moved it to server-commons, but it is not included in the jar.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] 3.6.0 RC1

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the second candidate for the release of Apache Kafka 3.6.0. Some of
the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Saturday, September 23, 8am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc1

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest at
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue
running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Thanks Luke for reviewing and running the changes
https://github.com/apache/kafka/pull/14409 for merging into 3.6.

On Wed, 20 Sept 2023 at 06:09, Satish Duggana  wrote:
>
> Hi David,
> Thanks for the update. Please get it reviewed sooner. I plan to create
> RC1 in the next 5 hrs. As it is not a release blocker, I will merge it
> if this PR is approved and jenkins job looks good within that time.
>
> Thanks,
> Satish.
>
> On Wed, 20 Sept 2023 at 01:42, David Arthur
>  wrote:
> >
> > As part of validating this RC, I found that most of the ZK migration system
> > tests were failing. I investigated this today and found that it was just
> > problems in the test code. This is not a blocker, but since we're doing
> > RC1, I'd like to include it https://github.com/apache/kafka/pull/14409
> >
> > On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
> > wrote:
> >
> > > Hi All,
> > > I will start creating a new RC with the fix and send it for a vote by
> > > tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> > > determined to be a release blocker.
> > >
> > > Thanks.
> > > Satish.
> > >
> > > On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> > > wrote:
> > > >
> > > > Thanks Chris for your comment on the reported issue.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Given the regression raised by Greg Harris (
> > > > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> > > thread, I
> > > > > believe we should generate a new candidate with a fix.
> > > > >
> > > > > I'm -1 (binding) on this RC.
> > > > >
> > > > > Best,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  > > >
> > > > > wrote:
> > > > >
> > > > > > Heya,
> > > > > >
> > > > > > I have compiled and ran the test target successfully for the
> > > 3.6.0-rc0
> > > > > > branch on:
> > > > > >
> > > > > > Java 11, Scala 2.13 - ARM
> > > > > > Java 17, Scala 2.13 - ARM
> > > > > > Java 20, Scala 2.18 - ARM
> > > > > > Java 11, Scala 2.13 - Intel x86
> > > > > > Java 17, Scala 2.13 - Intel x86
> > > > > > Java 20, Scala 2.13 - Intel x86
> > > > > >
> > > > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> > > miss
> > > > > >
> > > > > > Best,
> > > > > > Christo
> > > > > >
> > > > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Satish
> > > > > > >
> > > > > > > Thank you for managing this release. I have a few comments:
> > > > > > >
> > > > > > > Documentation
> > > > > > >
> > > > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> > > "The
> > > > > > > current stable branch is 3.5. Kafka is regularly updated to 
> > > > > > > include
> > > > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> > > That
> > > > > > > needs an update since we are running Zk 3.8 now.
> > > > > > >
> > > > > > > 2. Section: Zookeeper/Migration - The documentation states
> > > "Migration
> > > > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > > > Preview and we expect it to be ready for production usage in
> > > version
> > > > > > > 3.6.". This probably needs an update on whether it is production
> > > ready
> > > > > > > or not in 3.6
> > > > > > >
> > > > > > > 3. Section: Kraft/missing features
> > > > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > > > believe that delegation token is now part of 3.6? I thin

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi David,
Thanks for the update. Please get it reviewed sooner. I plan to create
RC1 in the next 5 hrs. As it is not a release blocker, I will merge it
if this PR is approved and jenkins job looks good within that time.

Thanks,
Satish.

On Wed, 20 Sept 2023 at 01:42, David Arthur
 wrote:
>
> As part of validating this RC, I found that most of the ZK migration system
> tests were failing. I investigated this today and found that it was just
> problems in the test code. This is not a blocker, but since we're doing
> RC1, I'd like to include it https://github.com/apache/kafka/pull/14409
>
> On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
> wrote:
>
> > Hi All,
> > I will start creating a new RC with the fix and send it for a vote by
> > tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> > determined to be a release blocker.
> >
> > Thanks.
> > Satish.
> >
> > On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> > wrote:
> > >
> > > Thanks Chris for your comment on the reported issue.
> > >
> > > ~Satish.
> > >
> > > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Given the regression raised by Greg Harris (
> > > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> > thread, I
> > > > believe we should generate a new candidate with a fix.
> > > >
> > > > I'm -1 (binding) on this RC.
> > > >
> > > > Best,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  > >
> > > > wrote:
> > > >
> > > > > Heya,
> > > > >
> > > > > I have compiled and ran the test target successfully for the
> > 3.6.0-rc0
> > > > > branch on:
> > > > >
> > > > > Java 11, Scala 2.13 - ARM
> > > > > Java 17, Scala 2.13 - ARM
> > > > > Java 20, Scala 2.18 - ARM
> > > > > Java 11, Scala 2.13 - Intel x86
> > > > > Java 17, Scala 2.13 - Intel x86
> > > > > Java 20, Scala 2.13 - Intel x86
> > > > >
> > > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> > miss
> > > > >
> > > > > Best,
> > > > > Christo
> > > > >
> > > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  > >
> > > > > wrote:
> > > > >
> > > > > > Hey Satish
> > > > > >
> > > > > > Thank you for managing this release. I have a few comments:
> > > > > >
> > > > > > Documentation
> > > > > >
> > > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> > "The
> > > > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> > That
> > > > > > needs an update since we are running Zk 3.8 now.
> > > > > >
> > > > > > 2. Section: Zookeeper/Migration - The documentation states
> > "Migration
> > > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > > Preview and we expect it to be ready for production usage in
> > version
> > > > > > 3.6.". This probably needs an update on whether it is production
> > ready
> > > > > > or not in 3.6
> > > > > >
> > > > > > 3. Section: Kraft/missing features
> > > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > > believe that delegation token is now part of 3.6? I think this
> > > > > > probably needs an update.
> > > > > >
> > > > > > 4. Section: Configuration/rack.aware.assignment.strategy - there
> > seems
> > > > > > to be a formatting problem starting from here
> > > > > > (
> > > > > >
> > > > >
> > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > > > )
> > > > > >
> > > > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing
> >

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi All,
I will start creating a new RC with the fix and send it for a vote by
tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
determined to be a release blocker.

Thanks.
Satish.

On Tue, 19 Sept 2023 at 22:26, Satish Duggana  wrote:
>
> Thanks Chris for your comment on the reported issue.
>
> ~Satish.
>
> On Tue, 19 Sept 2023 at 22:16, Chris Egerton  wrote:
> >
> > Hi all,
> >
> > Given the regression raised by Greg Harris (
> > https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
> > believe we should generate a new candidate with a fix.
> >
> > I'm -1 (binding) on this RC.
> >
> > Best,
> >
> > Chris
> >
> > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
> > wrote:
> >
> > > Heya,
> > >
> > > I have compiled and ran the test target successfully for the 3.6.0-rc0
> > > branch on:
> > >
> > > Java 11, Scala 2.13 - ARM
> > > Java 17, Scala 2.13 - ARM
> > > Java 20, Scala 2.18 - ARM
> > > Java 11, Scala 2.13 - Intel x86
> > > Java 17, Scala 2.13 - Intel x86
> > > Java 20, Scala 2.13 - Intel x86
> > >
> > > I will update the Zookeeper KIP title in the KIPs page, that's my miss
> > >
> > > Best,
> > > Christo
> > >
> > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> > > wrote:
> > >
> > > > Hey Satish
> > > >
> > > > Thank you for managing this release. I have a few comments:
> > > >
> > > > Documentation
> > > >
> > > > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > the latest release in the 3.5 series." in the ZooKeeper section. That
> > > > needs an update since we are running Zk 3.8 now.
> > > >
> > > > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > Preview and we expect it to be ready for production usage in version
> > > > 3.6.". This probably needs an update on whether it is production ready
> > > > or not in 3.6
> > > >
> > > > 3. Section: Kraft/missing features
> > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > believe that delegation token is now part of 3.6? I think this
> > > > probably needs an update.
> > > >
> > > > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > > > to be a formatting problem starting from here
> > > > (
> > > >
> > > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > )
> > > >
> > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > > > documentation here.
> > > >
> > > > Release notes
> > > >
> > > > 1. I found a bunch of tickets which have not been marked with a
> > > > release version but have been resolved in last 6 months using the
> > > > query
> > > >
> > > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > . Are some of them targeted for 3.6 release?
> > > >
> > > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> > > >
> > > >
> > > > Additionally, I have verified the following:
> > > > 1. release tag is correctly made after the latest commit on the 3.6
> > > > branch at
> > > >
> > > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > > >
> > > > 2. protocol documentation contains the newly introduced error code as
> > > > part of tiered storage
> > > >
> > > > 3. verified that public keys for RM are available at
> > > > https://keys.openpgp.org/
> > > >
> > > > 4. verified that public keys for RM are available at
> > > > https://people.apache.or

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Thanks Chris for your comment on the reported issue.

~Satish.

On Tue, 19 Sept 2023 at 22:16, Chris Egerton  wrote:
>
> Hi all,
>
> Given the regression raised by Greg Harris (
> https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
> believe we should generate a new candidate with a fix.
>
> I'm -1 (binding) on this RC.
>
> Best,
>
> Chris
>
> On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
> wrote:
>
> > Heya,
> >
> > I have compiled and ran the test target successfully for the 3.6.0-rc0
> > branch on:
> >
> > Java 11, Scala 2.13 - ARM
> > Java 17, Scala 2.13 - ARM
> > Java 20, Scala 2.18 - ARM
> > Java 11, Scala 2.13 - Intel x86
> > Java 17, Scala 2.13 - Intel x86
> > Java 20, Scala 2.13 - Intel x86
> >
> > I will update the Zookeeper KIP title in the KIPs page, that's my miss
> >
> > Best,
> > Christo
> >
> > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> > wrote:
> >
> > > Hey Satish
> > >
> > > Thank you for managing this release. I have a few comments:
> > >
> > > Documentation
> > >
> > > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > > current stable branch is 3.5. Kafka is regularly updated to include
> > > the latest release in the 3.5 series." in the ZooKeeper section. That
> > > needs an update since we are running Zk 3.8 now.
> > >
> > > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > Preview and we expect it to be ready for production usage in version
> > > 3.6.". This probably needs an update on whether it is production ready
> > > or not in 3.6
> > >
> > > 3. Section: Kraft/missing features
> > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > believe that delegation token is now part of 3.6? I think this
> > > probably needs an update.
> > >
> > > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > > to be a formatting problem starting from here
> > > (
> > >
> > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > )
> > >
> > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > > documentation here.
> > >
> > > Release notes
> > >
> > > 1. I found a bunch of tickets which have not been marked with a
> > > release version but have been resolved in last 6 months using the
> > > query
> > >
> > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > . Are some of them targeted for 3.6 release?
> > >
> > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> > >
> > >
> > > Additionally, I have verified the following:
> > > 1. release tag is correctly made after the latest commit on the 3.6
> > > branch at
> > >
> > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > >
> > > 2. protocol documentation contains the newly introduced error code as
> > > part of tiered storage
> > >
> > > 3. verified that public keys for RM are available at
> > > https://keys.openpgp.org/
> > >
> > > 4. verified that public keys for RM are available at
> > > https://people.apache.org/keys/committer/
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Tue, Sep 19, 2023 at 12:41 PM Sagar 
> > wrote:
> > > >
> > > > Hey Satish,
> > > >
> > > > I have commented on KAFKA-15473. I think the changes in the PR look
> > > fine. I
> > > > also feel this need not be a release blocker given there are other
> > > > possibilities in which duplicates can manifest on the response of the
> > end
> > > > point in question (albeit we can potentially see more in number due to
> > > > this).
> > > >
> > > > Would like to hear others' thoughts as w

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi Greg,
Thanks for reporting the KafkaConnect issue. I replied to this issue
on "Apache Kafka 3.6.0 release" email thread and on
https://issues.apache.org/jira/browse/KAFKA-15473.

I would like to hear other KafkaConnect experts' opinions on whether
this issue is really a release blocker.

Thanks,
Satish.




On Tue, 19 Sept 2023 at 00:27, Greg Harris  wrote:
>
> Hey all,
>
> I noticed this regression in RC0:
> https://issues.apache.org/jira/browse/KAFKA-15473
> I've mentioned it in the release thread, and I'm working on a fix.
>
> I'm -1 (non-binding) until we determine if this regression is a blocker.
>
> Thanks!
>
> On Mon, Sep 18, 2023 at 10:56 AM Josep Prat  
> wrote:
> >
> > Hi Satish,
> > Thanks for running the release.
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Verified Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> > - Checked License-binary against libs and matched them
> >
> > I +1 this release (non-binding)
> >
> > Best,
> >
> > On Mon, Sep 18, 2023 at 6:02 PM David Arthur  wrote:
> >
> > > Hey Satish, thanks for getting the RC underway!
> > >
> > > I noticed that the PR for the 3.6 blog post is merged. This makes the blog
> > > post live on the Kafka website https://kafka.apache.org/blog.html. The
> > > blog
> > > post (along with other public announcements) is usually the last thing we
> > > do as part of the release. I think we should probably take this down until
> > > we're done with the release, otherwise users stumbling on this post could
> > > get confused. It also contains some broken links.
> > >
> > > Thanks!
> > > David
> > >
> > > On Sun, Sep 17, 2023 at 1:31 PM Satish Duggana 
> > > wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for the release of Apache Kafka 3.6.0. Some
> > > of
> > > > the major features include:
> > > >
> > > > * KIP-405 : Kafka Tiered Storage
> > > > * KIP-868 : KRaft Metadata Transactions
> > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > * KIP-898: Modernize Connect plugin discovery
> > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > * KIP-917: Additional custom metadata for remote log segment
> > > >
> > > > Release notes for the 3.6.0 release:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Wednesday, September 21, 12pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc0
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/36/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/36/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.6 branch:
> > > > There are a few runs of unit/integration tests. You can see the latest 
> > > > at
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > > > continue
> > > > running a few more iterations.
> > > > System tests:
> > > > We will send an update once we have the results.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
> >   <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Greg,
Is this API documented that it does not return duplicate entries?

Can we also get an opinion from PMC/Committers who have KafkaConnect
expertise on whether this issue is a release blocker?

If we agree that it is not a release blocker then we can have a
release note clarifying this behaviour and add a reference to the JIRA
that follows up on the possible solutions.

Thanks,
Satish.


On Tue, 19 Sept 2023 at 03:29, Greg Harris  wrote:
>
> Hey Satish,
>
> After investigating further, I believe that this is a regression, but
> mostly a cosmetic one.
> I don't think there is significant risk of breaking clients with this
> change, but it would be confusing for users, so I'd still like to get
> the fix into the next RC.
> I've opened a PR here: https://github.com/apache/kafka/pull/14398 and
> I'll work to get it merged promptly.
>
> Thanks!
>
> On Mon, Sep 18, 2023 at 11:54 AM Greg Harris  wrote:
> >
> > Hi Satish,
> >
> > While validating 3.6.0-rc0, I noticed this regression as compared to
> > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> >
> > Impact: The `connector-plugins` endpoint lists duplicates which may
> > cause confusion for users, or poor behavior in clients.
> > Using the other REST API endpoints appears unaffected.
> > I'll open a PR for this later today.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> >  wrote:
> > >
> > > Thanks Justine for the update. I saw in the morning that these changes
> > > are pushed to trunk and 3.6.
> > >
> > > ~Satish.
> > >
> > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > >  wrote:
> > > >
> > > > Hi Satish,
> > > > We were able to merge
> > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > and pick to 3.6.
> > > >
> > > > Hopefully nothing more from me on this release.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana 
> > > > 
> > > > wrote:
> > > >
> > > > > Thanks Luke for the update.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > Since this PR:
> > > > > > https://github.com/apache/kafka/pull/14366 only changes the doc, 
> > > > > > I've
> > > > > > backported to 3.6 branch. FYI.
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
> > > > > >  wrote:
> > > > > >
> > > > > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > > > > > > PR should be finalized soon.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > > > > > > migration, that we would like to backport to 3.5 and 3.6 
> > > > > > > > branches.
> > > > > Are
> > > > > > > > you ok with that?
> > > > > > > >
> > > > > > > > https://github.com/apache/kafka/pull/14366
> > > > > > > >
> > > > > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks David for the quick resolution.
> > > > > > > > >
> > > > > > > > > ~Satish.
> > > > > > > > >
> > > > > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > Satish,
> > >

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

On Sun, 17 Sept 2023 at 23:00, Satish Duggana 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for the release of Apache Kafka 3.6.0. Some of
> the major features include:
>
> * KIP-405 : Kafka Tiered Storage
> * KIP-868 : KRaft Metadata Transactions
> * KIP-875: First-class offsets support in Kafka Connect
> * KIP-898: Modernize Connect plugin discovery
> * KIP-938: Add more metrics for measuring KRaft performance
> * KIP-902: Upgrade Zookeeper to 3.8.1
> * KIP-917: Additional custom metadata for remote log segment
>
> Release notes for the 3.6.0 release:
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Wednesday, September 21, 12pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/
>
> * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> https://github.com/apache/kafka/releases/tag/3.6.0-rc0
>
> * Documentation:
> https://kafka.apache.org/36/documentation.html
>
> * Protocol:
> https://kafka.apache.org/36/protocol.html
>
> * Successful Jenkins builds for the 3.6 branch:
> There are a few runs of unit/integration tests. You can see the latest at
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> System tests:
> We will send an update once we have the results.
>
> Thanks,
> Satish.
>

[VOTE] 3.6.0 RC0

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the first candidate for the release of Apache Kafka 3.6.0. Some of
the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Wednesday, September 21, 12pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc0

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest at
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue
running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Justine for the update. I saw in the morning that these changes
are pushed to trunk and 3.6.

~Satish.

On Thu, 14 Sept 2023 at 21:54, Justine Olshan
 wrote:
>
> Hi Satish,
> We were able to merge
> https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> and pick to 3.6.
>
> Hopefully nothing more from me on this release.
>
> Thanks,
> Justine
>
> On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana 
> wrote:
>
> > Thanks Luke for the update.
> >
> > ~Satish.
> >
> > On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > Since this PR:
> > > https://github.com/apache/kafka/pull/14366 only changes the doc, I've
> > > backported to 3.6 branch. FYI.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
> > >  wrote:
> > >
> > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > > > PR should be finalized soon.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > > > migration, that we would like to backport to 3.5 and 3.6 branches.
> > Are
> > > > > you ok with that?
> > > > >
> > > > > https://github.com/apache/kafka/pull/14366
> > > > >
> > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Thanks David for the quick resolution.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > > > >  wrote:
> > > > > > >
> > > > > > > Satish,
> > > > > > >
> > > > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> > > > > > >
> > > > > > > Thanks!
> > > > > > > David
> > > > > > >
> > > > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma 
> > > > > wrote:
> > > > > > >
> > > > > > > > Justine,
> > > > > > > >
> > > > > > > > Probably best to have the conversation in the JIRA ticket vs
> > the
> > > > > release
> > > > > > > > thread. Generally, we want to only include low risk bug fixes
> > that
> > > > > are
> > > > > > > > fully compatible in patch releases.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > > > > > > 
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Satish. I understand.
> > > > > > > > > Just curious, is this something that could be added to
> > 3.6.1? It
> > > > > would be
> > > > > > > > > nice to say that hanging transactions are fully covered in a
> > 3.6
> > > > > release.
> > > > > > > > > I'm not as familiar with the rules around minor releases, but
> > > > > adding it
> > > > > > > > > there would give more time to ensure stability.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Justine,
> > > > > > > > > > We can skip this change into 3.6 now as it is not a
> > blocker or
> > > > > > > > > > regression and it involves changes to the API
> > implementation.
> > > > > Let us
> > > > > > > > > > plan t

[jira] [Resolved] (KAFKA-7739) Kafka Tiered Storage

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-7739.
---
Resolution: Fixed

> Kafka Tiered Storage
> 
>
> Key: KAFKA-7739
> URL: https://issues.apache.org/jira/browse/KAFKA-7739
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Harsha
>    Assignee: Satish Duggana
>Priority: Major
>  Labels: needs-kip
> Fix For: 3.6.0
>
>
> KIP: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Luke for the update.

~Satish.

On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
>
> Hi Satish,
>
> Since this PR:
> https://github.com/apache/kafka/pull/14366 only changes the doc, I've
> backported to 3.6 branch. FYI.
>
> Thanks.
> Luke
>
> On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
>  wrote:
>
> > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > PR should be finalized soon.
> >
> > Thanks,
> > Justine
> >
> > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > wrote:
> >
> > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > migration, that we would like to backport to 3.5 and 3.6 branches. Are
> > > you ok with that?
> > >
> > > https://github.com/apache/kafka/pull/14366
> > >
> > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana  > >
> > > wrote:
> > > >
> > > > Thanks David for the quick resolution.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > >  wrote:
> > > > >
> > > > > Satish,
> > > > >
> > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> > > > >
> > > > > Thanks!
> > > > > David
> > > > >
> > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Justine,
> > > > > >
> > > > > > Probably best to have the conversation in the JIRA ticket vs the
> > > release
> > > > > > thread. Generally, we want to only include low risk bug fixes that
> > > are
> > > > > > fully compatible in patch releases.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Satish. I understand.
> > > > > > > Just curious, is this something that could be added to 3.6.1? It
> > > would be
> > > > > > > nice to say that hanging transactions are fully covered in a 3.6
> > > release.
> > > > > > > I'm not as familiar with the rules around minor releases, but
> > > adding it
> > > > > > > there would give more time to ensure stability.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana <
> > > satish.dugg...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Justine,
> > > > > > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > > > > > regression and it involves changes to the API implementation.
> > > Let us
> > > > > > > > plan to add the gap in the release notes as you mentioned.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Satish.
> > > > > > > >
> > > > > > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hey Satish,
> > > > > > > > >
> > > > > > > > > We just discovered a gap in KIP-890 part 1. We currently
> > don't
> > > verify
> > > > > > > on
> > > > > > > > > txn offset commits, so it is still possible to have hanging
> > > > > > > transactions
> > > > > > > > on
> > > > > > > > > the consumer offsets partitions.
> > > > > > > > > I've opened a jira to wire the verification in that request.
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > > > > > >
> > > > > > > > > This also isn't a regression, but it would be nice to have
> > > part 1
> > > > > > fully
> > > > > > > > > complete. I have opened a PR with the fix:
> >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks David for the quick resolution.

~Satish.

On Tue, 12 Sept 2023 at 22:51, David Arthur
 wrote:
>
> Satish,
>
> KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
>
> Thanks!
> David
>
> On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma  wrote:
>
> > Justine,
> >
> > Probably best to have the conversation in the JIRA ticket vs the release
> > thread. Generally, we want to only include low risk bug fixes that are
> > fully compatible in patch releases.
> >
> > Ismael
> >
> > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > 
> > wrote:
> >
> > > Thanks Satish. I understand.
> > > Just curious, is this something that could be added to 3.6.1? It would be
> > > nice to say that hanging transactions are fully covered in a 3.6 release.
> > > I'm not as familiar with the rules around minor releases, but adding it
> > > there would give more time to ensure stability.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana  > >
> > > wrote:
> > >
> > > > Hi Justine,
> > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > regression and it involves changes to the API implementation. Let us
> > > > plan to add the gap in the release notes as you mentioned.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > >  wrote:
> > > > >
> > > > > Hey Satish,
> > > > >
> > > > > We just discovered a gap in KIP-890 part 1. We currently don't verify
> > > on
> > > > > txn offset commits, so it is still possible to have hanging
> > > transactions
> > > > on
> > > > > the consumer offsets partitions.
> > > > > I've opened a jira to wire the verification in that request.
> > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > >
> > > > > This also isn't a regression, but it would be nice to have part 1
> > fully
> > > > > complete. I have opened a PR with the fix:
> > > > > https://github.com/apache/kafka/pull/14370.
> > > > >
> > > > > I understand if there are concerns about last minute changes to this
> > > API
> > > > > and we can hold off if that makes the most sense.
> > > > > If we take that route, I think we should still keep verification for
> > > the
> > > > > data partitions since it still provides full protection there and
> > > > improves
> > > > > the transactions experience. We will need to call out the gap in the
> > > > > release notes for consumer offsets partitions
> > > > >
> > > > > Let me know what you think.
> > > > > Justine
> > > > >
> > > > >
> > > > > On Mon, Sep 11, 2023 at 12:29 PM David Arthur
> > > > >  wrote:
> > > > >
> > > > > > Another (small) ZK migration issue was identified. This one isn't a
> > > > > > regression (it has existed since 3.4), but I think it's reasonable
> > to
> > > > > > include. It's a small configuration check that could potentially
> > save
> > > > end
> > > > > > users from some headaches down the line.
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15450
> > > > > > https://github.com/apache/kafka/pull/14367
> > > > > >
> > > > > > I think we can get this one committed to trunk today.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > That sounds great. I think we should aim to only allow blockers
> > > > > > > (regressions, impactful security issues, etc.) on the 3.6 branch
> > > > until
> > > > > > > 3.6.0 is out.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > >
> > > > > > > On Sat, 

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Justine,
Thanks for the update. From JIRA[1] it looks like it affects only
3.6.0 but not for the earlier releases. Is that right?

https://issues.apache.org/jira/browse/KAFKA-15459

~Satish.

On Wed, 13 Sept 2023 at 00:42, Justine Olshan
 wrote:
>
> It's me again. 😅
> While reviewing the previous PR, it was discovered we had a potential
> breaking return code for non-java clients.
> This unfortunately seems like a blocker.
> https://issues.apache.org/jira/browse/KAFKA-15459
>
> I will be able to get a PR open today.
>
> Apologies for all the noise in the thread,
> Justine
>
> On Tue, Sep 12, 2023 at 10:21 AM David Arthur
>  wrote:
>
> > Satish,
> >
> > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> >
> > Thanks!
> > David
> >
> > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma  wrote:
> >
> > > Justine,
> > >
> > > Probably best to have the conversation in the JIRA ticket vs the release
> > > thread. Generally, we want to only include low risk bug fixes that are
> > > fully compatible in patch releases.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Thanks Satish. I understand.
> > > > Just curious, is this something that could be added to 3.6.1? It would
> > be
> > > > nice to say that hanging transactions are fully covered in a 3.6
> > release.
> > > > I'm not as familiar with the rules around minor releases, but adding it
> > > > there would give more time to ensure stability.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > > regression and it involves changes to the API implementation. Let us
> > > > > plan to add the gap in the release notes as you mentioned.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > We just discovered a gap in KIP-890 part 1. We currently don't
> > verify
> > > > on
> > > > > > txn offset commits, so it is still possible to have hanging
> > > > transactions
> > > > > on
> > > > > > the consumer offsets partitions.
> > > > > > I've opened a jira to wire the verification in that request.
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > > >
> > > > > > This also isn't a regression, but it would be nice to have part 1
> > > fully
> > > > > > complete. I have opened a PR with the fix:
> > > > > > https://github.com/apache/kafka/pull/14370.
> > > > > >
> > > > > > I understand if there are concerns about last minute changes to
> > this
> > > > API
> > > > > > and we can hold off if that makes the most sense.
> > > > > > If we take that route, I think we should still keep verification
> > for
> > > > the
> > > > > > data partitions since it still provides full protection there and
> > > > > improves
> > > > > > the transactions experience. We will need to call out the gap in
> > the
> > > > > > release notes for consumer offsets partitions
> > > > > >
> > > > > > Let me know what you think.
> > > > > > Justine
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 11, 2023 at 12:29 PM David Arthur
> > > > > >  wrote:
> > > > > >
> > > > > > > Another (small) ZK migration issue was identified. This one
> > isn't a
> > > > > > > regression (it has existed since 3.4), but I think it's
> > reasonable
> > > to
> > > > > > > include. It's a small configuration check that could potentially
> > > save
> > > > > end
> > > > > > > users from some headaches down the

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Justine,
We can skip this change into 3.6 now as it is not a blocker or
regression and it involves changes to the API implementation. Let us
plan to add the gap in the release notes as you mentioned.

Thanks,
Satish.

On Tue, 12 Sept 2023 at 04:44, Justine Olshan
 wrote:
>
> Hey Satish,
>
> We just discovered a gap in KIP-890 part 1. We currently don't verify on
> txn offset commits, so it is still possible to have hanging transactions on
> the consumer offsets partitions.
> I've opened a jira to wire the verification in that request.
> https://issues.apache.org/jira/browse/KAFKA-15449
>
> This also isn't a regression, but it would be nice to have part 1 fully
> complete. I have opened a PR with the fix:
> https://github.com/apache/kafka/pull/14370.
>
> I understand if there are concerns about last minute changes to this API
> and we can hold off if that makes the most sense.
> If we take that route, I think we should still keep verification for the
> data partitions since it still provides full protection there and improves
> the transactions experience. We will need to call out the gap in the
> release notes for consumer offsets partitions
>
> Let me know what you think.
> Justine
>
>
> On Mon, Sep 11, 2023 at 12:29 PM David Arthur
>  wrote:
>
> > Another (small) ZK migration issue was identified. This one isn't a
> > regression (it has existed since 3.4), but I think it's reasonable to
> > include. It's a small configuration check that could potentially save end
> > users from some headaches down the line.
> >
> > https://issues.apache.org/jira/browse/KAFKA-15450
> > https://github.com/apache/kafka/pull/14367
> >
> > I think we can get this one committed to trunk today.
> >
> > -David
> >
> >
> >
> > On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma  wrote:
> >
> > > Hi Satish,
> > >
> > > That sounds great. I think we should aim to only allow blockers
> > > (regressions, impactful security issues, etc.) on the 3.6 branch until
> > > 3.6.0 is out.
> > >
> > > Ismael
> > >
> > >
> > > On Sat, Sep 9, 2023, 12:20 AM Satish Duggana 
> > > wrote:
> > >
> > > > Hi Ismael,
> > > > It looks like we will publish RC0 by 14th Sep.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Fri, 8 Sept 2023 at 19:23, Ismael Juma  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > Do you have a sense of when we'll publish RC0?
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > > > On Fri, Sep 8, 2023 at 6:27 AM David Arthur
> > > > >  wrote:
> > > > >
> > > > > > Quick update on my two blockers: KAFKA-15435 is merged to trunk and
> > > > > > cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will
> > > > hopefully
> > > > > > get it merged today.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko 
> > > wrote:
> > > > > >
> > > > > > > Hi Satish and all,
> > > > > > >
> > > > > > > I wonder if https://issues.apache.org/jira/browse/KAFKA-14993
> > > > should be
> > > > > > > included in the 3.6 release plan. I'm thinking that when
> > > > implemented, it
> > > > > > > would be a small, but still a change in the RSM contract: throw
> > an
> > > > > > > exception instead of returning an empty InputStream. Maybe it
> > > should
> > > > be
> > > > > > > included right away to save the migration later? What do you
> > think?
> > > > > > >
> > > > > > > Best,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > > > > > > > Hi Jose,
> > > > > > > > Thanks for looking into this issue and resolving it with a
> > quick
> > > > fix.
> > > > > > > >
> > > > > > > > ~Satish.
> > > > > > > >
> > > > > > > > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > >
> > > > > > > > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana <
> > > > > > > satish.dugg...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273
> > > has
> > > > > > been
> > > > > > > > > > there in 3.5.x too.
> > > > > > > > >
> > > > > > > > > I also agree that it should be a blocker for 3.6.0. It should
> > > > have
> > > > > > > > > been a blocker for those previous releases. I didn't fix it
> > > > because,
> > > > > > > > > unfortunately, I wasn't aware of the issue and jira.
> > > > > > > > > I'll create a PR with a fix in case the original author
> > doesn't
> > > > > > > respond in time.
> > > > > > > > >
> > > > > > > > > Satish, do you agree?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > --
> > > > > > > > > -José
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -David
> > > > > >
> > > >
> > >
> >
> >
> > --
> > -David
> >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi David,
It looks like the PR is already accepted with a minor config change
validation, and you mentioned that the failed tests are not related.
Please merge it to 3.6 by today.

Thanks,
Satish.

On Tue, 12 Sept 2023 at 00:59, David Arthur
 wrote:
>
> Another (small) ZK migration issue was identified. This one isn't a
> regression (it has existed since 3.4), but I think it's reasonable to
> include. It's a small configuration check that could potentially save end
> users from some headaches down the line.
>
> https://issues.apache.org/jira/browse/KAFKA-15450
> https://github.com/apache/kafka/pull/14367
>
> I think we can get this one committed to trunk today.
>
> -David
>
>
>
> On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma  wrote:
>
> > Hi Satish,
> >
> > That sounds great. I think we should aim to only allow blockers
> > (regressions, impactful security issues, etc.) on the 3.6 branch until
> > 3.6.0 is out.
> >
> > Ismael
> >
> >
> > On Sat, Sep 9, 2023, 12:20 AM Satish Duggana 
> > wrote:
> >
> > > Hi Ismael,
> > > It looks like we will publish RC0 by 14th Sep.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Fri, 8 Sept 2023 at 19:23, Ismael Juma  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > Do you have a sense of when we'll publish RC0?
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > > > On Fri, Sep 8, 2023 at 6:27 AM David Arthur
> > > >  wrote:
> > > >
> > > > > Quick update on my two blockers: KAFKA-15435 is merged to trunk and
> > > > > cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will
> > > hopefully
> > > > > get it merged today.
> > > > >
> > > > > -David
> > > > >
> > > > > On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko 
> > wrote:
> > > > >
> > > > > > Hi Satish and all,
> > > > > >
> > > > > > I wonder if https://issues.apache.org/jira/browse/KAFKA-14993
> > > should be
> > > > > > included in the 3.6 release plan. I'm thinking that when
> > > implemented, it
> > > > > > would be a small, but still a change in the RSM contract: throw an
> > > > > > exception instead of returning an empty InputStream. Maybe it
> > should
> > > be
> > > > > > included right away to save the migration later? What do you think?
> > > > > >
> > > > > > Best,
> > > > > > Ivan
> > > > > >
> > > > > > On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > > > > > > Hi Jose,
> > > > > > > Thanks for looking into this issue and resolving it with a quick
> > > fix.
> > > > > > >
> > > > > > > ~Satish.
> > > > > > >
> > > > > > > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana <
> > > > > > satish.dugg...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273
> > has
> > > > > been
> > > > > > > > > there in 3.5.x too.
> > > > > > > >
> > > > > > > > I also agree that it should be a blocker for 3.6.0. It should
> > > have
> > > > > > > > been a blocker for those previous releases. I didn't fix it
> > > because,
> > > > > > > > unfortunately, I wasn't aware of the issue and jira.
> > > > > > > > I'll create a PR with a fix in case the original author doesn't
> > > > > > respond in time.
> > > > > > > >
> > > > > > > > Satish, do you agree?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > --
> > > > > > > > -José
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -David
> > > > >
> > >
> >
>
>
> --
> -David