Re: [VOTE] KIP-950: Tiered Storage Disablement

[Satish Duggana]

+1
Thanks Christo for addressing the review comments. We can update the
KIP for any minor comments/clarifications.


On Thu, 16 May 2024 at 15:21, Luke Chen  wrote:
>
> Thanks Chia-Ping!
> Since ZK is going to be removed, I agree the KRaft part has higher priority.
> But if Christo or the community contributor has spare time, it's good to
> have ZK part, too!
>
> Thanks.
> Luke
>
> On Thu, May 16, 2024 at 5:45 PM Chia-Ping Tsai  wrote:
>
> > +1 but I prefer to ship it to KRaft only.
> >
> > I do concern that community have enough time to accept more feature in 3.8
> > :(
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/05/14 15:20:50 Christo Lolov wrote:
> > > Heya!
> > >
> > > I would like to start a vote on KIP-950: Tiered Storage Disablement in
> > > order to catch the last Kafka release targeting Zookeeper -
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> > >
> > > Best,
> > > Christo
> > >
> >

Re: [VOTE] KIP-1018: Introduce max remote fetch timeout config

[Satish Duggana]

Thanks Kamal for the KIP.

+1 from me.

~Satish.

On Thu, 9 May 2024 at 17:52, Christo Lolov  wrote:
>
> Heya Kamal,
>
> Thanks for the KIP and the answers in the discussion!
>
> +1 from me :)
>
> Best,
> Christo
>
> On Thu, 9 May 2024 at 11:11, Federico Valeri  wrote:
>
> > +1 non binding
> >
> > Thanks
> >
> > On Thu, May 9, 2024 at 12:05 PM Luke Chen  wrote:
> > >
> > > Hi Kamal,
> > >
> > > Thanks for the KIP!
> > > +1 from me.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Mon, May 6, 2024 at 5:03 PM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > We would like to start a voting thread for KIP-1018: Introduce
> > > > max remote fetch timeout config for DelayedRemoteFetch requests.
> > > >
> > > > The KIP is available on
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1018%3A+Introduce+max+remote+fetch+timeout+config+for+DelayedRemoteFetch+requests
> > > >
> > > > If you have any suggestions, feel free to participate in the discussion
> > > > thread:
> > > > https://lists.apache.org/thread/9x21hzpxzmrt7xo4vozl17d70fkg3chk
> > > >
> > > > --
> > > > Kamal
> > > >
> >

Re: [VOTE] KIP-932: Queues for Kafka

[Satish Duggana]

Hi Andrew,
Thanks for the nice KIP, it will allow other messaging use cases to be
onboarded to Kafka.

+1 from me.

Satish.

On Tue, 7 May 2024 at 03:41, Jun Rao  wrote:
>
> Hi, Andrew,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Mon, Mar 18, 2024 at 11:00 AM Edoardo Comar 
> wrote:
>
> > Thanks Andrew,
> >
> > +1 (binding)
> >
> > Edo
> >
> > On Mon, 18 Mar 2024 at 16:32, Kenneth Eversole
> >  wrote:
> > >
> > > Hi Andrew
> > >
> > > + 1 (Non-Binding)
> > >
> > > This will be great addition to Kafka
> > >
> > > On Mon, Mar 18, 2024 at 8:27 AM Apoorv Mittal 
> > > wrote:
> > >
> > > > Hi Andrew,
> > > > Thanks for writing the KIP. This is indeed going to be a valuable
> > addition
> > > > to the Kafka, excited to see the KIP.
> > > >
> > > > + 1 (Non-Binding)
> > > >
> > > > Regards,
> > > > Apoorv Mittal
> > > > +44 7721681581
> > > >
> > > >
> > > > On Sun, Mar 17, 2024 at 11:16 PM Andrew Schofield <
> > > > andrew_schofield_j...@outlook.com> wrote:
> > > >
> > > > > Hi,
> > > > > I’ve been working to complete KIP-932 over the past few months and
> > > > > discussions have quietened down.
> > > > >
> > > > > I’d like to open the voting for KIP-932:
> > > > >
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
> > > > >
> > > > > Thanks,
> > > > > Andrew
> > > >
> >

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

[Satish Duggana]

Thanks Abhijeet for the KIP.
+1 from me.

~Satish

On Fri, 26 Apr 2024 at 8:35 PM, Jun Rao  wrote:

> Hi, Abhijeet,
>
> Thanks for the KIP. +1
>
> Jun
>
> On Thu, Apr 25, 2024 at 10:30 PM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I would like to start the vote for KIP-1023 - Follower fetch from tiered
> > offset
> >
> > The KIP is here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+tiered+offset
> >
> > Regards.
> > Abhijeet.
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

[Satish Duggana]

Congratulations Greg!

On Mon, 15 Apr 2024 at 11:04, Claude Warren  wrote:
>
> Congrats Greg!  All the hard work paid off.
>
> On Mon, Apr 15, 2024 at 6:58 AM Ivan Yurchenko  wrote:
>
> > Congrats Greg!
> >
> > On Sun, Apr 14, 2024, at 22:51, Sophie Blee-Goldman wrote:
> > > Congrats Greg! Happy to have you
> > >
> > > On Sun, Apr 14, 2024 at 9:26 AM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > > > Congrats, Greg!!
> > > >
> > > > On Sun 14. Apr 2024 at 15.05, Josep Prat 
> > > > wrote:
> > > >
> > > > > Congrats Greg!!!
> > > > >
> > > > >
> > > > > Best,
> > > > >
> > > > > Josep Prat
> > > > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > > > +491715557497 | aiven.io
> > > > > Aiven Deutschland GmbH
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > On Sun, Apr 14, 2024, 12:30 Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > > Congratulations Greg!
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Apr 14, 2024 at 6:39 AM Kamal Chandraprakash <
> > > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > >
> > > > > > > Congratulations, Greg!
> > > > > > >
> > > > > > > On Sun, Apr 14, 2024 at 8:57 AM Yash Mayya  > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Congrats Greg!
> > > > > > > >
> > > > > > > > On Sun, 14 Apr, 2024, 05:56 Randall Hauch, 
> > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Greg!
> > > > > > > > >
> > > > > > > > > On Sat, Apr 13, 2024 at 6:36 PM Luke Chen  > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats, Greg!
> > > > > > > > > >
> > > > > > > > > > On Sun, Apr 14, 2024 at 7:05 AM Viktor Somogyi-Vass
> > > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > > Congrats Greg! :)
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Apr 14, 2024, 00:35 Bill Bejeck <
> > bbej...@gmail.com>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congrats Greg!
> > > > > > > > > > > >
> > > > > > > > > > > > -Bill
> > > > > > > > > > > >
> > > > > > > > > > > > On Sat, Apr 13, 2024 at 4:25 PM Boudjelda Mohamed Said
> > <
> > > > > > > > > > > bmsc...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations Greg
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sat 13 Apr 2024 at 20:42, Chris Egerton <
> > > > > > > ceger...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Greg Harris has been a Kafka committer since July
> > 2023.
> > > > > He
> > > > > > > has
> > > > > > > > > > > remained
> > > > > > > > > > > > > > very active and instructive in the community since
> > > > > > becoming a
> > > > > > > > > > > > committer.
> > > > > > > > > > > > > > It's my pleasure to announce that Greg is now a
> > member
> > > > of
> > > > > > > Kafka
> > > > > > > > > > PMC.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Congratulations, Greg!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Chris, on behalf of the Apache Kafka PMC
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> LinkedIn: http://www.linkedin.com/in/claudewarren

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

and where we
> > retain
> > > half of the remote log and delete the other half.*
> > >
> > > * 103. Do we plan to add any metrics related to this feature?*
> > >
> > >
> > >
> > > *Any recommendations?*
> > > *We may emit a gauge showing the enablement state of a topic but we could
> > > gather that info from the logs as well.*
> > > *The total duration for remote topic deletion could be added as well but
> > > this is more of a metric for the RemotePartitionRemover itself.*
> > >
> > >
> > >
> > > *104. Please add configuration details about copier thread pool,
> > expiration
> > > thread pool and the migration of the existing
> > > RemoteLogManagerScheduledThreadPool.*
> > >
> > > *Will add the details.*
> > >
> > > 105. How is the behaviour with topic or partition deletion request
> > > handled when tiered storage disablement request is still being
> > > processed on a topic?
> > >
> > > *If the disablement policy is Delete then a successive topic deletion
> > > request is going to be a NOOP because RemoteLogs is already being
> > deleted.*
> > > *If the disablement policy is Retain, then we only moved the
> > LogStartOffset
> > > and didn't touch RemoteLogs anyway, so the delete topic request will
> > > result*
> > >
> > > *in the initiation of RemoteLog deletion.*
> > >
> > >
> > > On Tue, 26 Mar 2024 at 18:21, Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for the KIP! Overall the KIP looks good and covered most of the
> > > > items.
> > > >
> > > > 1. Could you explain how the brokers will handle the DisableRemoteTopic
> > > API
> > > > request?
> > > >
> > > > 2. Who will initiate the controller interaction sequence? Does the
> > > > controller listens for
> > > > topic config updates and initiate the disablement?
> > > >
> > > > --
> > > > Kamal
> > > >
> > > > On Tue, Mar 26, 2024 at 4:40 PM Satish Duggana <
> > satish.dugg...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks Mehari, Divij, Christo etal for the KIP.
> > > > >
> > > > > I had an initial review of the KIP and left the below comments.
> > > > >
> > > > > 101. For remote.log.disable.policy=delete:
> > > > > Does it delete the remote log data immediately and the data in remote
> > > > > storage will not be taken into account by any replica? That means
> > > > > log-start-offset is moved to the earlier local-log-start-offset.
> > > > >
> > > > > 102. Can we update the remote.log.disable.policy after tiered storage
> > > > > is disabled on a topic?
> > > > >
> > > > > 103. Do we plan to add any metrics related to this feature?
> > > > >
> > > > > 104. Please add configuration details about copier thread pool,
> > > > > expiration thread pool and the migration of the existing
> > > > > RemoteLogManagerScheduledThreadPool.
> > > > >
> > > > > 105. How is the behaviour with topic or partition deletion request
> > > > > handled when tiered storage disablement request is still being
> > > > > processed on a topic?
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Mon, 25 Mar 2024 at 13:34, Doğuşcan Namal <
> > namal.dogus...@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Christo and Luke,
> > > > > >
> > > > > > I think the KRaft section of the KIP requires slight improvement.
> > The
> > > > > metadata propagation in KRaft is handled by the RAFT layer instead of
> > > > > sending Controller -> Broker RPCs. In fact, KIP-631 deprecated these
> > > > RPCs.
> > > > > >
> > > > > > I will come up with some recommendations on how we could improve
> > that
> > > > > one but until then, @Luke please feel free to review the KIP.
> > > > > >
> > > > > > @Satish, if we want this to make it to Kafka 3.8 I believe we need
> > to
> > > > > aim to get the KIP approved in th

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

[Satish Duggana]

+1 to Jun for adding the consumer fetching from a follower scenario
also to the existing section that talked about the drawback when a
node built with last-tiered-offset has become a leader. As Abhijeet
mentioned, we plan to have a follow-up KIP that will address these by
having a deprioritzation of these brokers. The deprioritization of
those brokers can be removed once they catchup until the local log
retention.

Thanks,
Satish.

On Tue, 9 Apr 2024 at 14:08, Luke Chen  wrote:
>
> Hi Abhijeet,
>
> Thanks for the KIP to improve the tiered storage feature!
>
> Questions:
> 1. We could also get the "pending-upload-offset" and epoch via remote log
> metadata, instead of adding a new API to fetch from the leader. Could you
> explain why you choose the later approach, instead of the former?
> 2.
> > We plan to have a follow-up KIP that will address both the
> deprioritization
> of these brokers from leadership, as well as
> for consumption (when fetching from followers is allowed).
>
> I agree with Jun that we might need to make it clear all possible drawbacks
> that could have. So, could we add the drawbacks that Jun mentioned about
> the performance issue when consumer fetch from follower?
>
> 3. Could we add "Rejected Alternatives" section to the end of the KIP to
> add some of them?
> Like the "ListOffsetRequest" approach VS "Earliest-Pending-Upload-Offset"
> approach, or getting the "Earliest-Pending-Upload-Offset" from remote log
> metadata... etc.
>
> Thanks.
> Luke
>
>
> On Tue, Apr 9, 2024 at 2:25 PM Abhijeet Kumar 
> wrote:
>
> > Hi Christo,
> >
> > Please find my comments inline.
> >
> > On Fri, Apr 5, 2024 at 12:36 PM Christo Lolov 
> > wrote:
> >
> > > Hello Abhijeet and Jun,
> > >
> > > I have been mulling this KIP over a bit more in recent days!
> > >
> > > re: Jun
> > >
> > > I wasn't aware we apply 2.1 and 2.2 for reserving new timestamps - in
> > > retrospect it should have been fairly obvious. I would need to go an
> > update
> > > KIP-1005 myself then, thank you for giving the useful reference!
> > >
> > > 4. I think Abhijeet wants to rebuild state from latest-tiered-offset and
> > > fetch from latest-tiered-offset + 1 only for new replicas (or replicas
> > > which experienced a disk failure) to decrease the time a partition spends
> > > in under-replicated state. In other words, a follower which has just
> > fallen
> > > out of ISR, but has local data will continue using today's Tiered Storage
> > > replication protocol (i.e. fetching from earliest-local). I further
> > believe
> > > he has taken this approach so that local state of replicas which have
> > just
> > > fallen out of ISR isn't forcefully wiped thus leading to situation 1.
> > > Abhijeet, have I understood (and summarised) what you are proposing
> > > correctly?
> > >
> > > Yes, your understanding is correct. We want to limit the behavior changes
> > only to new replicas.
> >
> >
> > > 5. I think in today's Tiered Storage we know the leader's
> > log-start-offset
> > > from the FetchResponse and we can learn its local-log-start-offset from
> > the
> > > ListOffsets by asking for earliest-local timestamp (-4). But granted,
> > this
> > > ought to be added as an additional API call in the KIP.
> > >
> > >
> > Yes, I clarified this in my reply to Jun. I will add this missing detail in
> > the KIP.
> >
> >
> > > re: Abhijeet
> > >
> > > 101. I am still a bit confused as to why you want to include a new offset
> > > (i.e. pending-upload-offset) when you yourself mention that you could use
> > > an already existing offset (i.e. last-tiered-offset + 1). In essence, you
> > > end your Motivation with "In this KIP, we will focus only on the follower
> > > fetch protocol using the *last-tiered-offset*" and then in the following
> > > sections you talk about pending-upload-offset. I understand this might be
> > > classified as an implementation detail, but if you introduce a new offset
> > > (i.e. pending-upload-offset) you have to make a change to the ListOffsets
> > > API (i.e. introduce -6) and thus document it in this KIP as such.
> > However,
> > > the last-tiered-offset ought to already be exposed as part of KIP-1005
> > > (under implementation). Am I misunderstanding something here?
> > >
> >
> > I have tried to clarify this in my reply to Jun.
> >
> > > The follower needs to build the local data starting from the offset
> > > Earliest-Pending-Upload-Offset. Hence it needs the offset and the
> > > corresponding leader-epoch.
> > > There are two ways to do this:
> > >1. We add support in ListOffsetRequest to be able to fetch this offset
> > > (and leader epoch) from the leader.
> > >2. Or, fetch the tiered-offset (which is already supported). From this
> > > offset, we can get the Earliest-Pending-Upload-Offset. We can just add 1
> > to
> > > the tiered-offset.
> > >   However, we still need the leader epoch for offset, since there is
> > > no guarantee that the leader epoch for Earliest-Pending-Upload-Offset
> > will
> 

Re: [ANNOUNCE] New committer: Christo Lolov

[Satish Duggana]

Congratulations Christo!

On Tue, 26 Mar 2024 at 19:20, Ivan Yurchenko  wrote:
>
> Congrats!
>
> On Tue, Mar 26, 2024, at 14:48, Lucas Brutschy wrote:
> > Congrats!
> >
> > On Tue, Mar 26, 2024 at 2:44 PM Federico Valeri  
> > wrote:
> > >
> > > Congrats!
> > >
> > > On Tue, Mar 26, 2024 at 2:27 PM Mickael Maison  
> > > wrote:
> > > >
> > > > Congratulations Christo!
> > > >
> > > > On Tue, Mar 26, 2024 at 2:26 PM Chia-Ping Tsai  
> > > > wrote:
> > > > >
> > > > > Congrats Christo!
> > > > >
> > > > > Chia-Ping
> >

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

Thanks Mehari, Divij, Christo etal for the KIP.

I had an initial review of the KIP and left the below comments.

101. For remote.log.disable.policy=delete:
Does it delete the remote log data immediately and the data in remote
storage will not be taken into account by any replica? That means
log-start-offset is moved to the earlier local-log-start-offset.

102. Can we update the remote.log.disable.policy after tiered storage
is disabled on a topic?

103. Do we plan to add any metrics related to this feature?

104. Please add configuration details about copier thread pool,
expiration thread pool and the migration of the existing
RemoteLogManagerScheduledThreadPool.

105. How is the behaviour with topic or partition deletion request
handled when tiered storage disablement request is still being
processed on a topic?

~Satish.

On Mon, 25 Mar 2024 at 13:34, Doğuşcan Namal  wrote:
>
> Hi Christo and Luke,
>
> I think the KRaft section of the KIP requires slight improvement. The 
> metadata propagation in KRaft is handled by the RAFT layer instead of sending 
> Controller -> Broker RPCs. In fact, KIP-631 deprecated these RPCs.
>
> I will come up with some recommendations on how we could improve that one but 
> until then, @Luke please feel free to review the KIP.
>
> @Satish, if we want this to make it to Kafka 3.8 I believe we need to aim to 
> get the KIP approved in the following weeks otherwise it will slip and we can 
> not support it in Zookeeper mode.
>
> I also would like to better understand what is the community's stand for 
> adding a new feature for Zookeeper since it is marked as deprecated already.
>
> Thanks.
>
>
>
> On Mon, 18 Mar 2024 at 13:42, Christo Lolov  wrote:
>>
>> Heya,
>>
>> I do have some time to put into this, but to be honest I am still after
>> reviews of the KIP itself :)
>>
>> After the latest changes it ought to be detailing both a Zookeeper approach
>> and a KRaft approach.
>>
>> Do you have any thoughts on how it could be improved or should I start a
>> voting thread?
>>
>> Best,
>> Christo
>>
>> On Thu, 14 Mar 2024 at 06:12, Luke Chen  wrote:
>>
>> > Hi Christo,
>> >
>> > Any update with this KIP?
>> > If you don't have time to complete it, I can collaborate with you to work
>> > on it.
>> >
>> > Thanks.
>> > Luke
>> >
>> > On Wed, Jan 17, 2024 at 11:38 PM Satish Duggana 
>> > wrote:
>> >
>> > > Hi Christo,
>> > > Thanks for volunteering to contribute to the KIP discussion. I suggest
>> > > considering this KIP for both ZK and KRaft as it will be helpful for
>> > > this feature to be available in 3.8.0 running with ZK clusters.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > > On Wed, 17 Jan 2024 at 19:04, Christo Lolov 
>> > > wrote:
>> > > >
>> > > > Hello!
>> > > >
>> > > > I volunteer to get this KIP moving forward and implemented in Apache
>> > > Kafka
>> > > > 3.8.
>> > > >
>> > > > I have caught up with Mehari offline and we have agreed that given
>> > Apache
>> > > > Kafka 4.0 being around the corner we would like to propose this feature
>> > > > only for KRaft clusters.
>> > > >
>> > > > Any and all reviews and comments are welcome!
>> > > >
>> > > > Best,
>> > > > Christo
>> > > >
>> > > > On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
>> > > > wrote:
>> > > >
>> > > > > Hi everyone, any progress on the status of this KIP? Overall looks
>> > > good to
>> > > > > me but I wonder whether we still need to support it for Zookeeper
>> > mode
>> > > > > given that it will be deprecated in the next 3 months.
>> > > > >
>> > > > > On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
>> > > > > > Hi everyone,
>> > > > > > I would like to start a discussion on KIP-950: Tiered Storage
>> > > Disablement
>> > > > > (
>> > > > >
>> > > > >
>> > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
>> > > > > ).
>> > > > > >
>> > > > > > This KIP proposes adding the ability to disable and re-enable
>> > tiered
>> > > > > storage on a topic.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Mehari
>> > > > > >
>> > > > >
>> > >
>> >

Re: [DISCUSS] Apache Kafka 3.6.2 release

[Satish Duggana]

+1, Thanks Mani for volunteering.

On Thu, 14 Mar 2024 at 06:01, Luke Chen  wrote:
>
> +1, Thanks Manikumar!
>
> On Thu, Mar 14, 2024 at 3:40 AM Bruno Cadonna  wrote:
>
> > Thanks Manikumar!
> >
> > +1
> >
> > Best,
> > Bruno
> >
> > On 3/13/24 5:56 PM, Josep Prat wrote:
> > > +1 thanks for volunteering!
> > >
> > > Best
> > > ---
> > >
> > > Josep Prat
> > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > On Wed, Mar 13, 2024, 17:17 Divij Vaidya 
> > wrote:
> > >
> > >> +1
> > >>
> > >> Thank you for volunteering.
> > >>
> > >> --
> > >> Divij Vaidya
> > >>
> > >>
> > >>
> > >> On Wed, Mar 13, 2024 at 4:58 PM Justine Olshan
> > >> 
> > >> wrote:
> > >>
> > >>> Thanks Manikumar!
> > >>> +1 from me
> > >>>
> > >>> Justine
> > >>>
> > >>> On Wed, Mar 13, 2024 at 8:52 AM Manikumar 
> > >>> wrote:
> > >>>
> >  Hi,
> > 
> >  I'd like to volunteer to be the release manager for a bug fix release
> > >> of
> >  the 3.6 line.
> >  If there are no objections, I'll send out the release plan soon.
> > 
> >  Thanks,
> >  Manikumar
> > 
> > >>>
> > >>
> > >
> >

Re: [VOTE] KIP-956: Tiered Storage Quotas

[Satish Duggana]

Thanks Abhijeet for the KIP, +1 from me.


On Sat, 9 Mar 2024 at 1:51 AM, Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> +1 (non-binding), Thanks for the KIP, Abhijeet!
>
> --
> Kamal
>
> On Fri, Mar 8, 2024 at 11:02 PM Jun Rao  wrote:
>
> > Hi, Abhijeet,
> >
> > Thanks for the KIP. +1
> >
> > Jun
> >
> > On Fri, Mar 8, 2024 at 3:44 AM Abhijeet Kumar <
> abhijeet.cse@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I would like to start the vote for KIP-956 - Tiered Storage Quotas
> > >
> > > The KIP is here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas
> > >
> > > Regards.
> > > Abhijeet.
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Satish Duggana]

Thanks Josep, +1.

On Tue, 27 Feb 2024 at 17:29, Divij Vaidya  wrote:
>
> Thank you for volunteering Josep. +1 from me.
>
> --
> Divij Vaidya
>
>
>
> On Tue, Feb 27, 2024 at 9:35 AM Bruno Cadonna  wrote:
>
> > Thanks Josep!
> >
> > +1
> >
> > Best,
> > Bruno
> >
> > On 2/26/24 9:53 PM, Chris Egerton wrote:
> > > Thanks Josep, I'm +1 as well.
> > >
> > > On Mon, Feb 26, 2024 at 12:32 PM Justine Olshan
> > >  wrote:
> > >
> > >> Thanks Joesp. +1 from me.
> > >>
> > >> On Mon, Feb 26, 2024 at 3:37 AM Josep Prat  > >
> > >> wrote:
> > >>
> > >>> Hi all,
> > >>>
> > >>> I'd like to volunteer as release manager for the Apache Kafka 3.8.0
> > >>> release.
> > >>> If there are no objections, I'll start building a release plan (or
> > >> adapting
> > >>> the one Colin made some weeks ago) in the wiki in the next days.
> > >>>
> > >>> Thank you.
> > >>>
> > >>> --
> > >>> [image: Aiven] 
> > >>>
> > >>> *Josep Prat*
> > >>> Open Source Engineering Director, *Aiven*
> > >>> josep.p...@aiven.io   |   +491715557497
> > >>> aiven.io    |   <
> > >> https://www.facebook.com/aivencloud
> > 
> > >>>   <
> > >>> https://twitter.com/aiven_io>
> > >>> *Aiven Deutschland GmbH*
> > >>> Alexanderufer 3-7, 10117 Berlin
> > >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > >>>
> > >>
> > >
> >

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 3.7.0

[Satish Duggana]

Thanks Stanislav for all your hard work on running the release. Thanks
to all the contributors to this release.


On Wed, 28 Feb 2024 at 13:43, Bruno Cadonna  wrote:
>
> Thanks Stan and all contributors for the release!
>
> Best,
> Bruno
>
> On 2/27/24 7:01 PM, Stanislav Kozlovski wrote:
> > The Apache Kafka community is pleased to announce the release of
> > Apache Kafka 3.7.0
> >
> > This is a minor release that includes new features, fixes, and
> > improvements from 296 JIRAs
> >
> > An overview of the release and its notable changes can be found in the
> > release blog post:
> > https://kafka.apache.org/blog#apache_kafka_370_release_announcement
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html
> >
> > You can download the source and binary release (Scala 2.12, 2.13) from:
> > https://kafka.apache.org/downloads#3.7.0
> >
> > ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you to the following 146 contributors to this release!
> > (Please report an unintended omission)
> >
> > Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra,
> > Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew
> > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal,
> > Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj,
> > ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna,
> > Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin
> > Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David
> > Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, Dongnuo
> > Lyu, Eaugene Thomas, Eduwer Camacaro, Eike Thaden, Federico Valeri,
> > Florin Akermann, Gantigmaa Selenge, Gaurav Narula, gongzhongqiang,
> > Greg Harris, Guozhang Wang, Gyeongwon, Do, Hailey Ni, Hanyu Zheng, Hao
> > Li, Hector Geraldino, hudeqi, Ian McDonald, Iblis Lin, Igor Soarez,
> > iit2009060, Ismael Juma, Jakub Scholz, James Cheng, Jason Gustafson,
> > Jay Wang, Jeff Kim, Jim Galasyn, John Roesler, Jorge Esteban Quilcate
> > Otoya, Josep Prat, José Armando García Sancio, Jotaniya Jeel, Jouni
> > Tenhunen, Jun Rao, Justine Olshan, Kamal Chandraprakash, Kirk True,
> > kpatelatwork, kumarpritam863, Laglangyue, Levani Kokhreidze, Lianet
> > Magrans, Liu Zeyu, Lucas Brutschy, Lucia Cerchie, Luke Chen, maniekes,
> > Manikumar Reddy, mannoopj, Maros Orsak, Matthew de Detrich, Matthias
> > J. Sax, Max Riedel, Mayank Shekhar Narula, Mehari Beyene, Michael
> > Westerby, Mickael Maison, Nick Telford, Nikhil Ramakrishnan, Nikolay,
> > Okada Haruki, olalamichelle, Omnia G.H Ibrahim, Owen Leung, Paolo
> > Patierno, Philip Nee, Phuc-Hong-Tran, Proven Provenzano, Purshotam
> > Chauhan, Qichao Chu, Matthias J. Sax, Rajini Sivaram, Renaldo Baur
> > Filho, Ritika Reddy, Robert Wagner, Rohan, Ron Dagostino, Roon, runom,
> > Ruslan Krivoshein, rykovsi, Sagar Rao, Said Boudjelda, Satish Duggana,
> > shuoer86, Stanislav Kozlovski, Taher Ghaleb, Tang Yunzi, TapDa

[jira] [Created] (KAFKA-16161) Avoid creating remote log metadata snapshot file in partition data directory.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-16161:
--

 Summary: Avoid creating remote log metadata snapshot file in 
partition data directory.
 Key: KAFKA-16161
 URL: https://issues.apache.org/jira/browse/KAFKA-16161
 Project: Kafka
  Issue Type: Improvement
Reporter: Satish Duggana


Avoid creating remote log metadata snapshot file in a partition data directory. 
This can be added when the snapshots implementation related functionality is 
enabled end to end. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-950: Tiered Storage Disablement

[Satish Duggana]

Hi Christo,
Thanks for volunteering to contribute to the KIP discussion. I suggest
considering this KIP for both ZK and KRaft as it will be helpful for
this feature to be available in 3.8.0 running with ZK clusters.

Thanks,
Satish.

On Wed, 17 Jan 2024 at 19:04, Christo Lolov  wrote:
>
> Hello!
>
> I volunteer to get this KIP moving forward and implemented in Apache Kafka
> 3.8.
>
> I have caught up with Mehari offline and we have agreed that given Apache
> Kafka 4.0 being around the corner we would like to propose this feature
> only for KRaft clusters.
>
> Any and all reviews and comments are welcome!
>
> Best,
> Christo
>
> On Tue, 9 Jan 2024 at 09:44, Doğuşcan Namal 
> wrote:
>
> > Hi everyone, any progress on the status of this KIP? Overall looks good to
> > me but I wonder whether we still need to support it for Zookeeper mode
> > given that it will be deprecated in the next 3 months.
> >
> > On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> > > Hi everyone,
> > > I would like to start a discussion on KIP-950: Tiered Storage Disablement
> > (
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> > ).
> > >
> > > This KIP proposes adding the ability to disable and re-enable tiered
> > storage on a topic.
> > >
> > > Thanks,
> > > Mehari
> > >
> >

Re: [VOTE] KIP-1005: Expose EarliestLocalOffset and TieredOffset

[Satish Duggana]

+1 (binding)

Thanks,
Satish.

On Thu, 11 Jan 2024 at 17:52, Divij Vaidya  wrote:
>
> +1 (binding)
>
> Divij Vaidya
>
>
>
> On Tue, Dec 26, 2023 at 7:05 AM Kamal Chandraprakash <
> kamal.chandraprak...@gmail.com> wrote:
>
> > +1 (non-binding). Thanks for the KIP!
> >
> > --
> > Kamal
> >
> > On Thu, Dec 21, 2023 at 2:23 PM Christo Lolov 
> > wrote:
> >
> > > Heya all!
> > >
> > > KIP-1005 (
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Expose+EarliestLocalOffset+and+TieredOffset
> > > )
> > > has been open for around a month with no further comments - I would like
> > to
> > > start a voting round on it!
> > >
> > > Best,
> > > Christo
> > >
> >

Re: [VOTE] KIP-1007: Introduce Remote Storage Not Ready Exception

[Satish Duggana]

Thanks Kamal for the KIP.

+1 (binding)

On Fri, 5 Jan 2024 at 17:04, Divij Vaidya  wrote:
>
> +1 (binding)
>
> --
> Divij Vaidya
>
>
>
> On Thu, Dec 21, 2023 at 10:30 AM Luke Chen  wrote:
>
> > Hi Kamal,
> >
> > Thanks for the KIP.
> > +1 (binding) from me.
> >
> > Luke
> >
> > On Thu, Dec 21, 2023 at 4:51 PM Christo Lolov 
> > wrote:
> >
> > > Heya Kamal,
> > >
> > > The proposed change makes sense to me as it will be a more explicit
> > > behaviour than what Kafka does today - I am happy with it!
> > >
> > > +1 (non-binding) from me
> > >
> > > Best,
> > > Christo
> > >
> > > On Tue, 12 Dec 2023 at 09:01, Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to call a vote for KIP-1007
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1007%3A+Introduce+Remote+Storage+Not+Ready+Exception
> > > > >.
> > > > This KIP aims to introduce a new error code for retriable remote
> > storage
> > > > errors. Thanks to everyone who reviewed the KIP!
> > > >
> > > > --
> > > > Kamal
> > > >
> > >
> >

Re: [DISCUSS] KIP-1007: Introduce Remote Storage Not Ready Exception

[Satish Duggana]

Thanks for the KIP Kamal, LGTM.

On Tue, 26 Dec 2023 at 10:23, Kamal Chandraprakash
 wrote:
>
> Hi Divij,
>
> Thanks for reviewing the KIP! I've updated the KIP with the below
> documentation. Let me know if it needs to be changed:
>
> The consumer can read the local data as long as it knows the offset from
> where to fetch the data from.
> When there is no initial offset, the consumer decides the offset based on
> the below config:
>
> ```
> auto.offset.reset = earliest / latest / none
> ```
>
>- For `earliest` offset policy and any offset that lies in the remote
>storage, the consumer (FETCH request)
>cannot be able to make progress until the remote log metadata gets
>synced.
>- In a FETCH request, when there are multiple partitions where a subset
>of them are consuming from local
>and others from remote, then only the partitions which are consuming
>from the remote cannot make
>progress and the partitions that fetch data from local storage should be
>able to make progress.
>- In a FETCH request, when the fetch-offset for a partition is within
>the local-storage, then it should be able
>to consume the messages.
>- All the calls to LIST_OFFETS will fail until the remote log metadata
>gets synced.
>
>
> The code link that is mentioned is referring to the `LIST_OFFSETS` handler.
> Usually, consumers don't use this API
> unless it's explicitly called by the user.
>
>
> On Fri, Dec 22, 2023 at 4:10 PM Divij Vaidya 
> wrote:
>
> > Thanks for the KIP, Kamal.
> >
> > The change looks good to me, though, I think we can do a better job at
> > documenting what the error means for the clients and users.
> >
> > Correct me if I'm wrong, when remote metadata is being synced on a new
> > leader, we cannot fetch even the local data (as per [1]), hence, partition
> > is considered "unreadable" but writes (and all other operations such as
> > admin operations) can continue to work on that partition. If my
> > understanding is correct, perhaps, please clarify this in the error
> > description. In absence of it, it is difficult to determine what this error
> > means for operations that can be performed on a partition.
> >
> > [1]
> >
> > https://github.com/apache/kafka/blob/82808873cbf6a95611243c2e7984c4aa6ff2cfff/core/src/main/scala/kafka/log/UnifiedLog.scala#L1336
> >
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Tue, Dec 12, 2023 at 9:58 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Thanks Luke for reviewing this KIP!
> > >
> > > If there are no more comments from others, I'll start the VOTE since this
> > > is a minor KIP.
> > >
> > > On Mon, Dec 11, 2023 at 1:01 PM Luke Chen  wrote:
> > >
> > > > Hi Kamal,
> > > >
> > > > Thanks for the KIP!
> > > > LGTM.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Wed, Nov 22, 2023 at 7:28 PM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would like to start a discussion to introduce a new error code for
> > > > > retriable remote storage errors. Please take a look at the proposal:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1007%3A+Introduce+Remote+Storage+Not+Ready+Exception
> > > > >
> > > >
> > >
> >

Re: [VOTE] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

[Satish Duggana]

Thanks Ismael for the proposal.

Adopting JDK 17 enhances developer productivity and has reached a
level of maturity that has led to its adoption by several other major
projects, signifying its reliability and effectiveness.

+1 (binding)


~Satish.

On Wed, 3 Jan 2024 at 06:59, Justine Olshan
 wrote:
>
> Thanks for driving this.
>
> +1 (binding) from me.
>
> Justine
>
> On Tue, Jan 2, 2024 at 4:30 PM Ismael Juma  wrote:
>
> > Hi all,
> >
> > I would like to start a vote on KIP-1013.
> >
> > As stated in the discussion thread, this KIP was proposed after the KIP
> > freeze for Apache Kafka 3.7, but it is purely a documentation update (if we
> > decide to adopt it) and I believe it would serve our users best if we
> > communicate the deprecation for removal sooner (i.e. 3.7) rather than later
> > (i.e. 3.8).
> >
> > Please take a look and cast your vote.
> >
> > Link:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510
> >
> > Ismael
> >

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

[Satish Duggana]

Congratulations Divij!

On Thu, 28 Dec 2023 at 07:21, Kamal Chandraprakash
 wrote:
>
> Congrats Divij!
>
> On Thu, Dec 28, 2023, 07:09 Kirk True  wrote:
>
> > Congrats Divij!!!
> >
> > On Wed, Dec 27, 2023, at 1:44 PM, Jorge Esteban Quilcate Otoya wrote:
> > > Congratulations Divij!!
> > >
> > > On Wed 27. Dec 2023 at 14.56, Tom Bentley  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > On Thu, 28 Dec 2023 at 06:17, Philip Nee  wrote:
> > > >
> > > > > congrats divij!
> > > > >
> > > > > On Wed, Dec 27, 2023 at 8:55 AM Justine Olshan
> > > > > 
> > > > > wrote:
> > > > >
> > > > > > Congratulations Divij!
> > > > > >
> > > > > > On Wed, Dec 27, 2023 at 4:20 AM Gaurav Narula 
> > > > wrote:
> > > > > >
> > > > > > > Congratulations Divij!
> > > > > > >
> > > > > > > Regards,
> > > > > > > Gaurav
> > > > > > >
> > > > > > > > On 27-Dec-2023, at 17:44, Mickael Maison <
> > mickael.mai...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Congratulations Divij!
> > > > > > > >
> > > > > > > >> On Wed, Dec 27, 2023 at 1:05 PM Sagar <
> > sagarmeansoc...@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Congrats Divij! Absolutely well deserved !
> > > > > > > >>
> > > > > > > >> Thanks!
> > > > > > > >> Sagar.
> > > > > > > >>
> > > > > > > >>> On Wed, Dec 27, 2023 at 5:15 PM Luke Chen  > >
> > > > > wrote:
> > > > > > > >>>
> > > > > > > >>> Hi, Everyone,
> > > > > > > >>>
> > > > > > > >>> Divij has been a Kafka committer since June, 2023. He has
> > > > remained
> > > > > > very
> > > > > > > >>> active and instructive in the community since becoming a
> > > > committer.
> > > > > > > It's my
> > > > > > > >>> pleasure to announce that Divij is now a member of Kafka PMC.
> > > > > > > >>>
> > > > > > > >>> Congratulations Divij!
> > > > > > > >>>
> > > > > > > >>> Luke
> > > > > > > >>> on behalf of Apache Kafka PMC
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: New Release Branch 3.7

[Satish Duggana]

Hi Stanislav,
+1 to what Luke mentioned in the earlier email. I have also replied to your
comment on https://issues.apache.org/jira/browse/KAFKA-15147

These are minor changes and we plan to target the remaining changes to be
merged in the next few days(before 20th Dec).

Thanks,
Satish.

On Thu, 14 Dec 2023 at 17:54, Luke Chen  wrote:

> Hi Stanislav,
>
> For KIP-963, there are only 1 remaining task and 2 PRs which should be
> close to get merged. (ref:
> https://issues.apache.org/jira/browse/KAFKA-15147)
> Since this is an improvement for tiered storage metrics, not a huge
> change, I'd like to backport them into 3.7 branch if no objection from
> you.
>
> Thanks.
> Luke
>
> On Thu, Dec 14, 2023 at 5:15 PM Stanislav Kozlovski
>  wrote:
> >
> > Hey all,
> >
> > (thanks to Josep for reviewing the 3.8 bump PR)
> >
> > I have two more PRs to get reviewed regarding the release:
> > - targeting trunk: MINOR: Update documentation.html with the 3.7 release
> > #15010 
> > - targeting 3.7: MINOR: Update documentation.html with the 3.7 release
> > #15011 
> >
> > Additionally, I have one ask:
> > - if you are reading this message, can you double-check the list of KIPs
> > being released in the Release Page
> > 
> and
> > if you recognize any KIP you are involved it - can you ensure that the
> data
> > in the page & the associated JIRA/KIP (target release, merge status, vote
> > status) is up to date?
> >
> > I have myself went over the list of KIPs and a bit of the commit
> history. I
> > think KIPs:
> > - KIP-998: Give ProducerConfig(props, doLog) constructor protected access
> > - KIP-938: Add more metrics for measuring KRaft performance
> > are slipping this release.
> >
> > By far our biggest feature this release, KIP-848 The Next Generation of
> the
> > Consumer Rebalance Protocol, is a bit on the border. The main PR that KIP
> > is dependent on is this KAFKA-15456: Commit/Fetch error handling
> > improvements and V9 support #14557
> > .
> > While it's a gray area, I am weighing this change more on the
> stabilization
> > side as well as allowing it to come in the middle of feature freeze for
> two
> > chief reasons: a) it fixes issues that were recently discovered, hence
> can
> > be marked as stabilization work and b) it only touches the code path used
> > for Early Access, and not the existing production code path (hence
> doesn't
> > substantially risk the release)
> >
> > Additionally, KIP-858 has one minor but important change pending
> > .
> >
> > With that, I remind you that there are only 6 days to code freeze! It's
> not
> > long until we will have our very first Apache Kafka 3.7 RC! Let's get
> this
> > shipped.
> >
> > Best,
> > Stanislav
> >
> > On Tue, Dec 12, 2023 at 2:56 PM Stanislav Kozlovski <
> stanis...@confluent.io>
> > wrote:
> >
> > > Hello Kafka developers and friends,
> > >
> > > As promised, we now have a release branch for 3.7 release.
> > > Trunk is being bumped to 3.8.0-SNAPSHOT (please help review the PR <
> https://github.com/apache/kafka/pull/14993>).
> > >
> > > I'll be going over the JIRAs to move every non-blocker from this
> release to the next release.
> > >
> > > From this point, most changes should go to trunk.
> > > *- Blockers (existing and new that we discover while testing the
> release) will be double-committed.*
> > > *- Please discuss with your reviewer whether your PR should go to
> trunk or to trunk+release so they can merge accordingly.*
> > > *- Please help us test the release!*
> > >
> > > Thanks!
> > >
> > > --
> > > Best,
> > > Stanislav
> > >
> >
> >
> > --
> > Best,
> > Stanislav
>

[jira] [Created] (KAFKA-15864) Add more tests asserting the log-start-offset, local-log-start-offset, and HW/LSO/LEO in rolling over segments with tiered storage.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15864:
--

 Summary: Add more tests asserting the log-start-offset, 
local-log-start-offset, and HW/LSO/LEO in rolling over segments with tiered 
storage.
 Key: KAFKA-15864
 URL: https://issues.apache.org/jira/browse/KAFKA-15864
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Satish Duggana
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1005: Add EarliestLocalOffset to GetOffsetShell

[Satish Duggana]

Thanks Christo for starting the discussion on the KIP.

As mentioned in KAFKA-15857[1], the goal is to add new entries for
local-log-start-offset and tierd-offset in OffsetSpec. This will be
used in AdminClient APIs and also to be added as part of
GetOffsetShell. This was also raised by Kamal in the earlier email.

OffsetSpec related changes for these entries also need to be mentioned
as part of the PublicInterfaces section because these are exposed to
users as public APIs through Admin#listOffsets() APIs[2, 3].

Please update the KIP with the above details.

1. https://issues.apache.org/jira/browse/KAFKA-15857
2.  
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/Admin.java#L1238
3. 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/Admin.java#L1226

~Satish.

On Mon, 20 Nov 2023 at 18:35, Kamal Chandraprakash
 wrote:
>
> Hi Christo,
>
> Thanks for the KIP!
>
> Similar to the earliest-local-log offset, can we also expose the
> highest-copied-remote-offset via
> GetOffsetShell tool? This will be useful during the debugging session.
>
>
> On Mon, Nov 20, 2023 at 5:38 PM Christo Lolov 
> wrote:
>
> > Hello all!
> >
> > I would like to start a discussion for
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1005%3A+Add+EarliestLocalOffset+to+GetOffsetShell
> > .
> >
> > A new offset called local log start offset was introduced as part of
> > KIP-405: Kafka Tiered Storage. KIP-1005 aims to expose this offset by
> > changing the AdminClient and in particular the GetOffsetShell tool.
> >
> > I am looking forward to your suggestions for improvement!
> >
> > Best,
> > Christo
> >

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

[Satish Duggana]

+1 (binding)
Thanks for the KIP and the discussion.

Discussion mail thread for the KIP:
https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45



On Tue, 21 Nov 2023 at 05:21, Kamal Chandraprakash
 wrote:
>
> +1 (non-binding). Thanks for the KIP!
>
> On Tue, Nov 21, 2023, 03:04 Divij Vaidya  wrote:
>
> > + 1 (binding)
> >
> > This Kip will greatly improve Tiered Storage troubleshooting. Thank you
> > Christo.
> >
> > On Mon 20. Nov 2023 at 17:21, Christo Lolov 
> > wrote:
> >
> > > Hello all!
> > >
> > > Now that the discussion for KIP-963 has winded down, I would like to open
> > > it for a vote targeting 3.7.0 as the release. You can find the current
> > > version of the KIP at
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Additional+metrics+in+Tiered+Storage
> > >
> > > Best,
> > > Christo
> > >
> >

[jira] [Created] (KAFKA-15857) Introduce LocalLogStartOffset and TieredOffset in OffsetSpec.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15857:
--

 Summary: Introduce LocalLogStartOffset and TieredOffset in 
OffsetSpec.
 Key: KAFKA-15857
 URL: https://issues.apache.org/jira/browse/KAFKA-15857
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Satish Duggana


Introduce  EarliestLocalOffset and TieredOffset in OffsetSpec which will help 
in finding respective offsets while using AdminClient#listOffsets().

EarliestLocalOffset - local log start offset of a topic partition.
TieredOffset - Highest offset up to which the segments were copied to remote 
storage.

We can discuss further on naming and semantics of these offset specs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Satish Duggana]

Hi Christo,
I think we can start the vote thread on the KIP which is updated with the
finalized metrics. We can have followup KIPs with other metrics if needed
in future.

Thanks,
Satish.


On Fri, 17 Nov 2023 at 22:16, Christo Lolov  wrote:

> Heya all!
>
> I have updated the KIP so please have another read through when you have
> the time. I know we are cutting it a bit close, but I would be grateful if
> I could start a vote early next week in order to get this in 3.7.
>
> re: Satish
>
> 104. I envision that ideally we will compare this metric with
> *RemoteCopyLagSegments*, yes.
>
> re: Jorge
>
> I have been thinking about your suggestion for RemoteDeleteBytesPerSec. The
> *BytesPerSec make sense on the Copy and Read paths because there we
> actually write a whole segment or read it, but on the delete path we tend
> to just mark it for deletion, as such we don't really have deleted bytes
> per sec. Or am I misunderstanding why you want this metric added?
>
> I have also thought a bit more about the LocalDeleteLag and your
> description. If I understand correctly you propose this metric to monitor
> the segments expired due to local retention, have the .deleted suffix, but
> haven't yet been actually deleted by the LogCleaner. This will serve as a
> proxy for how much data which we should be serving from remote but we are
> serving from local? However, I believe that the moment we add the .delete
> suffix we stop serving traffic from those segments, hence we will be
> serving requests for those from remote. Am I wrong?
>
> Best,
> Christo
>
> On Thu, 16 Nov 2023 at 08:45, Satish Duggana 
> wrote:
>
> > Thanks Christo for your reply.
> >
> > 101 and 102 We have conclusion on them.
> >
> > 103. I am not strongly opinionated on this. I am fine if it is helpful
> > for your scenarios.
> >
> > 104. It seems you want to compare this metric with the number of
> > segments that are copied. Do you have such a metric now?
> >
> > Kamal and Luke,
> > I agree some of the metrics are needed outside of RSM layer in remote
> > fetch path. Can we take those fine grained remote fetch flow sequence
> > metrics separately later?
> >
> > Thanks,
> > Satish.
> >
> > On Tue, 14 Nov 2023 at 22:07, Christo Lolov 
> > wrote:
> > >
> > > Heya everyone,
> > >
> > > Apologies for the delay in my response and thank you very much for all
> > your
> > > comments! I will start answering in reverse:
> > >
> > > *re: Satish*
> > >
> > > 101. I am happy to scope down this KIP and start off by emitting those
> > > metrics on a topic level. I had a preference to emit them on a
> partition
> > > level because I have ran into situations where data wasn't evenly
> spread
> > > across partitions and not having that granularity made it harder to
> > > discover.
> > >
> > > 102. Fair enough, others have expressed the same preference. I will
> scope
> > > down the KIP to only bytes-based and segment-based metrics.
> > >
> > > 103. I agree that we could do this, but I personally prefer this to be
> a
> > > metric. At the very least a newcomer might not know to look for the log
> > > line, while most metric-collection systems allow you to explore the
> whole
> > > namespace. For example, I really dislike that while log loading happens
> > > Kafka emits log lines of "X/Y logs loaded" rather than just show me the
> > > progress via a metric. If you are strongly against this, however, I am
> > > happy to scope down on this as well.
> > >
> > > 104. Ideally we have only one metadata in remote storage for every
> > segment
> > > of the correct lineage. Due to leadership changes, however, this is not
> > > always the case. I envisioned that exposing such a metric will showcase
> > if
> > > there are problems with too many metadata records not part of the
> correct
> > > lineage of a log.
> > >
> > > *re: Luke*
> > >
> > > 1. I am a bit conflicted on this one. As discussed earlier with Jorge,
> in
> > > my head such metrics are better left to plugin implementations. If you
> > and
> > > Kamal feel strongly about this being included I will add it to the KIP.
> > >
> > > 2. After running tiered storage in production for a while I ran into
> > > problems where a partition-level metric would have allowed me to zone
> in
> > on
> > > the problem sooner. I tried balancing this with not exposing everything
> 

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Satish Duggana]

Thanks Christo for your reply.

101 and 102 We have conclusion on them.

103. I am not strongly opinionated on this. I am fine if it is helpful
for your scenarios.

104. It seems you want to compare this metric with the number of
segments that are copied. Do you have such a metric now?

Kamal and Luke,
I agree some of the metrics are needed outside of RSM layer in remote
fetch path. Can we take those fine grained remote fetch flow sequence
metrics separately later?

Thanks,
Satish.

On Tue, 14 Nov 2023 at 22:07, Christo Lolov  wrote:
>
> Heya everyone,
>
> Apologies for the delay in my response and thank you very much for all your
> comments! I will start answering in reverse:
>
> *re: Satish*
>
> 101. I am happy to scope down this KIP and start off by emitting those
> metrics on a topic level. I had a preference to emit them on a partition
> level because I have ran into situations where data wasn't evenly spread
> across partitions and not having that granularity made it harder to
> discover.
>
> 102. Fair enough, others have expressed the same preference. I will scope
> down the KIP to only bytes-based and segment-based metrics.
>
> 103. I agree that we could do this, but I personally prefer this to be a
> metric. At the very least a newcomer might not know to look for the log
> line, while most metric-collection systems allow you to explore the whole
> namespace. For example, I really dislike that while log loading happens
> Kafka emits log lines of "X/Y logs loaded" rather than just show me the
> progress via a metric. If you are strongly against this, however, I am
> happy to scope down on this as well.
>
> 104. Ideally we have only one metadata in remote storage for every segment
> of the correct lineage. Due to leadership changes, however, this is not
> always the case. I envisioned that exposing such a metric will showcase if
> there are problems with too many metadata records not part of the correct
> lineage of a log.
>
> *re: Luke*
>
> 1. I am a bit conflicted on this one. As discussed earlier with Jorge, in
> my head such metrics are better left to plugin implementations. If you and
> Kamal feel strongly about this being included I will add it to the KIP.
>
> 2. After running tiered storage in production for a while I ran into
> problems where a partition-level metric would have allowed me to zone in on
> the problem sooner. I tried balancing this with not exposing everything on
> a partition level so not to explode the cardinality too much (point 101
> from Satish). I haven't ran into a situation where knowing the
> RemoteLogSizeComputationTime on a partition level helped me, but this
> doesn't mean there isn't one.
>
> 3. I was thinking that the metric can be emitted while reading of those
> records is happening i.e. if it takes a long time then it will just
> gradually increase as we read. What do you think?
>
> *re: Jorge*
>
> 3.5. Sure, I will aim to add my thoughts to the KIP
>
> 4. Let me check and come back to you on this one. I have a vague memory
> this wasn't as easy to calculate, but if it is, I will include
> RemoteDeleteBytesPerSec as well.
>
> 7. Yes, this is I believe a better explanation than the one I have in the
> KIP, so I will aim to update it with your one. Thank you! LocalDeleteLag
> makes sense to me as well, I will aim to include it.
>
> *re: Kamal*
>
> 1. I can add this to the KIP, but similar to what I have mentioned earlier,
> I believe these are better left to plugin implementations, no?
>
> 2. Yeah, this makes sense.
>
> Best,
> Christo
>
> On Fri, 10 Nov 2023 at 09:33, Satish Duggana 
> wrote:
>
> > Thanks Christo for the KIP and the interesting discussion.
> >
> > 101. Adding metrics at partition level may increase the cardinality of
> > these metrics. We should be cautious of that and see whether they are
> > really needed. RLM related operations do not generally affect based on
> > partition(s) but it is mostly because of the remote storage or broker
> > level issues.
> >
> > 102. I am not sure whether the records metric is much useful when we
> > have other bytes and segments related metrics available. If needed,
> > records level information can be derived once we have segments/bytes
> > metrics.
> >
> > 103. Regarding RemoteLogSizeComputationTime, we can add logs for
> > debugging purposes to print the required duration while computing size
> > instead of generating a metric. If there is any degradation in remote
> > log size computation, it will have an effect on RLM task leading to
> > remote log copy and delete lags.
> >
> > RLMM and RSM implementations can always add more metrics for
> > observ

Re: [VOTE] KIP-997: Partition-Level Throughput Metrics

[Satish Duggana]

Thanks Qichao for the KIP.

+1 (binding)

~Satish.

On Thu, 16 Nov 2023 at 02:20, Jorge Esteban Quilcate Otoya
 wrote:
>
> Qichao, thanks again for leading this proposal!
>
> +1 (non-binding)
>
> Cheers,
> Jorge.
>
> On Wed, 15 Nov 2023 at 19:17, Divij Vaidya  wrote:
>
> > +1 (binding)
> >
> > I was involved in the discussion thread for this KIP and support it in its
> > current form.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Wed, Nov 15, 2023 at 10:55 AM Qichao Chu 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to call a vote for KIP-977: Partition-Level Throughput Metrics.
> > >
> > > Please take a look here:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> > >
> > > Best,
> > > Qichao Chu
> > >
> >

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Satish Duggana]

Thanks Christo for the KIP and the interesting discussion.

101. Adding metrics at partition level may increase the cardinality of
these metrics. We should be cautious of that and see whether they are
really needed. RLM related operations do not generally affect based on
partition(s) but it is mostly because of the remote storage or broker
level issues.

102. I am not sure whether the records metric is much useful when we
have other bytes and segments related metrics available. If needed,
records level information can be derived once we have segments/bytes
metrics.

103. Regarding RemoteLogSizeComputationTime, we can add logs for
debugging purposes to print the required duration while computing size
instead of generating a metric. If there is any degradation in remote
log size computation, it will have an effect on RLM task leading to
remote log copy and delete lags.

RLMM and RSM implementations can always add more metrics for
observability based on the respective implementations.

104. What is the purpose of RemoteLogMetadataCount as a metric?

Thanks,
Satish.

On Fri, 10 Nov 2023 at 04:10, Jorge Esteban Quilcate Otoya
 wrote:
>
> Hi Christo,
>
> I'd like to add another suggestion:
>
> 7. Adding on TS lag formulas, my understanding is that per pertition:
> - RemoteCopyLag: difference between: latest local segment candidate for
> upload - latest remote segment
>   - Represents how Remote Log Manager task is handling backlog of segments.
>   - Ideally, this lag is zero -- grows when upload is slower than the
> increase on candidate segments to upload
>
> - RemoteDeleteLag: difference between: latest remote candidate segment to
> keep based on retention - oldest remote segment
>   - Represents how many segments Remote Log Manager task is missing to
> delete at a given point in time
>   - Ideally, this lag is zero -- grows when retention condition changes but
> RLM task is not able to schedule deletion yet.
>
> Is my understanding of these lags correct?
>
> I'd like to also consider an additional lag:
> - LocalDeleteLag: difference between: latest local candidate segment to
> keep based on local retention - oldest local segment
>   - Represents how many segments are still available locally when they are
> candidate for deletion. This usually happens when log cleaner has not been
> scheduled yet. It's important because it represents how much data is stored
> locally when it could be removed, and it represents how much data that can
> be sourced from remote tier is still been sourced from local tier.
>   - Ideally, this lag returns to zero when log cleaner runs; but could be
> growing if there are issues uploading data (other lag) or removing data
> internally.
>
> Thanks,
> Jorge.
>
> On Thu, 9 Nov 2023 at 10:51, Luke Chen  wrote:
>
> > Hi Christo,
> >
> > Thanks for the KIP!
> >
> > Some comments:
> > 1. I agree with Kamal that a metric to cover the time taken to read data
> > from remote storage is helpful.
> >
> > 2. I can see there are some metrics are only on topic level, but some are
> > on partition level.
> > Could you explain why some of them are only on topic level?
> > Like RemoteLogSizeComputationTime, it's different from partition to
> > partition, will it be better to be exposed as partition metric?
> >
> > 3. `RemoteLogSizeBytes` metric hanging.
> > To compute the RemoteLogSizeBytes, we need to wait until all records in the
> > metadata topic loaded.
> > What will happen if it takes long to load the data from metadata topic?
> > Should we instead return -1 or something to indicate it's still loading?
> >
> > Thanks.
> > Luke
> >
> > On Fri, Nov 3, 2023 at 1:53 AM Kamal Chandraprakash <
> > kamal.chandraprak...@gmail.com> wrote:
> >
> > > Hi Christo,
> > >
> > > Thanks for expanding the scope of the KIP!  We should also cover the time
> > > taken to
> > > read data from remote storage. This will give our users a fair idea about
> > > the P99, P95,
> > > and P50 Fetch latency to read data from remote storage.
> > >
> > > The Fetch API request metrics currently provides a breakdown of the time
> > > spent on each item:
> > >
> > >
> > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/RequestChannel.scala#L517
> > > Should we also provide `RemoteStorageTimeMs` item (only for FETCH API) so
> > > that users can
> > > understand the overall and per-step time taken?
> > >
> > > Regarding the Remote deletion metrics, should we also emit a metric to
> > > expose the oldest segment time?
> > > Users can configure the topic retention either by size (or) time. If time
> > > is configured, then emitting
> > > the oldest segment time allows the user to configure an alert on top of
> > it
> > > and act accordingly.
> > >
> > > On Wed, Nov 1, 2023 at 7:07 PM Jorge Esteban Quilcate Otoya <
> > > quilcate.jo...@gmail.com> wrote:
> > >
> > > > Thanks, Christo!
> > > >
> > > > 1. Agree. Having a further look into how many latency metrics are
> > > included
> > > > on the broker side 

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Satish Duggana]

Thank you, everyone!

~Satish.

On Mon, 30 Oct 2023 at 16:14, Josep Prat  wrote:
>
> Congrats Satish!
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Oct 30, 2023, 11:37 Bruno Cadonna  wrote:
>
> > Congrats, Satish!
> >
> > Bruno
> >
> > On 10/29/23 2:42 PM, John Roesler wrote:
> > > Congratulations, Satish!
> > > -John
> > >
> > > On Sun, Oct 29, 2023, at 08:09, Randall Hauch wrote:
> > >> Congratulations, Satish!
> > >>
> > >> On Sun, Oct 29, 2023 at 1:47 AM Tom Bentley 
> > wrote:
> > >>
> > >>> Congratulations!
> > >>>
> > >>> On Sun, 29 Oct 2023 at 5:41 PM, Guozhang Wang <
> > guozhang.wang...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Congratulations Satish!
> > >>>>
> > >>>> On Sat, Oct 28, 2023 at 12:59 AM Luke Chen  wrote:
> > >>>>>
> > >>>>> Congrats Satish!
> > >>>>>
> > >>>>> Luke
> > >>>>>
> > >>>>> On Sat, Oct 28, 2023 at 11:16 AM ziming deng <
> > dengziming1...@gmail.com
> > >>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Congratulations Satish!
> > >>>>>>
> > >>>>>>> On Oct 27, 2023, at 23:03, Jun Rao 
> > >>> wrote:
> > >>>>>>>
> > >>>>>>> Hi, Everyone,
> > >>>>>>>
> > >>>>>>> Satish Duggana has been a Kafka committer since 2022. He has been
> > >>>> very
> > >>>>>>> instrumental to the community since becoming a committer. It's my
> > >>>>>> pleasure
> > >>>>>>> to announce that Satish is now a member of Kafka PMC.
> > >>>>>>>
> > >>>>>>> Congratulations Satish!
> > >>>>>>>
> > >>>>>>> Jun
> > >>>>>>> on behalf of Apache Kafka PMC
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>
> >

Re: [DISCUSS] Apache Kafka 3.5.2 release

[Satish Duggana]

Thanks Luke for volunteering for 3.5.2 release.

On Tue, 17 Oct 2023 at 11:58, Josep Prat  wrote:
>
> Hi Luke,
>
> Thanks for taking this one!
>
> Best,
>
> On Tue, Oct 17, 2023 at 8:12 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to volunteer as release manager for the Apache Kafka 3.5.2, to
> > have an important bug/vulnerability fix release for 3.5.1.
> >
> > If there are no objections, I'll start building a release plan in thewiki
> > in the next couple of weeks.
> >
> > Thanks,
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-15612) Followup on whether the indexes need to be materialized before they are passed to RSM for writing them to tiered storage.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15612:
--

 Summary: Followup on whether the indexes need to be materialized 
before they are passed to RSM for writing them to tiered storage. 
 Key: KAFKA-15612
 URL: https://issues.apache.org/jira/browse/KAFKA-15612
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana


Followup on the [PR 
comment|https://github.com/apache/kafka/pull/14529#discussion_r1355263700]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Apache Kafka 3.6.0

[Satish Duggana]

I'd like to clarify calling out 3.6.0 as a minor release.

We use semantic versioning*(Major.Minor.Patch) in Apache Kafka
releases. Releases like 3.5.0 or 3.6.0 are called minor releases
according to the semantic versioning. The next major release will be
4.0.0.

3.6.0 release is packed with big features and improvements as
mentioned in the blog post[1], and release notes[2], even though it is
called a minor release :)

1. https://kafka.apache.org/blog
2. https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html
* https://semver.org/

Thanks,
Satish.


On Wed, 11 Oct 2023 at 12:00, Luke Chen  wrote:
>
> Thanks for running the release, Satish!
>
> BTW, 3.6.0 should be a major release, not a minor one. :)
>
> Luke
>
> On Wed, Oct 11, 2023 at 1:39 PM Satish Duggana  wrote:
>
> > The Apache Kafka community is pleased to announce the release for
> > Apache Kafka 3.6.0
> >
> > This is a minor release and it includes fixes and improvements from 238
> > JIRAs.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html
> >
> > An overview of the release can be found in our announcement blog post:
> > https://kafka.apache.org/blog
> >
> > You can download the source and binary release (Scala 2.12 and Scala 2.13)
> > from:
> > https://kafka.apache.org/downloads#3.6.0
> >
> >
> > ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 139 contributors to this release!
> > (Please report an unintended omission)
> >
> > This was a community effort, so thank you to everyone who contributed
> > to this release, including all our users and our 139 contributors:
> > A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
> > Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
> > Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
> > Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
> > bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
> > Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
> > Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
> > Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
> > David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
> > Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
> > Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
> > Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
> > Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
> > iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
> > Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
> > Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
> > Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
> > Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas
> > Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, M

[jira] [Created] (KAFKA-15594) Add 3.6.0 to streams upgrade/compatibility tests

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15594:
--

 Summary: Add 3.6.0 to streams upgrade/compatibility tests
 Key: KAFKA-15594
 URL: https://issues.apache.org/jira/browse/KAFKA-15594
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15593) Add 3.6.0 to broker/client upgrade/compatibility tests

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15593:
--

 Summary: Add 3.6.0 to broker/client upgrade/compatibility tests
 Key: KAFKA-15593
 URL: https://issues.apache.org/jira/browse/KAFKA-15593
 Project: Kafka
  Issue Type: Sub-task
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15576) Add 3.2.0 to broker/client and streams upgrade/compatibility tests

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15576:
--

 Summary: Add 3.2.0 to broker/client and streams 
upgrade/compatibility tests
 Key: KAFKA-15576
 URL: https://issues.apache.org/jira/browse/KAFKA-15576
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Kafka 3.6.0

[Satish Duggana]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.6.0

This is a minor release and it includes fixes and improvements from 238 JIRAs.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html

An overview of the release can be found in our announcement blog post:
https://kafka.apache.org/blog

You can download the source and binary release (Scala 2.12 and Scala 2.13) from:
https://kafka.apache.org/downloads#3.6.0

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 139 contributors to this release!
(Please report an unintended omission)

This was a community effort, so thank you to everyone who contributed
to this release, including all our users and our 139 contributors:
A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas
Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, Manikumar Reddy,
Manyanda Chitimbo, Maros Orsak, Matthew de Detrich, Matthias J. Sax,
maulin-vasavada, Max Riedel, Mehari Beyene, Michal Cabak (@miccab),
Mickael Maison, Milind Mantri, minjian.cai, mojh7, Nikolay, Okada
Haruki, Omnia G H Ibrahim, Owen Leung, Philip Nee, prasanthV, Proven
Provenzano, Purshotam Chauhan, Qichao Chu, Rajini Sivaram, Randall
Hauch, Renaldo Baur Filho, Ritika Reddy, Rittika Adhikari, Rohan, Ron
Dagostino, Sagar Rao, Said Boudjelda, Sambhav Jain, Satish Duggana,
sciclon2, Shekhar Rajak, Sungyun Hur, Sushant Mahajan, Tanay
Karmarkar, tison, Tom Bentley, vamossagar12, Victoria Xia, Vincent
Jiang, vveicc, Walker Carlson, Yash Mayya, Yi-Sheng Lien, Ziming Deng,
蓝士钦

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Satish Duggana

[jira] [Resolved] (KAFKA-15535) Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property.

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15535.

Resolution: Fixed

> Add documentation of "remote.log.index.file.cache.total.size.bytes" 
> configuration property. 
> 
>
> Key: KAFKA-15535
> URL: https://issues.apache.org/jira/browse/KAFKA-15535
> Project: Kafka
>  Issue Type: Task
>  Components: documentation
>Reporter: Satish Duggana
>Assignee: hudeqi
>Priority: Major
>  Labels: tiered-storage
> Fix For: 3.7.0
>
>
> Add documentation of "remote.log.index.file.cache.total.size.bytes" 
> configuration property. 
> Please double check all the existing public tiered storage configurations. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

+1 to add it as part of upgrade page. Users refer to the upgrade page
to access any information related to upgrades for a release as it is
expected to contain all the relevant text and references regarding the
upgrade in that particular release.

David,

On Thu, 5 Oct 2023 at 17:22, Ismael Juma  wrote:
>
> The upgrade page is the right place to add it, in my opinion.
>
> Ismael
>
> On Thu, Oct 5, 2023 at 5:02 PM David Arthur
>  wrote:
>
> > Hey folks, we found a significant ZK migration bug today
> > https://issues.apache.org/jira/browse/KAFKA-15552. Since the release is
> > already voted in, the fix will wait for 3.6.1. In the meantime, do we have
> > a way to indicate known significant issues in the release notes? Maybe just
> > a note in the blog?
> >
> > Thanks!
> > David
> >
> > On Fri, Sep 29, 2023 at 1:42 PM Ismael Juma  wrote:
> >
> > > Thanks Satish!
> > >
> > > Ismael
> > >
> > > On Thu, Sep 28, 2023 at 6:39 AM Satish Duggana  > >
> > > wrote:
> > >
> > > > We do not have any pending release blockers for now. RC1 release
> > > > blocker KAFKA-15498[1] is merged to 3.6.
> > > >  I will create RC2 by 29th Sep 12:00 pm PT and start a new RC thread.
> > > >
> > > > 1. https://github.com/apache/kafka/pull/14434
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > >
> > > > On Wed, 27 Sept 2023 at 13:36, Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > Ismael,
> > > > > Thank you for checking.
> > > > > Multiple other folks have validated after I left the comment here
> > that
> > > > > it doesn't impact log truncation and hence won't lead to data loss. I
> > > > > agree that it's not a blocker.
> > > > >
> > > > > (ref: https://github.com/apache/kafka/pull/14457)
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > > On Wed, Sep 27, 2023 at 8:50 PM Ismael Juma 
> > wrote:
> > > > > >
> > > > > > Doesn't look like a blocker to me.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Wed, Sep 27, 2023 at 2:36 AM Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hey team
> > > > > > >
> > > > > > > I need help in determining whether
> > > > > > > https://github.com/apache/kafka/pull/14457 is a release blocker
> > > bug
> > > > or
> > > > > > > not. If someone is familiar with replication protocol (on the log
> > > > > > > diverange and reconciliation process), please add your comments
> > on
> > > > the
> > > > > > > PR.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > > On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > A community member reported another bug in TS feature in 3.6 -
> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15511
> > > > > > > >
> > > > > > > > I don't consider it as a blocker for release because the bug
> > > > occurs in
> > > > > > > > rare situations when the index on disk or in a remote store is
> > > > > > > > corrupted and fails a sanity check.
> > > > > > > > Sharing it here as an FYI.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Divij Vaidya
> > > > > > > >
> > > > > > > > On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Found a bug while testing TS feature in 3.6 -
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15481
> > > > > > > > >
> > > > > > > > > I don't consider it as a b

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

Hi,
Thank you all for participating in the 3.6.0 release candidate vote threads.

This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes.

+1 votes
PMC Members:
* Luke Chen
* Bill Bejeck
* Chris Egerton
* Justine Olshan

Community:
* Jakub Scholz
* Federico Valeri
* Divij Vaidya
* Kamal Chandraprakash
* Josep Prat
* Proven Provenzano
* Greg Harris

0 votes
* No votes

-1 votes
* No votes

I will continue with the release process and the release announcement
will follow in the next few days.

Thanks,
Satish.

On Tue, 3 Oct 2023 at 12:35, Satish Duggana  wrote:
>
> Hi,
> Thank you all for participating in the 3.6.0 release candidate vote threads.
>
> This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes.
>
> +1 votes
> PMC Members:
> * Luke Chen
> * Bill Bejeck
> * Chris Egerton
> * Justine Olshan
>
> Community:
> * Jakub Scholz
> * Federico Valeri
> * Divij Vaidya
> * Kamal Chandraprakash
> * Josep Prat
> * Proven Provenzano
> * Greg Harris
>
> 0 votes
> * No votes
>
> -1 votes
> * No votes
>
> I will continue with the release process and the release announcement
> will follow in the next few days.
>
> Thanks,
> Satish.
>
> On Tue, 3 Oct 2023 at 09:54, Justine Olshan
>  wrote:
> >
> > Thanks folks for following up. Given my previous testing and the results
> > you've provided, I'm +1 (binding)
> >
> > I will also follow up with the non-blocking metrics documentation.
> >
> > Thanks!
> > Justine
> >
> > On Tue, Oct 3, 2023 at 8:17 AM Chris Egerton 
> > wrote:
> >
> > > Hi Satish,
> > >
> > > Thanks for running this release!
> > >
> > > To verify, I:
> > > - Built from source using Java 11 with both:
> > > - - the 3.6.0-rc2 tag on GitHub
> > > - - the kafka-3.6.0-src.tgz artifact from
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > - Checked signatures and checksums
> > > - Ran the quickstart using the kafka_2.13-3.6.0.tgz artifact from
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/ with Java 11 and Scala
> > > 13
> > > in KRaft mode
> > > - Ran all unit tests
> > > - Ran all integration tests for Connect and MM2
> > > - Verified that the connect-test-plugins module is present in the staging
> > > Maven artifacts (https://issues.apache.org/jira/browse/KAFKA-15249)
> > >
> > > Everything looks good to me!
> > >
> > > +1 (binding)
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Oct 3, 2023 at 6:43 AM Satish Duggana 
> > > wrote:
> > >
> > > > Thanks Luke for helping on running system tests on RCs and updating
> > > > the status on this email thread.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 3 Oct 2023 at 05:04, Luke Chen  wrote:
> > > > >
> > > > > Hi Justine and all,
> > > > >
> > > > > The system test result for 3.6.0 RC2 can be found below.
> > > > > In short, no failed tests. The flaky tests will pass in the 2nd run.
> > > > >
> > > >
> > > https://drive.google.com/drive/folders/1qwIKg-B4CBrswUeo5fBRv65KWpDsGUiS?usp=sharing
> > > > >
> > > > > Thank you.
> > > > > Luke
> > > > >
> > > > > On Tue, Oct 3, 2023 at 7:08 AM Justine Olshan
> > > > 
> > > > > wrote:
> > > > >
> > > > > > I realized Luke shared the results here for RC1
> > > > > >
> > > > https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC
> > > > > > Given we had some runs that looked reasonable, and we made a small
> > > > change,
> > > > > > I'm ok with this. But I wouldn't be upset if we had another set of
> > > > runs :)
> > > > > >
> > > > > > As for the validation:
> > > > > >
> > > > > >- I've compiled from source with java 17, 2.13, run the
> > > > transactional
> > > > > >produce bench
> > > > > >- Run unit tests
> > > > > >- Validated the checksums
> > > > > >- Downloaded and ran the 2.12 version of the release
> > > > > >- Briefly took a look at the documentation
> > > > > >- I was browsing through the site html files and I noticed the
> > > html
> > > > for
> &

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

Hi,
Thank you all for participating in the 3.6.0 release candidate vote threads.

This vote passes with 11 +1 votes(4 binding) and no 0 or -1 votes.

+1 votes
PMC Members:
* Luke Chen
* Bill Bejeck
* Chris Egerton
* Justine Olshan

Community:
* Jakub Scholz
* Federico Valeri
* Divij Vaidya
* Kamal Chandraprakash
* Josep Prat
* Proven Provenzano
* Greg Harris

0 votes
* No votes

-1 votes
* No votes

I will continue with the release process and the release announcement
will follow in the next few days.

Thanks,
Satish.

On Tue, 3 Oct 2023 at 09:54, Justine Olshan
 wrote:
>
> Thanks folks for following up. Given my previous testing and the results
> you've provided, I'm +1 (binding)
>
> I will also follow up with the non-blocking metrics documentation.
>
> Thanks!
> Justine
>
> On Tue, Oct 3, 2023 at 8:17 AM Chris Egerton 
> wrote:
>
> > Hi Satish,
> >
> > Thanks for running this release!
> >
> > To verify, I:
> > - Built from source using Java 11 with both:
> > - - the 3.6.0-rc2 tag on GitHub
> > - - the kafka-3.6.0-src.tgz artifact from
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > - Checked signatures and checksums
> > - Ran the quickstart using the kafka_2.13-3.6.0.tgz artifact from
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/ with Java 11 and Scala
> > 13
> > in KRaft mode
> > - Ran all unit tests
> > - Ran all integration tests for Connect and MM2
> > - Verified that the connect-test-plugins module is present in the staging
> > Maven artifacts (https://issues.apache.org/jira/browse/KAFKA-15249)
> >
> > Everything looks good to me!
> >
> > +1 (binding)
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Oct 3, 2023 at 6:43 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Luke for helping on running system tests on RCs and updating
> > > the status on this email thread.
> > >
> > > ~Satish.
> > >
> > > On Tue, 3 Oct 2023 at 05:04, Luke Chen  wrote:
> > > >
> > > > Hi Justine and all,
> > > >
> > > > The system test result for 3.6.0 RC2 can be found below.
> > > > In short, no failed tests. The flaky tests will pass in the 2nd run.
> > > >
> > >
> > https://drive.google.com/drive/folders/1qwIKg-B4CBrswUeo5fBRv65KWpDsGUiS?usp=sharing
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > > On Tue, Oct 3, 2023 at 7:08 AM Justine Olshan
> > > 
> > > > wrote:
> > > >
> > > > > I realized Luke shared the results here for RC1
> > > > >
> > > https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC
> > > > > Given we had some runs that looked reasonable, and we made a small
> > > change,
> > > > > I'm ok with this. But I wouldn't be upset if we had another set of
> > > runs :)
> > > > >
> > > > > As for the validation:
> > > > >
> > > > >- I've compiled from source with java 17, 2.13, run the
> > > transactional
> > > > >produce bench
> > > > >- Run unit tests
> > > > >- Validated the checksums
> > > > >- Downloaded and ran the 2.12 version of the release
> > > > >- Briefly took a look at the documentation
> > > > >- I was browsing through the site html files and I noticed the
> > html
> > > for
> > > > >documentation.html seemed to be for 3.4. Not sure if this is a
> > > blocker,
> > > > > but
> > > > >wanted to flag it. This seems to be the case for the previous
> > > release
> > > > >candidates as well. (As well as 3.5 release it seems)
> > > > >
> > > > >
> > > > > I will hold off on voting until we figure that part out. I will also
> > > follow
> > > > > up with the documentation Divij mentioned outside this thread.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Mon, Oct 2, 2023 at 3:05 PM Greg Harris
> > > 
> > > > > wrote:
> > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > I verified KIP-898 functionality and the KAFKA-15473 patch.
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks!
> > > > > >
> > >

[jira] [Created] (KAFKA-15535) Add documentation of "remote.log.index.file.cache.total.size.bytes" configuration property.

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15535:
--

 Summary: Add documentation of 
"remote.log.index.file.cache.total.size.bytes" configuration property. 
 Key: KAFKA-15535
 URL: https://issues.apache.org/jira/browse/KAFKA-15535
 Project: Kafka
  Issue Type: Task
  Components: documentation
Reporter: Satish Duggana
 Fix For: 3.7.0


Add documentation of "remote.log.index.file.cache.total.size.bytes" 
configuration property. 

Please double check all the existing public tiered storage configurations. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

gt; > manager
> > > > > > public keys are available at
> > > > > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > > > > >
> > > > > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are
> > also
> > > > > > being emitted in 3.6.0 (with Zk).
> > > > > >
> > > > > > Problems (but not blockers):
> > > > > > 1. Metrics added in
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > > > > aren't available in the documentation (cc: Justine). I don't
> > consider
> > > > > this
> > > > > > as a release blocker but we should add it as a fast follow-up.
> > > > > >
> > > > > > 2. Metric added in
> > > > > >
> > > > > >
> > > > >
> > > >
> > https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > > > > isn't available in documentation (cc: David). I don't consider
> > this as
> > > > a
> > > > > > release blocker but we should add it as a fast follow-up.
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri <
> > fedeval...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Satish, I did the following to verify the release:
> > > > > > >
> > > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > - Ran all unit and integration tests
> > > > > > > - Spot checked documentation
> > > > > > > - Ran custom client applications using staging artifacts on a
> > 3-nodes
> > > > > > > cluster
> > > > > > > - Tested tiered storage with one of the available RSM
> > implementations
> > > > > > >
> > > > > > > +1 (non binding)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Fede
> > > > > > >
> > > > > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen 
> > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I verified with:
> > > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > > 2. Making sure the checksum are correct
> > > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > > > 4. Verified the tiered storage feature works well.
> > > > > > > >
> > > > > > > > +1 (binding).
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > Luke
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz 
> > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and the
> > staged
> > > > > Maven
> > > > > > > > > artifacts and run my tests. Everything seems to work fine
> > for me.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Jakub
> > > > > > > > >
> > > > > > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > > > > > satish.dugg...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > > >
> > > > > > > > > > This is the third candidate for the release of Apache Kafka
> > > > > 3.6.0.
> > > > > > > > > > Some of the major features include:
> > > > > > > > > >
> > > > > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > > > > > * KIP-917: Additional custom metadata for remote log
> > segment
> > > > > > > > > >
> > > > > > > > > > Release notes for the 3.6.0 release:
> > > > > > > > > >
> > > > > >
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > > > > > > >
> > > > > > > > > > *** Please download, test and vote by Tuesday, October 3,
> > 12pm
> > > > PT
> > > > > > > > > >
> > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > release:
> > > > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > > > >
> > > > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > > > > > > >
> > > > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > > > >
> > > > > > >
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > >
> > > > > > > > > > * Javadoc:
> > > > > > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > > > > > > > > >
> > > > > > > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2
> > tag:
> > > > > > > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > > > > > > > > >
> > > > > > > > > > * Documentation:
> > > > > > > > > > https://kafka.apache.org/36/documentation.html
> > > > > > > > > >
> > > > > > > > > > * Protocol:
> > > > > > > > > > https://kafka.apache.org/36/protocol.html
> > > > > > > > > >
> > > > > > > > > > * Successful Jenkins builds for the 3.6 branch:
> > > > > > > > > > There are a few runs of unit/integration tests. You can
> > see the
> > > > > > > latest
> > > > > > > > > > at
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/.
> > > > We
> > > > > > > will
> > > > > > > > > > continue running a few more iterations.
> > > > > > > > > > System tests:
> > > > > > > > > > We will send an update once we have the results.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Satish.
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

> > overall lower P99.8 latencies.
> > > > > > >
> > > > > > > 2. I have verified that detached signature is correct using
> > > > > > > https://www.apache.org/info/verification.html and the release
> > > manager
> > > > > > > public keys are available at
> > > > > > > https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
> > > > > > >
> > > > > > > 3. I have verified that all metrics emitted in 3.5.1 (with Zk)
> > are
> > > also
> > > > > > > being emitted in 3.6.0 (with Zk).
> > > > > > >
> > > > > > > Problems (but not blockers):
> > > > > > > 1. Metrics added in
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> > > > > > > aren't available in the documentation (cc: Justine). I don't
> > > consider
> > > > > > this
> > > > > > > as a release blocker but we should add it as a fast follow-up.
> > > > > > >
> > > > > > > 2. Metric added in
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> > > > > > > isn't available in documentation (cc: David). I don't consider
> > > this as
> > > > > a
> > > > > > > release blocker but we should add it as a fast follow-up.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri <
> > > fedeval...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish, I did the following to verify the release:
> > > > > > > >
> > > > > > > > - Built from source with Java 17 and Scala 2.13
> > > > > > > > - Ran all unit and integration tests
> > > > > > > > - Spot checked documentation
> > > > > > > > - Ran custom client applications using staging artifacts on a
> > > 3-nodes
> > > > > > > > cluster
> > > > > > > > - Tested tiered storage with one of the available RSM
> > > implementations
> > > > > > > >
> > > > > > > > +1 (non binding)
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Fede
> > > > > > > >
> > > > > > > > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen 
> > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > >
> > > > > > > > > I verified with:
> > > > > > > > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > > > > > > > 2. Making sure the checksum are correct
> > > > > > > > > 3. Browsing release notes, documents, javadocs, protocols.
> > > > > > > > > 4. Verified the tiered storage feature works well.
> > > > > > > > >
> > > > > > > > > +1 (binding).
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and the
> > > staged
> > > > > > Maven
> > > > > > > > > > artifacts and run my tests. Everything seems to work fine
> > > for me.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Jakub
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > > > > > > > satish.dugg...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > > > >
> > > > > > > > > > > This is the third candidate for the release of Apache
> > Kafka
> > > > > > 3.6.0.
> > > > > > > > > > > Some of the major features include:
> > > > > > > > > > >
> > > > > > > > > > > * KIP-405 : Kafka Tiered Storage
> > > > > > > > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > > > > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > > > > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > > > > > > > * KIP-938: Add more metrics for measuring KRaft
> > performance
> > > > > > > > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > > > > > > > * KIP-917: Additional custom metadata for remote log
> > > segment
> > > > > > > > > > >
> > > > > > > > > > > Release notes for the 3.6.0 release:
> > > > > > > > > > >
> > > > > > >
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > > > > > > > >
> > > > > > > > > > > *** Please download, test and vote by Tuesday, October 3,
> > > 12pm
> > > > > PT
> > > > > > > > > > >
> > > > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > > release:
> > > > > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > > > > >
> > > > > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > > > > > > > >
> > > > > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > > > >
> > > > > > > > > > > * Javadoc:
> > > > > > > > > > >
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> > > > > > > > > > >
> > > > > > > > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2
> > > tag:
> > > > > > > > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> > > > > > > > > > >
> > > > > > > > > > > * Documentation:
> > > > > > > > > > > https://kafka.apache.org/36/documentation.html
> > > > > > > > > > >
> > > > > > > > > > > * Protocol:
> > > > > > > > > > > https://kafka.apache.org/36/protocol.html
> > > > > > > > > > >
> > > > > > > > > > > * Successful Jenkins builds for the 3.6 branch:
> > > > > > > > > > > There are a few runs of unit/integration tests. You can
> > > see the
> > > > > > > > latest
> > > > > > > > > > > at
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/.
> > > > > We
> > > > > > > > will
> > > > > > > > > > > continue running a few more iterations.
> > > > > > > > > > > System tests:
> > > > > > > > > > > We will send an update once we have the results.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Satish.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >

Re: [VOTE] 3.6.0 RC2

[Satish Duggana]

Thanks Divij for the observations.

KAFKA-15483 is added to address the missing metrics related to KRaft
and ZK to KRaft migration. Please leave a comment at
https://github.com/apache/kafka-site/pull/548 in which related metrics
documentation is added.

https://issues.apache.org/jira/browse/KAFKA-15530 is raised to
followup on the missing documentation of metrics introduced in
KIP-890.

~Satish.

On Mon, 2 Oct 2023 at 05:37, Divij Vaidya  wrote:
>
> + 1 (non-binding)
>
> Verifications:
> 1. I ran a produce-consume workload with plaintext auth, JDK17, zstd
> compression using an open messaging benchmark and found 3.6 to be better
> than or equal to 3.5.1 across all dimensions. Notably, 3.6 had consistently
> 6-7% lower CPU utilization, lesser spikes on P99 produce latencies and
> overall lower P99.8 latencies.
>
> 2. I have verified that detached signature is correct using
> https://www.apache.org/info/verification.html and the release manager
> public keys are available at
> https://keys.openpgp.org/search?q=F65DC3423D4CD7B9
>
> 3. I have verified that all metrics emitted in 3.5.1 (with Zk) are also
> being emitted in 3.6.0 (with Zk).
>
> Problems (but not blockers):
> 1. Metrics added in
> https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e
> aren't available in the documentation (cc: Justine). I don't consider this
> as a release blocker but we should add it as a fast follow-up.
>
> 2. Metric added in
> https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18
> isn't available in documentation (cc: David). I don't consider this as a
> release blocker but we should add it as a fast follow-up.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Oct 2, 2023 at 9:50 AM Federico Valeri  wrote:
>
> > Hi Satish, I did the following to verify the release:
> >
> > - Built from source with Java 17 and Scala 2.13
> > - Ran all unit and integration tests
> > - Spot checked documentation
> > - Ran custom client applications using staging artifacts on a 3-nodes
> > cluster
> > - Tested tiered storage with one of the available RSM implementations
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> > On Mon, Oct 2, 2023 at 8:50 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I verified with:
> > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > 2. Making sure the checksum are correct
> > > 3. Browsing release notes, documents, javadocs, protocols.
> > > 4. Verified the tiered storage feature works well.
> > >
> > > +1 (binding).
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > >
> > > On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  wrote:
> > >
> > > > +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven
> > > > artifacts and run my tests. Everything seems to work fine for me.
> > > >
> > > > Thanks
> > > > Jakub
> > > >
> > > > On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the third candidate for the release of Apache Kafka 3.6.0.
> > > > > Some of the major features include:
> > > > >
> > > > > * KIP-405 : Kafka Tiered Storage
> > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > >
> > > > > Release notes for the 3.6.0 release:
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Tuesday, October 3, 12pm PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://reposit

[jira] [Created] (KAFKA-15530) Add missing documentation of metrics introduced as part of KAFKA-15196

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15530:
--

 Summary: Add missing documentation of metrics introduced as part 
of KAFKA-15196
 Key: KAFKA-15530
 URL: https://issues.apache.org/jira/browse/KAFKA-15530
 Project: Kafka
  Issue Type: Task
Reporter: Satish Duggana


This is a followup to the 3.6.0 RC2 verification email 
[thread|https://lists.apache.org/thread/js2nmq3ggn46qg122h4jg5p2fcq5hr2s]. Add 
the missing documentation of a few metrics added as part of the 
[change|https://github.com/apache/kafka/commit/2f71708955b293658cec3b27e9a5588d39c38d7e].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] 3.6.0 RC2

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the third candidate for the release of Apache Kafka 3.6.0.
Some of the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, October 3, 12pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc2

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest
at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
continue running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

We do not have any pending release blockers for now. RC1 release
blocker KAFKA-15498[1] is merged to 3.6.
 I will create RC2 by 29th Sep 12:00 pm PT and start a new RC thread.

1. https://github.com/apache/kafka/pull/14434

Thanks,
Satish.



On Wed, 27 Sept 2023 at 13:36, Divij Vaidya  wrote:
>
> Ismael,
> Thank you for checking.
> Multiple other folks have validated after I left the comment here that
> it doesn't impact log truncation and hence won't lead to data loss. I
> agree that it's not a blocker.
>
> (ref: https://github.com/apache/kafka/pull/14457)
>
> --
> Divij Vaidya
>
> On Wed, Sep 27, 2023 at 8:50 PM Ismael Juma  wrote:
> >
> > Doesn't look like a blocker to me.
> >
> > Ismael
> >
> > On Wed, Sep 27, 2023 at 2:36 AM Divij Vaidya 
> > wrote:
> >
> > > Hey team
> > >
> > > I need help in determining whether
> > > https://github.com/apache/kafka/pull/14457 is a release blocker bug or
> > > not. If someone is familiar with replication protocol (on the log
> > > diverange and reconciliation process), please add your comments on the
> > > PR.
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Wed, Sep 27, 2023 at 10:43 AM Divij Vaidya 
> > > wrote:
> > > >
> > > > A community member reported another bug in TS feature in 3.6 -
> > > > https://issues.apache.org/jira/browse/KAFKA-15511
> > > >
> > > > I don't consider it as a blocker for release because the bug occurs in
> > > > rare situations when the index on disk or in a remote store is
> > > > corrupted and fails a sanity check.
> > > > Sharing it here as an FYI.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > > On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya 
> > > wrote:
> > > > >
> > > > > Found a bug while testing TS feature in 3.6 -
> > > > > https://issues.apache.org/jira/browse/KAFKA-15481
> > > > >
> > > > > I don't consider it as a blocker for release since it's a concurrency
> > > > > bug that should occur rarely for a feature which is early access.
> > > > > Sharing it here as FYI in case someone else thinks differently.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > > On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana <
> > > satish.dugg...@gmail.com> wrote:
> > > > > >
> > > > > > Thanks Divij for raising a PR for doc formatting issue.
> > > > > >
> > > > > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya, 
> > > > > > 
> > > wrote:
> > > > > >
> > > > > > > Hey Satish
> > > > > > >
> > > > > > > I filed a PR to fix the website formatting bug in 3.6
> > > documentation -
> > > > > > > https://github.com/apache/kafka/pull/14419
> > > > > > > Please take a look when you get a chance.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > > On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton
> > > 
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > I think this qualifies as a blocker. This API has been around
> > > for years
> > > > > > > now
> > > > > > > > and, while we don't document it as not exposing duplicates*, it
> > > has come
> > > > > > > > with that implicit contract since its inception. More
> > > importantly, it has
> > > > > > > > also never exposed plugins that cannot be used on the worker.
> > > This change
> > > > > > > > in behavior not only introduces duplicates*, it causes
> > > unreachable
> > > > > > > plugins
> > > > > > > > to be displayed. With this in mind, it seems to qualify pretty
> > > clearly
> > > > > > > as a
> > > > > > > > regression and we should not put out a release that includes it.
> > > > > > > >
> > > > > > > > * - Really, these aren't duplicates; rather, they're multiple
> > > copies of
> >

[jira] [Created] (KAFKA-15503) CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 10.0.16, 11.0.16, 12.0.1

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15503:
--

 Summary: CVE-2023-40167, CVE-2023-36479 - Upgrade jetty to 9.4.52, 
10.0.16, 11.0.16, 12.0.1
 Key: KAFKA-15503
 URL: https://issues.apache.org/jira/browse/KAFKA-15503
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.7.0, 2.6.1, 3.4.1, 3.6.0, 3.5.1
Reporter: Rafael Rios Saavedra
Assignee: Divij Vaidya
 Fix For: 2.8.0, 2.7.1, 2.6.2, 3.0.0


CVE-2023-40167 and CVE-2023-36479 vulnerabilities affects Jetty version 
{*}9.4.51{*}. For more information see 
[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-40167] 
[https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-364749] 

Upgrading to Jetty version *9.4.52, 10.0.16, 11.0.16, 12.0.1* should address 
this issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Luke for reporting the High CVE issue. Ismael also confirmed
that High CVEs are generally considered as release blockers.

Agree with all that we should consider this CVE as a release blocker.
We will include other CVE also with the next RC.

~Satish.


On Mon, 25 Sept 2023 at 03:04, Luke Chen  wrote:
>
> Hi Divij,
>
> About the system tests, it's me helping Satish working on it since our team
> has internal jenkins pipeline for it.
> Here's the result:
> https://drive.google.com/drive/folders/1S2XYd79f6_AeWj9f9qEkliRg7JtL04AC?usp=sharing
>
> I'm mainly focusing on the failed tests.
> For kraft_upgrade_test, I've fixed in this PR
> <https://github.com/apache/kafka/pull/14424>. After the fix, the rerun
> results look good.
> For quota_test, it's a known issue in our environment. After rerun, all
> passed.
>
> Thanks.
> Luke
>
>
>
> On Mon, Sep 25, 2023 at 4:09 PM Divij Vaidya 
> wrote:
>
> > Correction: posted the wrong JIRA in my previous email. Instead of
> > https://issues.apache.org/jira/browse/KAFKA-15001, please consider
> > this https://issues.apache.org/jira/browse/KAFKA-15487
> >
> > --
> > Divij Vaidya
> >
> > On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya 
> > wrote:
> > >
> > > Hi Satish
> > >
> > > 1. I agree with Luke. It's a "high" severity vulnerability and we
> > > should create another RC with the upgraded Snappy version. If we
> > > create another RC, we should also fix a different CVE resported in
> > > https://issues.apache.org/jira/browse/KAFKA-15001
> > >
> > > 2. I was hoping you could post the results of system tests before I
> > > vote on this. I am particularly interested in looking at
> > > producer/consumer performance results since we have quite a few
> > > changes in this release. What is the plan on the system tests?
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Mon, Sep 25, 2023 at 9:10 AM Luke Chen  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > Snappy-java published a new vulnerability
> > > > <
> > https://github.com/xerial/snappy-java/security/advisories/GHSA-55g7-9cwv-5qfv
> > >
> > > > that will cause OOM error in the server.
> > > > Kafka is also impacted by this vulnerability since it's like
> > CVE-2023-34455
> > > > <https://nvd.nist.gov/vuln/detail/CVE-2023-34455>.
> > > > We'd better bump the snappy-java version to bypass this vulnerability.
> > > > PR <https://github.com/apache/kafka/pull/14434> is created to run the
> > CI
> > > > build.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > >
> > > > On Mon, Sep 25, 2023 at 2:38 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks to everyone who voted for this release.
> > > > >
> > > > > We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
> > > > > deadline. Please try RC1 and send your vote to this email thread.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > >
> > > > > On Sun, 24 Sept 2023 at 13:23, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I've done the following:
> > > > > > - Verified signature
> > > > > > - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> > > > > > - Run unit + integration tests
> > > > > > - Ran a shorter Trogdor transactional-produce-bench on a single
> > broker
> > > > > > cluster (KRaft and ZK) to verify transactional workloads worked
> > > > > reasonably
> > > > > >
> > > > > > Minor thing (we can discuss elsewhere and is non-blocking for the
> > > > > release)
> > > > > > but if ZK has been deprecated since 3.5 we should move up the
> > Kraft setup
> > > > > > in the quickstart guide  <http://goog_2103708782>here
> > > > > > <https://kafka.apache.org/quickstart>.
> > > > > >
> > > > > > +1 (binding) from me.
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Sun, Sep 24, 2023 at 7:09 A

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks to everyone who voted for this release.

We have 2 +1 PMC votes and 3 +1 non-binding votes. We are past the
deadline. Please try RC1 and send your vote to this email thread.

Thanks,
Satish.


On Sun, 24 Sept 2023 at 13:23, Justine Olshan
 wrote:
>
> Hi Satish,
>
> I've done the following:
> - Verified signature
> - Built from Java 17/Scala 2.13 and Java 8/Scala 2.11
> - Run unit + integration tests
> - Ran a shorter Trogdor transactional-produce-bench on a single broker
> cluster (KRaft and ZK) to verify transactional workloads worked reasonably
>
> Minor thing (we can discuss elsewhere and is non-blocking for the release)
> but if ZK has been deprecated since 3.5 we should move up the Kraft setup
> in the quickstart guide  <http://goog_2103708782>here
> <https://kafka.apache.org/quickstart>.
>
> +1 (binding) from me.
>
> Justine
>
> On Sun, Sep 24, 2023 at 7:09 AM Federico Valeri 
> wrote:
>
> > Hi Satish, I did the following to verify the release:
> >
> > - Verified signature and checksum
> > - Built from source with Java 17 and Scala 2.13
> > - Ran all unit and integration tests
> > - Spot checked release notes and documentation
> > - Ran a custom client using staging artifacts on a 3-nodes cluster
> > - Tested tiered storage with one of the available RSM implementations
> >
> > +1 (non binding)
> >
> > Thanks
> > Fede
> >
> >
> > On Sun, Sep 24, 2023 at 8:49 AM Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I verified with:
> > > 1. Ran quick start in KRaft for scala 2.12 artifact
> > > 2. Making sure the checksum are correct
> > > 3. Browsing release notes, documents, javadocs, protocols.
> > >
> > > I filed KAFKA-15491 <https://issues.apache.org/jira/browse/KAFKA-15491
> > >for
> > > log output improvement while testing stream application.
> > > It won't be blocker in v3.6.0.
> > >
> > > For KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489>, I'm
> > > fine if we decide to fix it in v3.6.1/v3.7.0.
> > >
> > > +1 (binding) from me.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:
> > >
> > > > Given that this is not a regression and there have been no reports for
> > over
> > > > a year, I think it's ok for this to land in 3.6.1.
> > > >
> > > > Ismael
> > > >
> > > > On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Luke for reporting KRaft issue[1].
> > > > >
> > > > > I am not sure whether it is a release blocker for 3.6.0. Need input
> > > > > from other KRaft experts also to finalize the decision. Even if we
> > > > > adopt a fix, do not we need to bake it for some time before it is
> > > > > pushed to production to avoid any regressions as this change is in
> > the
> > > > > critical paths?
> > > > >
> > > > > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > I found the current KRaft implementation will have "split brain"
> > issue
> > > > > when
> > > > > > network partition happens, which will cause inconsistent metadata
> > > > > returned
> > > > > > from the controller.
> > > > > > Filed KAFKA-15489 <
> > https://issues.apache.org/jira/browse/KAFKA-15489>
> > > > > for
> > > > > > this issue, and PR <https://github.com/apache/kafka/pull/14428> is
> > > > ready
> > > > > > for review.
> > > > > >
> > > > > > Even though this is not a regression issue (this has already
> > existed
> > > > > since
> > > > > > the 1st release of KRaft feature), I think this is an important
> > issue
> > > > > since
> > > > > > KRaft is announced production ready.
> > > > > > Not sure what other people's thoughts are.
> > > > > >
> > > > > >

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[Satish Duggana]

Congratulations Justine!!

On Sat, 23 Sept 2023 at 15:46, Bill Bejeck  wrote:
>
> Congrats Justine!
>
> -Bill
>
> On Sat, Sep 23, 2023 at 6:23 PM Greg Harris 
> wrote:
>
> > Congratulations Justine!
> >
> > On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
> >  wrote:
> > >
> > > Congrats Justin !
> > >
> > > On Sat 23 Sep 2023 at 14:44, Randall Hauch  wrote:
> > >
> > > > Congratulations, Justine!
> > > >
> > > > On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Congrats Justine!
> > > > >
> > > > > On Sat, Sep 23, 2023, 13:28 Divij Vaidya 
> > > > wrote:
> > > > >
> > > > > > Congratulations Justine!
> > > > > >
> > > > > > On Sat 23. Sep 2023 at 07:06, Chris Egerton <
> > fearthecel...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats Justine!
> > > > > > > On Fri, Sep 22, 2023, 20:47 Guozhang Wang <
> > > > guozhang.wang...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations!
> > > > > > > >
> > > > > > > > On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <
> > > > > > tzuli...@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Congratulations Justine!
> > > > > > > > >
> > > > > > > > > On Fri, Sep 22, 2023, 19:25 Philip Nee 
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats Justine!
> > > > > > > > > >
> > > > > > > > > > On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <
> > show...@gmail.com>
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi, Everyone,
> > > > > > > > > > >
> > > > > > > > > > > Justine Olshan has been a Kafka committer since Dec.
> > 2022.
> > > > She
> > > > > > has
> > > > > > > > been
> > > > > > > > > > > very active and instrumental to the community since
> > becoming
> > > > a
> > > > > > > > committer.
> > > > > > > > > > > It's my pleasure to announce that Justine is now a
> > member of
> > > > > > Kafka
> > > > > > > > PMC.
> > > > > > > > > > >
> > > > > > > > > > > Congratulations Justine!
> > > > > > > > > > >
> > > > > > > > > > > Luke
> > > > > > > > > > > on behalf of Apache Kafka PMC
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Luke for reporting KRaft issue[1].

I am not sure whether it is a release blocker for 3.6.0. Need input
from other KRaft experts also to finalize the decision. Even if we
adopt a fix, do not we need to bake it for some time before it is
pushed to production to avoid any regressions as this change is in the
critical paths?

1. https://issues.apache.org/jira/browse/KAFKA-15489

Thanks,
Satish.

On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
>
> Hi Satish,
>
> I found the current KRaft implementation will have "split brain" issue when
> network partition happens, which will cause inconsistent metadata returned
> from the controller.
> Filed KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489> for
> this issue, and PR <https://github.com/apache/kafka/pull/14428> is ready
> for review.
>
> Even though this is not a regression issue (this has already existed since
> the 1st release of KRaft feature), I think this is an important issue since
> KRaft is announced production ready.
> Not sure what other people's thoughts are.
>
> Thank you.
> Luke
>
> On Thu, Sep 21, 2023 at 6:33 PM Josep Prat 
> wrote:
>
> > Hi Satish,
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Verified Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> >
> >
> > I +1 this release (non-binding)
> >
> > Thanks for your efforts!
> >
> > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana 
> > wrote:
> >
> > > Thanks Greg for verifying the release including the earlier
> > > blocker(KAFKA-15473) verification.
> > >
> > > ~Satish.
> > >
> > > On Wed, 20 Sept 2023 at 22:30, Greg Harris  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I verified the functionality of KIP-898 and the recent fix for
> > > > KAFKA-15473 with the following steps:
> > > >
> > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > publicly available plugins installed
> > > > 2. I captured the output of /connector-plugins
> > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > 4. I captured the output of /connector-plugins with various settings
> > > > of plugin.discovery
> > > > 5. I ran the migration script to add manifests to my plugins
> > > > 6. I captured the output of /connector-plugins with various settings
> > > > of plugin.discovery
> > > > 7. I downgraded the worker to 3.5.1
> > > > 8. I diffed the output of /connector-plugins across the different
> > > > cases and observed the expected changes.
> > > > a. When plugins are migrated for 3.6.0, all modes produce identical
> > > > results.
> > > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > > service_load is missing plugins
> > > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > > constructors are hidden, AK plugins now have versions, multi-interface
> > > > plugins now show each interface type, and plugins using AppInfoParser
> > > > change versions.
> > > > d. The startup logs now include descriptive errors for invalid
> > > > plugins that otherwise would have been thrown at runtime
> > > > d. The fix for KAFKA-15473 prevents duplicates
> > > > e. The output for 3.5.1 after downgrading is identical to before.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks Satish for running the release!
> > > >
> > > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> > wrote:
> > > > >
> > > > > Hey Satish
> > > > >
> > > > > My comments about documentation misses from RC0 vote thread [1] are
> > > > > still not addressed (such as missing metric documentation, formatting
> > > > > problems etc). Could you please mention why we shouldn't consider
> > them
> > > > > as blockers to make RC1 as the final release?
> > > > >
> > > > > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > > > >
> > > > > On Wed, Sep 20, 2023 at 4:53 PM Satish Dugga

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Divij for raising a PR for doc formatting issue.

On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya,  wrote:

> Hey Satish
>
> I filed a PR to fix the website formatting bug in 3.6 documentation -
> https://github.com/apache/kafka/pull/14419
> Please take a look when you get a chance.
>
> --
> Divij Vaidya
>
> On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton 
> wrote:
> >
> > Hi Satish,
> >
> > I think this qualifies as a blocker. This API has been around for years
> now
> > and, while we don't document it as not exposing duplicates*, it has come
> > with that implicit contract since its inception. More importantly, it has
> > also never exposed plugins that cannot be used on the worker. This change
> > in behavior not only introduces duplicates*, it causes unreachable
> plugins
> > to be displayed. With this in mind, it seems to qualify pretty clearly
> as a
> > regression and we should not put out a release that includes it.
> >
> > * - Really, these aren't duplicates; rather, they're multiple copies of
> the
> > same plugin that come from different locations on the worker
> >
> > Best,
> >
> > Chris
> >
> > On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana  >
> > wrote:
> >
> > > Hi Greg,
> > > Is this API documented that it does not return duplicate entries?
> > >
> > > Can we also get an opinion from PMC/Committers who have KafkaConnect
> > > expertise on whether this issue is a release blocker?
> > >
> > > If we agree that it is not a release blocker then we can have a
> > > release note clarifying this behaviour and add a reference to the JIRA
> > > that follows up on the possible solutions.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > > On Tue, 19 Sept 2023 at 03:29, Greg Harris
> 
> > > wrote:
> > > >
> > > > Hey Satish,
> > > >
> > > > After investigating further, I believe that this is a regression, but
> > > > mostly a cosmetic one.
> > > > I don't think there is significant risk of breaking clients with this
> > > > change, but it would be confusing for users, so I'd still like to get
> > > > the fix into the next RC.
> > > > I've opened a PR here: https://github.com/apache/kafka/pull/14398
> and
> > > > I'll work to get it merged promptly.
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris 
> > > wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > While validating 3.6.0-rc0, I noticed this regression as compared
> to
> > > > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> > > > >
> > > > > Impact: The `connector-plugins` endpoint lists duplicates which may
> > > > > cause confusion for users, or poor behavior in clients.
> > > > > Using the other REST API endpoints appears unaffected.
> > > > > I'll open a PR for this later today.
> > > > >
> > > > > Thanks,
> > > > > Greg
> > > > >
> > > > > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> > > > >  wrote:
> > > > > >
> > > > > > Thanks Justine for the update. I saw in the morning that these
> > > changes
> > > > > > are pushed to trunk and 3.6.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > > We were able to merge
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > > > > and pick to 3.6.
> > > > > > >
> > > > > > > Hopefully nothing more from me on this release.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Luke for the update.
> > > > > > > >
> > > > > > > > ~Satish.
> > > > > > > &g

Re: [ANNOUNCE] New committer: Lucas Brutschy

[Satish Duggana]

Congratulations Lucas!

On Thu, 21 Sept 2023 at 22:58, Viktor Somogyi-Vass
 wrote:
>
> Congrats Lucas!
>
> On Thu, Sep 21, 2023 at 7:12 PM Alexander Sorokoumov
>  wrote:
>
> > Congratulations, Lucas!
> >
> > On Thu, Sep 21, 2023 at 10:09 AM Walker Carlson
> >  wrote:
> >
> > > Congrats Lucas!
> > >
> > > On Thu, Sep 21, 2023 at 11:42 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Congrats Lucas!
> > > >
> > > > On Thu, Sep 21, 2023, 22:05 Boudjelda Mohamed Said 
> > > > wrote:
> > > >
> > > > > Congratulations, Lucas!!
> > > > >
> > > > > On Thu 21 Sep 2023 at 18:34, Lianet M.  wrote:
> > > > >
> > > > > > Congratulations Lucas!
> > > > > >
> > > > > > On Thu, Sept 21, 2023, 11:45 a.m. Bruno Cadonna <
> > cado...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > committer
> > > > > > > Lucas Brutschy.
> > > > > > >
> > > > > > > Lucas' major contributions are around Kafka Streams.
> > > > > > >
> > > > > > > Lucas' significantly contributed to the state updater
> > > > > > > (https://issues.apache.org/jira/browse/KAFKA-10199) and he
> > drives
> > > > the
> > > > > > > implementation of the new threading model for Kafka Streams
> > > > > > > (https://issues.apache.org/jira/browse/KAFKA-15326).
> > > > > > >
> > > > > > > Lucas' contributions to KIP discussions and PR reviews are very
> > > > > > thoughtful.
> > > > > > >
> > > > > > > Congratulations, Lucas!
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Bruno (on behalf of the Apache Kafka PMC)
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [ANNOUNCE] New committer: Yash Mayya

[Satish Duggana]

Congratulations Yash!!

On Thu, 21 Sept 2023 at 22:57, Viktor Somogyi-Vass
 wrote:
>
> Congrats Yash!
>
> On Thu, Sep 21, 2023 at 7:04 PM Josep Prat 
> wrote:
>
> > Congrats Yash!
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Thu, Sep 21, 2023, 18:55 Raymond Ng  wrote:
> >
> > > Congrats Yash! Well-deserved!
> > >
> > > /Ray
> > >
> > > On Thu, Sep 21, 2023 at 9:40 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > Congratulations Yash!
> > > >
> > > > On Thu, Sep 21, 2023, 22:03 Bill Bejeck  wrote:
> > > >
> > > > > Congrats Yash!
> > > > >
> > > > > On Thu, Sep 21, 2023 at 12:26 PM Divij Vaidya <
> > divijvaidy...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Congratulations Yash!
> > > > > >
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 21, 2023 at 6:18 PM Sagar 
> > > > wrote:
> > > > > > >
> > > > > > > Congrats Yash !
> > > > > > > On Thu, 21 Sep 2023 at 9:38 PM, Ashwin
> > >  > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Awesome ! Congratulations Yash !!
> > > > > > > >
> > > > > > > > On Thu, Sep 21, 2023 at 9:25 PM Edoardo Comar <
> > > > edoardli...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Yash
> > > > > > > > >
> > > > > > > > > On Thu, 21 Sept 2023 at 16:28, Bruno Cadonna <
> > > cado...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > > > committer
> > > > > > > > > > Yash Mayya.
> > > > > > > > > >
> > > > > > > > > > Yash's major contributions are around Connect.
> > > > > > > > > >
> > > > > > > > > > Yash authored the following KIPs:
> > > > > > > > > >
> > > > > > > > > > KIP-793: Allow sink connectors to be used with
> > topic-mutating
> > > > > SMTs
> > > > > > > > > > KIP-882: Kafka Connect REST API configuration validation
> > > > timeout
> > > > > > > > > > improvements
> > > > > > > > > > KIP-970: Deprecate and remove Connect's redundant task
> > > > > > configurations
> > > > > > > > > > endpoint
> > > > > > > > > > KIP-980: Allow creating connectors in a stopped state
> > > > > > > > > >
> > > > > > > > > > Overall, Yash is known for insightful and friendly input to
> > > > > > discussions
> > > > > > > > > > and his high quality contributions.
> > > > > > > > > >
> > > > > > > > > > Congratulations, Yash!
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Bruno (on behalf of the Apache Kafka PMC)
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Thanks Greg for verifying the release including the earlier
blocker(KAFKA-15473) verification.

~Satish.

On Wed, 20 Sept 2023 at 22:30, Greg Harris 
wrote:

> Hi all,
>
> I verified the functionality of KIP-898 and the recent fix for
> KAFKA-15473 with the following steps:
>
> 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> publicly available plugins installed
> 2. I captured the output of /connector-plugins
> 3. I upgraded the worker to 3.6.0-rc1
> 4. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 5. I ran the migration script to add manifests to my plugins
> 6. I captured the output of /connector-plugins with various settings
> of plugin.discovery
> 7. I downgraded the worker to 3.5.1
> 8. I diffed the output of /connector-plugins across the different
> cases and observed the expected changes.
> a. When plugins are migrated for 3.6.0, all modes produce identical
> results.
> b. When plugins are not migrated for 3.6.0, only_scan and
> hybrid_warn produce identical results, hybrid_fail crashes, and
> service_load is missing plugins
> c. When upgrading from 3.5.1 I see that plugins with invalid
> constructors are hidden, AK plugins now have versions, multi-interface
> plugins now show each interface type, and plugins using AppInfoParser
> change versions.
> d. The startup logs now include descriptive errors for invalid
> plugins that otherwise would have been thrown at runtime
> d. The fix for KAFKA-15473 prevents duplicates
> e. The output for 3.5.1 after downgrading is identical to before.
>
> +1 (non-binding)
>
> Thanks Satish for running the release!
>
> On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya  wrote:
> >
> > Hey Satish
> >
> > My comments about documentation misses from RC0 vote thread [1] are
> > still not addressed (such as missing metric documentation, formatting
> > problems etc). Could you please mention why we shouldn't consider them
> > as blockers to make RC1 as the final release?
> >
> > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> >
> > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for the release of Apache Kafka 3.6.0.
> Some of the major features include:
> > >
> > > * KIP-405 : Kafka Tiered Storage
> > > * KIP-868 : KRaft Metadata Transactions
> > > * KIP-875: First-class offsets support in Kafka Connect
> > > * KIP-898: Modernize Connect plugin discovery
> > > * KIP-938: Add more metrics for measuring KRaft performance
> > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > * KIP-917: Additional custom metadata for remote log segment
> > >
> > > Release notes for the 3.6.0 release:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Saturday, September 23, 8am PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/36/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/36/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.6 branch:
> > > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > > System tests:
> > > We will send an update once we have the results.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Satish Duggana]

Hi Divij,
Thanks for calling out documentation issues. Some of them are being
addressed in the existing documentation PRs. We will keep addressing the
remaining documentation issues by making respective changes in kafka-site
repo for 3.6.0 release.

I think documentation is important but I do not consider documentation
issues as release blockers. I want others to review and test the release
candidate sooner to finalize the RC for 3.6.0 release.

~Satish.

On Wed, 20 Sept 2023 at 21:04, Divij Vaidya  wrote:

> Hey Satish
>
> My comments about documentation misses from RC0 vote thread [1] are
> still not addressed (such as missing metric documentation, formatting
> problems etc). Could you please mention why we shouldn't consider them
> as blockers to make RC1 as the final release?
>
> [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
>
> On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana 
> wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for the release of Apache Kafka 3.6.0. Some
> of the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Saturday, September 23, 8am PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest
> at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "kafka-clients" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to kafka-clients+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAM-aUZ%3DuJ-SKeVFtBZwBjhLHKw4CbxF_ws%2BvQqaymGHFsC%2Bmdg%40mail.gmail.com
> .
>

[jira] [Created] (KAFKA-15483) Update metrics documentation for the new metrics implemented as part of KIP-938

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15483:
--

 Summary: Update metrics documentation for the new metrics 
implemented as part of KIP-938
 Key: KAFKA-15483
 URL: https://issues.apache.org/jira/browse/KAFKA-15483
 Project: Kafka
  Issue Type: Task
  Components: docs, documentation
Reporter: Satish Duggana






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15092) KafkaClusterTestKit in test jar depends on MockFaultHandler

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15092.

Resolution: Invalid

> KafkaClusterTestKit in test jar depends on MockFaultHandler
> ---
>
> Key: KAFKA-15092
> URL: https://issues.apache.org/jira/browse/KAFKA-15092
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.5.0
>Reporter: Gary Russell
>Priority: Major
>
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/kafka/server/fault/MockFaultHandler
>   at 
> kafka.testkit.KafkaClusterTestKit$SimpleFaultHandlerFactory.(KafkaClusterTestKit.java:119)
>   at 
> kafka.testkit.KafkaClusterTestKit$Builder.(KafkaClusterTestKit.java:143)
> {noformat}
> MockFaultHandler is missing from the test jar.
> This PR https://github.com/apache/kafka/pull/13375/files seems to work around 
> it by adding the {code}server-common sourcesets.test.output{code} to the 
> class path.
> The class needs to be available for third parties to create an embedded KRaft 
> broker.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15482) kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15482.

Resolution: Invalid

> kafka.utils.TestUtils Depends on MockTime Which is Not in Any Jar
> -
>
> Key: KAFKA-15482
> URL: https://issues.apache.org/jira/browse/KAFKA-15482
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.6.0
>Reporter: Gary Russell
>Priority: Major
>
> Commit 
> 7eea2a3908fdcee1627c18827e6dcb5ed0089fdd 
> Moved it to server-commons, but it is not included in the jar.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] 3.6.0 RC1

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the second candidate for the release of Apache Kafka 3.6.0. Some of
the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Saturday, September 23, 8am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc1

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest at
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue
running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Thanks Luke for reviewing and running the changes
https://github.com/apache/kafka/pull/14409 for merging into 3.6.

On Wed, 20 Sept 2023 at 06:09, Satish Duggana  wrote:
>
> Hi David,
> Thanks for the update. Please get it reviewed sooner. I plan to create
> RC1 in the next 5 hrs. As it is not a release blocker, I will merge it
> if this PR is approved and jenkins job looks good within that time.
>
> Thanks,
> Satish.
>
> On Wed, 20 Sept 2023 at 01:42, David Arthur
>  wrote:
> >
> > As part of validating this RC, I found that most of the ZK migration system
> > tests were failing. I investigated this today and found that it was just
> > problems in the test code. This is not a blocker, but since we're doing
> > RC1, I'd like to include it https://github.com/apache/kafka/pull/14409
> >
> > On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
> > wrote:
> >
> > > Hi All,
> > > I will start creating a new RC with the fix and send it for a vote by
> > > tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> > > determined to be a release blocker.
> > >
> > > Thanks.
> > > Satish.
> > >
> > > On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> > > wrote:
> > > >
> > > > Thanks Chris for your comment on the reported issue.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Given the regression raised by Greg Harris (
> > > > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> > > thread, I
> > > > > believe we should generate a new candidate with a fix.
> > > > >
> > > > > I'm -1 (binding) on this RC.
> > > > >
> > > > > Best,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  > > >
> > > > > wrote:
> > > > >
> > > > > > Heya,
> > > > > >
> > > > > > I have compiled and ran the test target successfully for the
> > > 3.6.0-rc0
> > > > > > branch on:
> > > > > >
> > > > > > Java 11, Scala 2.13 - ARM
> > > > > > Java 17, Scala 2.13 - ARM
> > > > > > Java 20, Scala 2.18 - ARM
> > > > > > Java 11, Scala 2.13 - Intel x86
> > > > > > Java 17, Scala 2.13 - Intel x86
> > > > > > Java 20, Scala 2.13 - Intel x86
> > > > > >
> > > > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> > > miss
> > > > > >
> > > > > > Best,
> > > > > > Christo
> > > > > >
> > > > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hey Satish
> > > > > > >
> > > > > > > Thank you for managing this release. I have a few comments:
> > > > > > >
> > > > > > > Documentation
> > > > > > >
> > > > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> > > "The
> > > > > > > current stable branch is 3.5. Kafka is regularly updated to 
> > > > > > > include
> > > > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> > > That
> > > > > > > needs an update since we are running Zk 3.8 now.
> > > > > > >
> > > > > > > 2. Section: Zookeeper/Migration - The documentation states
> > > "Migration
> > > > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > > > Preview and we expect it to be ready for production usage in
> > > version
> > > > > > > 3.6.". This probably needs an update on whether it is production
> > > ready
> > > > > > > or not in 3.6
> > > > > > >
> > > > > > > 3. Section: Kraft/missing features
> > > > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > > > believe that delegation token is now part of 3.6? I think this
> > >

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi David,
Thanks for the update. Please get it reviewed sooner. I plan to create
RC1 in the next 5 hrs. As it is not a release blocker, I will merge it
if this PR is approved and jenkins job looks good within that time.

Thanks,
Satish.

On Wed, 20 Sept 2023 at 01:42, David Arthur
 wrote:
>
> As part of validating this RC, I found that most of the ZK migration system
> tests were failing. I investigated this today and found that it was just
> problems in the test code. This is not a blocker, but since we're doing
> RC1, I'd like to include it https://github.com/apache/kafka/pull/14409
>
> On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
> wrote:
>
> > Hi All,
> > I will start creating a new RC with the fix and send it for a vote by
> > tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> > determined to be a release blocker.
> >
> > Thanks.
> > Satish.
> >
> > On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> > wrote:
> > >
> > > Thanks Chris for your comment on the reported issue.
> > >
> > > ~Satish.
> > >
> > > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Given the regression raised by Greg Harris (
> > > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> > thread, I
> > > > believe we should generate a new candidate with a fix.
> > > >
> > > > I'm -1 (binding) on this RC.
> > > >
> > > > Best,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  > >
> > > > wrote:
> > > >
> > > > > Heya,
> > > > >
> > > > > I have compiled and ran the test target successfully for the
> > 3.6.0-rc0
> > > > > branch on:
> > > > >
> > > > > Java 11, Scala 2.13 - ARM
> > > > > Java 17, Scala 2.13 - ARM
> > > > > Java 20, Scala 2.18 - ARM
> > > > > Java 11, Scala 2.13 - Intel x86
> > > > > Java 17, Scala 2.13 - Intel x86
> > > > > Java 20, Scala 2.13 - Intel x86
> > > > >
> > > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> > miss
> > > > >
> > > > > Best,
> > > > > Christo
> > > > >
> > > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  > >
> > > > > wrote:
> > > > >
> > > > > > Hey Satish
> > > > > >
> > > > > > Thank you for managing this release. I have a few comments:
> > > > > >
> > > > > > Documentation
> > > > > >
> > > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> > "The
> > > > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> > That
> > > > > > needs an update since we are running Zk 3.8 now.
> > > > > >
> > > > > > 2. Section: Zookeeper/Migration - The documentation states
> > "Migration
> > > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > > Preview and we expect it to be ready for production usage in
> > version
> > > > > > 3.6.". This probably needs an update on whether it is production
> > ready
> > > > > > or not in 3.6
> > > > > >
> > > > > > 3. Section: Kraft/missing features
> > > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > > believe that delegation token is now part of 3.6? I think this
> > > > > > probably needs an update.
> > > > > >
> > > > > > 4. Section: Configuration/rack.aware.assignment.strategy - there
> > seems
> > > > > > to be a formatting problem starting from here
> > > > > > (
> > > > > >
> > > > >
> > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > > > )
> > > > > >
> > > > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing
> > from the
> >

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi All,
I will start creating a new RC with the fix and send it for a vote by
tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
determined to be a release blocker.

Thanks.
Satish.

On Tue, 19 Sept 2023 at 22:26, Satish Duggana  wrote:
>
> Thanks Chris for your comment on the reported issue.
>
> ~Satish.
>
> On Tue, 19 Sept 2023 at 22:16, Chris Egerton  wrote:
> >
> > Hi all,
> >
> > Given the regression raised by Greg Harris (
> > https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
> > believe we should generate a new candidate with a fix.
> >
> > I'm -1 (binding) on this RC.
> >
> > Best,
> >
> > Chris
> >
> > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
> > wrote:
> >
> > > Heya,
> > >
> > > I have compiled and ran the test target successfully for the 3.6.0-rc0
> > > branch on:
> > >
> > > Java 11, Scala 2.13 - ARM
> > > Java 17, Scala 2.13 - ARM
> > > Java 20, Scala 2.18 - ARM
> > > Java 11, Scala 2.13 - Intel x86
> > > Java 17, Scala 2.13 - Intel x86
> > > Java 20, Scala 2.13 - Intel x86
> > >
> > > I will update the Zookeeper KIP title in the KIPs page, that's my miss
> > >
> > > Best,
> > > Christo
> > >
> > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> > > wrote:
> > >
> > > > Hey Satish
> > > >
> > > > Thank you for managing this release. I have a few comments:
> > > >
> > > > Documentation
> > > >
> > > > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > the latest release in the 3.5 series." in the ZooKeeper section. That
> > > > needs an update since we are running Zk 3.8 now.
> > > >
> > > > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > Preview and we expect it to be ready for production usage in version
> > > > 3.6.". This probably needs an update on whether it is production ready
> > > > or not in 3.6
> > > >
> > > > 3. Section: Kraft/missing features
> > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > believe that delegation token is now part of 3.6? I think this
> > > > probably needs an update.
> > > >
> > > > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > > > to be a formatting problem starting from here
> > > > (
> > > >
> > > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > )
> > > >
> > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > > > documentation here.
> > > >
> > > > Release notes
> > > >
> > > > 1. I found a bunch of tickets which have not been marked with a
> > > > release version but have been resolved in last 6 months using the
> > > > query
> > > >
> > > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > . Are some of them targeted for 3.6 release?
> > > >
> > > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> > > >
> > > >
> > > > Additionally, I have verified the following:
> > > > 1. release tag is correctly made after the latest commit on the 3.6
> > > > branch at
> > > >
> > > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > > >
> > > > 2. protocol documentation contains the newly introduced error code as
> > > > part of tiered storage
> > > >
> > > > 3. verified that public keys for RM are available at
> > > > https://keys.openpgp.org/
> > > >
> > > > 4. verified that public keys for RM are available at
> > > > https://people.apache.org/keys/com

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Thanks Chris for your comment on the reported issue.

~Satish.

On Tue, 19 Sept 2023 at 22:16, Chris Egerton  wrote:
>
> Hi all,
>
> Given the regression raised by Greg Harris (
> https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
> believe we should generate a new candidate with a fix.
>
> I'm -1 (binding) on this RC.
>
> Best,
>
> Chris
>
> On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
> wrote:
>
> > Heya,
> >
> > I have compiled and ran the test target successfully for the 3.6.0-rc0
> > branch on:
> >
> > Java 11, Scala 2.13 - ARM
> > Java 17, Scala 2.13 - ARM
> > Java 20, Scala 2.18 - ARM
> > Java 11, Scala 2.13 - Intel x86
> > Java 17, Scala 2.13 - Intel x86
> > Java 20, Scala 2.13 - Intel x86
> >
> > I will update the Zookeeper KIP title in the KIPs page, that's my miss
> >
> > Best,
> > Christo
> >
> > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> > wrote:
> >
> > > Hey Satish
> > >
> > > Thank you for managing this release. I have a few comments:
> > >
> > > Documentation
> > >
> > > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > > current stable branch is 3.5. Kafka is regularly updated to include
> > > the latest release in the 3.5 series." in the ZooKeeper section. That
> > > needs an update since we are running Zk 3.8 now.
> > >
> > > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > Preview and we expect it to be ready for production usage in version
> > > 3.6.". This probably needs an update on whether it is production ready
> > > or not in 3.6
> > >
> > > 3. Section: Kraft/missing features
> > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > believe that delegation token is now part of 3.6? I think this
> > > probably needs an update.
> > >
> > > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > > to be a formatting problem starting from here
> > > (
> > >
> > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > )
> > >
> > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > > documentation here.
> > >
> > > Release notes
> > >
> > > 1. I found a bunch of tickets which have not been marked with a
> > > release version but have been resolved in last 6 months using the
> > > query
> > >
> > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > . Are some of them targeted for 3.6 release?
> > >
> > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> > >
> > >
> > > Additionally, I have verified the following:
> > > 1. release tag is correctly made after the latest commit on the 3.6
> > > branch at
> > >
> > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > >
> > > 2. protocol documentation contains the newly introduced error code as
> > > part of tiered storage
> > >
> > > 3. verified that public keys for RM are available at
> > > https://keys.openpgp.org/
> > >
> > > 4. verified that public keys for RM are available at
> > > https://people.apache.org/keys/committer/
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Tue, Sep 19, 2023 at 12:41 PM Sagar 
> > wrote:
> > > >
> > > > Hey Satish,
> > > >
> > > > I have commented on KAFKA-15473. I think the changes in the PR look
> > > fine. I
> > > > also feel this need not be a release blocker given there are other
> > > > possibilities in which duplicates can manifest on the response of the
> > end
> > > > point in question (albeit we can potentially see more in number due to
> > > > this).
> > > >
> > > > Would like to hear others' thoughts as well.
> > 

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi Greg,
Thanks for reporting the KafkaConnect issue. I replied to this issue
on "Apache Kafka 3.6.0 release" email thread and on
https://issues.apache.org/jira/browse/KAFKA-15473.

I would like to hear other KafkaConnect experts' opinions on whether
this issue is really a release blocker.

Thanks,
Satish.




On Tue, 19 Sept 2023 at 00:27, Greg Harris  wrote:
>
> Hey all,
>
> I noticed this regression in RC0:
> https://issues.apache.org/jira/browse/KAFKA-15473
> I've mentioned it in the release thread, and I'm working on a fix.
>
> I'm -1 (non-binding) until we determine if this regression is a blocker.
>
> Thanks!
>
> On Mon, Sep 18, 2023 at 10:56 AM Josep Prat  
> wrote:
> >
> > Hi Satish,
> > Thanks for running the release.
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Verified Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> > - Checked License-binary against libs and matched them
> >
> > I +1 this release (non-binding)
> >
> > Best,
> >
> > On Mon, Sep 18, 2023 at 6:02 PM David Arthur  wrote:
> >
> > > Hey Satish, thanks for getting the RC underway!
> > >
> > > I noticed that the PR for the 3.6 blog post is merged. This makes the blog
> > > post live on the Kafka website https://kafka.apache.org/blog.html. The
> > > blog
> > > post (along with other public announcements) is usually the last thing we
> > > do as part of the release. I think we should probably take this down until
> > > we're done with the release, otherwise users stumbling on this post could
> > > get confused. It also contains some broken links.
> > >
> > > Thanks!
> > > David
> > >
> > > On Sun, Sep 17, 2023 at 1:31 PM Satish Duggana 
> > > wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for the release of Apache Kafka 3.6.0. Some
> > > of
> > > > the major features include:
> > > >
> > > > * KIP-405 : Kafka Tiered Storage
> > > > * KIP-868 : KRaft Metadata Transactions
> > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > * KIP-898: Modernize Connect plugin discovery
> > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > * KIP-917: Additional custom metadata for remote log segment
> > > >
> > > > Release notes for the 3.6.0 release:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Wednesday, September 21, 12pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc0
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/36/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/36/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.6 branch:
> > > > There are a few runs of unit/integration tests. You can see the latest 
> > > > at
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > > > continue
> > > > running a few more iterations.
> > > > System tests:
> > > > We will send an update once we have the results.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> >
> > --
> > [image: Aiven] <https://www.aiven.io>
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud>
> >   <https://www.linkedin.com/company/aiven/>   <https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Greg,
Is this API documented that it does not return duplicate entries?

Can we also get an opinion from PMC/Committers who have KafkaConnect
expertise on whether this issue is a release blocker?

If we agree that it is not a release blocker then we can have a
release note clarifying this behaviour and add a reference to the JIRA
that follows up on the possible solutions.

Thanks,
Satish.


On Tue, 19 Sept 2023 at 03:29, Greg Harris  wrote:
>
> Hey Satish,
>
> After investigating further, I believe that this is a regression, but
> mostly a cosmetic one.
> I don't think there is significant risk of breaking clients with this
> change, but it would be confusing for users, so I'd still like to get
> the fix into the next RC.
> I've opened a PR here: https://github.com/apache/kafka/pull/14398 and
> I'll work to get it merged promptly.
>
> Thanks!
>
> On Mon, Sep 18, 2023 at 11:54 AM Greg Harris  wrote:
> >
> > Hi Satish,
> >
> > While validating 3.6.0-rc0, I noticed this regression as compared to
> > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> >
> > Impact: The `connector-plugins` endpoint lists duplicates which may
> > cause confusion for users, or poor behavior in clients.
> > Using the other REST API endpoints appears unaffected.
> > I'll open a PR for this later today.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> >  wrote:
> > >
> > > Thanks Justine for the update. I saw in the morning that these changes
> > > are pushed to trunk and 3.6.
> > >
> > > ~Satish.
> > >
> > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > >  wrote:
> > > >
> > > > Hi Satish,
> > > > We were able to merge
> > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > and pick to 3.6.
> > > >
> > > > Hopefully nothing more from me on this release.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana 
> > > > 
> > > > wrote:
> > > >
> > > > > Thanks Luke for the update.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > Since this PR:
> > > > > > https://github.com/apache/kafka/pull/14366 only changes the doc, 
> > > > > > I've
> > > > > > backported to 3.6 branch. FYI.
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
> > > > > >  wrote:
> > > > > >
> > > > > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > > > > > > PR should be finalized soon.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > > > > > > migration, that we would like to backport to 3.5 and 3.6 
> > > > > > > > branches.
> > > > > Are
> > > > > > > > you ok with that?
> > > > > > > >
> > > > > > > > https://github.com/apache/kafka/pull/14366
> > > > > > > >
> > > > > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks David for the quick resolution.
> > > > > > > > >
> > > > > > > > > ~Satish.
> > > > > > > > >
> > > > > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > Satish,
> > > > > > > > >

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

On Sun, 17 Sept 2023 at 23:00, Satish Duggana 
wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the first candidate for the release of Apache Kafka 3.6.0. Some of
> the major features include:
>
> * KIP-405 : Kafka Tiered Storage
> * KIP-868 : KRaft Metadata Transactions
> * KIP-875: First-class offsets support in Kafka Connect
> * KIP-898: Modernize Connect plugin discovery
> * KIP-938: Add more metrics for measuring KRaft performance
> * KIP-902: Upgrade Zookeeper to 3.8.1
> * KIP-917: Additional custom metadata for remote log segment
>
> Release notes for the 3.6.0 release:
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html
>
> *** Please download, test and vote by Wednesday, September 21, 12pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>
> * Javadoc:
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/
>
> * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> https://github.com/apache/kafka/releases/tag/3.6.0-rc0
>
> * Documentation:
> https://kafka.apache.org/36/documentation.html
>
> * Protocol:
> https://kafka.apache.org/36/protocol.html
>
> * Successful Jenkins builds for the 3.6 branch:
> There are a few runs of unit/integration tests. You can see the latest at
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> continue running a few more iterations.
> System tests:
> We will send an update once we have the results.
>
> Thanks,
> Satish.
>

[VOTE] 3.6.0 RC0

[Satish Duggana]

Hello Kafka users, developers and client-developers,

This is the first candidate for the release of Apache Kafka 3.6.0. Some of
the major features include:

* KIP-405 : Kafka Tiered Storage
* KIP-868 : KRaft Metadata Transactions
* KIP-875: First-class offsets support in Kafka Connect
* KIP-898: Modernize Connect plugin discovery
* KIP-938: Add more metrics for measuring KRaft performance
* KIP-902: Upgrade Zookeeper to 3.8.1
* KIP-917: Additional custom metadata for remote log segment

Release notes for the 3.6.0 release:
https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html

*** Please download, test and vote by Wednesday, September 21, 12pm PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~satishd/kafka-3.6.0-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/

* Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
https://github.com/apache/kafka/releases/tag/3.6.0-rc0

* Documentation:
https://kafka.apache.org/36/documentation.html

* Protocol:
https://kafka.apache.org/36/protocol.html

* Successful Jenkins builds for the 3.6 branch:
There are a few runs of unit/integration tests. You can see the latest at
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will continue
running a few more iterations.
System tests:
We will send an update once we have the results.

Thanks,
Satish.

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Justine for the update. I saw in the morning that these changes
are pushed to trunk and 3.6.

~Satish.

On Thu, 14 Sept 2023 at 21:54, Justine Olshan
 wrote:
>
> Hi Satish,
> We were able to merge
> https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> and pick to 3.6.
>
> Hopefully nothing more from me on this release.
>
> Thanks,
> Justine
>
> On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana 
> wrote:
>
> > Thanks Luke for the update.
> >
> > ~Satish.
> >
> > On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > Since this PR:
> > > https://github.com/apache/kafka/pull/14366 only changes the doc, I've
> > > backported to 3.6 branch. FYI.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
> > >  wrote:
> > >
> > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > > > PR should be finalized soon.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > > > migration, that we would like to backport to 3.5 and 3.6 branches.
> > Are
> > > > > you ok with that?
> > > > >
> > > > > https://github.com/apache/kafka/pull/14366
> > > > >
> > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Thanks David for the quick resolution.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > > > >  wrote:
> > > > > > >
> > > > > > > Satish,
> > > > > > >
> > > > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> > > > > > >
> > > > > > > Thanks!
> > > > > > > David
> > > > > > >
> > > > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma 
> > > > > wrote:
> > > > > > >
> > > > > > > > Justine,
> > > > > > > >
> > > > > > > > Probably best to have the conversation in the JIRA ticket vs
> > the
> > > > > release
> > > > > > > > thread. Generally, we want to only include low risk bug fixes
> > that
> > > > > are
> > > > > > > > fully compatible in patch releases.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > > > > > > 
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Satish. I understand.
> > > > > > > > > Just curious, is this something that could be added to
> > 3.6.1? It
> > > > > would be
> > > > > > > > > nice to say that hanging transactions are fully covered in a
> > 3.6
> > > > > release.
> > > > > > > > > I'm not as familiar with the rules around minor releases, but
> > > > > adding it
> > > > > > > > > there would give more time to ensure stability.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Justine
> > > > > > > > >
> > > > > > > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Justine,
> > > > > > > > > > We can skip this change into 3.6 now as it is not a
> > blocker or
> > > > > > > > > > regression and it involves changes to the API
> > implementation.
> > > > > Let us
> > > > > > > > > > plan to add the 

[jira] [Resolved] (KAFKA-7739) Kafka Tiered Storage

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-7739.
---
Resolution: Fixed

> Kafka Tiered Storage
> 
>
> Key: KAFKA-7739
> URL: https://issues.apache.org/jira/browse/KAFKA-7739
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Harsha
>    Assignee: Satish Duggana
>Priority: Major
>  Labels: needs-kip
> Fix For: 3.6.0
>
>
> KIP: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Luke for the update.

~Satish.

On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
>
> Hi Satish,
>
> Since this PR:
> https://github.com/apache/kafka/pull/14366 only changes the doc, I've
> backported to 3.6 branch. FYI.
>
> Thanks.
> Luke
>
> On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
>  wrote:
>
> > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > PR should be finalized soon.
> >
> > Thanks,
> > Justine
> >
> > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > wrote:
> >
> > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > migration, that we would like to backport to 3.5 and 3.6 branches. Are
> > > you ok with that?
> > >
> > > https://github.com/apache/kafka/pull/14366
> > >
> > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana  > >
> > > wrote:
> > > >
> > > > Thanks David for the quick resolution.
> > > >
> > > > ~Satish.
> > > >
> > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > >  wrote:
> > > > >
> > > > > Satish,
> > > > >
> > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> > > > >
> > > > > Thanks!
> > > > > David
> > > > >
> > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Justine,
> > > > > >
> > > > > > Probably best to have the conversation in the JIRA ticket vs the
> > > release
> > > > > > thread. Generally, we want to only include low risk bug fixes that
> > > are
> > > > > > fully compatible in patch releases.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > > > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Satish. I understand.
> > > > > > > Just curious, is this something that could be added to 3.6.1? It
> > > would be
> > > > > > > nice to say that hanging transactions are fully covered in a 3.6
> > > release.
> > > > > > > I'm not as familiar with the rules around minor releases, but
> > > adding it
> > > > > > > there would give more time to ensure stability.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana <
> > > satish.dugg...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Justine,
> > > > > > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > > > > > regression and it involves changes to the API implementation.
> > > Let us
> > > > > > > > plan to add the gap in the release notes as you mentioned.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Satish.
> > > > > > > >
> > > > > > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hey Satish,
> > > > > > > > >
> > > > > > > > > We just discovered a gap in KIP-890 part 1. We currently
> > don't
> > > verify
> > > > > > > on
> > > > > > > > > txn offset commits, so it is still possible to have hanging
> > > > > > > transactions
> > > > > > > > on
> > > > > > > > > the consumer offsets partitions.
> > > > > > > > > I've opened a jira to wire the verification in that request.
> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > > > > > >
> > > > > > > > > This also isn't a regression, but it would be nice to have
> > > part 1
> > > > > > fully
> > > > > > > > > complete. I have opened a PR with the fix:
> > > > > > >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks David for the quick resolution.

~Satish.

On Tue, 12 Sept 2023 at 22:51, David Arthur
 wrote:
>
> Satish,
>
> KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
>
> Thanks!
> David
>
> On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma  wrote:
>
> > Justine,
> >
> > Probably best to have the conversation in the JIRA ticket vs the release
> > thread. Generally, we want to only include low risk bug fixes that are
> > fully compatible in patch releases.
> >
> > Ismael
> >
> > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > 
> > wrote:
> >
> > > Thanks Satish. I understand.
> > > Just curious, is this something that could be added to 3.6.1? It would be
> > > nice to say that hanging transactions are fully covered in a 3.6 release.
> > > I'm not as familiar with the rules around minor releases, but adding it
> > > there would give more time to ensure stability.
> > >
> > > Thanks,
> > > Justine
> > >
> > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana  > >
> > > wrote:
> > >
> > > > Hi Justine,
> > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > regression and it involves changes to the API implementation. Let us
> > > > plan to add the gap in the release notes as you mentioned.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > >  wrote:
> > > > >
> > > > > Hey Satish,
> > > > >
> > > > > We just discovered a gap in KIP-890 part 1. We currently don't verify
> > > on
> > > > > txn offset commits, so it is still possible to have hanging
> > > transactions
> > > > on
> > > > > the consumer offsets partitions.
> > > > > I've opened a jira to wire the verification in that request.
> > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > >
> > > > > This also isn't a regression, but it would be nice to have part 1
> > fully
> > > > > complete. I have opened a PR with the fix:
> > > > > https://github.com/apache/kafka/pull/14370.
> > > > >
> > > > > I understand if there are concerns about last minute changes to this
> > > API
> > > > > and we can hold off if that makes the most sense.
> > > > > If we take that route, I think we should still keep verification for
> > > the
> > > > > data partitions since it still provides full protection there and
> > > > improves
> > > > > the transactions experience. We will need to call out the gap in the
> > > > > release notes for consumer offsets partitions
> > > > >
> > > > > Let me know what you think.
> > > > > Justine
> > > > >
> > > > >
> > > > > On Mon, Sep 11, 2023 at 12:29 PM David Arthur
> > > > >  wrote:
> > > > >
> > > > > > Another (small) ZK migration issue was identified. This one isn't a
> > > > > > regression (it has existed since 3.4), but I think it's reasonable
> > to
> > > > > > include. It's a small configuration check that could potentially
> > save
> > > > end
> > > > > > users from some headaches down the line.
> > > > > >
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15450
> > > > > > https://github.com/apache/kafka/pull/14367
> > > > > >
> > > > > > I think we can get this one committed to trunk today.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma 
> > > wrote:
> > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > That sounds great. I think we should aim to only allow blockers
> > > > > > > (regressions, impactful security issues, etc.) on the 3.6 branch
> > > > until
> > > > > > > 3.6.0 is out.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 9, 2023, 12:20 AM Satish Duggan

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Justine,
Thanks for the update. From JIRA[1] it looks like it affects only
3.6.0 but not for the earlier releases. Is that right?

https://issues.apache.org/jira/browse/KAFKA-15459

~Satish.

On Wed, 13 Sept 2023 at 00:42, Justine Olshan
 wrote:
>
> It's me again. 
> While reviewing the previous PR, it was discovered we had a potential
> breaking return code for non-java clients.
> This unfortunately seems like a blocker.
> https://issues.apache.org/jira/browse/KAFKA-15459
>
> I will be able to get a PR open today.
>
> Apologies for all the noise in the thread,
> Justine
>
> On Tue, Sep 12, 2023 at 10:21 AM David Arthur
>  wrote:
>
> > Satish,
> >
> > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 3.4)
> >
> > Thanks!
> > David
> >
> > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma  wrote:
> >
> > > Justine,
> > >
> > > Probably best to have the conversation in the JIRA ticket vs the release
> > > thread. Generally, we want to only include low risk bug fixes that are
> > > fully compatible in patch releases.
> > >
> > > Ismael
> > >
> > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > 
> > > wrote:
> > >
> > > > Thanks Satish. I understand.
> > > > Just curious, is this something that could be added to 3.6.1? It would
> > be
> > > > nice to say that hanging transactions are fully covered in a 3.6
> > release.
> > > > I'm not as familiar with the rules around minor releases, but adding it
> > > > there would give more time to ensure stability.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Tue, Sep 12, 2023 at 5:49 AM Satish Duggana <
> > satish.dugg...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Justine,
> > > > > We can skip this change into 3.6 now as it is not a blocker or
> > > > > regression and it involves changes to the API implementation. Let us
> > > > > plan to add the gap in the release notes as you mentioned.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Tue, 12 Sept 2023 at 04:44, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > We just discovered a gap in KIP-890 part 1. We currently don't
> > verify
> > > > on
> > > > > > txn offset commits, so it is still possible to have hanging
> > > > transactions
> > > > > on
> > > > > > the consumer offsets partitions.
> > > > > > I've opened a jira to wire the verification in that request.
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15449
> > > > > >
> > > > > > This also isn't a regression, but it would be nice to have part 1
> > > fully
> > > > > > complete. I have opened a PR with the fix:
> > > > > > https://github.com/apache/kafka/pull/14370.
> > > > > >
> > > > > > I understand if there are concerns about last minute changes to
> > this
> > > > API
> > > > > > and we can hold off if that makes the most sense.
> > > > > > If we take that route, I think we should still keep verification
> > for
> > > > the
> > > > > > data partitions since it still provides full protection there and
> > > > > improves
> > > > > > the transactions experience. We will need to call out the gap in
> > the
> > > > > > release notes for consumer offsets partitions
> > > > > >
> > > > > > Let me know what you think.
> > > > > > Justine
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 11, 2023 at 12:29 PM David Arthur
> > > > > >  wrote:
> > > > > >
> > > > > > > Another (small) ZK migration issue was identified. This one
> > isn't a
> > > > > > > regression (it has existed since 3.4), but I think it's
> > reasonable
> > > to
> > > > > > > include. It's a small configuration check that could potentially
> > > save
> > > > > end
> > > > > > > users from some headaches down the line.
> > > > > > >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Justine,
We can skip this change into 3.6 now as it is not a blocker or
regression and it involves changes to the API implementation. Let us
plan to add the gap in the release notes as you mentioned.

Thanks,
Satish.

On Tue, 12 Sept 2023 at 04:44, Justine Olshan
 wrote:
>
> Hey Satish,
>
> We just discovered a gap in KIP-890 part 1. We currently don't verify on
> txn offset commits, so it is still possible to have hanging transactions on
> the consumer offsets partitions.
> I've opened a jira to wire the verification in that request.
> https://issues.apache.org/jira/browse/KAFKA-15449
>
> This also isn't a regression, but it would be nice to have part 1 fully
> complete. I have opened a PR with the fix:
> https://github.com/apache/kafka/pull/14370.
>
> I understand if there are concerns about last minute changes to this API
> and we can hold off if that makes the most sense.
> If we take that route, I think we should still keep verification for the
> data partitions since it still provides full protection there and improves
> the transactions experience. We will need to call out the gap in the
> release notes for consumer offsets partitions
>
> Let me know what you think.
> Justine
>
>
> On Mon, Sep 11, 2023 at 12:29 PM David Arthur
>  wrote:
>
> > Another (small) ZK migration issue was identified. This one isn't a
> > regression (it has existed since 3.4), but I think it's reasonable to
> > include. It's a small configuration check that could potentially save end
> > users from some headaches down the line.
> >
> > https://issues.apache.org/jira/browse/KAFKA-15450
> > https://github.com/apache/kafka/pull/14367
> >
> > I think we can get this one committed to trunk today.
> >
> > -David
> >
> >
> >
> > On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma  wrote:
> >
> > > Hi Satish,
> > >
> > > That sounds great. I think we should aim to only allow blockers
> > > (regressions, impactful security issues, etc.) on the 3.6 branch until
> > > 3.6.0 is out.
> > >
> > > Ismael
> > >
> > >
> > > On Sat, Sep 9, 2023, 12:20 AM Satish Duggana 
> > > wrote:
> > >
> > > > Hi Ismael,
> > > > It looks like we will publish RC0 by 14th Sep.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > > On Fri, 8 Sept 2023 at 19:23, Ismael Juma  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > Do you have a sense of when we'll publish RC0?
> > > > >
> > > > > Thanks,
> > > > > Ismael
> > > > >
> > > > > On Fri, Sep 8, 2023 at 6:27 AM David Arthur
> > > > >  wrote:
> > > > >
> > > > > > Quick update on my two blockers: KAFKA-15435 is merged to trunk and
> > > > > > cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will
> > > > hopefully
> > > > > > get it merged today.
> > > > > >
> > > > > > -David
> > > > > >
> > > > > > On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko 
> > > wrote:
> > > > > >
> > > > > > > Hi Satish and all,
> > > > > > >
> > > > > > > I wonder if https://issues.apache.org/jira/browse/KAFKA-14993
> > > > should be
> > > > > > > included in the 3.6 release plan. I'm thinking that when
> > > > implemented, it
> > > > > > > would be a small, but still a change in the RSM contract: throw
> > an
> > > > > > > exception instead of returning an empty InputStream. Maybe it
> > > should
> > > > be
> > > > > > > included right away to save the migration later? What do you
> > think?
> > > > > > >
> > > > > > > Best,
> > > > > > > Ivan
> > > > > > >
> > > > > > > On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > > > > > > > Hi Jose,
> > > > > > > > Thanks for looking into this issue and resolving it with a
> > quick
> > > > fix.
> > > > > > > >
> > > > > > > > ~Satish.
> > > > > > > >
> > > > > > > > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > > Hi Satish,
> > > > > > > > >
> > > > > > > > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana <
> > > > > > > satish.dugg...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273
> > > has
> > > > > > been
> > > > > > > > > > there in 3.5.x too.
> > > > > > > > >
> > > > > > > > > I also agree that it should be a blocker for 3.6.0. It should
> > > > have
> > > > > > > > > been a blocker for those previous releases. I didn't fix it
> > > > because,
> > > > > > > > > unfortunately, I wasn't aware of the issue and jira.
> > > > > > > > > I'll create a PR with a fix in case the original author
> > doesn't
> > > > > > > respond in time.
> > > > > > > > >
> > > > > > > > > Satish, do you agree?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > > --
> > > > > > > > > -José
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > -David
> > > > > >
> > > >
> > >
> >
> >
> > --
> > -David
> >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi David,
It looks like the PR is already accepted with a minor config change
validation, and you mentioned that the failed tests are not related.
Please merge it to 3.6 by today.

Thanks,
Satish.

On Tue, 12 Sept 2023 at 00:59, David Arthur
 wrote:
>
> Another (small) ZK migration issue was identified. This one isn't a
> regression (it has existed since 3.4), but I think it's reasonable to
> include. It's a small configuration check that could potentially save end
> users from some headaches down the line.
>
> https://issues.apache.org/jira/browse/KAFKA-15450
> https://github.com/apache/kafka/pull/14367
>
> I think we can get this one committed to trunk today.
>
> -David
>
>
>
> On Sun, Sep 10, 2023 at 7:50 PM Ismael Juma  wrote:
>
> > Hi Satish,
> >
> > That sounds great. I think we should aim to only allow blockers
> > (regressions, impactful security issues, etc.) on the 3.6 branch until
> > 3.6.0 is out.
> >
> > Ismael
> >
> >
> > On Sat, Sep 9, 2023, 12:20 AM Satish Duggana 
> > wrote:
> >
> > > Hi Ismael,
> > > It looks like we will publish RC0 by 14th Sep.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Fri, 8 Sept 2023 at 19:23, Ismael Juma  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > Do you have a sense of when we'll publish RC0?
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > > > On Fri, Sep 8, 2023 at 6:27 AM David Arthur
> > > >  wrote:
> > > >
> > > > > Quick update on my two blockers: KAFKA-15435 is merged to trunk and
> > > > > cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will
> > > hopefully
> > > > > get it merged today.
> > > > >
> > > > > -David
> > > > >
> > > > > On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko 
> > wrote:
> > > > >
> > > > > > Hi Satish and all,
> > > > > >
> > > > > > I wonder if https://issues.apache.org/jira/browse/KAFKA-14993
> > > should be
> > > > > > included in the 3.6 release plan. I'm thinking that when
> > > implemented, it
> > > > > > would be a small, but still a change in the RSM contract: throw an
> > > > > > exception instead of returning an empty InputStream. Maybe it
> > should
> > > be
> > > > > > included right away to save the migration later? What do you think?
> > > > > >
> > > > > > Best,
> > > > > > Ivan
> > > > > >
> > > > > > On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > > > > > > Hi Jose,
> > > > > > > Thanks for looking into this issue and resolving it with a quick
> > > fix.
> > > > > > >
> > > > > > > ~Satish.
> > > > > > >
> > > > > > > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > Hi Satish,
> > > > > > > >
> > > > > > > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana <
> > > > > > satish.dugg...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273
> > has
> > > > > been
> > > > > > > > > there in 3.5.x too.
> > > > > > > >
> > > > > > > > I also agree that it should be a blocker for 3.6.0. It should
> > > have
> > > > > > > > been a blocker for those previous releases. I didn't fix it
> > > because,
> > > > > > > > unfortunately, I wasn't aware of the issue and jira.
> > > > > > > > I'll create a PR with a fix in case the original author doesn't
> > > > > > respond in time.
> > > > > > > >
> > > > > > > > Satish, do you agree?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > > --
> > > > > > > > -José
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -David
> > > > >
> > >
> >
>
>
> --
> -David

[jira] [Resolved] (KAFKA-14993) Improve TransactionIndex instance handling while copying to and fetching from RSM.

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-14993.

Resolution: Fixed

> Improve TransactionIndex instance handling while copying to and fetching from 
> RSM.
> --
>
> Key: KAFKA-14993
> URL: https://issues.apache.org/jira/browse/KAFKA-14993
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>    Reporter: Satish Duggana
>Assignee: Abhijeet Kumar
>Priority: Blocker
>  Labels: tiered-storage
> Fix For: 3.6.0
>
>
> RSM should throw a ResourceNotFoundException if it does not have 
> TransactionIndex. Currently, it expects an empty InputStream and creates an 
> unnecessary file in the cache. This can be avoided by catching 
> ResourceNotFoundException and not creating an instance. There are minor 
> cleanups needed in RemoteIndexCache and other TransactionIndex usages.
> Also, update the LocalTieredStorage, see 
> [this|https://github.com/apache/kafka/pull/13837#discussion_r1258917584] 
> comment.
> Note, please remember to update the javadoc in RSM after the fix. See: 
> https://github.com/apache/kafka/pull/14352



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15453) Enable `testFencingOnTransactionExpiration` in TransactionsWithTieredStoreTest

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15453:
--

 Summary: Enable `testFencingOnTransactionExpiration` in 
TransactionsWithTieredStoreTest
 Key: KAFKA-15453
 URL: https://issues.apache.org/jira/browse/KAFKA-15453
 Project: Kafka
  Issue Type: Test
  Components: core
Reporter: Satish Duggana
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15352) Ensure consistency while deleting the remote log segments

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15352.

Resolution: Fixed

> Ensure consistency while deleting the remote log segments
> -
>
> Key: KAFKA-15352
> URL: https://issues.apache.org/jira/browse/KAFKA-15352
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Kamal Chandraprakash
>Assignee: Christo Lolov
>Priority: Blocker
> Fix For: 3.6.0
>
>
> In Kafka-14888, the remote log segments are deleted which breaches the 
> retention time/size before updating the log-start-offset. In middle of 
> deletion, if the consumer starts to read from the beginning of the topic, 
> then it will fail to read the messages and UNKNOWN_SERVER_ERROR will be 
> thrown back to the consumer.
> To ensure consistency, similar to local log segments where the actual 
> segments are deleted after {{{}segment.delete.delay.ms{}}}, we should update 
> the log-start-offset first before deleting the remote log segment.
> See the [PR#13561|https://github.com/apache/kafka/pull/13561] and 
> [comment|https://github.com/apache/kafka/pull/13561#discussion_r1293086543] 
> for more details.
> Case-2:
> The old-leader (follower) can delete the remote log segment in the middle of 
> leader election. We need to update the log-start-offset metadata for this 
> case.
> See this comment 
> [https://github.com/apache/kafka/pull/13561#discussion_r1293081560]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Ismael,
It looks like we will publish RC0 by 14th Sep.

Thanks,
Satish.

On Fri, 8 Sept 2023 at 19:23, Ismael Juma  wrote:
>
> Hi Satish,
>
> Do you have a sense of when we'll publish RC0?
>
> Thanks,
> Ismael
>
> On Fri, Sep 8, 2023 at 6:27 AM David Arthur
>  wrote:
>
> > Quick update on my two blockers: KAFKA-15435 is merged to trunk and
> > cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will hopefully
> > get it merged today.
> >
> > -David
> >
> > On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko  wrote:
> >
> > > Hi Satish and all,
> > >
> > > I wonder if https://issues.apache.org/jira/browse/KAFKA-14993 should be
> > > included in the 3.6 release plan. I'm thinking that when implemented, it
> > > would be a small, but still a change in the RSM contract: throw an
> > > exception instead of returning an empty InputStream. Maybe it should be
> > > included right away to save the migration later? What do you think?
> > >
> > > Best,
> > > Ivan
> > >
> > > On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > > > Hi Jose,
> > > > Thanks for looking into this issue and resolving it with a quick fix.
> > > >
> > > > ~Satish.
> > > >
> > > > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> > > >  wrote:
> > > > >
> > > > > Hi Satish,
> > > > >
> > > > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana <
> > > satish.dugg...@gmail.com> wrote:
> > > > > >
> > > > > > Hi Greg,
> > > > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273 has
> > been
> > > > > > there in 3.5.x too.
> > > > >
> > > > > I also agree that it should be a blocker for 3.6.0. It should have
> > > > > been a blocker for those previous releases. I didn't fix it because,
> > > > > unfortunately, I wasn't aware of the issue and jira.
> > > > > I'll create a PR with a fix in case the original author doesn't
> > > respond in time.
> > > > >
> > > > > Satish, do you agree?
> > > > >
> > > > > Thanks!
> > > > > --
> > > > > -José
> > > >
> > >
> >
> >
> > --
> > -David
> >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks David for the fixes and the update.

~Satish.

On Fri, 8 Sept 2023 at 18:57, David Arthur
 wrote:
>
> Quick update on my two blockers: KAFKA-15435 is merged to trunk and
> cherry-picked to 3.6. I have a PR open for KAFKA-15441 and will hopefully
> get it merged today.
>
> -David
>
> On Fri, Sep 8, 2023 at 5:26 AM Ivan Yurchenko  wrote:
>
> > Hi Satish and all,
> >
> > I wonder if https://issues.apache.org/jira/browse/KAFKA-14993 should be
> > included in the 3.6 release plan. I'm thinking that when implemented, it
> > would be a small, but still a change in the RSM contract: throw an
> > exception instead of returning an empty InputStream. Maybe it should be
> > included right away to save the migration later? What do you think?
> >
> > Best,
> > Ivan
> >
> > On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > > Hi Jose,
> > > Thanks for looking into this issue and resolving it with a quick fix.
> > >
> > > ~Satish.
> > >
> > > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> > >  wrote:
> > > >
> > > > Hi Satish,
> > > >
> > > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana <
> > satish.dugg...@gmail.com> wrote:
> > > > >
> > > > > Hi Greg,
> > > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273 has been
> > > > > there in 3.5.x too.
> > > >
> > > > I also agree that it should be a blocker for 3.6.0. It should have
> > > > been a blocker for those previous releases. I didn't fix it because,
> > > > unfortunately, I wasn't aware of the issue and jira.
> > > > I'll create a PR with a fix in case the original author doesn't
> > respond in time.
> > > >
> > > > Satish, do you agree?
> > > >
> > > > Thanks!
> > > > --
> > > > -José
> > >
> >
>
>
> --
> -David

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Ivan,
This is a reasonable request. I think it is good to address now
because of the confusion the API is creating. Abhijeet agreed to
volunteer to raise PR today to address KAFKA-14993.

Thanks,
Satish.

On Fri, 8 Sept 2023 at 14:54, Ivan Yurchenko  wrote:
>
> Hi Satish and all,
>
> I wonder if https://issues.apache.org/jira/browse/KAFKA-14993 should be 
> included in the 3.6 release plan. I'm thinking that when implemented, it 
> would be a small, but still a change in the RSM contract: throw an exception 
> instead of returning an empty InputStream. Maybe it should be included right 
> away to save the migration later? What do you think?
>
> Best,
> Ivan
>
> On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> > Hi Jose,
> > Thanks for looking into this issue and resolving it with a quick fix.
> >
> > ~Satish.
> >
> > On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
> >  wrote:
> > >
> > > Hi Satish,
> > >
> > > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana  
> > > wrote:
> > > >
> > > > Hi Greg,
> > > > It seems https://issues.apache.org/jira/browse/KAFKA-14273 has been
> > > > there in 3.5.x too.
> > >
> > > I also agree that it should be a blocker for 3.6.0. It should have
> > > been a blocker for those previous releases. I didn't fix it because,
> > > unfortunately, I wasn't aware of the issue and jira.
> > > I'll create a PR with a fix in case the original author doesn't respond 
> > > in time.
> > >
> > > Satish, do you agree?
> > >
> > > Thanks!
> > > --
> > > -José
> >

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Jose,
Thanks for looking into this issue and resolving it with a quick fix.

~Satish.

On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
 wrote:
>
> Hi Satish,
>
> On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana  
> wrote:
> >
> > Hi Greg,
> > It seems https://issues.apache.org/jira/browse/KAFKA-14273 has been
> > there in 3.5.x too.
>
> I also agree that it should be a blocker for 3.6.0. It should have
> been a blocker for those previous releases. I didn't fix it because,
> unfortunately, I wasn't aware of the issue and jira.
> I'll create a PR with a fix in case the original author doesn't respond in 
> time.
>
> Satish, do you agree?
>
> Thanks!
> --
> -José

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Greg,
It seems https://issues.apache.org/jira/browse/KAFKA-14273 has been
there in 3.5.x too.

David/Colin,
Do you think it is a blocker for 3.6.0?

Thanks,
Satish.

On Wed, 6 Sept 2023 at 21:34, Greg Harris  wrote:
>
> Hey all,
>
> A user just experienced this problem that has already been reported
> for several versions and has an open PR:
> https://issues.apache.org/jira/browse/KAFKA-14273
> https://github.com/apache/kafka/pull/12763
>
> Impact: In KRaft mode on Windows, Kafka crashes on startup with an 
> IOException.
>
> Does this qualify as a release blocker?
>
> Thanks,
> Greg
>
> On Wed, Sep 6, 2023 at 7:22 AM David Arthur  wrote:
> >
> > Thanks, Satish! Here's another blocker
> > https://issues.apache.org/jira/browse/KAFKA-15441 :)
> >
> > For the 3.6 release notes and announcement, I'd like to include a special
> > note about ZK to KRaft migrations being GA (Generally Available). We have
> > finished closing all the gaps from the earlier releases of ZK migrations
> > (e.g., ACLs, SCRAM), so it is now possible to migrate all metadata to
> > KRaft. We have also made the migration more reliable and fault
> > tolerant with the inclusion of KIP-868 transactions. I'd be happy to write
> > something for the release notes when the time comes, if it's helpful.
> >
> > Thanks!
> > David
> >
> > On Tue, Sep 5, 2023 at 8:13 PM Satish Duggana 
> > wrote:
> >
> > > Hi David,
> > > Thanks for bringing this issue to this thread.
> > > I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Tue, 5 Sept 2023 at 21:29, David Arthur  wrote:
> > > >
> > > > Hi Satish. Thanks for running the release!
> > > >
> > > > I'd like to raise this as a blocker for 3.6
> > > > https://issues.apache.org/jira/browse/KAFKA-15435.
> > > >
> > > > It's a very quick fix, so I should be able to post a PR soon.
> > > >
> > > > Thanks!
> > > > David
> > > >
> > > > On Mon, Sep 4, 2023 at 11:44 PM Justine Olshan
> > > 
> > > > wrote:
> > > >
> > > > > Thanks Satish. This is done 
> > > > >
> > > > > Justine
> > > > >
> > > > > On Mon, Sep 4, 2023 at 5:16 PM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hey Justine,
> > > > > > I went through KAFKA-15424 and the PR[1]. It seems there are no
> > > > > > dependent changes missing in 3.6 branch. They seem to be low risk as
> > > > > > you mentioned. Please merge it to the 3.6 branch as well.
> > > > > >
> > > > > > 1. https://github.com/apache/kafka/pull/14324.
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > > > On Tue, 5 Sept 2023 at 05:06, Justine Olshan
> > > > > >  wrote:
> > > > > > >
> > > > > > > Sorry I meant to add the jira as well.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-15424
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Mon, Sep 4, 2023 at 4:34 PM Justine Olshan <
> > > jols...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hey Satish,
> > > > > > > >
> > > > > > > > I was working on adding dynamic configuration for
> > > > > > > > transaction verification. The PR is approved and ready to merge
> > > into
> > > > > > trunk.
> > > > > > > > I was thinking I could also add it to 3.6 since it is fairly low
> > > > > risk.
> > > > > > > > What do you think?
> > > > > > > >
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Sat, Sep 2, 2023 at 6:21 PM Sophie Blee-Goldman <
> > > > > > ableegold...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks Satish! The fix has been merged and cherrypicked to 3.6
> > > > > 

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi David,
Thanks for bringing the KAFKA-15441 to this discussion thread, it
seems to be a blocker and updated the JIRA. What is the plan to
address that?

https://issues.apache.org/jira/browse/KAFKA-15411, which is about
KRaft and delegation token related test failures. It seems to be a
test env issue but it is better to check and confirm it is a
non-blocker for 3.6.0. Please take a look and update the JIRA.

You can add a special section in the release note about KRaft
migrations being GA.

Thanks,
Satish.

On Wed, 6 Sept 2023 at 19:52, David Arthur  wrote:
>
> Thanks, Satish! Here's another blocker
> https://issues.apache.org/jira/browse/KAFKA-15441 :)
>
> For the 3.6 release notes and announcement, I'd like to include a special
> note about ZK to KRaft migrations being GA (Generally Available). We have
> finished closing all the gaps from the earlier releases of ZK migrations
> (e.g., ACLs, SCRAM), so it is now possible to migrate all metadata to
> KRaft. We have also made the migration more reliable and fault
> tolerant with the inclusion of KIP-868 transactions. I'd be happy to write
> something for the release notes when the time comes, if it's helpful.
>
> Thanks!
> David
>
> On Tue, Sep 5, 2023 at 8:13 PM Satish Duggana 
> wrote:
>
> > Hi David,
> > Thanks for bringing this issue to this thread.
> > I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker.
> >
> > Thanks,
> > Satish.
> >
> > On Tue, 5 Sept 2023 at 21:29, David Arthur  wrote:
> > >
> > > Hi Satish. Thanks for running the release!
> > >
> > > I'd like to raise this as a blocker for 3.6
> > > https://issues.apache.org/jira/browse/KAFKA-15435.
> > >
> > > It's a very quick fix, so I should be able to post a PR soon.
> > >
> > > Thanks!
> > > David
> > >
> > > On Mon, Sep 4, 2023 at 11:44 PM Justine Olshan
> > 
> > > wrote:
> > >
> > > > Thanks Satish. This is done 
> > > >
> > > > Justine
> > > >
> > > > On Mon, Sep 4, 2023 at 5:16 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hey Justine,
> > > > > I went through KAFKA-15424 and the PR[1]. It seems there are no
> > > > > dependent changes missing in 3.6 branch. They seem to be low risk as
> > > > > you mentioned. Please merge it to the 3.6 branch as well.
> > > > >
> > > > > 1. https://github.com/apache/kafka/pull/14324.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > > On Tue, 5 Sept 2023 at 05:06, Justine Olshan
> > > > >  wrote:
> > > > > >
> > > > > > Sorry I meant to add the jira as well.
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15424
> > > > > >
> > > > > > Justine
> > > > > >
> > > > > > On Mon, Sep 4, 2023 at 4:34 PM Justine Olshan <
> > jols...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > > Hey Satish,
> > > > > > >
> > > > > > > I was working on adding dynamic configuration for
> > > > > > > transaction verification. The PR is approved and ready to merge
> > into
> > > > > trunk.
> > > > > > > I was thinking I could also add it to 3.6 since it is fairly low
> > > > risk.
> > > > > > > What do you think?
> > > > > > >
> > > > > > > Justine
> > > > > > >
> > > > > > > On Sat, Sep 2, 2023 at 6:21 PM Sophie Blee-Goldman <
> > > > > ableegold...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks Satish! The fix has been merged and cherrypicked to 3.6
> > > > > > >>
> > > > > > >> On Sat, Sep 2, 2023 at 6:02 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Sophie,
> > > > > > >> > Please feel free to add that to 3.6 branch as you say this is
> > a
> > > > > minor
> > > > > > >> > change and will not cause any regressions.
> > > > > > >> >
> > > > &g

[jira] [Resolved] (KAFKA-15245) Improve Tiered Storage Metrics

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15245.

Resolution: Fixed

> Improve Tiered Storage Metrics
> --
>
> Key: KAFKA-15245
> URL: https://issues.apache.org/jira/browse/KAFKA-15245
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Abhijeet Kumar
>Assignee: Abhijeet Kumar
>Priority: Major
> Fix For: 3.7.0
>
>
> Rename existing tiered storage metrics to remove ambiguity and add metrics 
> for the RemoteIndexCache.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15439) Add transaction tests enabled with tiered storage

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15439:
--

 Summary: Add transaction tests enabled with tiered storage
 Key: KAFKA-15439
 URL: https://issues.apache.org/jira/browse/KAFKA-15439
 Project: Kafka
  Issue Type: Test
  Components: core
Reporter: Satish Duggana
Assignee: Kamal Chandraprakash






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi David,
Thanks for bringing this issue to this thread.
I marked https://issues.apache.org/jira/browse/KAFKA-15435 as a blocker.

Thanks,
Satish.

On Tue, 5 Sept 2023 at 21:29, David Arthur  wrote:
>
> Hi Satish. Thanks for running the release!
>
> I'd like to raise this as a blocker for 3.6
> https://issues.apache.org/jira/browse/KAFKA-15435.
>
> It's a very quick fix, so I should be able to post a PR soon.
>
> Thanks!
> David
>
> On Mon, Sep 4, 2023 at 11:44 PM Justine Olshan 
> wrote:
>
> > Thanks Satish. This is done 
> >
> > Justine
> >
> > On Mon, Sep 4, 2023 at 5:16 PM Satish Duggana 
> > wrote:
> >
> > > Hey Justine,
> > > I went through KAFKA-15424 and the PR[1]. It seems there are no
> > > dependent changes missing in 3.6 branch. They seem to be low risk as
> > > you mentioned. Please merge it to the 3.6 branch as well.
> > >
> > > 1. https://github.com/apache/kafka/pull/14324.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Tue, 5 Sept 2023 at 05:06, Justine Olshan
> > >  wrote:
> > > >
> > > > Sorry I meant to add the jira as well.
> > > > https://issues.apache.org/jira/browse/KAFKA-15424
> > > >
> > > > Justine
> > > >
> > > > On Mon, Sep 4, 2023 at 4:34 PM Justine Olshan 
> > > wrote:
> > > >
> > > > > Hey Satish,
> > > > >
> > > > > I was working on adding dynamic configuration for
> > > > > transaction verification. The PR is approved and ready to merge into
> > > trunk.
> > > > > I was thinking I could also add it to 3.6 since it is fairly low
> > risk.
> > > > > What do you think?
> > > > >
> > > > > Justine
> > > > >
> > > > > On Sat, Sep 2, 2023 at 6:21 PM Sophie Blee-Goldman <
> > > ableegold...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Thanks Satish! The fix has been merged and cherrypicked to 3.6
> > > > >>
> > > > >> On Sat, Sep 2, 2023 at 6:02 AM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Sophie,
> > > > >> > Please feel free to add that to 3.6 branch as you say this is a
> > > minor
> > > > >> > change and will not cause any regressions.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Satish.
> > > > >> >
> > > > >> > On Sat, 2 Sept 2023 at 08:44, Sophie Blee-Goldman
> > > > >> >  wrote:
> > > > >> > >
> > > > >> > > Hey Satish, someone reported a minor bug in the Streams
> > > application
> > > > >> > > shutdown which was a recent regression, though not strictly a
> > new
> > > one:
> > > > >> > was
> > > > >> > > introduced in 3.4 I believe.
> > > > >> > >
> > > > >> > > The fix seems to be super lightweight and low-risk so I was
> > > hoping to
> > > > >> > slip
> > > > >> > > it into 3.6 if that's ok with you? They plan to have the patch
> > > > >> tonight.
> > > > >> > >
> > > > >> > > https://issues.apache.org/jira/browse/KAFKA-15429
> > > > >> > >
> > > > >> > > On Thu, Aug 31, 2023 at 5:45 PM Satish Duggana <
> > > > >> satish.dugg...@gmail.com
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Thanks Chris for bringing this issue here and filing the new
> > > JIRA
> > > > >> for
> > > > >> > > > 3.6.0[1]. It seems to be a blocker for 3.6.0.
> > > > >> > > >
> > > > >> > > > Please help review https://github.com/apache/kafka/pull/14314
> > > as
> > > > >> Chris
> > > > >> > > > requested.
> > > > >> > > >
> > > > >> > > > 1. https://issues.apache.org/jira/browse/KAFKA-15425
> > > > >> > > >
> > > > >> > > > ~Satish.
> > > > >> > > >
>

[jira] [Resolved] (KAFKA-15293) Update metrics doc to add tiered storage metrics

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15293.

Resolution: Fixed

> Update metrics doc to add tiered storage metrics
> 
>
> Key: KAFKA-15293
> URL: https://issues.apache.org/jira/browse/KAFKA-15293
> Project: Kafka
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Abhijeet Kumar
>Assignee: Abhijeet Kumar
>Priority: Critical
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15434) Tiered Storage Quotas

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15434:
--

 Summary: Tiered Storage Quotas 
 Key: KAFKA-15434
 URL: https://issues.apache.org/jira/browse/KAFKA-15434
 Project: Kafka
  Issue Type: Improvement
Reporter: Satish Duggana
Assignee: Abhijeet Kumar






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15433) Follower fetch from tiered offset

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15433:
--

 Summary: Follower fetch from tiered offset 
 Key: KAFKA-15433
 URL: https://issues.apache.org/jira/browse/KAFKA-15433
 Project: Kafka
  Issue Type: Improvement
Reporter: Satish Duggana
Assignee: Abhijeet Kumar






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hey Justine,
I went through KAFKA-15424 and the PR[1]. It seems there are no
dependent changes missing in 3.6 branch. They seem to be low risk as
you mentioned. Please merge it to the 3.6 branch as well.

1. https://github.com/apache/kafka/pull/14324.

Thanks,
Satish.

On Tue, 5 Sept 2023 at 05:06, Justine Olshan
 wrote:
>
> Sorry I meant to add the jira as well.
> https://issues.apache.org/jira/browse/KAFKA-15424
>
> Justine
>
> On Mon, Sep 4, 2023 at 4:34 PM Justine Olshan  wrote:
>
> > Hey Satish,
> >
> > I was working on adding dynamic configuration for
> > transaction verification. The PR is approved and ready to merge into trunk.
> > I was thinking I could also add it to 3.6 since it is fairly low risk.
> > What do you think?
> >
> > Justine
> >
> > On Sat, Sep 2, 2023 at 6:21 PM Sophie Blee-Goldman 
> > wrote:
> >
> >> Thanks Satish! The fix has been merged and cherrypicked to 3.6
> >>
> >> On Sat, Sep 2, 2023 at 6:02 AM Satish Duggana 
> >> wrote:
> >>
> >> > Hi Sophie,
> >> > Please feel free to add that to 3.6 branch as you say this is a minor
> >> > change and will not cause any regressions.
> >> >
> >> > Thanks,
> >> > Satish.
> >> >
> >> > On Sat, 2 Sept 2023 at 08:44, Sophie Blee-Goldman
> >> >  wrote:
> >> > >
> >> > > Hey Satish, someone reported a minor bug in the Streams application
> >> > > shutdown which was a recent regression, though not strictly a new one:
> >> > was
> >> > > introduced in 3.4 I believe.
> >> > >
> >> > > The fix seems to be super lightweight and low-risk so I was hoping to
> >> > slip
> >> > > it into 3.6 if that's ok with you? They plan to have the patch
> >> tonight.
> >> > >
> >> > > https://issues.apache.org/jira/browse/KAFKA-15429
> >> > >
> >> > > On Thu, Aug 31, 2023 at 5:45 PM Satish Duggana <
> >> satish.dugg...@gmail.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Thanks Chris for bringing this issue here and filing the new JIRA
> >> for
> >> > > > 3.6.0[1]. It seems to be a blocker for 3.6.0.
> >> > > >
> >> > > > Please help review https://github.com/apache/kafka/pull/14314 as
> >> Chris
> >> > > > requested.
> >> > > >
> >> > > > 1. https://issues.apache.org/jira/browse/KAFKA-15425
> >> > > >
> >> > > > ~Satish.
> >> > > >
> >> > > > On Fri, 1 Sept 2023 at 03:59, Chris Egerton  >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > Hi all,
> >> > > > >
> >> > > > > Quick update: I've filed a separate ticket,
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-15425, to track the
> >> > behavior
> >> > > > > change in Admin::listOffsets. For the full history of the ticket,
> >> > it's
> >> > > > > worth reading the comment thread on the old ticket at
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-12879.
> >> > > > >
> >> > > > > I've also published https://github.com/apache/kafka/pull/14314
> >> as a
> >> > > > fairly
> >> > > > > lightweight PR to revert the behavior of Admin::listOffsets
> >> without
> >> > also
> >> > > > > reverting the refactoring to use the internal admin driver API.
> >> Would
> >> > > > > appreciate a review on that if anyone can spare the cycles.
> >> > > > >
> >> > > > > Cheers,
> >> > > > >
> >> > > > > Chris
> >> > > > >
> >> > > > > On Wed, Aug 30, 2023 at 1:01 PM Chris Egerton 
> >> > wrote:
> >> > > > >
> >> > > > > > Hi Satish,
> >> > > > > >
> >> > > > > > Wanted to let you know that KAFKA-12879 (
> >> > > > > > https://issues.apache.org/jira/browse/KAFKA-12879), a breaking
> >> > change
> >> > > > in
> >> > > > > > Admin::listOffsets, has been reintroduced into the code base.
> >> > Since we
> >>

[jira] [Resolved] (KAFKA-15260) RLM Task should wait until RLMM is initialized before copying segments to remote

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-15260.

Resolution: Fixed

> RLM Task should wait until RLMM is initialized before copying segments to 
> remote
> 
>
> Key: KAFKA-15260
> URL: https://issues.apache.org/jira/browse/KAFKA-15260
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Abhijeet Kumar
>Assignee: Abhijeet Kumar
>Priority: Blocker
> Fix For: 3.6.0
>
>
> The RLM Task uploads segment to the remote storage for its leader partitions 
> and after each upload it sends a message 'COPY_SEGMENT_STARTED' to the Topic 
> based RLMM topic and then waits for the TBRLMM to consume the message before 
> continuing.
> If the RLMM is not initialized, TBRLMM may not be able to consume the message 
> within the stipulated time and timeout and RLMM will repeat later. It make 
> take a few mins for the TBRLMM to initialize during which RLM Task will keep 
> timing out.
> Instead the RLM task should wait until RLMM is initialized before attempting 
> to copy segments to remote storage.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-9564) Integration Test framework for Tiered Storage

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-9564.
---
Resolution: Fixed

> Integration Test framework for Tiered Storage
> -
>
> Key: KAFKA-9564
> URL: https://issues.apache.org/jira/browse/KAFKA-9564
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Alexandre Dupriez
>Assignee: Alexandre Dupriez
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Sophie,
Please feel free to add that to 3.6 branch as you say this is a minor
change and will not cause any regressions.

Thanks,
Satish.

On Sat, 2 Sept 2023 at 08:44, Sophie Blee-Goldman
 wrote:
>
> Hey Satish, someone reported a minor bug in the Streams application
> shutdown which was a recent regression, though not strictly a new one: was
> introduced in 3.4 I believe.
>
> The fix seems to be super lightweight and low-risk so I was hoping to slip
> it into 3.6 if that's ok with you? They plan to have the patch tonight.
>
> https://issues.apache.org/jira/browse/KAFKA-15429
>
> On Thu, Aug 31, 2023 at 5:45 PM Satish Duggana 
> wrote:
>
> > Thanks Chris for bringing this issue here and filing the new JIRA for
> > 3.6.0[1]. It seems to be a blocker for 3.6.0.
> >
> > Please help review https://github.com/apache/kafka/pull/14314 as Chris
> > requested.
> >
> > 1. https://issues.apache.org/jira/browse/KAFKA-15425
> >
> > ~Satish.
> >
> > On Fri, 1 Sept 2023 at 03:59, Chris Egerton 
> > wrote:
> > >
> > > Hi all,
> > >
> > > Quick update: I've filed a separate ticket,
> > > https://issues.apache.org/jira/browse/KAFKA-15425, to track the behavior
> > > change in Admin::listOffsets. For the full history of the ticket, it's
> > > worth reading the comment thread on the old ticket at
> > > https://issues.apache.org/jira/browse/KAFKA-12879.
> > >
> > > I've also published https://github.com/apache/kafka/pull/14314 as a
> > fairly
> > > lightweight PR to revert the behavior of Admin::listOffsets without also
> > > reverting the refactoring to use the internal admin driver API. Would
> > > appreciate a review on that if anyone can spare the cycles.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Wed, Aug 30, 2023 at 1:01 PM Chris Egerton  wrote:
> > >
> > > > Hi Satish,
> > > >
> > > > Wanted to let you know that KAFKA-12879 (
> > > > https://issues.apache.org/jira/browse/KAFKA-12879), a breaking change
> > in
> > > > Admin::listOffsets, has been reintroduced into the code base. Since we
> > > > haven't yet published a release with this change (at least, not the
> > more
> > > > recent instance of it), I was hoping we could treat it as a blocker for
> > > > 3.6.0. I'd also like to solicit the input of people familiar with the
> > admin
> > > > client to weigh in on the Jira ticket about whether we should continue
> > to
> > > > preserve the current behavior (if the consensus is that we should, I'm
> > > > happy to file a fix).
> > > >
> > > > Please let me know if you agree that this qualifies as a blocker. I
> > plan
> > > > on publishing a potential fix sometime this week.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Wed, Aug 30, 2023 at 9:19 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >> Please plan to continue merging pull requests associated with any
> > > >> outstanding minor features and stabilization changes to 3.6 branch
> > > >> before September 3rd. Kindly update the KIP's implementation status in
> > > >> the 3.6.0 release notes.
> > > >>
> > > >> Thanks,
> > > >> Satish.
> > > >>
> > > >> On Fri, 25 Aug 2023 at 21:37, Justine Olshan
> > > >>  wrote:
> > > >> >
> > > >> > Hey Satish,
> > > >> > Everything should be in 3.6, and I will update the release plan
> > wiki.
> > > >> > Thanks!
> > > >> >
> > > >> > On Fri, Aug 25, 2023 at 4:08 AM Satish Duggana <
> > > >> satish.dugg...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Justine,
> > > >> > > Adding KIP-890 part-1 to 3.6.0 seems reasonable to me. This part
> > looks
> > > >> > > to be addressing a critical issue of consumers getting stuck.
> > Please
> > > >> > > update the release plan wiki and merge all the required changes
> > to 3.6
> > > >> > > branch.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > Satish.
> > > >> > >
> > &g

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Thanks Chris for bringing this issue here and filing the new JIRA for
3.6.0[1]. It seems to be a blocker for 3.6.0.

Please help review https://github.com/apache/kafka/pull/14314 as Chris
requested.

1. https://issues.apache.org/jira/browse/KAFKA-15425

~Satish.

On Fri, 1 Sept 2023 at 03:59, Chris Egerton  wrote:
>
> Hi all,
>
> Quick update: I've filed a separate ticket,
> https://issues.apache.org/jira/browse/KAFKA-15425, to track the behavior
> change in Admin::listOffsets. For the full history of the ticket, it's
> worth reading the comment thread on the old ticket at
> https://issues.apache.org/jira/browse/KAFKA-12879.
>
> I've also published https://github.com/apache/kafka/pull/14314 as a fairly
> lightweight PR to revert the behavior of Admin::listOffsets without also
> reverting the refactoring to use the internal admin driver API. Would
> appreciate a review on that if anyone can spare the cycles.
>
> Cheers,
>
> Chris
>
> On Wed, Aug 30, 2023 at 1:01 PM Chris Egerton  wrote:
>
> > Hi Satish,
> >
> > Wanted to let you know that KAFKA-12879 (
> > https://issues.apache.org/jira/browse/KAFKA-12879), a breaking change in
> > Admin::listOffsets, has been reintroduced into the code base. Since we
> > haven't yet published a release with this change (at least, not the more
> > recent instance of it), I was hoping we could treat it as a blocker for
> > 3.6.0. I'd also like to solicit the input of people familiar with the admin
> > client to weigh in on the Jira ticket about whether we should continue to
> > preserve the current behavior (if the consensus is that we should, I'm
> > happy to file a fix).
> >
> > Please let me know if you agree that this qualifies as a blocker. I plan
> > on publishing a potential fix sometime this week.
> >
> > Cheers,
> >
> > Chris
> >
> > On Wed, Aug 30, 2023 at 9:19 AM Satish Duggana 
> > wrote:
> >
> >> Hi,
> >> Please plan to continue merging pull requests associated with any
> >> outstanding minor features and stabilization changes to 3.6 branch
> >> before September 3rd. Kindly update the KIP's implementation status in
> >> the 3.6.0 release notes.
> >>
> >> Thanks,
> >> Satish.
> >>
> >> On Fri, 25 Aug 2023 at 21:37, Justine Olshan
> >>  wrote:
> >> >
> >> > Hey Satish,
> >> > Everything should be in 3.6, and I will update the release plan wiki.
> >> > Thanks!
> >> >
> >> > On Fri, Aug 25, 2023 at 4:08 AM Satish Duggana <
> >> satish.dugg...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Justine,
> >> > > Adding KIP-890 part-1 to 3.6.0 seems reasonable to me. This part looks
> >> > > to be addressing a critical issue of consumers getting stuck. Please
> >> > > update the release plan wiki and merge all the required changes to 3.6
> >> > > branch.
> >> > >
> >> > > Thanks,
> >> > > Satish.
> >> > >
> >> > > On Thu, 24 Aug 2023 at 22:19, Justine Olshan
> >> > >  wrote:
> >> > > >
> >> > > > Hey Satish,
> >> > > > Does it make sense to include KIP-890 part 1? It prevents hanging
> >> > > > transactions for older clients. (An optimization and stronger EOS
> >> > > > guarantees will be included in part 2)
> >> > > >
> >> > > > Thanks,
> >> > > > Justine
> >> > > >
> >> > > > On Mon, Aug 21, 2023 at 3:29 AM Satish Duggana <
> >> satish.dugg...@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > > 3.6 branch is created. Please make sure any PRs targeted for 3.6.0
> >> > > > > should be merged to 3.6 branch once those are merged to trunk.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Satish.
> >> > > > >
> >> > > > > On Wed, 16 Aug 2023 at 15:58, Satish Duggana <
> >> satish.dugg...@gmail.com
> >> > > >
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > Hi,
> >> > > > > > Please plan to merge PRs(including the major features) targeted
> >> for
> >> > > > > > 3.6.0 by the end of Aug 20th UTC. Starting from August 21st,
> >> a

[jira] [Resolved] (KAFKA-9565) Implementation of Tiered Storage SPI to integrate with S3

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-9565.
---
Resolution: Won't Fix

> Implementation of Tiered Storage SPI to integrate with S3
> -
>
> Key: KAFKA-9565
> URL: https://issues.apache.org/jira/browse/KAFKA-9565
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Alexandre Dupriez
>Assignee: Ivan Yurchenko
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12458) Implementation of Tiered Storage Integration with Azure Storage (ADLS + Blob Storage)

[Satish Duggana (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-12458.

Resolution: Won't Do

> Implementation of Tiered Storage Integration with Azure Storage (ADLS + Blob 
> Storage)
> -
>
> Key: KAFKA-12458
> URL: https://issues.apache.org/jira/browse/KAFKA-12458
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Israel Ekpo
>Assignee: Israel Ekpo
>Priority: Major
>
> Task to cover integration support for Azure Storage
>  * Azure Blob Storage
>  * Azure Data Lake Store
> Will split task up later into distinct tracks and components



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15420) Kafka Tiered Storage V1

[Satish Duggana (Jira)]

Satish Duggana created KAFKA-15420:
--

 Summary: Kafka Tiered Storage V1
 Key: KAFKA-15420
 URL: https://issues.apache.org/jira/browse/KAFKA-15420
 Project: Kafka
  Issue Type: Improvement
Reporter: Satish Duggana
Assignee: Satish Duggana
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)