[jira] [Created] (KAFKA-16706) Refactor ReplicationQuotaManager/RLMQuotaManager to eliminate code duplication

Abhijeet Kumar created KAFKA-16706: -- Summary: Refactor ReplicationQuotaManager/RLMQuotaManager to eliminate code duplication Key: KAFKA-16706 URL: https://issues.apache.org/jira/browse/KAFKA-16706

Re: [VOTE] KIP-1023: Follower fetch from tiered offset

gt; On Fri, 26 Apr 2024 at 8:35 PM, Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the KIP. +1 > > > > Jun > > > > On Thu, Apr 25, 2024 at 10:30 PM Abhijeet Kumar < > > abhijeet.cse@gmail.com> > > wrote: > > > > >

[VOTE] KIP-1023: Follower fetch from tiered offset

Hi All, I would like to start the vote for KIP-1023 - Follower fetch from tiered offset The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Follower+fetch+from+tiered+offset Regards. Abhijeet.

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

ments. > > Luke > > On Thu, Apr 25, 2024 at 4:21 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the updated KIP. It looks good to me. > > > > Jun > > > > On Mon, Apr 22, 2024 at 12:08 PM Abhijeet Kumar < > > abhijeet.cs

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

ower? > > > > > > 3. Could we add "Rejected Alternatives" section to the end of the KIP > to > > > add some of them? > > > Like the "ListOffsetRequest" approach VS > "Earliest-Pending-Upload-Offset" > > > approach, or ge

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

we add "Rejected Alternatives" section to the end of the KIP to > add some of them? > Like the "ListOffsetRequest" approach VS "Earliest-Pending-Upload-Offset" > approach, or getting the "Earliest-Pending-Upload-Offset" from remote log > metadata

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

ng epoch. > > > > 4. "Check if the follower replica is empty and if the feature to use > > last-tiered-offset is enabled." > > Why do we need to check if the follower replica is empty? > > > > 5. It can be confirmed by checking if the leader's Log-Sta

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

Local-Log-Start-Offset? > Missed this detail. The follower will need to call the leader APi to fetch the EarliestLocal offset for this. > Jun > > On Sat, Mar 30, 2024 at 5:51 AM Abhijeet Kumar > > wrote: > > > Hi Christo, > > > > Thanks for rev

Re: [DISCUSS] KIP-1023: Follower fetch from tiered offset

> back to a follower - earliest-pending-upload-offset or the > latest-tiered-offset? If it is the second, then I believe KIP-1005 ought to > already be exposing that offset as part of the ListOffsets API, no? > > Best, > Christo > > On Wed, 27 Mar 2024 at 18:23, Abhijeet Kumar >

[DISCUSS] KIP-1023: Follower fetch from tiered offset

Hi All, I have created KIP-1023 to introduce follower fetch from tiered offset. This feature will be helpful in significantly reducing Kafka rebalance/rebuild times when the cluster is enabled with tiered storage.

Re: [VOTE] KIP-956: Tiered Storage Quotas

t; > > > > -- > > > > Kamal > > > > > > > > On Fri, Mar 8, 2024 at 11:02 PM Jun Rao > > > wrote: > > > > > > > > > Hi, Abhijeet, > > > > > > > > > > Thanks for the KIP.

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

the configs: > - RLMCopyQuotaManager > - RLMFetchQuotaManager > > > On Fri, 8 Mar 2024 at 13:43, Abhijeet Kumar > wrote: > > > Thank you all for your comments. As all the comments in the thread are > > addressed, I am starting a Vote thread for the KIP. Please have a look. >

[VOTE] KIP-956: Tiered Storage Quotas

Hi All, I would like to start the vote for KIP-956 - Tiered Storage Quotas The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas Regards. Abhijeet.

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

another look, and it LGTM now! > > Thanks. > Luke > > On Tue, Mar 5, 2024 at 2:50 AM Jun Rao wrote: > > > Hi, Abhijeet, > > > > Thanks for the reply. Sounds good to me. > > > > Jun > > > > > > On Sat, Mar 2, 2024 at 7:40 PM Abhijeet Kumar &

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

/write whereas the metrics use fetch/copy. Could > we make them consistent? > > Jun > > On Wed, Feb 28, 2024 at 6:49 AM Abhijeet Kumar > > wrote: > > > Hi Jun, > > > > Clarified the meaning of the two metrics. Also updated the KIP. > > > > kafka.lo

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

ger, name=RemoteFetchThrottleTime > kafka.log.remote:type=RemoteLogManager, name=RemoteCopyThrottleTime > > Jun > > On Tue, Feb 27, 2024 at 1:39 AM Abhijeet Kumar > > wrote: > > > Hi Jun, > > > > The existing quota system for consumers is

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

tive Kafka topics.". However, this > can happen with or without tier storage and the current quota system for > consumers is designed for solving this exact problem. Could you explain the > usage of this additional quota? > > Thanks, > > Jun > > On Mon, Feb 12, 2024 at

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

we handle the situation when a segment is still waiting for > > offloading while this segment is expired and eligible to be deleted? > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and > > just check it each time the RLMTask runs? > > > >

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

maller tasks is already being proposed in KIP-950. The two tasks will need some coordination at a topic-partition level to handle such cases as pointed out. For this particular case, the upload task could skip uploading the segment if it is already eligible for deletion. > Thank you. > Luke

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

uation when a segment is still waiting for > > > offloading while this segment is expired and eligible to be deleted? > > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and > > > just check it each time the RLMTask runs? > > > > >

[DISCUSS] KIP-956: Tiered Storage Quotas

Hi All, I have created KIP-956 for defining read and write quota for tiered storage. https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas Feedback and suggestions are welcome. Regards, Abhijeet.

[jira] [Resolved] (KAFKA-15181) Race condition on partition assigned to TopicBasedRemoteLogMetadataManager

[ https://issues.apache.org/jira/browse/KAFKA-15181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijeet Kumar resolved KAFKA-15181. Resolution: Fixed > Race condition on partition assig

[jira] [Resolved] (KAFKA-15261) ReplicaFetcher thread should not block if RLMM is not initialized

[ https://issues.apache.org/jira/browse/KAFKA-15261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhijeet Kumar resolved KAFKA-15261. Resolution: Fixed > ReplicaFetcher thread should not block if RLMM is not initiali

[jira] [Created] (KAFKA-15405) Create a new error code to indicate a resource is not ready yet

Abhijeet Kumar created KAFKA-15405: -- Summary: Create a new error code to indicate a resource is not ready yet Key: KAFKA-15405 URL: https://issues.apache.org/jira/browse/KAFKA-15405 Project: Kafka

[jira] [Created] (KAFKA-15293) Update metrics doc to add tiered storage metrics

Abhijeet Kumar created KAFKA-15293: -- Summary: Update metrics doc to add tiered storage metrics Key: KAFKA-15293 URL: https://issues.apache.org/jira/browse/KAFKA-15293 Project: Kafka Issue

Re: [VOTE] KIP-930: Tiered Storage Metrics

to it yet. > > > > -- > > Divij Vaidya > > > > > > > > On Tue, Jul 25, 2023 at 6:45 PM Satish Duggana > wrote: > > > > > > +1 for the KIP. > > > > > > Thanks, > > > Satish. > > > > > > On Tue

[jira] [Created] (KAFKA-15261) ReplicaFetcher thread should not block if RLMM is not initialized

Abhijeet Kumar created KAFKA-15261: -- Summary: ReplicaFetcher thread should not block if RLMM is not initialized Key: KAFKA-15261 URL: https://issues.apache.org/jira/browse/KAFKA-15261 Project: Kafka

[jira] [Created] (KAFKA-15260) RLM Task should wait until RLMM is initialized before copying segments to remote

Abhijeet Kumar created KAFKA-15260: -- Summary: RLM Task should wait until RLMM is initialized before copying segments to remote Key: KAFKA-15260 URL: https://issues.apache.org/jira/browse/KAFKA-15260

Re: [VOTE] KIP-930: Tiered Storage Metrics

Please find the updated link to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics Updated the KIP as per our conversation on the discussion thread. On Tue, Jul 25, 2023 at 11:29 AM Abhijeet Kumar wrote: > Hi All, > > I w

Re: [DISCUSS] KIP-930: Tiered Storage Metrics

set and log-end-offset metrics. Should we > expose local-log-start-offset and highest-offset-uploaded-to-remote-storage > as metric? > > Thanks, > Kamal > > On Mon, Jul 24, 2023 at 2:08 PM Abhijeet Kumar > > wrote: > > > Hi All, > > > > I created K

[VOTE] KIP-930: Tiered Storage Metrics

Hi All, I would like to start the vote for KIP-930 Tiered Storage Metrics. The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Tiered+Storage+Metrics Regards Abhijeet.

[jira] [Created] (KAFKA-15245) Improve Tiered Storage Metrics

Abhijeet Kumar created KAFKA-15245: -- Summary: Improve Tiered Storage Metrics Key: KAFKA-15245 URL: https://issues.apache.org/jira/browse/KAFKA-15245 Project: Kafka Issue Type: Sub-task

[DISCUSS] KIP-930: Tiered Storage Metrics

Hi All, I created KIP-930 for adding RemoteIndexCache stats and also to rename some tiered storage metrics added as part of KIP-405 to remove ambiguity.

[DISCUSS] KIP-930: Tiered Storage Metrics

Hi All, I created KIP-930 for adding RemoteIndexCache stats and also to rename some tiered storage metrics added as part of KIP-405 to remove ambiguity.

[jira] [Created] (KAFKA-15236) Rename Remote Storage metrics to remove ambiguity

Abhijeet Kumar created KAFKA-15236: -- Summary: Rename Remote Storage metrics to remove ambiguity Key: KAFKA-15236 URL: https://issues.apache.org/jira/browse/KAFKA-15236 Project: Kafka Issue