Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, If there are no more comments, I'll open a VOTE thread. -- Kamal On Sat, May 4, 2024 at 8:39 AM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Luke, > > Thanks for the review! > > DelayedFetch and DelayedRemoteFetch are orthogonal calls >

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Luke, Thanks for the review! DelayedFetch and DelayedRemoteFetch are orthogonal calls . Only one of them will be active in a given FETCH request. The fetch request might take more than 500

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi Kamal, Thanks for the KIP! Sorry for the late review. Overall LGTM! Just 1 question: If one fetch request contains 2 partitions: [p1, p2] fetch.max.wait.ms: 500, remote.fetch.max.wait.ms: 1000 And now, p1 fetch offset is the log end offset and has no new data coming, and p2 fetch offset is

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Christo, We have localTimeMs, remoteTimeMs, and totalTimeMs as part of the FetchConsumer request metric. kafka.network:type=RequestMetrics,name={LocalTimeMs|RemoteTimeMs|TotalTimeMs},request={Produce|FetchConsumer|FetchFollower} RemoteTimeMs refers to the amount of time spent in the purgatory

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Heya! Is it difficult to instead add the metric at kafka.network:type=RequestMetrics,name=TieredStorageMs (or some other name=*)? Alternatively, if it is difficult to add it there, is it possible to add 2 metrics, one at the RequestMetrics level (even if it is total-time-ms - (all other times))

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Christo, Updated the KIP with the remote fetch latency metric. Please take another look! -- Kamal On Sun, Apr 28, 2024 at 12:23 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Federico, > > Thanks for the suggestion! Updated the config name to " >

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi Federico, Thanks for the suggestion! Updated the config name to " remote.fetch.max.wait.ms". Christo, Good point. We don't have the remote-read latency metrics to measure the performance of the remote read requests. I'll update the KIP to emit this metric. -- Kamal On Sat, Apr 27, 2024 at

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Heya Kamal, I quite like the proposal and would support it! However, today I don't think we have a metric which shows the latency of fetch requests which are served from remote, am I wrong? I looked at both

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi Kamal, it looks like all TS configurations starts with "remote." prefix, so I was wondering if we should name it "remote.fetch.max.wait.ms". On Fri, Apr 26, 2024 at 7:07 PM Kamal Chandraprakash wrote: > > Hi all, > > If there are no more comments, I'll start a vote thread by tomorrow. >

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, If there are no more comments, I'll start a vote thread by tomorrow. Please review the KIP. Thanks, Kamal On Sat, Mar 30, 2024 at 11:08 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi all, > > Bumping the thread. Please review this KIP. Thanks! > > On Thu, Feb 1,

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, Bumping the thread. Please review this KIP. Thanks! On Thu, Feb 1, 2024 at 9:11 PM Kamal Chandraprakash < kamal.chandraprak...@gmail.com> wrote: > Hi Jorge, > > Thanks for the review! Added your suggestions to the KIP. PTAL. > > The `fetch.max.wait.ms` config will be also applicable for

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi Jorge, Thanks for the review! Added your suggestions to the KIP. PTAL. The `fetch.max.wait.ms` config will be also applicable for topics enabled with remote storage. Updated the description to: ``` The maximum amount of time the server will block before answering the fetch request when it is

Re: [DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi Kamal, Thanks for this KIP! It should help to solve one of the main issues with tiered storage at the moment that is dealing with individual consumer configurations to avoid flooding logs with interrupted exceptions. One of the topics discussed in [1][2] was on the semantics of `

[DISCUSS] KIP-1018: Introduce max remote fetch timeout config

Hi all, I have opened a KIP-1018 to introduce dynamic max-remote-fetch-timeout broker config to give more control to the operator.