[jira] [Commented] (KAFKA-12713) Report "REAL" broker/consumer fetch latency

[Ming Liu (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331590#comment-17331590
 ] 

Ming Liu commented on KAFKA-12713:
--

The idea is like this,  we can:
1. Add waitTimeMS in FetchResponse.
2. If the fetch has to wait in purgatory due to either 
replica.fetch.wait.max.ms or fetch.min.bytes, then it will fill the waitTimeMS 
in FetchResponse.
3. In updateRequestMetrics() function, we will special-process the Fetch 
response, and remove the waitTimeMS out of RemoteTime and TotalTime.

> Report "REAL" broker/consumer fetch latency
> ---
>
> Key: KAFKA-12713
> URL: https://issues.apache.org/jira/browse/KAFKA-12713
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ming Liu
>Priority: Major
>
> The fetch latency is an important metrics to monitor for the cluster 
> performance. With ACK=ALL, the produce latency is affected primarily by 
> broker fetch latency.
> However, currently the reported fetch latency didn't reflect the true fetch 
> latency because it sometimes need to stay in purgatory and wait for 
> replica.fetch.wait.max.ms when data is not available. This greatly affect the 
> real P50, P99 etc. 
> I like to propose a KIP to be able track the real fetch latency for both 
> broker follower and consumer. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-12713) Report "REAL" broker/consumer fetch latency

[Ismael Juma (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-12713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331311#comment-17331311
 ] 

Ismael Juma commented on KAFKA-12713:
-

It would definitely be useful to have a better fetch latency metric. It's not 
obvious how it should work, however. Do you have suggestions?

> Report "REAL" broker/consumer fetch latency
> ---
>
> Key: KAFKA-12713
> URL: https://issues.apache.org/jira/browse/KAFKA-12713
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ming Liu
>Priority: Major
>
> The fetch latency is an important metrics to monitor for the cluster 
> performance. With ACK=ALL, the produce latency is affected primarily by 
> broker fetch latency.
> However, currently the reported fetch latency didn't reflect the true fetch 
> latency because it sometimes need to stay in purgatory and wait for 
> replica.fetch.wait.max.ms when data is not available. This greatly affect the 
> real P50, P99 etc. 
> I like to propose a KIP to be able track the real fetch latency for both 
> broker follower and consumer. 
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)