Hi Philip, Thanks for your vote and interest in the KIP. KIP-714 does not introduce any new client metrics, and that’s intentional. It does tell how that all of the client metrics can have their names transformed into equivalent "telemetry metric names”, and then potentially used in metrics subscriptions.
I am interested in the idea of client’s leader epoch in this context, but I don’t have an immediate plan for how best to do this, and it would take another KIP to enhance existing metrics or introduce some new ones. Those would then naturally be applicable to the metrics push introduced in KIP-714. In a similar vein, there are no existing client metrics specifically for auto-commit. We could add them to Kafka, but I really think this is just an example of asynchronous commit in which the application has decided not to specify when the commit should begin. It is possible to increase the cadence of pushing by modifying the interval.ms configuration property of the CLIENT_METRICS resource. There is an “assigned-partitions” metric for each consumer, but not one for active partitions. We could add one, again as a follow-on KIP. I take your point about holding on to a connection in a channel which might experience congestion. Do you have a suggestion for how to improve on this? For example, the client does have the concept of a least-loaded node. Maybe this is something we should investigate in the implementation and decide on the best approach. In general, I think sticking with the same node for consecutive pushes is best, but if you choose the “wrong” node to start with, it’s not ideal. Thanks, Andrew > On 8 Sep 2023, at 19:29, Philip Nee <philip...@gmail.com> wrote: > > Hey Andrew - > > +1 but I don't have a binding vote! > > It took me a while to go through the KIP. Here are some of my notes during > the reading: > > *Metrics* > - Should we care about the client's leader epoch? There is a case where the > user recreates the topic, but the consumer thinks it is still the same > topic and therefore, attempts to start from an offset that doesn't exist. > KIP-848 addresses this issue, but I can still see some potential benefits > from knowing the client's epoch information. > - I assume poll idle is similar to poll interval: I needed to read the > description a few times. > - I don't have a clear use case in mind for the commit latency, but I do > think sometimes people lack clarity about how much progress was tracked by > the auto-commit. Would tracking auto-commit-related metrics be useful? I > was thinking: the last offset committed or the actual cadence in ms. > - Are there cases when we need to increase the cadence of telemetry data > push? i.e. variable interval. > - Thanks for implementing the randomized initial metric push; I think it is > really important. > - Is there a potential use case for tracking the number of active > partitions? The consumer can pause partitions via API, during revocation, > or during offset reset for the stream. > > *Connections*: > - The KIP stated that it will keep the same connection until the connection > is disconnected. I wonder if that could potentially cause congestion if it > is already a busy channel, which leads to connection timeout and > subsequently disconnection. > > Thanks, > P > > On Fri, Sep 8, 2023 at 4:15 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > >> Bumping the voting thread for KIP-714. >> >> So far, we have: >> Non-binding +2 (Milind and Kirk), non-binding -1 (Ryanne) >> >> Thanks, >> Andrew >> >>> On 4 Aug 2023, at 09:45, Andrew Schofield <andrew_schofi...@live.com> >> wrote: >>> >>> Hi, >>> After almost 2 1/2 years in the making, I would like to call a vote for >> KIP-714 ( >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability >> ). >>> >>> This KIP aims to improve monitoring and troubleshooting of client >> performance by enabling clients to push metrics to brokers. >>> >>> I’d like to thank everyone that participated in the discussion, >> especially the librdkafka team since one of the aims of the KIP is to >> enable any client to participate, not just the Apache Kafka project’s Java >> clients. >>> >>> Thanks, >>> Andrew
