Re 1: We don't expose the reuse* methods, because the proposed
OperatorIOMetricGroup is a separate interface from the existing
implementations (which will be renamed and implement the new interface).
Re 2: Currently the plan is to re-use the "new" numByesIn/Out counters
for tasks ("new" because all we are doing is exposing already existing
metrics). We may however change this in the future if we want to report
the byte metrics on an operator level, which is primarily interesting
for async IO or other external connectivity outside of sinks/sources.
On 13/07/2021 12:38, Becket Qin wrote:
Hi Arvid,
Thanks for the proposal. I like the idea of exposing concrete metric group
class so that users can access the predefined metrics.
A few questions are following:
1. When exposing the OperatorIOMetrics to the users, we are also exposing
the reuseInputMetricsForTask to the users. Should we hide these two methods
because users won't have enough information to decide whether the records
IO metrics should be reused by the task or not.
2. Similar to question 1, in the OperatorIOMetricGroup, we are adding
numBytesInCounter and numBytesOutCounter. Should these metrics be reusing
the task level metrics by default?
3. Regarding SourceMetricGroup#setLastFetchTimeGauge(), I am not sure how
it works with the FetchLag. Typically there are two cases when reporting
the fetch lag.
A. The EventTime is known at the point when the record is fetched in
the SplitFetcher, so the fetch lag can be derived and reported immediately.
B. The EventTime is known only after the fetched record was parsed in
the RecordEmitter. In this case, the RecordEmitter needs to get the fetch
time of that particular record.
I am not sure when users set the LastFetchTime in the above two cases. Can
you help elaborate on how users should use it?
Thanks,
Jiangjie (Becket) Qin
On Thu, Jul 8, 2021 at 10:25 PM Arvid Heise <ar...@apache.org> wrote:
Dear devs,
As a continuation and generalization of FLIP-33 (Standardize Connector
Metrics) [1], we'd like to discuss how we actually expose the standardized
operator metrics to users in terms of changes to the API.
Please check out the FLIP [2] and provide feedback.
Best,
Arvid
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-179%3A+Expose+Standardized+Operator+Metrics
