Hi Mason, We always appreciate contributions but I think that this metric reporter would need to be beefed up quite a bit to be of general use. In this case, it works because there is an existing relatively narrow group to deny, but there are plenty of other cases where this isn't the case (like most Flink-provided job/task/operator metrics): - "I don't want the KafkaConsumer metrics, except that exact once" (so also an include list?), or - "I only want Kafka metrics from the JM" (rules depending on the scope?), or - "I don't care about perSecond IO metrics, just give me the raw counts" (support for patterns?).
To support those we'd first need some form of categorization (which MetricGroups could be, but currently aren't); we can't expect users to setup a deny list for 50 metrics as it just doesn't scale nor is it maintainable for them, especially if we add more metrics in the future. If you are still interested: the first step would be a ticket on https://issues.apache.org/jira/ where we would discuss some options until we reach an agreement. Then, you would be assigned and can start to implement. Alternatively, you could also write a (Flink) blog post where you outline your solution and provide your code. Then other advanced users could follow your approach and implement their own version of reporter that fits their needs. On Mon, Aug 23, 2021 at 7:07 PM Mason Chen <mas.chen6...@gmail.com> wrote: > Sweet, I suspected it but I thought I might ask anyway. > > Consequently, I've implemented a deny list feature for my reporter (based > on groupNameKey and metricName). The reporter will skip reporting metrics > if a metric's variables set contains keys that map to the groupNameKey and > if the metric has a name equal to the specified metricName. > > Configurations are specified as follows > `groupNameKey1:metricName1;groupNameKey2:metricName2`. Thus, I can deny > list KafkaConsumer (group name key) and committed_offsets (metric name) > which correspond to the legacy kafka metrics. > > Would Flink appreciate this as a contribution? I can see this being used > generically over all reporters. > > Best, > Mason > > On Mon, Aug 23, 2021 at 8:21 AM Arvid Heise <ar...@apache.org> wrote: > >> Hi Mason, >> >> I'm afraid it's an all-or-nothing. Either you get the proxied metrics >> with all partitions or none. >> >> You could also implement a custom MetricReporter that delegates to your >> actual reporter and filters the respective metrics. >> >> Best, >> >> Arvid >> >> On Fri, Aug 20, 2021 at 8:16 AM Mason Chen <mas.chen6...@gmail.com> >> wrote: >> >>> FYI, I'm referring to the legacy offsets metric gauges. >>> >>> On Thu, Aug 19, 2021 at 4:53 PM Mason Chen <mas.chen6...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> We have found that the per partition Kafka metrics contributes to a lot >>>> of metrics being indexed by our metrics system. >>>> >>>> We would still like to have the proxied kafka metrics from the kafka >>>> clients library. Is there a flag to only exclude Flink's additional Kafka >>>> metrics? >>>> >>>> Best, >>>> Mason >>>> >>>>
