Hello. Related metric issue. I am looking for a way to expose flink metrics via opentelemerty to the gcp could monitoring dashboard. Does anyone has experience with that?
Thanks בתאריך יום ו׳, 18 בפבר׳ 2022, 21:55, מאת Chesnay Schepler < ches...@apache.org>: > As I said, this is not possible. In the custom reporter you can filter out > all the metrics you do not want. > > On 18/02/2022 19:44, Diwakar Jha wrote: > > Thank you. I understand that filtering metrics is not possible but i can > configure it for only the memory metrics listed here? : > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#memory > > Currently, it's publishing all the container/operator metrics which is > what I want to avoid. Thanks! > > > On Wed, Feb 16, 2022 at 11:35 PM Chesnay Schepler <ches...@apache.org> > wrote: > >> It is currently not possible to select metrics. >> >> What you can do however is create a custom reporter that wraps the StatsD >> reporter which does this filtering. >> >> On 16/02/2022 17:41, Diwakar Jha wrote: >> >> >> Hello, >> >> Could someone please help! I'm trying to publish only these three >> metrics per tasknode >> Status.JVM.Memory.Heap.Used >> Status.JVM.Memory.Heap.Committed >> Status.JVM.Memory.NonHeap.Max >> >> But, with my current setting I see all Flink metrics getting published. >> Please let me know if I need to provide any other information. >> >> Thank you! >> >> >> ---------- Forwarded message --------- >> From: Diwakar Jha <diwakar.n...@gmail.com> >> Date: Tue, Feb 15, 2022 at 1:31 PM >> Subject: How to get memory specific metrics for tasknodes >> To: user <user@flink.apache.org> >> >> >> Hello, >> >> I'm running Flink 1.11 on AWS EMR using the Yarn application. I'm trying >> to access memory metrics(Heap.Max, Heap.Used) per tasknode in CloudWatch. I >> have 50 tasknodes and it creates Millions of metrics(including per >> operator) though I need only a few metrics per tasknode (Heap.Max, >> Heap.Used). It is way too much than my current cloudwatch limit and I also >> don't need so many metrics. >> Could someone please help me how to get only the tasknode memory specific >> metrics ? >> I'm referring to this doc : >> https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html#memory >> >> I used the following approach to enable Flink metrics. >> 1. Enable Flink Metrics >> copy /opt/flink-metrics-statsd-x.x.jar into the /lib folder of your >> Flink distribution >> 2. Add StatsD metric reporter in Flink-conf to send to CloudWatch >> Agent's StatsD interface >> metrics.reporters: stsd >> metrics.reporter.stsd.factory.class: >> org.apache.flink.metrics.statsd.StatsDReporterFactory >> metrics.reporter.stsd.host: localhost >> metrics.reporter.stsd.port: 8125 >> 3. Setup tasknode scope >> metrics.scope.tm: taskmanager >> 4. setup Cloudwatch agent to publish the metrics >> "metrics":{ >> "namespace": "CustomeNamespace/FlinkMemoryMetrics", >> "metrics_collected":{ >> "statsd":{ >> "service_address":":8125", >> "metrics_collection_interval":60, >> "metrics_aggregation_interval":300 >> } >> } >> }, >> >> Thanks! >> >> >> >> >
