Thank you. I understand that filtering metrics is not possible but i can configure it for only the memory metrics listed here? : https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#memory
Currently, it's publishing all the container/operator metrics which is what I want to avoid. Thanks! On Wed, Feb 16, 2022 at 11:35 PM Chesnay Schepler <ches...@apache.org> wrote: > It is currently not possible to select metrics. > > What you can do however is create a custom reporter that wraps the StatsD > reporter which does this filtering. > > On 16/02/2022 17:41, Diwakar Jha wrote: > > > Hello, > > Could someone please help! I'm trying to publish only these three metrics > per tasknode > Status.JVM.Memory.Heap.Used > Status.JVM.Memory.Heap.Committed > Status.JVM.Memory.NonHeap.Max > > But, with my current setting I see all Flink metrics getting published. > Please let me know if I need to provide any other information. > > Thank you! > > > ---------- Forwarded message --------- > From: Diwakar Jha <diwakar.n...@gmail.com> > Date: Tue, Feb 15, 2022 at 1:31 PM > Subject: How to get memory specific metrics for tasknodes > To: user <user@flink.apache.org> > > > Hello, > > I'm running Flink 1.11 on AWS EMR using the Yarn application. I'm trying > to access memory metrics(Heap.Max, Heap.Used) per tasknode in CloudWatch. I > have 50 tasknodes and it creates Millions of metrics(including per > operator) though I need only a few metrics per tasknode (Heap.Max, > Heap.Used). It is way too much than my current cloudwatch limit and I also > don't need so many metrics. > Could someone please help me how to get only the tasknode memory specific > metrics ? > I'm referring to this doc : > https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html#memory > > I used the following approach to enable Flink metrics. > 1. Enable Flink Metrics > copy /opt/flink-metrics-statsd-x.x.jar into the /lib folder of your Flink > distribution > 2. Add StatsD metric reporter in Flink-conf to send to CloudWatch Agent's > StatsD interface > metrics.reporters: stsd > metrics.reporter.stsd.factory.class: > org.apache.flink.metrics.statsd.StatsDReporterFactory > metrics.reporter.stsd.host: localhost > metrics.reporter.stsd.port: 8125 > 3. Setup tasknode scope > metrics.scope.tm: taskmanager > 4. setup Cloudwatch agent to publish the metrics > "metrics":{ > "namespace": "CustomeNamespace/FlinkMemoryMetrics", > "metrics_collected":{ > "statsd":{ > "service_address":":8125", > "metrics_collection_interval":60, > "metrics_aggregation_interval":300 > } > } > }, > > Thanks! > > > >
