Hi Chesnay, You saved my day! Yes, the WordCount example <https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/wordcount/WordCount.java> runs too fast, about 20ms. I extended its running time, and also set 'metrics.reporter.stsd.interval: 1'. Guess what?! Metrics show up in Datadog!
I will open a PR to add this tip to Metrics page <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html> to help other newbies ramp up faster. Thank you! Bowen On Mon, Mar 13, 2017 at 1:48 AM, Chesnay Schepler <ches...@apache.org> wrote: > Hello, > > yes, Flink can handle custom metrics. It is quite odd that you can see the > system metrics but not your > own; I don't see a problem with your code. > > How long is the job running? Since you create a metric for the flatMap > operation the metric will only > be exposed as long as the flatMap operation is active. Thus, if this > operation takes less than 10 seconds > (the default interval for all reporters) then it may never report it. > > Regards, > Chesnay > > On 13.03.2017 08:48, Bowen Li wrote: > >> Hi guys, >> I'm retrying to send some app related custom metrics from Flink to >> Datadog via StatsD. >> >> I followed https://ci.apache.org/projects/flink/flink-docs- >> release-1.2/monitoring/metrics.html to set up flink-conf.yaml and test >> code >> >> like this >> >> // flink-conf.yaml >> >> metrics.reporters: stsd >> metrics.reporter.stsd.class: org.apache.flink.metrics.stats >> d.StatsDReporter >> metrics.reporter.stsd.host: localhost >> metrics.reporter.stsd.port: 8125 >> >> metrics.scope.jm: bowen.jobmanager >> metrics.scope.jm.job: bowen.jobmanager.bowen.job_name >> metrics.scope.tm: bowen.taskmanager.<tm_id> >> >> metrics.scope.tm.job: bowen.taskmanager.<tm_id>.<job_name> >> metrics.scope.task: bowen.taskmanager.<tm_id>.<job_name>.<task_name>.< >> subtask_index> >> metrics.scope.operator: bowen.taskmanager.<tm_id>.< >> job_name>.<operator_name>.<subtask_index> >> >> >> // test code, by modifying WordCount example >> >> >> public static final class Tokenizer extends >> RichFlatMapFunction<String, Tuple2<String, Integer>> { >> private static final long serialVersionUID = 1L; >> >> private Counter counter; >> >> @Override >> public void open(Configuration config) { >> getRuntimeContext() >> .getMetricGroup() >> .addGroup("bowen.test") >> .gauge("bowen.test.flink", new Gauge<Integer>() { >> @Override >> public Integer getValue() { >> return 100; >> } >> }); // test custom metrics >> >> counter = getRuntimeContext() >> .getMetricGroup() >> .addGroup("bowen.test") >> .counter("bowen.flink.metric"); // test custom metrics >> counter.inc(100); >> } >> >> @Override >> public void flatMap(String value, Collector<Tuple2<String, Integer>> >> out) >> throws Exception { >> // normalize and split the line >> String[] tokens = value.toLowerCase().split("\\W+"); >> >> // emit the pairs >> for (String token : tokens) { >> if (token.length() > 0) { >> out.collect(new Tuple2<String, Integer>(token, 1)); >> } >> } >> >> counter.inc(100); >> } >> } >> >> I found my Datadog received all system scope metrics, but non of my >> custom metric. I researched all night but gained no progress. What did I >> do >> wrong? Flink is able to handle custom metrics right? I'd really appreciate >> some guidance on sending custom metrics! >> >> Thank you very much! >> Bowen >> >> >