To answer your question on your debugging code, your reporter has a bug:
log.info("STATSD SENDING: ", name, value);
should be ->
log.info("STATSD SENDING: {} {}", name, value);
-
On Sat, Jun 1, 2019 at 7:30 PM Padarn Wilson <pad...@gmail.com> wrote:
> Thanks both: Using the the inbuilt Slf4j reporter is a great idea, I will
> do this.
>
> @Peter.Groesbeck - appreciate the config. This looks very similar to what
> I had, but if it is working for you perhaps there is something else missing
> from our EMR setup. Will go back and double check the connectivity from all
> the instances.
>
>
>
> On Thu, May 30, 2019 at 9:42 PM Peter Groesbeck <peter.groesb...@gmail.com>
> wrote:
>
>> Hi Padarn for what it's worth I am using DataDog metrics on EMR with
>> Flink 1.7.1 and this here my flink-conf configuration:
>>
>> - Classification: flink-conf
>> ConfigurationProperties:
>> metrics.reporter.dghttp.class:
>> org.apache.flink.metrics.datadog.DatadogHttpReporter
>> metrics.reporter.dghttp.apikey: <your-datadog-key>
>> metrics.reporter.dghttp.tags:
>> 'service:myservice,env:prod,region:us-east-1'
>> metrics.scope.jm: 'jobmanager'
>> metrics.scope.jm.job: 'jobmanager'
>> metrics.scope.operator: 'taskmanager.<job_name>.<operator_name>'
>> metrics.scope.task: 'taskmanager.<job_name>.<task_name>'
>> metrics.scope.tm: 'taskmanager'
>> metrics.scope.tm.job: 'taskmanager'
>> Configurations: []
>>
>>
>> On Thu, May 30, 2019 at 6:46 AM Yun Tang <myas...@live.com> wrote:
>>
>>> Hi Padarn
>>>
>>> If you want to verify why no metrics sending out, how about using the
>>> built-in Slf4j reporter [1] which would record metrics in logs.
>>> If you could view the metrics after enabled slf4j-reporter, you could
>>> then compare the configurations.
>>>
>>> Best
>>> Yun Tang
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#slf4j-orgapacheflinkmetricsslf4jslf4jreporter
>>>
>>> ------------------------------
>>> *From:* Padarn Wilson <padarn.wil...@grab.com>
>>> *Sent:* Thursday, May 30, 2019 18:20
>>> *To:* user
>>> *Subject:* [External] Flink 1.7.1 on EMR metrics
>>>
>>> Hello all,
>>>
>>> I am trying to run Flink 1.7.1 on EMR and having some trouble with
>>> metric reporting.
>>>
>>> I was using the DataDogHttpReporter, but have also tried the
>>> StatsDReporter, but with both was seeing no metrics being collected.
>>>
>>> To debug this I implemented my own reporter (based on StatsDReporter)
>>> and logged the name of the metric being sent:
>>>
>>> private void send(final String name, final String value) {
>>> log.info("STATSD SENDING: ", name, value);
>>> try {
>>> String formatted = String.format("%s:%s|g", name, value);
>>> byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
>>> socket.send(new DatagramPacket(data, data.length, this.address));
>>> }
>>> catch (IOException e) {
>>> LOG.error("unable to send packet to statsd at '{}:{}'",
>>> address.getHostName(), address.getPort());
>>> }
>>> }
>>>
>>>
>>> This code is certainly reached, because in my log I see a lot of this:
>>>
>>> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>> 2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>> 2019-05-30 10:18:40,352 INFO com.grab.statsd.StatsDReporter
>>> - STATSD SENDING:
>>>
>>> As you can see, the name and value for the metric being reported is empty.
>>>
>>>
>>> And the logs show everything initialized fine with no error:
>>>
>>> 2019-05-30 10:18:30,342 INFO
>>> org.apache.flink.runtime.metrics.MetricRegistryImpl - Configuring
>>> stsd with {port=8125, host=127.0.0.1, class=com.grab.statsd.StatsDReporter}.
>>> 2019-05-30 10:18:30,344 INFO com.grab.statsd.StatsDReporter
>>> - Configured StatsDReporter with {host:127.0.0.1, port:8125}
>>> 2019-05-30 10:18:30,344 INFO
>>> org.apache.flink.runtime.metrics.MetricRegistryImpl -
>>> Periodically reporting metrics in intervals of 10 SECONDS for reporter stsd
>>> of type com.grab.statsd.StatsDReporter.
>>>
>>>
>>>
>>>
>>> Has anyone else tried to work with Flink and metrics on EMR 1.7.1 (latest
>>> version on EMR). If so, any pointers as to what could be set up incorrectly?
>>>
>>>
>>>
>>>
>>> *Grab is hiring. Learn more at https://grab.careers
>>> <https://grab.careers/>*
>>>
>>> By communicating with Grab Inc and/or its subsidiaries, associate
>>> companies and jointly controlled entities (“Grab Group”), you are deemed to
>>> have consented to the processing of your personal data as set out in the
>>> Privacy Notice which can be viewed at https://grab.com/privacy/
>>>
>>> This email contains confidential information and is only for the
>>> intended recipient(s). If you are not the intended recipient(s), please do
>>> not disseminate, distribute or copy this email Please notify Grab Group
>>> immediately if you have received this by mistake and delete this email from
>>> your system. Email transmission cannot be guaranteed to be secure or
>>> error-free as any information therein could be intercepted, corrupted,
>>> lost, destroyed, delayed or incomplete, or contain viruses. Grab Group do
>>> not accept liability for any errors or omissions in the contents of this
>>> email arises as a result of email transmission. All intellectual property
>>> rights in this email and attachments therein shall remain vested in Grab
>>> Group, unless otherwise provided by law.
>>>
>>
