Programmatically get live values of accumulators

[Gwenhael Pasquiers]

Hi, and best wishes for the year to come :)

I'd like to be able to programmatically get the (live) values of accumulators 
in order to send them using a statsd (or another) client in the JobManager of a 
yarn-deployed application. I say live because I'd like to use that in streaming 
(24/7) applications, and send live stats, I cannot way for the application to 
end.

I've seen that there is a json API (I'd prefer no to have my app poll itself).
I've seen some code on github (tests files) where it's done using the 
underlying akka framework, I don't mind doing it the same way and creating an 
actor to get notifications messages, but I don't know the best way, and there 
probably is a better one.

Thanks in advance,

Gwenhaël PASQUIERS

Re: Programmatically get live values of accumulators

[Jamie Grier]

Hi Gwenhael,

I think what you actually want is to use the Apache Flink metrics
interface.  See the following:
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html

Sending metrics to StatsD is supported out-of-the-box.

-Jamie


On Mon, Jan 2, 2017 at 1:34 AM, Gwenhael Pasquiers <
gwenhael.pasqui...@ericsson.com> wrote:

> Hi, and best wishes for the year to come J
>
>
>
> I’d like to be able to programmatically get the (live) values of
> accumulators in order to send them using a statsd (or another) client in
> the JobManager of a yarn-deployed application. I say live because I’d like
> to use that in streaming (24/7) applications, and send live stats, I cannot
> way for the application to end.
>
>
>
> I’ve seen that there is a json API (I’d prefer no to have my app poll
> itself).
>
> I’ve seen some code on github (tests files) where it’s done using the
> underlying akka framework, I don’t mind doing it the same way and creating
> an actor to get notifications messages, but I don’t know the best way, and
> there probably is a better one.
>
>
>
> Thanks in advance,
>
>
>
> Gwenhaël PASQUIERS
>



-- 

Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier 
ja...@data-artisans.com

RE: Programmatically get live values of accumulators

[Gwenhael Pasquiers]

Indeed, that looks like what we need. We currently rely on flink 1.0.1, that 
feature might be a good reason to update our flink version. I’ll test it. Many 
thanks ! !

From: Jamie Grier [mailto:ja...@data-artisans.com]
Sent: lundi 2 janvier 2017 20:56
To: user@flink.apache.org
Subject: Re: Programmatically get live values of accumulators

Hi Gwenhael,

I think what you actually want is to use the Apache Flink metrics interface.  
See the following: 
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/metrics.html

Sending metrics to StatsD is supported out-of-the-box.

-Jamie


On Mon, Jan 2, 2017 at 1:34 AM, Gwenhael Pasquiers 
mailto:gwenhael.pasqui...@ericsson.com>> wrote:
Hi, and best wishes for the year to come ☺

I’d like to be able to programmatically get the (live) values of accumulators 
in order to send them using a statsd (or another) client in the JobManager of a 
yarn-deployed application. I say live because I’d like to use that in streaming 
(24/7) applications, and send live stats, I cannot way for the application to 
end.

I’ve seen that there is a json API (I’d prefer no to have my app poll itself).
I’ve seen some code on github (tests files) where it’s done using the 
underlying akka framework, I don’t mind doing it the same way and creating an 
actor to get notifications messages, but I don’t know the best way, and there 
probably is a better one.

Thanks in advance,

Gwenhaël PASQUIERS



--

Jamie Grier
data Artisans, Director of Applications Engineering
@jamiegrier<https://twitter.com/jamiegrier>
ja...@data-artisans.com<mailto:ja...@data-artisans.com>