1) None that I'm aware of.
2) You should use counters.
3) No, counters are not checkpointed, but you could store the value in
state yourself.
4) None that I'm aware of that doesn't require modifications to the
application logic.
How long does your job run for, and how do you access metrics?
On 27/06/2019 17:36, M Singh wrote:
Hi:
I need to collect application metrics which are counts (per unit of
time eg: minute) for certain events. There are two ways of doing this:
1. Create separate streams (using split stream etc) in the application
explicitly, then aggregate the counts in a window and save them. This
mixes metrics collection with application logic and making the
application logic complex.
2. Use Flink metrics framework (counter, guage, etc) to save metrics
I have a very small test with 2 events but when I run the application
the counters are not getting saved (they show value 0) even though
that part of the code is being executed. I do see the numRecordsIn
counters being updated from the source operator. I've also tried
incrementing the count by 10 (instead of 1) every time the function
gets execute but still the counts remain 0.
Here is snippet of the code:
dataStream.map(new RichMapFunction<String, String>() {
protected Counter counter;
public void open(Configuration parameters) {
counter =
getRuntimeContext().getMetricGroup().addGroup("test",
"split").counter("success");
}
@Override
public String map(String value) throws Exception {
counter.inc();
return value;
}
});
As I mentioned, I do get the success metric count but the value is
always 0, even though the above map function was executed.
My questions are:
1. Are there any issues regarding counters being approximate ?
2. If I want to collect accurate counts, is it recommended to use
counters or should I do it explicitly (which is making the code too
complex) ?
3. Do counters participate in flink's failure/checkpointing/recovery ?
4. Is there any better way of collecting application metric counts ?
Thanks
Mans
