Hi all,
Recently, I was working on adding some custom metrics to a Flink job that
required the use of dynamic labels (i.e. capturing various counters that
were "slicable" by things like tenant / source, etc.).
I ended up handling it in a very naive fashion that would just keep a
dictionary of met
Was there anything in the logs (ideally on debug)?
Have you debugged the execution and followed the counter() calls all the
way to the reporter?
Do you only see JobManager metrics, or is there somewhere also something
about the TaskManager?
I can see several issues with your code, but none tha
Hi Chesnay,
Thanks for the prompt response and feedback, it's very much appreciated.
Please see the inline responses below to your questions:
*Was there anything in the logs (ideally on debug)?*
I didn't see anything within the logs that seemed to indicate anything out
of the ordinary. I'm curr
Are you actually running a job, or are you using a harness for testing
your function?
On 3/16/2021 3:24 PM, Rion Williams wrote:
Hi Chesnay,
Thanks for the prompt response and feedback, it's very much
appreciated. Please see the inline responses below to your questions:
*Was there anyth
In this case, I was using a harness to test the function. Although, I could
honestly care less about the unit-test surrounding metrics, I'm much more
concerned with having something that will actually run and work as intended
within a job. The only real concern I have or problem that I want to solv
The test harness is fully independent of the MiniClusterResource; it
isn't actually running a job. That's why your metrics never arrive at
the reporter.
You can either:
a) use the test harness with a custom MetricGroup implementation that
intercepts registered metrics, set in the MockEnvironme
Actually you'd have to further subclass the operatorMetricGroup such
that addGroup works as expected.
This is admittedly a bit of a drag :/
On 3/16/2021 4:35 PM, Chesnay Schepler wrote:
The test harness is fully independent of the MiniClusterResource; it
isn't actually running a job. That's why
I've made a handful of tweaks to it to try and get them to pick up as
expected (i.e. adding logging to every available overload for the
interceptors, etc) using something similar to the following:
fun create(): InterceptingTaskMetricGroup {
val operatorGroup = object: InterceptingOperatorMetri
