The config looks OK to me. On the Flink side I cannot find an
explanation why only /some /metrics disappear.
The only explanation I could come up with at the moment is that
FLINK-8946 is triggered, all metrics are (officially) unregistered, but
the reporter isn't removing some metrics (i.e.
Fabian,
It does look like it may be related. I'll add a comment. After digging a
bit more I found that the crash and lack of metrics were precipitated by the
JobManager instance crashing and cycling, which caused the job to restart.
Chesnay,
I didn't see anything interesting in our logs. Our
Can you show us the metrics-related configuration parameters in
flink-conf.yaml?
Please also check the logs for any warnings from the MetricGroup and
MetricRegistry classes.
On 04.06.2018 10:44, Fabian Hueske wrote:
Hi Nik,
Can you have a look at this JIRA ticket [1] and check if it is
Hi Nik,
Can you have a look at this JIRA ticket [1] and check if it is related to
the problems your are facing?
If so, would you mind leaving a comment there?
Thank you,
Fabian
[1] https://issues.apache.org/jira/browse/FLINK-8946
2018-05-31 4:41 GMT+02:00 Nikolas Davis :
> We keep track of
We keep track of metrics by using the value of
MetricGroup::getMetricIdentifier, which returns the fully qualified metric
name. The query that we use to monitor metrics filters for metrics IDs that
match '%Status.JVM.Memory%'. As long as the new metrics come online via the
MetricReporter interface
How are your metrics dimensionalized/named? Task managers often have UIDs
generated for them. The task id dimension will change on restart. If you
name your metric based on this 'task_id' there would be a discontinuity
with the old metric.
On Wed, May 30, 2018 at 4:49 PM, Nikolas Davis wrote:
>
Howdy,
We are seeing our task manager JVM metrics disappear over time. This last
time we correlated it to our job crashing and restarting. I wasn't able to
grab the failing exception to share. Any thoughts?
We track metrics through the MetricReporter interface. As far as I can tell
this more or
