Task Manager metrics per job on Flink 0.9.1

Hi people! A lot of metrics are gathered for each TaskManager every few seconds. The web UI shows nice graphs for some of these metrics too. I would like to make graphs of the memory and cpu usage, and the time spent on garbage collection for each job. Because of this I am wondering if the metric

Re: Task Manager metrics per job on Flink 0.9.1

Going by the list in the latest documentation for Flink 0.10.1 release, memory and cpu stats are not stored. Neither is the time spent on garbage collection stored anywhere. In my opinion, trying to s

Re: Task Manager metrics per job on Flink 0.9.1

Hi Ritesh, thanks for the response! The metrics are already being gathered though, so I think it would be nice to have a configuration/option to log them somewhere. It doesnt have to be enabled by default, and I dont think it should degrade the performance very much. It looks like the metrics are

Re: Task Manager metrics per job on Flink 0.9.1

I didn't know these stats were collected. Thanks for telling :) In that case, it should definitely be a feature which can be enabled via config files. *Ritesh Kumar Singh,* *https://riteshtoday.wordpress.com/* On Tue, Jan 26, 2016 at 8:22 PM, Pieter Hameete w

Re: Task Manager metrics per job on Flink 0.9.1

Hi, it is correct that the metrics are collected from the task managers. In Flink 0.9.1 the metrics are visualized as charts in the web dashboard. This visualization was removed when the dashboard was redesigned and updated for 0.10. but will be hopefully be added again. For Flink 0.9.1, the metr

Re: Task Manager metrics per job on Flink 0.9.1

Hi Pieter, you're right that it would be nice to record the metrics for a later analysis. However, at the moment this is not supported. You could use the REST interface to obtain the JSON representation of the shown data in the web interface. By doing this repeatedly and parsing the metric data yo

Re: Task Manager metrics per job on Flink 0.9.1

Hi Fabian and Till, thanks for the tips i'll see if I can work with the REST interface for now. I'll make a JIRA ticket as well. I might even be able to develop this feature but I wont have time to do that in the coming 2 months. It would be nice to be able to make a first contribution though. Kee