Hi, it is correct that the metrics are collected from the task managers. In Flink 0.9.1 the metrics are visualized as charts in the web dashboard. This visualization was removed when the dashboard was redesigned and updated for 0.10. but will be hopefully be added again.
For Flink 0.9.1, the metrics are cached in memory for plotting and not persisted anywhere. The JobManager exposes the stats via a REST-like interface. You would need to check the code of the web dashboard to get the correct URL. With the updated web UI of 0.10.1 all job information and stats are exposed via well defined REST interfaces: https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html In both cases, you can periodically poll the interfaces to collect stats. Best, Fabian 2016-01-26 21:22 GMT+01:00 Ritesh Kumar Singh <riteshoneinamill...@gmail.com >: > I didn't know these stats were collected. Thanks for telling :) > In that case, it should definitely be a feature which can be enabled via > config files. > > *Ritesh Kumar Singh,* > *https://riteshtoday.wordpress.com/* <https://riteshtoday.wordpress.com/> > > On Tue, Jan 26, 2016 at 8:22 PM, Pieter Hameete <phame...@gmail.com> > wrote: > >> Hi Ritesh, >> >> thanks for the response! The metrics are already being gathered though, >> so I think it would be nice to have a configuration/option to log them >> somewhere. It doesnt have to be enabled by default, and I dont think it >> should degrade the performance very much. It looks like the metrics are >> currently sent with each heartbeat by default already. Your Web UI probably >> hangs because it has to update all the graphs on every heartbeat, when you >> have many task managers that will be heavy on your computer :-) >> >> - Pieter >> >> 2016-01-26 20:17 GMT+01:00 Ritesh Kumar Singh < >> riteshoneinamill...@gmail.com>: >> >>> Going by the list in the latest documentation >>> <https://ci.apache.org/projects/flink/flink-docs-release-0.10/internals/monitoring_rest_api.html> >>> for Flink 0.10.1 release, memory and cpu stats are not stored. Neither is >>> the time spent on garbage collection stored anywhere. >>> >>> In my opinion, trying to store these metrics will degrade the >>> performance of jobs too. And so its basically a trade off between >>> performance and computation cost. For me, the web ui hangs even for the >>> current set of parameters :( >>> >>> *Ritesh Kumar Singh,* >>> *https://riteshtoday.wordpress.com/* >>> <https://riteshtoday.wordpress.com/> >>> >>> On Tue, Jan 26, 2016 at 7:16 PM, Pieter Hameete <phame...@gmail.com> >>> wrote: >>> >>>> Hi people! >>>> >>>> A lot of metrics are gathered for each TaskManager every few seconds. >>>> The web UI shows nice graphs for some of these metrics too. >>>> >>>> I would like to make graphs of the memory and cpu usage, and the time >>>> spent on garbage collection for each job. Because of this I am wondering if >>>> the metrics are also stored somewhere, or if there is an option to enable >>>> storing the metrics per job. >>>> >>>> In the configuration documentation I could not find such an option. Is >>>> this possible in version 0.9.1 of Flink? If not: is it possible in Flink >>>> 0.10.1 or is it possible to request or develop such a feature? >>>> >>>> Thank you for your help and kind regards, >>>> >>>> Pieter >>>> >>>> >>>> >>> >> >
