Fabian told me that he recently looked into the Java JMX infrastructure. Once we have the metrics collection integrated into the JobManager, we can expose these numbers via a JMX service by the JM. Ganglia (and other tools) can connect to the JMX interface to retrieve metrics.
But I the first step is certainly to get the infrastructure inside our system in place. On Tue, Dec 2, 2014 at 10:28 PM, Márton Balassi <balassi.mar...@gmail.com> wrote: > It would be nice to have integration with the existing tools, e.g. Ganglia. > [1] These already cover system statistics, (CPU, network, I/O...) and one > can define own stats to monitor. > Hadoop is nicely integrated with it. > > [1] http://ganglia.sourceforge.net/ > > On Tue, Dec 2, 2014 at 9:37 PM, Fabian Hueske <fhue...@apache.org> wrote: > > > I see mainly two use cases to locally collect data on TMs and send it > (and > > aggregate it) on the JM. > > > > 1) Monitoring of the system and running jobs: This might include system > > stats (CPU, disk usage, network traffic & buffer usage, internal memory > > utilization, ...) but also progress information (number of processed > > elements, histogram of UDF in/out ratio, UDF exec times, etc.). > > 2) Statistics collection for optimization: Stats would include key > counts & > > distributions, record count & sizes, UDF stats (in/out ratio, exec times, > > ...). Depending on the expertise of the user, this information could also > > be valuable monitoring information. > > > > In both cases, we need a service to ship collected data from the TMs to > the > > JM and aggregated and store it there. > > Once this service is in place, the collection of metrics could be > > independently implemented. > > > > 2014-12-02 14:57 GMT+01:00 Alexander Alexandrov < > > alexander.s.alexand...@gmail.com>: > > > > > This is another way to do it. > > > > > > I just created a JIRA issue for that: > > > > > > https://issues.apache.org/jira/browse/FLINK-1297 > > > > > > If you can give me some pointers and suggest implementation strategies > I > > > can try to prototype something in a feature branch over the weekend and > > > share it for review. > > > > > > > > > > > > 2014-12-02 14:43 GMT+01:00 Ufuk Celebi <u...@apache.org>: > > > > > > > Have you also thought about adding the statistics collection with the > > > > writers, i.e. the collector or record writer? > > > > > > > > If all you care about is the data that the user emits from her code, > > that > > > > should be fine. > > > > > > > > On Tue, Dec 2, 2014 at 2:33 PM, Robert Metzger <rmetz...@apache.org> > > > > wrote: > > > > > > > > > Yes. I also got the impression that you are looking for something > > > > slightly > > > > > different. > > > > > > > > > > It is probably easier for you right now to "hack" something into > the > > > > system > > > > > to get these statistics. > > > > > > > > > > On Tue, Dec 2, 2014 at 2:25 PM, Alexander Alexandrov < > > > > > alexander.s.alexand...@gmail.com> wrote: > > > > > > > > > > > I checked the thread. I am not sure whether this is aligned with > > > what I > > > > > > want to contribute. > > > > > > > > > > > > The discussion in the other thread seems to be going in the > > direction > > > > of > > > > > > general-purpose monitoring (you are talking about Disk + Network > > IO, > > > > > input > > > > > > splits). > > > > > > > > > > > > I would like to have a very thin code base that can be (1) > > > > transparently > > > > > > injected in UDFs (if you can manipulate the AST), or wrapped in > > > > identity > > > > > > mappers (if you cannot) in order to gather collection statistics > > > (min, > > > > > max, > > > > > > distinct, maybe some histograms) to facilitate incremental > > > > optimization. > > > > > > > > > > > > I agree that this should be based on existing infrastructure > (Akka) > > > and > > > > > > should not be over over-engineered. > > > > > > > > > > > > I will announce this in the other branch and create a JIRA ticket > > to > > > > fix > > > > > > the parameters of what has to be done and the best way to > implement > > > it > > > > > with > > > > > > the other contributors. > > > > > > > > > > > > > > > > > > > > > > > > 2014-12-02 14:12 GMT+01:00 Kostas Tzoumas <ktzou...@apache.org>: > > > > > > > > > > > > > From the status of that thread and absence of a JIRA (as far > as I > > > > could > > > > > > > tell), I would suggest that you start working on this and > > announce > > > it > > > > > on > > > > > > > the other thread, perhaps Nils would be interested in jumping > in. > > > > > > > > > > > > > > On Tue, Dec 2, 2014 at 2:06 PM, Ufuk Celebi <u...@apache.org> > > > wrote: > > > > > > > > > > > > > > > Very nice to hear :) > > > > > > > > > > > > > > > > See this thread: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Enhance-Flink-s-monitoring-capabilities-td2573.html > > > > > > > > > > > > > > > > On Tue, Dec 2, 2014 at 2:00 PM, Alexander Alexandrov < > > > > > > > > alexander.s.alexand...@gmail.com> wrote: > > > > > > > > > > > > > > > > > Just a quick shout to check whether somebody is already > > working > > > > on > > > > > a > > > > > > > > > statistics collection component? > > > > > > > > > > > > > > > > > > If yes, can you point me to previous discussions in the > > mailing > > > > > list > > > > > > > and > > > > > > > > a > > > > > > > > > WIP branch -- I want to bring myself up to date with the > > > ongoing > > > > > > > efforts. > > > > > > > > > > > > > > > > > > If not, I would like to start working on that component and > > > > ideally > > > > > > > > > integrate some parts of it in the 0.8 release. > > > > > > > > > > > > > > > > > > Cheers! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >