Hi Stephan, Thank you for you answer. I would love to contribute but currently I have no capacity as I am buried with my thesis.
I will reach out after graduating :) Bestr regards. Filip Pozdrawiam, Filip Łęczycki 2016-01-05 10:35 GMT+01:00 Stephan Ewen <se...@apache.org>: > Hi Filip! > > There are thoughts and efforts to extend Flink to push the result > statistics of Flink jobs to the YARN timeline server. That way, you can > explore jobs that are completed. > > Since the whole web dashboard in Flink has a pure REST design, this is a > quite straightforward fix. > > From the capacities I see in the community, I can not promise that to be > fixed immediately. Let me know, though, if you are interested in > contributing an addition there, and I can walk you through the steps that > would be needed. > > Greetings, > Stephan > > > On Mon, Jan 4, 2016 at 9:17 PM, Filip Łęczycki <filipleczy...@gmail.com> > wrote: > >> Hi Till, >> >> Thank you for you answer however I am sorry to hear that. I was reluctant >> to execute jobs with long running Flink cluster due to the fact that >> multiple jobs would cloud yarn statistics regarding cpu and memory time as >> well as Flink's garbage collector statistics in log, as they would be >> stored for the whole Flink cluster, instead of a single job. >> >> Do you know whether is there a way to extract mentioned stats (cpu time, >> mem time, gc time) for a single job ran on long running Flink cluster? >> >> I will be very grateful for an answer:) >> >> Best regards, >> Filip >> >> Pozdrawiam, >> Filip Łęczycki >> >> 2016-01-04 10:05 GMT+01:00 Till Rohrmann <till.rohrm...@gmail.com>: >> >>> Hi Filip, >>> >>> at the moment it is not possible to retrieve the job statistics after >>> the job has finished with flink run -m yarn-cluster. The reason is that >>> the YARN cluster is only alive as long as the job is executed. Thus, I >>> would recommend you to execute your jobs with a long running Flink cluster >>> on YARN. >>> >>> Cheers, >>> Till >>> >>> >>> On Fri, Jan 1, 2016 at 11:29 PM, Filip Łęczycki <filipleczy...@gmail.com >>> > wrote: >>> >>>> Hi all, >>>> >>>> I am running filnk aps on YARN cluster and I am trying to get some >>>> benchmarks. When I start a long-running flink cluster on my YARN cluster I >>>> have an access to web UI and rest API that provide me statistics of the >>>> deployed jobs (as desribed here: >>>> https://ci.apache.org/projects/flink/flink-docs-master/internals/monitoring_rest_api.html). >>>> I was wondering is this possible to get such information about a single run >>>> job trigerred with 'flink run -m yarn-cluster ...'? After the job is >>>> finished there is no flink client running so I cannot use rest api to get >>>> stats. >>>> >>>> Thanks for any help:) >>>> >>>> >>>> Best regards/Pozdrawiam, >>>> Filip Łęczycki >>>> >>> >>> >> >
