Hi Andreas, Maybe zeppelin's source code would be helpful for you. Here's how zeppelin get jobmanager url (jmWebUrl) based on different execution mode https://github.com/apache/zeppelin/blob/master/flink/flink-scala-parent/src/main/scala/org/apache/zeppelin/flink/FlinkScalaInterpreter.scala#L297
David Morávek <d...@apache.org> 于2021年8月3日周二 下午11:13写道： > Hi Andreas, > > you should be able to discover running JobManagers using YarnClient > (looking at the API, "ApplicationReport#getTrackingUrl()" [1] may be what > you're looking for). I don't think it would be easy to have a generic > support for this in Flink without making assumptions about the underlying > infrastructure. > > [1] > https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html#getTrackingUrl-- > > Best, > D. > > On Tue, Aug 3, 2021 at 5:00 AM Yangze Guo <karma...@gmail.com> wrote: > >> From my understanding, what you want is actually a management system >> for Flink jobs. I think it might be good to submit the job(with `flink >> run`) and retrieve the WebUI in another process. >> >> Best, >> Yangze Guo >> >> On Mon, Aug 2, 2021 at 10:39 PM Hailu, Andreas [Engineering] >> <andreas.ha...@gs.com> wrote: >> > >> > Hi Yangze, sure! >> > >> > After a submitted Flink app is complete, our client app polls the >> RESTful interface to pull job metrics -- operator start/end times, >> duration, records + bytes read/written etc... All of these metrics are all >> published to a database for analytical purposes, again both programmatic >> and ad-hoc. >> > >> > There was no clear exposure of ClusterClient, so we had originally >> worked around this by extending the CliFrontend class with a bit of a >> façade class that grabbed the ClusterClient from the executeProgram() >> method: >> > >> > @Override >> > protected void executeProgram(PackagedProgram program, ClusterClient >> client, int parallelism) throws ProgramMissingJobException, >> ProgramInvocationException { >> > logAndSysout("Starting execution of program"); >> > System.setProperty(JOB_MANAGER_WEB_INTERFACE_PROPERTY, >> client.getWebInterfaceURL()); // <- Used elsewhere in application >> > ... >> > } >> > >> > These metrics prove immensely valuable as they help us optimize >> performance, diagnose issues, as well as predict resource requirements for >> applications. >> > >> > // ah >> > >> > -----Original Message----- >> > From: Yangze Guo <karma...@gmail.com> >> > Sent: Sunday, August 1, 2021 10:38 PM >> > To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com> >> > Cc: user@flink.apache.org >> > Subject: Re: Obtain JobManager Web Interface URL >> > >> > AFAIK, the ClusterClient should not be exposed through the public API. >> > Would you like to explain your use case and why you need to get the web >> UI programmatically? >> > >> > Best, >> > Yangze Guo >> > >> > On Fri, Jul 30, 2021 at 9:54 PM Hailu, Andreas [Engineering] < >> andreas.ha...@gs.com> wrote: >> > > >> > > Hello Yangze, thanks for responding. >> > > >> > > I'm attempting to perform this programmatically on YARN, so looking >> at a log just won't do :) What's the appropriate way to get an instance of >> a ClusterClient? Do you know of any examples I can look at? >> > > >> > > // ah >> > > >> > > -----Original Message----- >> > > From: Yangze Guo <karma...@gmail.com> >> > > Sent: Thursday, July 29, 2021 11:17 PM >> > > To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com> >> > > Cc: user@flink.apache.org >> > > Subject: Re: Obtain JobManager Web Interface URL >> > > >> > > Hi, Hailu >> > > >> > > AFAIK, the ClusterClient#getWebInterfaceURL has been available since >> 1.10. >> > > >> > > Regarding the JobManager web interface, it will be print in the logs >> when staring a native Kubernetes or Yarn cluster. In standalone mode, it is >> configured by yourself[1]. >> > > >> > > [1] >> > > >> https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_pro >> > > jects_flink_flink-2Ddocs-2Dmaster_docs_deployment_resource-2Dproviders >> > > _standalone_overview_-23starting-2Dand-2Dstopping-2Da-2Dcluster&d=DwIF >> > > aQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VD >> > > hfisy2OJ1ZAzai-pcCC6TFXM&m=FVv2XIIuWzaAGdj6tz9whXTJ5GQ_xgAqIgesdgtEjG4 >> > > &s=Cu-w4-hIu8MGtvnq2Ob8StpWCZhbFmwN4knnt35NqOM&e= >> > > >> > > Best, >> > > Yangze Guo >> > > >> > > On Fri, Jul 30, 2021 at 1:41 AM Hailu, Andreas [Engineering] < >> andreas.ha...@gs.com> wrote: >> > > > >> > > > Hi team, >> > > > >> > > > >> > > > >> > > > Is there a method available to obtain the JobManager’s REST url? We >> originally overloaded CliFrontend#executeProgram and nabbed it from the >> ClusterClient#getWebInterfaceUrl method, but it seems this method’s >> signature has been changed and no longer available as of 1.10.0. >> > > > >> > > > >> > > > >> > > > Best, >> > > > >> > > > Andreas >> > > > >> > > > >> > > > >> > > > >> > > > ________________________________ >> > > > >> > > > Your Personal Data: We may collect and process information about you >> > > > that may be subject to data protection laws. For more information >> > > > about how we use and disclose your personal data, how we protect >> > > > your information, our legal basis to use your information, your >> > > > rights and who you can contact, please refer to: >> > > > http://www.gs.com/privacy-notices >> > > >> > > ________________________________ >> > > >> > > Your Personal Data: We may collect and process information about you >> > > that may be subject to data protection laws. For more information >> > > about how we use and disclose your personal data, how we protect your >> > > information, our legal basis to use your information, your rights and >> > > who you can contact, please refer to: >> > > www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> >> > >> > ________________________________ >> > >> > Your Personal Data: We may collect and process information about you >> that may be subject to data protection laws. For more information about how >> we use and disclose your personal data, how we protect your information, >> our legal basis to use your information, your rights and who you can >> contact, please refer to: www.gs.com/privacy-notices< >> http://www.gs.com/privacy-notices> >> > -- Best Regards Jeff Zhang
