Hi Yangze, sure!
After a submitted Flink app is complete, our client app polls the RESTful
interface to pull job metrics -- operator start/end times, duration, records +
bytes read/written etc... All of these metrics are all published to a database
for analytical purposes, again both programmatic and ad-hoc.
There was no clear exposure of ClusterClient, so we had originally worked
around this by extending the CliFrontend class with a bit of a façade class
that grabbed the ClusterClient from the executeProgram() method:
@Override
protected void executeProgram(PackagedProgram program, ClusterClient client,
int parallelism) throws ProgramMissingJobException, ProgramInvocationException {
logAndSysout("Starting execution of program");
System.setProperty(JOB_MANAGER_WEB_INTERFACE_PROPERTY,
client.getWebInterfaceURL()); // <- Used elsewhere in application
...
}
These metrics prove immensely valuable as they help us optimize performance,
diagnose issues, as well as predict resource requirements for applications.
// ah
-----Original Message-----
From: Yangze Guo <karma...@gmail.com>
Sent: Sunday, August 1, 2021 10:38 PM
To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com>
Cc: user@flink.apache.org
Subject: Re: Obtain JobManager Web Interface URL
AFAIK, the ClusterClient should not be exposed through the public API.
Would you like to explain your use case and why you need to get the web UI
programmatically?
Best,
Yangze Guo
On Fri, Jul 30, 2021 at 9:54 PM Hailu, Andreas [Engineering]
<andreas.ha...@gs.com> wrote:
>
> Hello Yangze, thanks for responding.
>
> I'm attempting to perform this programmatically on YARN, so looking at a log
> just won't do :) What's the appropriate way to get an instance of a
> ClusterClient? Do you know of any examples I can look at?
>
> // ah
>
> -----Original Message-----
> From: Yangze Guo <karma...@gmail.com>
> Sent: Thursday, July 29, 2021 11:17 PM
> To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com>
> Cc: user@flink.apache.org
> Subject: Re: Obtain JobManager Web Interface URL
>
> Hi, Hailu
>
> AFAIK, the ClusterClient#getWebInterfaceURL has been available since 1.10.
>
> Regarding the JobManager web interface, it will be print in the logs when
> staring a native Kubernetes or Yarn cluster. In standalone mode, it is
> configured by yourself[1].
>
> [1]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_pro
> jects_flink_flink-2Ddocs-2Dmaster_docs_deployment_resource-2Dproviders
> _standalone_overview_-23starting-2Dand-2Dstopping-2Da-2Dcluster&d=DwIF
> aQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VD
> hfisy2OJ1ZAzai-pcCC6TFXM&m=FVv2XIIuWzaAGdj6tz9whXTJ5GQ_xgAqIgesdgtEjG4
> &s=Cu-w4-hIu8MGtvnq2Ob8StpWCZhbFmwN4knnt35NqOM&e=
>
> Best,
> Yangze Guo
>
> On Fri, Jul 30, 2021 at 1:41 AM Hailu, Andreas [Engineering]
> <andreas.ha...@gs.com> wrote:
> >
> > Hi team,
> >
> >
> >
> > Is there a method available to obtain the JobManager’s REST url? We
> > originally overloaded CliFrontend#executeProgram and nabbed it from the
> > ClusterClient#getWebInterfaceUrl method, but it seems this method’s
> > signature has been changed and no longer available as of 1.10.0.
> >
> >
> >
> > Best,
> >
> > Andreas
> >
> >
> >
> >
> > ________________________________
> >
> > Your Personal Data: We may collect and process information about you
> > that may be subject to data protection laws. For more information
> > about how we use and disclose your personal data, how we protect
> > your information, our legal basis to use your information, your
> > rights and who you can contact, please refer to:
> > http://www.gs.com/privacy-notices
>
> ________________________________
>
> Your Personal Data: We may collect and process information about you
> that may be subject to data protection laws. For more information
> about how we use and disclose your personal data, how we protect your
> information, our legal basis to use your information, your rights and
> who you can contact, please refer to:
> www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
________________________________
Your Personal Data: We may collect and process information about you that may
be subject to data protection laws. For more information about how we use and
disclose your personal data, how we protect your information, our legal basis
to use your information, your rights and who you can contact, please refer to:
www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
