Re: [EXTERNAL] Re: Flink REST API
Hi Gary, Thanks a lot. I am able to use REST API now. As you informed, I am able to query REST API, by capturing the tracking-url, I get by using the command “yarn application -list” But, however as I observe in the YARN Resource manager UI, I am not able to query using the tracking url I am observing in YARN Resource Manager. Not sure, If the Hadoop environment I am on, is using some proxies or etc. Do you have any thoughts on why this is happening different in YARN command line Vs YARN Resource Manager ? Thanks a lot again. Regards, Raja. From: Gary Yao Date: Friday, February 2, 2018 at 10:20 AM To: Raja Aravapalli Cc: "user@flink.apache.org" Subject: [EXTERNAL] Re: Flink REST API Hi Raja, The registered tracking URL of the YARN application can be used to issue HTTP requests against the REST API. You can retrieve the URL by using the YARN client: yarn application -list In the output, the rightmost column shows the URL, e.g., Application-Id ... Tracking-URL application_1517585200464_0001 ... http://ip-172-31-40-165.eu-central-1.compute.internal:38045 You can also find the tracking URL in the Web UI of the YARN resource manager. Best, Gary On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli mailto:raja.aravapa...@target.com>> wrote: Hi, I have a triggered a Flink YARN Session on Hadoop yarn. While I was able to trigger applications and run them. I wish to find the URL of REST API for the Flink YARN Sesssion I launched. Can someone please help me point out on how to find the REST API Url for the Flink on YARN? Thanks a lot. Regards, Raja.
Re: [EXTERNAL] Re: Flink REST API
Hi Raja, Can you tell me the API call that you are trying to issue? If it is not a GET request, it could be that you are suffering from this bug: https://issues.apache.org/jira/browse/YARN-2031 In my case the tracking url shown on the resource manager ui is indeed one that targets the YARN proxy, i.e., POST/DELETE/PUT requests are not supported. However, the tracking url obtained from the YARN cli directly refers to the jobmanager. Best, Gary On Fri, Feb 9, 2018 at 3:54 PM, Raja.Aravapalli wrote: > > > Hi Gary, > > > > Thanks a lot. I am able to use REST API now. > > > > As you informed, I am able to query REST API, by capturing the > tracking-url, I get by using the command *“yarn application -list”* > > > > But, however as I observe in the YARN Resource manager UI, I am not able > to query using the tracking url I am observing in YARN Resource Manager. > Not sure, If the Hadoop environment I am on, is using some proxies or etc. > > Do you have any thoughts on why this is happening different in YARN > command line Vs YARN Resource Manager ? > > > > > > Thanks a lot again. > > > > > > Regards, > > Raja. > > > > *From: *Gary Yao > *Date: *Friday, February 2, 2018 at 10:20 AM > *To: *Raja Aravapalli > *Cc: *"user@flink.apache.org" > *Subject: *[EXTERNAL] Re: Flink REST API > > > > Hi Raja, > > > > The registered tracking URL of the YARN application can be used to issue > HTTP > > requests against the REST API. You can retrieve the URL by using the YARN > > client: > > > > yarn application -list > > > > In the output, the rightmost column shows the URL, e.g., > > > > Application-Id ... Tracking-URL > > application_1517585200464_0001 ... > > http://ip-172-31-40-165.eu-central-1.compute.internal:38045 > > > > You can also find the tracking URL in the Web UI of the YARN resource > manager. > > > > Best, > > > > Gary > > > > > > > > On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli < > raja.aravapa...@target.com> wrote: > > Hi, > > > > I have a triggered a Flink YARN Session on Hadoop yarn. > > > > While I was able to trigger applications and run them. I wish to find the > URL of REST API for the Flink YARN Sesssion I launched. > > > > Can someone please help me point out on how to find the REST API Url for > the Flink on YARN? > > > > Thanks a lot. > > > > > > Regards, > > Raja. > > >
Re: [EXTERNAL] Re: Flink REST API
I am issuing a GET call to list running jobs on Flink Session. Another quick question, is there a way to check the port on which my Flink YARN Session is exposing REST API ? Because, I could figure out on UI either in YARN Resource Manager / Flink Web UI of YARN Session the port number I received from Yarn Client using the command “yarn application -list” Thanks a lot. Regards, Raja. From: Gary Yao Date: Friday, February 9, 2018 at 9:25 AM To: Raja Aravapalli Cc: "user@flink.apache.org" Subject: Re: [EXTERNAL] Re: Flink REST API Hi Raja, Can you tell me the API call that you are trying to issue? If it is not a GET request, it could be that you are suffering from this bug: https://issues.apache.org/jira/browse/YARN-2031 In my case the tracking url shown on the resource manager ui is indeed one that targets the YARN proxy, i.e., POST/DELETE/PUT requests are not supported. However, the tracking url obtained from the YARN cli directly refers to the jobmanager. Best, Gary On Fri, Feb 9, 2018 at 3:54 PM, Raja.Aravapalli mailto:raja.aravapa...@target.com>> wrote: Hi Gary, Thanks a lot. I am able to use REST API now. As you informed, I am able to query REST API, by capturing the tracking-url, I get by using the command “yarn application -list” But, however as I observe in the YARN Resource manager UI, I am not able to query using the tracking url I am observing in YARN Resource Manager. Not sure, If the Hadoop environment I am on, is using some proxies or etc. Do you have any thoughts on why this is happening different in YARN command line Vs YARN Resource Manager ? Thanks a lot again. Regards, Raja. From: Gary Yao mailto:g...@data-artisans.com>> Date: Friday, February 2, 2018 at 10:20 AM To: Raja Aravapalli mailto:raja.aravapa...@target.com>> Cc: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: [EXTERNAL] Re: Flink REST API Hi Raja, The registered tracking URL of the YARN application can be used to issue HTTP requests against the REST API. You can retrieve the URL by using the YARN client: yarn application -list In the output, the rightmost column shows the URL, e.g., Application-Id ... Tracking-URL application_1517585200464_0001 ... http://ip-172-31-40-165.eu-central-1.compute.internal:38045 You can also find the tracking URL in the Web UI of the YARN resource manager. Best, Gary On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli mailto:raja.aravapa...@target.com>> wrote: Hi, I have a triggered a Flink YARN Session on Hadoop yarn. While I was able to trigger applications and run them. I wish to find the URL of REST API for the Flink YARN Sesssion I launched. Can someone please help me point out on how to find the REST API Url for the Flink on YARN? Thanks a lot. Regards, Raja.
Re: [EXTERNAL] Re: Flink REST API
Hi Raja, How does the HTTP request fail? Can you post the output of: curl -v http://host:port/jobs There are a couple of other ways to find out the port number. For example, when the YARN session is initially deployed, the url of the web ui is logged to stdout: https://github.com/apache/flink/blob/release-1.4/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L691 However, in a HA setup the JobManager may be restarted on a different host in case of a failure. You can also get the port number from the JobManager logs: https://github.com/apache/flink/blob/release-1.4/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/utils/WebFrontendBootstrap.java#L129 Best, Gary On Fri, Feb 9, 2018 at 4:46 PM, Raja.Aravapalli wrote: > I am issuing a GET call to list running jobs on Flink Session. > > > > Another quick question, is there a way to check the port on which my Flink > YARN Session is exposing REST API ? > > > > Because, I could figure out on UI either in YARN Resource Manager / Flink > Web UI of YARN Session the port number I received from Yarn Client using > the command *“yarn application -list”* > > > > Thanks a lot. > > > > Regards, > > Raja. > > > > *From: *Gary Yao > *Date: *Friday, February 9, 2018 at 9:25 AM > *To: *Raja Aravapalli > *Cc: *"user@flink.apache.org" > *Subject: *Re: [EXTERNAL] Re: Flink REST API > > > > Hi Raja, > > > > Can you tell me the API call that you are trying to issue? If it is not a > GET > > request, it could be that you are suffering from this bug: > > > > https://issues.apache.org/jira/browse/YARN-2031 > > > > In my case the tracking url shown on the resource manager ui is indeed one > that > > targets the YARN proxy, i.e., POST/DELETE/PUT requests are not supported. > > However, the tracking url obtained from the YARN cli directly refers to the > > jobmanager. > > > > Best, > > > > Gary > > > > On Fri, Feb 9, 2018 at 3:54 PM, Raja.Aravapalli < > raja.aravapa...@target.com> wrote: > > > > Hi Gary, > > > > Thanks a lot. I am able to use REST API now. > > > > As you informed, I am able to query REST API, by capturing the > tracking-url, I get by using the command *“yarn application -list”* > > > > But, however as I observe in the YARN Resource manager UI, I am not able > to query using the tracking url I am observing in YARN Resource Manager. > Not sure, If the Hadoop environment I am on, is using some proxies or etc. > > Do you have any thoughts on why this is happening different in YARN > command line Vs YARN Resource Manager ? > > > > > > Thanks a lot again. > > > > > > Regards, > > Raja. > > > > *From: *Gary Yao > *Date: *Friday, February 2, 2018 at 10:20 AM > *To: *Raja Aravapalli > *Cc: *"user@flink.apache.org" > *Subject: *[EXTERNAL] Re: Flink REST API > > > > Hi Raja, > > > > The registered tracking URL of the YARN application can be used to issue > HTTP > > requests against the REST API. You can retrieve the URL by using the YARN > > client: > > > > yarn application -list > > > > In the output, the rightmost column shows the URL, e.g., > > > > Application-Id ... Tracking-URL > > application_1517585200464_0001 ... > > http://ip-172-31-40-165.eu-central-1.compute.internal:38045 > > > > You can also find the tracking URL in the Web UI of the YARN resource > manager. > > > > Best, > > > > Gary > > > > > > > > On Thu, Feb 1, 2018 at 9:29 PM, Raja.Aravapalli < > raja.aravapa...@target.com> wrote: > > Hi, > > > > I have a triggered a Flink YARN Session on Hadoop yarn. > > > > While I was able to trigger applications and run them. I wish to find the > URL of REST API for the Flink YARN Sesssion I launched. > > > > Can someone please help me point out on how to find the REST API Url for > the Flink on YARN? > > > > Thanks a lot. > > > > > > Regards, > > Raja. > > > > >