Re: 1.1.1: JobManager config endpoint no longer supplies port

[Shannon Carey]

A followup (in case anyone is interested): we worked around this by making a 
request to the "/jars" endpoint of the UI. The response has an attribute called 
"address" which includes the DNS name and port where the UI is accessible.

Re: 1.1.1: JobManager config endpoint no longer supplies port

[Robert Metzger]

Hi Shannon,

the problem is that YARNs proxy only allows GET HTTP requests, but for
uploading files, a different request type is needed.

I've filed a JIRA for the problem you've reported:
https://issues.apache.org/jira/browse/FLINK-4432

Regards,
Robert

On Mon, Aug 15, 2016 at 6:03 PM, Shannon Carey  wrote:

> Thanks Ufuk. For now, we will use the Yarn AM proxy.
>
> About uploading JARs: the JobManager UI that is exposed via the Yarn AM
> proxy does not allow manually uploading Flink job jars for execution on the
> cluster (look for "Yarn's AM proxy doesn't allow file uploads." in the
> code). As I understand it, this is due choices Yarn has made about security.
>
>
>
> On 8/15/16, 9:25 AM, "Ufuk Celebi"  wrote:
>
> >I've verified this. I think this has likely accidentally changed with
> >the refactoring of the YARN setup for Flink 1.1. We probably wrote the
> >web monitor port explicitly to the config in 1.0 whereas we don't do
> >it in 1.1 anymore. I think this should be addressed with the next
> >bugfix release 1.1.2.
> >
> >What comes to mind to find out the port is to parse the logs, but I
> >don't think that's feasible, is it? Would also be interested what you
> >are referring to with the JARs.
> >
> >On Mon, Aug 15, 2016 at 4:04 PM, Ufuk Celebi  wrote:
> >> Hey Shannon! I just took a look at the code and it looks like the
> >> Flink REST handler for the config did _not_ change since last year. It
> >> could be that somehow the config is loaded differently. Can you verify
> >> that using the same config with Flink 1.0 and Flink 1.1 the port is
> >> printed correctly and as 0? I will look into it as well.
> >>
> >> The JAR uploading issue is a separate one, correct?
> >>
> >> On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey 
> wrote:
> >>> It appears that when running Flink 1.1.1 on Yarn, my previous method of
> >>> making a request to the yarn AM proxy on the master node at
> >>> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't
> work the
> >>> same as it did.
> >>>
> >>> Previously, the returned JSON value would include an accurate value for
> >>> "jobmanager.web.port". Now, however, it appears to return a zero.
> >>>
> >>> As a result, I cannot expose the JobManager UI itself. I can use the
> Yarn
> >>> proxy but it does not support the useful feature of uploading JARs.
> >>>
> >>> Is there another way of determining which port the JobManager is
> running on?
> >>> Thanks!
> >>>
> >>> -Shannon
> >
>

Re: 1.1.1: JobManager config endpoint no longer supplies port

[Shannon Carey]

Thanks Ufuk. For now, we will use the Yarn AM proxy.

About uploading JARs: the JobManager UI that is exposed via the Yarn AM proxy 
does not allow manually uploading Flink job jars for execution on the cluster 
(look for "Yarn's AM proxy doesn't allow file uploads." in the code). As I 
understand it, this is due choices Yarn has made about security.



On 8/15/16, 9:25 AM, "Ufuk Celebi"  wrote:

>I've verified this. I think this has likely accidentally changed with
>the refactoring of the YARN setup for Flink 1.1. We probably wrote the
>web monitor port explicitly to the config in 1.0 whereas we don't do
>it in 1.1 anymore. I think this should be addressed with the next
>bugfix release 1.1.2.
>
>What comes to mind to find out the port is to parse the logs, but I
>don't think that's feasible, is it? Would also be interested what you
>are referring to with the JARs.
>
>On Mon, Aug 15, 2016 at 4:04 PM, Ufuk Celebi  wrote:
>> Hey Shannon! I just took a look at the code and it looks like the
>> Flink REST handler for the config did _not_ change since last year. It
>> could be that somehow the config is loaded differently. Can you verify
>> that using the same config with Flink 1.0 and Flink 1.1 the port is
>> printed correctly and as 0? I will look into it as well.
>>
>> The JAR uploading issue is a separate one, correct?
>>
>> On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey  wrote:
>>> It appears that when running Flink 1.1.1 on Yarn, my previous method of
>>> making a request to the yarn AM proxy on the master node at
>>> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
>>> same as it did.
>>>
>>> Previously, the returned JSON value would include an accurate value for
>>> "jobmanager.web.port". Now, however, it appears to return a zero.
>>>
>>> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
>>> proxy but it does not support the useful feature of uploading JARs.
>>>
>>> Is there another way of determining which port the JobManager is running on?
>>> Thanks!
>>>
>>> -Shannon
>

Re: 1.1.1: JobManager config endpoint no longer supplies port

[Ufuk Celebi]

I've verified this. I think this has likely accidentally changed with
the refactoring of the YARN setup for Flink 1.1. We probably wrote the
web monitor port explicitly to the config in 1.0 whereas we don't do
it in 1.1 anymore. I think this should be addressed with the next
bugfix release 1.1.2.

What comes to mind to find out the port is to parse the logs, but I
don't think that's feasible, is it? Would also be interested what you
are referring to with the JARs.

On Mon, Aug 15, 2016 at 4:04 PM, Ufuk Celebi  wrote:
> Hey Shannon! I just took a look at the code and it looks like the
> Flink REST handler for the config did _not_ change since last year. It
> could be that somehow the config is loaded differently. Can you verify
> that using the same config with Flink 1.0 and Flink 1.1 the port is
> printed correctly and as 0? I will look into it as well.
>
> The JAR uploading issue is a separate one, correct?
>
> On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey  wrote:
>> It appears that when running Flink 1.1.1 on Yarn, my previous method of
>> making a request to the yarn AM proxy on the master node at
>> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
>> same as it did.
>>
>> Previously, the returned JSON value would include an accurate value for
>> "jobmanager.web.port". Now, however, it appears to return a zero.
>>
>> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
>> proxy but it does not support the useful feature of uploading JARs.
>>
>> Is there another way of determining which port the JobManager is running on?
>> Thanks!
>>
>> -Shannon

Re: 1.1.1: JobManager config endpoint no longer supplies port

[Ufuk Celebi]

Hey Shannon! I just took a look at the code and it looks like the
Flink REST handler for the config did _not_ change since last year. It
could be that somehow the config is loaded differently. Can you verify
that using the same config with Flink 1.0 and Flink 1.1 the port is
printed correctly and as 0? I will look into it as well.

The JAR uploading issue is a separate one, correct?

On Fri, Aug 12, 2016 at 6:03 PM, Shannon Carey  wrote:
> It appears that when running Flink 1.1.1 on Yarn, my previous method of
> making a request to the yarn AM proxy on the master node at
> http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the
> same as it did.
>
> Previously, the returned JSON value would include an accurate value for
> "jobmanager.web.port". Now, however, it appears to return a zero.
>
> As a result, I cannot expose the JobManager UI itself. I can use the Yarn
> proxy but it does not support the useful feature of uploading JARs.
>
> Is there another way of determining which port the JobManager is running on?
> Thanks!
>
> -Shannon

1.1.1: JobManager config endpoint no longer supplies port

[Shannon Carey]

It appears that when running Flink 1.1.1 on Yarn, my previous method of making 
a request to the yarn AM proxy on the master node at 
http://{master_node}:20888/proxy/{app_id}/jobmanager/config doesn't work the 
same as it did.

Previously, the returned JSON value would include an accurate value for 
"jobmanager.web.port". Now, however, it appears to return a zero.

As a result, I cannot expose the JobManager UI itself. I can use the Yarn proxy 
but it does not support the useful feature of uploading JARs.

Is there another way of determining which port the JobManager is running on? 
Thanks!

-Shannon