Setting Flink Monitoring API Port on YARN Cluster
Hi everyone,
I'm running a YARN session on a cluster with one master and one core and
would like to use the Monitoring API programmatically to submit jobs. I
have found that the configuration variables are read but ignored when
starting the session - it seems to choose a random port each run.
Here's a snippet from the startup logs:
2018-09-06 21:44:38,763 INFO
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: env.yarn.conf.dir, /etc/hadoop/conf
2018-09-06 21:44:38,764 INFO
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
2018-09-06 21:44:38,765 INFO
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: rest.port, 44477
2018-09-06 21:44:38,765 INFO
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: jobmanager.web.port, 44477
2018-09-06 21:44:38,765 INFO
org.apache.flink.configuration.GlobalConfiguration- Loading
configuration property: high-availability.jobmanager.port, 44477
2018-09-06 21:44:38,775 INFO
org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn
properties file under /tmp/.yarn-properties-hadoop.
2018-09-06 21:44:39,615 WARN org.apache.hadoop.util.NativeCodeLoader
- Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2018-09-06 21:44:39,799 INFO
org.apache.flink.runtime.security.modules.HadoopModule- Hadoop user
set to hadoop (auth:SIMPLE)
2018-09-06 21:44:40,045 INFO org.apache.hadoop.yarn.client.RMProxy
- Connecting to ResourceManager at
ip-10-2-3-71.ec2.internal/10.2.3.71:8032
2018-09-06 21:44:40,312 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster
specification: ClusterSpecification{masterMemoryMB=1024,
taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
2018-09-06 21:44:43,564 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting
application master application_1536250520330_0007
2018-09-06 21:44:43,802 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
application application_1536250520330_0007
2018-09-06 21:44:43,802 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for
the cluster to be allocated
2018-09-06 21:44:43,804 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying
cluster, current state ACCEPTED
2018-09-06 21:44:48,326 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN
application has been deployed successfully.
2018-09-06 21:44:48,326 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink
YARN client has been started in detached mode. In order to stop Flink on
YARN, use the following command or a YARN web interface to stop it:
yarn application -kill application_1536250520330_0007
Please also note that the temporary files of the YARN session in the home
directory will not be removed.
2018-09-06 21:44:48,821 INFO org.apache.flink.runtime.rest.RestClient
- Rest client endpoint started.
Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with
leader id ----.
JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683
I'm setting both the rest.port and jobmanager.web.port, but both are
ignored. Has anyone seen this before?
Thanks!
Re: Setting Flink Monitoring API Port on YARN Cluster
Hi Austin,
`rest.port` is the latest config option to configure "The port that the
server listens on / the client connects to.", with deprecated key
`web.port` which is with deprecated key `jobmanager.web.port`, so it is
enough to config `rest.port` only (at least for 1.6). However, in your case
the configuration should have worked.
Since Flink recognizes configuration from both flink-conf.yaml and
command-line, it would be helpful if you show us how you do the setting.
Best,
tison.
Austin Cawley-Edwards 于2018年9月7日周五 上午6:33写道：
> Hi everyone,
>
> I'm running a YARN session on a cluster with one master and one core and
> would like to use the Monitoring API programmatically to submit jobs. I
> have found that the configuration variables are read but ignored when
> starting the session - it seems to choose a random port each run.
>
> Here's a snippet from the startup logs:
>
> 2018-09-06 21:44:38,763 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: env.yarn.conf.dir, /etc/hadoop/conf
> 2018-09-06 21:44:38,764 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
> 2018-09-06 21:44:38,765 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: rest.port, 44477
> 2018-09-06 21:44:38,765 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: jobmanager.web.port, 44477
> 2018-09-06 21:44:38,765 INFO
> org.apache.flink.configuration.GlobalConfiguration- Loading
> configuration property: high-availability.jobmanager.port, 44477
> 2018-09-06 21:44:38,775 INFO
> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn
> properties file under /tmp/.yarn-properties-hadoop.
> 2018-09-06 21:44:39,615 WARN org.apache.hadoop.util.NativeCodeLoader
> - Unable to load native-hadoop library for your
> platform... using builtin-java classes where applicable
> 2018-09-06 21:44:39,799 INFO
> org.apache.flink.runtime.security.modules.HadoopModule- Hadoop user
> set to hadoop (auth:SIMPLE)
> 2018-09-06 21:44:40,045 INFO org.apache.hadoop.yarn.client.RMProxy
> - Connecting to ResourceManager at
> ip-10-2-3-71.ec2.internal/10.2.3.71:8032
> 2018-09-06 21:44:40,312 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster
> specification: ClusterSpecification{masterMemoryMB=1024,
> taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
> 2018-09-06 21:44:43,564 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting
> application master application_1536250520330_0007
> 2018-09-06 21:44:43,802 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
> application application_1536250520330_0007
> 2018-09-06 21:44:43,802 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for
> the cluster to be allocated
> 2018-09-06 21:44:43,804 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying
> cluster, current state ACCEPTED
> 2018-09-06 21:44:48,326 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN
> application has been deployed successfully.
> 2018-09-06 21:44:48,326 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink
> YARN client has been started in detached mode. In order to stop Flink on
> YARN, use the following command or a YARN web interface to stop it:
> yarn application -kill application_1536250520330_0007
> Please also note that the temporary files of the YARN session in the home
> directory will not be removed.
> 2018-09-06 21:44:48,821 INFO org.apache.flink.runtime.rest.RestClient
> - Rest client endpoint started.
> Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with
> leader id ----.
> JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683
>
>
> I'm setting both the rest.port and jobmanager.web.port, but both are
> ignored. Has anyone seen this before?
>
> Thanks!
>
Re: Setting Flink Monitoring API Port on YARN Cluster
Hi Austin,
The config options rest.port, jobmanager.web.port, etc. are intentionally
ignored on YARN. The port should be chosen randomly to avoid conflicts with
other containers [1]. I do not see a way how you can set a fixed port at the
moment but there is a related ticket for that [2]. The Flink CLI determines
the hostname and port from the YARN ApplicationReport [3][4] – you can do
the
same.
Best,
Gary
[1]
https://github.com/apache/flink/blob/d036417985d3e2b1ca63909007db9710e842abf4/flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnEntrypointUtils.java#L103
[2] https://issues.apache.org/jira/browse/FLINK-5758
[3]
https://github.com/apache/flink/blob/d036417985d3e2b1ca63909007db9710e842abf4/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L387
[4]
https://hadoop.apache.org/docs/r2.8.3/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html#getRpcPort()
On Fri, Sep 7, 2018 at 12:33 AM, Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:
> Hi everyone,
>
> I'm running a YARN session on a cluster with one master and one core and
> would like to use the Monitoring API programmatically to submit jobs. I
> have found that the configuration variables are read but ignored when
> starting the session - it seems to choose a random port each run.
>
> Here's a snippet from the startup logs:
>
> 2018-09-06 21:44:38,763 INFO
> org.apache.flink.configuration.GlobalConfiguration
> - Loading configuration property: env.yarn.conf.dir,
> /etc/hadoop/conf
> 2018-09-06 21:44:38,764 INFO
> org.apache.flink.configuration.GlobalConfiguration
> - Loading configuration property: env.hadoop.conf.dir,
> /etc/hadoop/conf
> 2018-09-06 21:44:38,765 INFO
> org.apache.flink.configuration.GlobalConfiguration
> - Loading configuration property: rest.port, 44477
> 2018-09-06 21:44:38,765 INFO
> org.apache.flink.configuration.GlobalConfiguration
> - Loading configuration property: jobmanager.web.port, 44477
> 2018-09-06 21:44:38,765 INFO
> org.apache.flink.configuration.GlobalConfiguration
> - Loading configuration property: high-availability.jobmanager.port,
> 44477
> 2018-09-06 21:44:38,775 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli
>- Found Yarn properties file under
> /tmp/.yarn-properties-hadoop.
> 2018-09-06 21:44:39,615 WARN org.apache.hadoop.util.NativeCodeLoader
>- Unable to load native-hadoop library for your
> platform... using builtin-java classes where applicable
> 2018-09-06 21:44:39,799 INFO
> org.apache.flink.runtime.security.modules.HadoopModule
> - Hadoop user set to hadoop (auth:SIMPLE)
> 2018-09-06 21:44:40,045 INFO org.apache.hadoop.yarn.client.RMProxy
>- Connecting to ResourceManager at
> ip-10-2-3-71.ec2.internal/10.2.3.71:8032
> 2018-09-06 21:44:40,312 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - Cluster specification: ClusterSpecification{masterMemoryMB=1024,
> taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
> 2018-09-06 21:44:43,564 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - Submitting application master application_1536250520330_0007
> 2018-09-06 21:44:43,802 INFO
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl
>- Submitted application application_1536250520330_0007
> 2018-09-06 21:44:43,802 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - Waiting for the cluster to be allocated
> 2018-09-06 21:44:43,804 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - Deploying cluster, current state ACCEPTED
> 2018-09-06 21:44:48,326 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - YARN application has been deployed successfully.
> 2018-09-06 21:44:48,326 INFO
> org.apache.flink.yarn.AbstractYarnClusterDescriptor
> - The Flink YARN client has been started in detached mode. In
> order to stop Flink on YARN, use the following command or a YARN web
> interface to stop it:
> yarn application -kill application_1536250520330_0007
> Please also note that the temporary files of the YARN session in the home
> directory will not be removed.
> 2018-09-06 21:44:48,821 INFO org.apache.flink.runtime.rest.RestClient
> - Rest client endpoint started.
> Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with
> leader id ----.
> JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683
>
>
> I'm setting both the rest.port and jobmanager.web.port, but both are
> ignored. Has anyone seen this before?
>
> Thanks!
>
Re: Setting Flink Monitoring API Port on YARN Cluster
Hi Gary,
Thank you so much for the detailed explanation and links. Extremely
helpful. For all others interested, this is also available through the YARN
CLI command `yarn application -status {appId}`.
Once again, thanks for your help!
Austin
On Fri, Sep 7, 2018, 2:24 AM Gary Yao wrote:
> Hi Austin,
>
> The config options rest.port, jobmanager.web.port, etc. are intentionally
> ignored on YARN. The port should be chosen randomly to avoid conflicts with
> other containers [1]. I do not see a way how you can set a fixed port at
> the
> moment but there is a related ticket for that [2]. The Flink CLI determines
> the hostname and port from the YARN ApplicationReport [3][4] – you can do
> the
> same.
>
> Best,
> Gary
>
> [1]
> https://github.com/apache/flink/blob/d036417985d3e2b1ca63909007db9710e842abf4/flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnEntrypointUtils.java#L103
>
> [2] https://issues.apache.org/jira/browse/FLINK-5758
>
> [3]
> https://github.com/apache/flink/blob/d036417985d3e2b1ca63909007db9710e842abf4/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L387
>
> [4]
> https://hadoop.apache.org/docs/r2.8.3/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html#getRpcPort()
>
> On Fri, Sep 7, 2018 at 12:33 AM, Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> I'm running a YARN session on a cluster with one master and one core and
>> would like to use the Monitoring API programmatically to submit jobs. I
>> have found that the configuration variables are read but ignored when
>> starting the session - it seems to choose a random port each run.
>>
>> Here's a snippet from the startup logs:
>>
>> 2018-09-06 21:44:38,763 INFO
>> org.apache.flink.configuration.GlobalConfiguration- Loading
>> configuration property: env.yarn.conf.dir, /etc/hadoop/conf
>> 2018-09-06 21:44:38,764 INFO
>> org.apache.flink.configuration.GlobalConfiguration- Loading
>> configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
>> 2018-09-06 21:44:38,765 INFO
>> org.apache.flink.configuration.GlobalConfiguration- Loading
>> configuration property: rest.port, 44477
>> 2018-09-06 21:44:38,765 INFO
>> org.apache.flink.configuration.GlobalConfiguration- Loading
>> configuration property: jobmanager.web.port, 44477
>> 2018-09-06 21:44:38,765 INFO
>> org.apache.flink.configuration.GlobalConfiguration- Loading
>> configuration property: high-availability.jobmanager.port, 44477
>> 2018-09-06 21:44:38,775 INFO
>> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn
>> properties file under /tmp/.yarn-properties-hadoop.
>> 2018-09-06 21:44:39,615 WARN org.apache.hadoop.util.NativeCodeLoader
>>- Unable to load native-hadoop library for your
>> platform... using builtin-java classes where applicable
>> 2018-09-06 21:44:39,799 INFO
>> org.apache.flink.runtime.security.modules.HadoopModule- Hadoop user
>> set to hadoop (auth:SIMPLE)
>> 2018-09-06 21:44:40,045 INFO org.apache.hadoop.yarn.client.RMProxy
>>- Connecting to ResourceManager at
>> ip-10-2-3-71.ec2.internal/10.2.3.71:8032
>> 2018-09-06 21:44:40,312 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster
>> specification: ClusterSpecification{masterMemoryMB=1024,
>> taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
>> 2018-09-06 21:44:43,564 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting
>> application master application_1536250520330_0007
>> 2018-09-06 21:44:43,802 INFO
>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
>> application application_1536250520330_0007
>> 2018-09-06 21:44:43,802 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for
>> the cluster to be allocated
>> 2018-09-06 21:44:43,804 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying
>> cluster, current state ACCEPTED
>> 2018-09-06 21:44:48,326 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN
>> application has been deployed successfully.
>> 2018-09-06 21:44:48,326 INFO
>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink
>> YARN client has been started in detached mode. In order to stop Flink on
>> YARN, use the following command or a YARN web interface to stop it:
>> yarn application -kill application_1536250520330_0007
>> Please also note that the temporary files of the YARN session in the home
>> directory will not be removed.
>> 2018-09-06 21:44:48,821 INFO org.apache.flink.runtime.rest.RestClient
>> - Rest client endpoint started.
>> Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with
>> leader id ----.
>> JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683
>>
>>
>> I'm setting both the r
