Hi everyone,
I'm running a YARN session on a cluster with one master and one core and
would like to use the Monitoring API programmatically to submit jobs. I
have found that the configuration variables are read but ignored when
starting the session - it seems to choose a random port each run.
Here's a snippet from the startup logs:
2018-09-06 21:44:38,763 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: env.yarn.conf.dir, /etc/hadoop/conf
2018-09-06 21:44:38,764 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: env.hadoop.conf.dir, /etc/hadoop/conf
2018-09-06 21:44:38,765 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.port, 44477
2018-09-06 21:44:38,765 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.web.port, 44477
2018-09-06 21:44:38,765 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: high-availability.jobmanager.port, 44477
2018-09-06 21:44:38,775 INFO
org.apache.flink.yarn.cli.FlinkYarnSessionCli - Found Yarn
properties file under /tmp/.yarn-properties-hadoop.
2018-09-06 21:44:39,615 WARN org.apache.hadoop.util.NativeCodeLoader
- Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
2018-09-06 21:44:39,799 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user
set to hadoop (auth:SIMPLE)
2018-09-06 21:44:40,045 INFO org.apache.hadoop.yarn.client.RMProxy
- Connecting to ResourceManager at
ip-10-2-3-71.ec2.internal/10.2.3.71:8032
2018-09-06 21:44:40,312 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster
specification: ClusterSpecification{masterMemoryMB=1024,
taskManagerMemoryMB=4096, numberTaskManagers=1, slotsPerTaskManager=1}
2018-09-06 21:44:43,564 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting
application master application_1536250520330_0007
2018-09-06 21:44:43,802 INFO
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted
application application_1536250520330_0007
2018-09-06 21:44:43,802 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for
the cluster to be allocated
2018-09-06 21:44:43,804 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying
cluster, current state ACCEPTED
2018-09-06 21:44:48,326 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN
application has been deployed successfully.
2018-09-06 21:44:48,326 INFO
org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink
YARN client has been started in detached mode. In order to stop Flink on
YARN, use the following command or a YARN web interface to stop it:
yarn application -kill application_1536250520330_0007
Please also note that the temporary files of the YARN session in the home
directory will not be removed.
2018-09-06 21:44:48,821 INFO org.apache.flink.runtime.rest.RestClient
- Rest client endpoint started.
Flink JobManager is now running on ip-10-2-3-25.ec2.internal:38683 with
leader id 00000000-0000-0000-0000-000000000000.
JobManager Web Interface: http://ip-10-2-3-25.ec2.internal:38683
I'm setting both the rest.port and jobmanager.web.port, but both are
ignored. Has anyone seen this before?
Thanks!
