Hi Craig! For YARN sessions, Flink will - (a) register the app master hostname/port/etc at Yarn, so you can get them from example from the yarn UI and tools - (b) it will create a .yarn-properties file that contain the hostname/ports info. Future calls to the command line pick up the info from there.
/cc Robert Greetings, Stephan On Thu, Aug 25, 2016 at 5:02 PM, Foster, Craig <foscr...@amazon.com> wrote: > I'm trying to understand Flink YARN configuration. The flink-conf.yaml > file is supposedly the way to configure Flink, except when you launch Flink > using YARN since that's determined for the AM. The following is > contradictory or not completely clear: > > > > "The system will use the configuration in conf/flink-config.yaml. Please > follow our configuration guide > <https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html> > if you want to change something. > > Flink on YARN will overwrite the following configuration parameters > jobmanager.rpc.address (because the JobManager is always allocated at > different machines), taskmanager.tmp.dirs (we are using the tmp > directories given by YARN) and parallelism.default if the number of slots > has been specified." > > > > OK, so it will use conf/flink-config.yaml, except for > jobmanager.rpc.address/port which will be decided by YARN and not > necessarily reported to the user since those are dynamically allocated by > YARN. That's fine with me, but if I want to make a "long-running" Flink > cluster available for more than one user, where do I check in Flink for the > Application Master hostname--or do I just have to scrape output of logs > (which would definitely be undesirable)? First, I thought this would be > written by Flink to conf/flink-config.yaml. It is not. Then I thought it > must surely be written to the HDFS configuration directory (under something > like hdfs://$USER/.flink/) for that application but that is merely copied > from the original conf/flink-config.yaml and doesn't have an accurate > configuration for the specified application. So is there an accurate config > somewhere in HDFS or on the ResourceManager--i.e. where could I > programmatically find that (outside of manipulating YARN app names or > scraping)? > > > > Thanks, > > Craig > > > > > > >
