What you describe seems to be correct and I am not aware of a "recommended approach". It's currently not well supported to programmatically create Flink on YARN clusters (like providing a YarnExecutionEnvironment or so).
I think that Stephan and Max (cc'd) are coordinating some work, which should add support for what you describe. Max worked on the current YARN client refactoring and he can probably share some of his insights regarding this. On Tue, Aug 2, 2016 at 11:09 PM, Bostow, Ben <[email protected]> wrote: > I’m currently trying to programmatically create a Flink cluster on a given > YARN cluster. I’m using the FlinkYarnClientBase class to do this currently > with some limitations (Flink version 1.0.3). > > > > I’m wanting to pass in my own YARN configuration so that I can deploy Flink > on different YARN clusters. Currently these appear to be picked up from an > environment variable YARN_CONF or if I add the yarn configuration files into > the classpath it works for that one. > > > > Is there a way I can dynamically build the YARN configuration and Flink > configuration without utilizing the file system? It would be nice if we had > the option to utilize the same Flink client code and have the CLI be a > wrapper that adds all the environment variable stuff, and then when we need > to do create these programmatically we can do that too. > > > > I’ve started looking at the new code in Flink master and it seems a little > easier than in 1.0.3 but seems that the YarnConfiguration is private in > AbstractYarnClusterDescriptor making it difficult to set the configuration > programmatically. I’m not sure if the new client code still throws an > exception if the Flink config doesn’t exist, even though I could build out > the Flink config programmatically. > > > > Is there a recommended approach to allow this? > > > > Benjamin
