Hi Jins, Do these settings have to be in the Jar File? Since you’re using Beam, you could also use PipelineOptions to make the options accessible to functions at runtime.
Best, Aljoscha > On 12. Jul 2017, at 20:21, Jins George <jins.geo...@aeris.net> wrote: > > Hi Aljoscha, > > I am still using Beam on Flink. I have one yarn session running multiple > streaming jobs. The application jar contains some environment specific run > time properties( like ip addresses, rest api end points etc). This adds > overhead in my usecase as we have to deploy this in multiple environments. I > was trying to decouple these properties files from the uber jar and provide > as as either a classpath resource or pass the path of the file as a system > property to the jvm. > > So far I noticed following options to achieve this. > put all properties in a file and use --classpath file:// <file:///><path to > file> option in flink run command . This needs the url to be accessible > from all nodes, something like NFS > use -D in yarn-session to pass each properties. This will need to restart the > yarn session if a new property gets added. > An ideal solution for me would to provide a local classpath to flink run > command and that gets propagated to other workers automatically :) > > Thanks, > Jins > On 07/12/2017 02:25 AM, Aljoscha Krettek wrote: >> Hi, >> >> Yes, setting the property using -D when creating the session should work to >> make it available on all workers. I think after that it cannot be changed >> since they JVMs are already running. >> >> If I may ask, what’s your use case for this? Are you still using Beam on >> Flink or are you using vanilla Flink with this? >> >> Best, >> Aljoscha >> >>> On 11. Jul 2017, at 07:24, Jins George <jins.geo...@aeris.net> >>> <mailto:jins.geo...@aeris.net> wrote: >>> >>> Thanks Nico. I am able to pass arguments to the main program, that works, >>> but not exactly that I was looking for. >>> >>> I guess to have all worker jvms the same system property, I have to set >>> it at yarn-session creation time using -D ( haven't tried it yet) >>> >>> Thanks, >>> Jins George >>> >>> On 07/10/2017 06:56 AM, Nico Kruber wrote: >>>> Hi Jins, >>>> I'm not sure whether you can define a system property, but you can include >>>> it >>>> in the program arguments of "flink run [OPTIONS] <jar-file> <arguments>" >>>> >>>> You may also be able to define system properties but these are probably >>>> only >>>> valid in your main() function executed within the flink run script, not >>>> any >>>> operators run on other JVM nodes. Have you tried that? >>>> >>>> >>>> Nico >>>> >>>> On Saturday, 8 July 2017 18:08:59 CEST Jins George wrote: >>>>> Hello, >>>>> >>>>> I want to set the path of a properties file as System property in my >>>>> application(something like -Dkey=value). >>>>> Is there a way to set it while submitting a flink job to running YARN >>>>> Session? I am using //bin/flink run/ to submit the job to a already >>>>> running YARN session. >>>>> >>>>> Thanks, >>>>> Jins George >> >
