FYI created https://github.com/apache/beam/pull/4683 about it
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> 2018-02-13 19:21 GMT+01:00 Romain Manni-Bucau <rmannibu...@gmail.com>: > Oki, will try a PR after the classloader one then. Thanks a lot. > > Le 13 févr. 2018 19:14, "Lukasz Cwik" <lc...@google.com> a écrit : > >> The one in Dataflow 1.x was one system property that contained all the >> JSON so it wasn't exactly what you were looking for. >> >> On Tue, Feb 13, 2018 at 9:51 AM, Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> I like your proposal Kenneth. Perfectly fits my use case and deployment >>> one as well - when ops configure the env without modifying the code. >>> >>> How do we move forward on that? Should I send a PR or do you want to >>> import what was in dataflow? >>> >>> >>> Romain Manni-Bucau >>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>> <https://rmannibucau.metawerx.net/> | Old Blog >>> <http://rmannibucau.wordpress.com> | Github >>> <https://github.com/rmannibucau> | LinkedIn >>> <https://www.linkedin.com/in/rmannibucau> | Book >>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>> >>> 2018-02-13 18:49 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>> >>>> PipelineOptionsFactory.fromSystemProperties did exist in Dataflow 1.x >>>> and was dropped for the reason that Ken mentioned. >>>> >>>> On Tue, Feb 13, 2018 at 9:46 AM, Kenneth Knowles <k...@google.com> >>>> wrote: >>>> >>>>> Pipeline options are not global - they are a property of a single job. >>>>> The TestPipeline reads them from a very particular system property because >>>>> it is a special testing rule. >>>>> >>>>> If you want a generic way to build pipeline options from a set of >>>>> system properties, it should be from an explicit prefix, not global >>>>> defaults. So if a user wants to put a bunch of options into properties, >>>>> they choose a namespace like "my.random.namespace" and everything under it >>>>> can be interpreted as a pipelineoption: >>>>> >>>>> my.random.namespace.SomeOption=bizzle >>>>> my.random.namespace.SomeOtherOption=whatever >>>>> >>>>> And you could do PipelineOptionsFactory.fromSys >>>>> temProperties("my.random.namespace") >>>>> >>>>> We should use the full name of the option not split it with dots - >>>>> dots represent hierarchy not words separation. >>>>> >>>>> interface FooOptions extends PipelineOptions { >>>>> getSomeOption() >>>>> getSomeOtherOption() >>>>> } >>>>> >>>>> I think I can be +0.5 on this. We might also reserve a namespace like >>>>> "beam.TestPipeline.options" and use it for specification of test config. >>>>> Could be easier than embedding JSON in a property, in some cases. Easier >>>>> to >>>>> override little pieces for sure. >>>>> >>>>> Kenn >>>>> >>>>> On Tue, Feb 13, 2018 at 9:29 AM, Romain Manni-Bucau < >>>>> rmannibu...@gmail.com> wrote: >>>>> >>>>>> makes sense, do we want beam.foo.bar -> --foo-bar conversion too? >>>>>> >>>>>> >>>>>> Romain Manni-Bucau >>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >>>>>> >>>>>> 2018-02-13 18:19 GMT+01:00 Eugene Kirpichov <kirpic...@google.com>: >>>>>> >>>>>>> Neutral about this one: haven't seen a case where this was needed, >>>>>>> but don't see anything wrong with it either. One thing I'd recommend if >>>>>>> you >>>>>>> go through with it, extract from system properties under "beam." rather >>>>>>> than all of them, to avoid clashes. >>>>>>> >>>>>>> On Tue, Feb 13, 2018, 7:53 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Romain, >>>>>>>> >>>>>>>> it sounds interesting to me, and doesn't break anything, so +1 from >>>>>>>> my side. >>>>>>>> >>>>>>>> Regards >>>>>>>> JB >>>>>>>> >>>>>>>> On 02/13/2018 03:42 PM, Romain Manni-Bucau wrote: >>>>>>>> > Hi guys, >>>>>>>> > >>>>>>>> > there are hacks in beam testing code to read the args from a >>>>>>>> system property but >>>>>>>> > I wonder if we shouldnt add a PipelineOptionsFactory.fromSys >>>>>>>> temProperties(). >>>>>>>> > >>>>>>>> > It would iterate over the system properties and take all >>>>>>>> --xxx=foo as potential >>>>>>>> > argument it tries to bind. >>>>>>>> > >>>>>>>> > Rational behind that is to enable users to wrap the pipeline API >>>>>>>> but still >>>>>>>> > expose the pipeline options to end users for advanced cases. >>>>>>>> > >>>>>>>> > Any discussion on this kind of usages already? What do you think >>>>>>>> of this proposal? >>>>>>>> > >>>>>>>> > Side note: we can think about a fromEnv() too. >>>>>>>> > >>>>>>>> > Romain Manni-Bucau >>>>>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>> > <http://rmannibucau.wordpress.com> | Github < >>>>>>>> https://github.com/rmannibucau> | >>>>>>>> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>> > <https://www.packtpub.com/application-development/java-ee-8- >>>>>>>> high-performance> >>>>>>>> >>>>>>>> -- >>>>>>>> Jean-Baptiste Onofré >>>>>>>> jbono...@apache.org >>>>>>>> http://blog.nanthrax.net >>>>>>>> Talend - http://www.talend.com >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
