PipelineOptionsFactory.fromSystemProperties did exist in Dataflow 1.x and
was dropped for the reason that Ken mentioned.
On Tue, Feb 13, 2018 at 9:46 AM, Kenneth Knowles <k...@google.com> wrote:
> Pipeline options are not global - they are a property of a single job. The
> TestPipeline reads them from a very particular system property because it
> is a special testing rule.
>
> If you want a generic way to build pipeline options from a set of system
> properties, it should be from an explicit prefix, not global defaults. So
> if a user wants to put a bunch of options into properties, they choose a
> namespace like "my.random.namespace" and everything under it can be
> interpreted as a pipelineoption:
>
> my.random.namespace.SomeOption=bizzle
> my.random.namespace.SomeOtherOption=whatever
>
> And you could do PipelineOptionsFactory.fromSystemProperties("my.
> random.namespace")
>
> We should use the full name of the option not split it with dots - dots
> represent hierarchy not words separation.
>
> interface FooOptions extends PipelineOptions {
> getSomeOption()
> getSomeOtherOption()
> }
>
> I think I can be +0.5 on this. We might also reserve a namespace like
> "beam.TestPipeline.options" and use it for specification of test config.
> Could be easier than embedding JSON in a property, in some cases. Easier to
> override little pieces for sure.
>
> Kenn
>
> On Tue, Feb 13, 2018 at 9:29 AM, Romain Manni-Bucau <rmannibu...@gmail.com
> > wrote:
>
>> makes sense, do we want beam.foo.bar -> --foo-bar conversion too?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> 2018-02-13 18:19 GMT+01:00 Eugene Kirpichov <kirpic...@google.com>:
>>
>>> Neutral about this one: haven't seen a case where this was needed, but
>>> don't see anything wrong with it either. One thing I'd recommend if you go
>>> through with it, extract from system properties under "beam." rather than
>>> all of them, to avoid clashes.
>>>
>>> On Tue, Feb 13, 2018, 7:53 AM Jean-Baptiste Onofré <j...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi Romain,
>>>>
>>>> it sounds interesting to me, and doesn't break anything, so +1 from my
>>>> side.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/13/2018 03:42 PM, Romain Manni-Bucau wrote:
>>>> > Hi guys,
>>>> >
>>>> > there are hacks in beam testing code to read the args from a system
>>>> property but
>>>> > I wonder if we shouldnt add a PipelineOptionsFactory.fromSys
>>>> temProperties().
>>>> >
>>>> > It would iterate over the system properties and take all --xxx=foo as
>>>> potential
>>>> > argument it tries to bind.
>>>> >
>>>> > Rational behind that is to enable users to wrap the pipeline API but
>>>> still
>>>> > expose the pipeline options to end users for advanced cases.
>>>> >
>>>> > Any discussion on this kind of usages already? What do you think of
>>>> this proposal?
>>>> >
>>>> > Side note: we can think about a fromEnv() too.
>>>> >
>>>> > Romain Manni-Bucau
>>>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> > <https://rmannibucau.metawerx.net/> | Old Blog
>>>> > <http://rmannibucau.wordpress.com> | Github <
>>>> https://github.com/rmannibucau> |
>>>> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>>>> > <https://www.packtpub.com/application-development/java-ee-8-
>>>> high-performance>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbono...@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>>
>>>
>>
>
