I know that 1) was suggested a long time ago in Google Cloud Dataflow. The
concerns were of how to make it generic for any type of PTransform instead
of just IO and also how to deal with the same PTransform being specified
multiple times and having each option still be specified. This is why the
current ValueProvider solution exists connecting any flag/PipelineOption
value to any PTransform.
Also, backwards compatibility is important so your suggestion needs to take
that fully into account or defer this kind of change till the next major
version.
On Fri, Jan 12, 2018 at 10:26 AM, Romain Manni-Bucau <rmannibu...@gmail.com>
wrote:
> Hi guys
>
> I'd like to discuss the IO configuration.
>
> My goal is to be able to instrospect (or equivalent) the IO to instantiate
> them programmatically in a generic manner from a generic config - this is
> not yet linked to the system property topic but can benefit beam on this
> other topic too.
>
> Auto value loosing the final fields ordering is impossible to use until
> you parse sources.
>
> In other words: auto value is nice for programmatic usage but is blocking
> for any automotion on top of it - even using unsafe is not an option ATM :(.
>
> Can we try to get something to solve that need please?
>
> Here are the solutions I see (pick just one ;)):
>
> 1. migrate IO to IOOptions (based on pipeline options kind of design).
> This is a breaking change - but I'm sure we can mitigate it in term of user
> compatibility - but it unifies the beam config and enables to not have IO
> specific configurations which can vary between the IO if not developped by
> the same guy.
> 2. Add an extension to @AutoValue to also generate the field names order
> in the create() (@Fields({"address","username","password"}). Cheap but
> the way to instantiate an IO from a config is still a pain (think
> Factory.createIO(clazz, properties))
> 3. Also generate a plain pojo from the abstract @AutoValue class - this
> requires to modify the source class to make it working but is feasible with
> a processor
> 4. drop autovalue and use plain pojo - writing it cause it is a technical
> option but it leads to break as much as 1 without getting all the benefit
> to have an agnostic config and the tooling we can build on top of it
> 5. probably others
>
>
> Wdyt?
>
> Personally I really like 1 cause it starts to create a clean programming
> model we can then build other features on.
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau>
>
