I created a ticket to track this: https://issues.apache.org/jira/browse/BEAM-11914. Thanks everyone for the comments!
Thanks, Xinyu On Mon, Mar 1, 2021 at 4:45 PM Xinyu Liu <xinyuliu...@gmail.com> wrote: > The reason for not passing it in directly is that we have a large amount > of configs here at LinkedIn so we use an internal config management > framework to hide the actual detailed configs needed to construct these > resources. Internally we put a config map inside the PipelineOptions and > then during @Setup, we would like to init the config framework with the > configs inside PipelineOptions. The user does not need to be aware of how > the configus are populated. They can use something like > > ConfigFramework.create(PipelineOptions).getInstance(SomeResourceFactory) > > to create a resource instance they need. > > On the other hand, even without this kind of use case, it seems still > simpler for the users to use parameters in PipelineOptions if we provide it > directly with @setup. > > Thanks, > Xinyu > > On Mon, Mar 1, 2021 at 4:14 PM Kenneth Knowles <k...@apache.org> wrote: > >> Why not just pass in the arguments to the DoFn constructor or as a >> variable in the containing scope? Do you only know the option after the >> pipeline is completely constructed so you need to make the switch at >> runtime? Makes sense. I think passing options to @Setup is useful and >> harmless. >> >> Kenn >> >> On Mon, Mar 1, 2021 at 3:42 PM Xinyu Liu <xinyuliu...@gmail.com> wrote: >> >>> Hi, all, >>> >>> Currently the @Setup method signature in DoFn does not support any >>> arguments. This is a bit cumbersome to use for use cases such as creating a >>> db connection, rest client or fetch some resources, where we would like to >>> read the configs from the PipelineOptions during setup. Shall we support >>> adding a DoFn SetupContext that can let the users specify the >>> PipelineOptions in the arguments, similar to @StartBundle? Seems the >>> PipelineOptions should always be available when the DoFnRunner is created. >>> Anyone seeing the downside of it? >>> >>> Thanks, >>> Xinyu >>> >>
