Re: PipelineOptions at execution time from DirectRunner
On Thu, Mar 21, 2019 at 5:29 PM Pablo Estrada wrote: > Thanks Ahmet! These are illustrative explanations. > > I still wonder about one question: > >> >>> Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is >>> a possiblity, but it seems like that's not ideal. Any other suggestions? >>> >> Is this an appropriate way of obtaining an option that is not explicitly > passed by the user? It prints a warning. > > "What's the context?" - I'm working on a transform that writes to big > query, and table destinations can come in the form "dataset.table", or > "project:dataset.table". Because these are parsed at runtime (destinations > are dynamic), the ptransform checks if a project was provided in > pipelineoptions via value providers. > Got it. We wanted to discourage this use and deprecated it. But we have not really offered a clean alternative solution either. I think we will need to figure this out but it does not need to happen within the context of your work. IMO, what you are doing is fine. If we want to think about solution, one option would be to pass all necessary arguments to the transform in the form of arguments. In this case could you could add an explicit project argument to the transform? (If we do that, how do we ensure that the same project argument is used as the project given as pipeline option?) > > Thanks! > -P. > > > >>> Should we simply support RuntimeValueProvider in direct runner? >>> >> >> This is a bit tricky for python. Because it is possible to run multiple >> pipeline with DirectRunner in the same process (e.g. call run and do not >> block on results.) RuntimeValueProvider works by setting a global variable >> and in the case of direct runner multiple pipelines could share the same >> process, it gets tricky to support this. >> >> >>> Best >>> -P. >>> >>
Re: PipelineOptions at execution time from DirectRunner
Thanks Ahmet! These are illustrative explanations. I still wonder about one question: > >> Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is a >> possiblity, but it seems like that's not ideal. Any other suggestions? >> > Is this an appropriate way of obtaining an option that is not explicitly passed by the user? It prints a warning. "What's the context?" - I'm working on a transform that writes to big query, and table destinations can come in the form "dataset.table", or "project:dataset.table". Because these are parsed at runtime (destinations are dynamic), the ptransform checks if a project was provided in pipelineoptions via value providers. Thanks! -P. >> Should we simply support RuntimeValueProvider in direct runner? >> > > This is a bit tricky for python. Because it is possible to run multiple > pipeline with DirectRunner in the same process (e.g. call run and do not > block on results.) RuntimeValueProvider works by setting a global variable > and in the case of direct runner multiple pipelines could share the same > process, it gets tricky to support this. > > >> Best >> -P. >> >
Re: PipelineOptions at execution time from DirectRunner
On Thu, Mar 21, 2019 at 4:20 PM Pablo Estrada wrote: > Hi all, > The DirectRunner does not seem to support RuntimeValueProvider. Is there a > suggestion for DirectRunner pipelines to access arguments passed in as > pipeline options(but not necessarily passed explicitly by users) at > pipeline execution time? > RuntimeValueProvider's are useful when some pipeline options are not set at the time of pipeline construction but available at execution time. In case of DirectRunner what is available at execution time is same as what is available at construction time. Am I missing something, why do we need RuntimeValueProvider support in DirectRunner? > > Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is a > possiblity, but it seems like that's not ideal. Any other suggestions? > > Should we simply support RuntimeValueProvider in direct runner? > This is a bit tricky for python. Because it is possible to run multiple pipeline with DirectRunner in the same process (e.g. call run and do not block on results.) RuntimeValueProvider works by setting a global variable and in the case of direct runner multiple pipelines could share the same process, it gets tricky to support this. > Best > -P. >
PipelineOptions at execution time from DirectRunner
Hi all, The DirectRunner does not seem to support RuntimeValueProvider. Is there a suggestion for DirectRunner pipelines to access arguments passed in as pipeline options(but not necessarily passed explicitly by users) at pipeline execution time? Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is a possiblity, but it seems like that's not ideal. Any other suggestions? Should we simply support RuntimeValueProvider in direct runner? Best -P.
