On Thu, Mar 21, 2019 at 4:20 PM Pablo Estrada <pabl...@google.com> wrote:
> Hi all, > The DirectRunner does not seem to support RuntimeValueProvider. Is there a > suggestion for DirectRunner pipelines to access arguments passed in as > pipeline options(but not necessarily passed explicitly by users) at > pipeline execution time? > RuntimeValueProvider's are useful when some pipeline options are not set at the time of pipeline construction but available at execution time. In case of DirectRunner what is available at execution time is same as what is available at construction time. Am I missing something, why do we need RuntimeValueProvider support in DirectRunner? > > Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is a > possiblity, but it seems like that's not ideal. Any other suggestions? > > Should we simply support RuntimeValueProvider in direct runner? > This is a bit tricky for python. Because it is possible to run multiple pipeline with DirectRunner in the same process (e.g. call run and do not block on results.) RuntimeValueProvider works by setting a global variable and in the case of direct runner multiple pipelines could share the same process, it gets tricky to support this. > Best > -P. >
