Thanks Ahmet! These are illustrative explanations. I still wonder about one question:
> >> Getting it as pcoll.pipeline.options in the expand(self, pcoll) call is a >> possiblity, but it seems like that's not ideal. Any other suggestions? >> > Is this an appropriate way of obtaining an option that is not explicitly passed by the user? It prints a warning. "What's the context?" - I'm working on a transform that writes to big query, and table destinations can come in the form "dataset.table", or "project:dataset.table". Because these are parsed at runtime (destinations are dynamic), the ptransform checks if a project was provided in pipelineoptions via value providers. Thanks! -P. >> Should we simply support RuntimeValueProvider in direct runner? >> > > This is a bit tricky for python. Because it is possible to run multiple > pipeline with DirectRunner in the same process (e.g. call run and do not > block on results.) RuntimeValueProvider works by setting a global variable > and in the case of direct runner multiple pipelines could share the same > process, it gets tricky to support this. > > >> Best >> -P. >> >
