Re: Runner-provided ValueProviders

Note that the Runner API makes the problem and solution cross-runner, but the problem exists separately for each runner today. So a near-term solution makes sense as long as it is forward-looking (broadcasting runner-provided info to user fns) or has a TTL (putting extra stuff into BoundedSource).

Re: Runner-provided ValueProviders

I think this was initially motivated by BEAM-758 . Copying from that issue: In the forthcoming runner API, a user will be able to save a pipeline to JSON and then run it repeatedly. Many pieces of code (e.g., BigQueryIO.Read or Write) rely o

Re: Runner-provided ValueProviders

Expecting runners to populate, or override, SDK-level pipeline options isn't a great thing, particularly in a scenario that would affect correctness. The main thing is discoverability of a subtle API like this -- there's little chance somebody writing a new runner would stumble across this and do

Runner-provided ValueProviders

Hi folks, I'm looking for feedback on whether the following is a reasonable approach to handling ValueProviders that are intended to be populated at runtime by a given Runner (e.g. a Dataflow job ID, which is a GUID from the service). Two potential pieces of a solution: 1. Annotate such parameter