I did some more testing this afternoon, and it looks like I'm incorrect on some details here.
Parameters implemented as ValueProviders can be provided at compile time and they do end up in the template. A value specified at runtime overrides. If not specified at runtime, the compile-time value is in effect. It still looks, though, like @Default has no effect for ValueProviders. If not value is specified at compile time or runtime, the ValueProvider is not available, even if it has a @Default configured. Does anybody have tips for documenting which params are overrideable at runtime? Or experience handling default values for ValueProviders? On Thu, Dec 20, 2018 at 3:49 PM Jeff Klukas <[email protected]> wrote: > I am deploying Beam pipelines with the DataflowRunner and would like to > move more of the pipeline options to use the ValueProvider interface so > they can be specified at runtime rather than at template compile time, but > running into various issues. > > First, it's been unclear to the operations engineers deploying pipelines > which options are runtime and which are compile-time. The engineers are > typically using the gcloud command-line interface to deploy rather than the > console, so I don't see much benefit from implementing a metadata json file. > > AFAICT, Dataflow happily accepts values for runtime parameters when > compiling the template, but those values are completely ignored and need to > be defined again at runtime, furthering the confusion. Should I just go > ahead and add "(runtime parameter)" to each of the @Description annotations > to at least document the distinction in --help output? > > Finally, it's not clear whether the @Default annotation supports runtime > parameters. The dataflow docs show an example where @Default is used on a > ValueProvider [0], but this doesn't appear to actually have any effect. If > I don't pass in a value for a runtime parameter when executing a template, > the pipeline throws a "Value only available at runtime" exception on calls > to .get(), rather than returning the default value. Have others encountered > this? Is there a pattern for providing defaults for runtime parameters? > > [0] > https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#using-valueprovider-in-your-pipeline-options >
