I am also now realizing that runtime params show up in --help with type of ValueProvider, so I can likely meet my documentation needs via a README for the project that explains how runtime parameters behave and that ValueProvider type is the flag that lets you know a parameter can be overridden at runtime.
On Thu, Dec 20, 2018 at 4:47 PM Jeff Klukas <[email protected]> wrote: > I did some more testing this afternoon, and it looks like I'm incorrect on > some details here. > > Parameters implemented as ValueProviders can be provided at compile time > and they do end up in the template. A value specified at runtime overrides. > If not specified at runtime, the compile-time value is in effect. > > It still looks, though, like @Default has no effect for ValueProviders. If > not value is specified at compile time or runtime, the ValueProvider is not > available, even if it has a @Default configured. > > Does anybody have tips for documenting which params are overrideable at > runtime? Or experience handling default values for ValueProviders? > > On Thu, Dec 20, 2018 at 3:49 PM Jeff Klukas <[email protected]> wrote: > >> I am deploying Beam pipelines with the DataflowRunner and would like to >> move more of the pipeline options to use the ValueProvider interface so >> they can be specified at runtime rather than at template compile time, but >> running into various issues. >> >> First, it's been unclear to the operations engineers deploying pipelines >> which options are runtime and which are compile-time. The engineers are >> typically using the gcloud command-line interface to deploy rather than the >> console, so I don't see much benefit from implementing a metadata json file. >> >> AFAICT, Dataflow happily accepts values for runtime parameters when >> compiling the template, but those values are completely ignored and need to >> be defined again at runtime, furthering the confusion. Should I just go >> ahead and add "(runtime parameter)" to each of the @Description annotations >> to at least document the distinction in --help output? >> >> Finally, it's not clear whether the @Default annotation supports runtime >> parameters. The dataflow docs show an example where @Default is used on a >> ValueProvider [0], but this doesn't appear to actually have any effect. If >> I don't pass in a value for a runtime parameter when executing a template, >> the pipeline throws a "Value only available at runtime" exception on calls >> to .get(), rather than returning the default value. Have others encountered >> this? Is there a pattern for providing defaults for runtime parameters? >> >> [0] >> https://cloud.google.com/dataflow/docs/guides/templates/creating-templates#using-valueprovider-in-your-pipeline-options >> >
