Hi Theodore! Display data is what's throwing the error, but the BigQuerySource does not support value providers even despite that issue because it's a Dataflow native source. Unfortunately, this is not currently possible. Currently, you could do this executing a BQ export job (using a DoFn), and use fileio to consume those newly exported files. We may prioritize building a source for that, but it is not there ATM. Best -P.
On Fri, Oct 18, 2019 at 6:42 AM Theodore Siu <theo...@google.com> wrote: > Additionally, for reference > https://stackoverflow.com/questions/46595149/dynamic-bigquery-query-in-dataflow-template > > On Fri, Oct 18, 2019 at 9:34 AM Theodore Siu <theo...@google.com> wrote: > >> Hi, >> >> We are attempting to build a Dataflow template in Beam Python and are >> running into issues with using a value provider specifically >> with beam.io.BigQuerySource which throws the following error. ValueError: >> Invalid DisplayDataItem. Value RuntimeValueProvider(option: input, type: >> str, default_value: 'test') is of an unsupported type. >> >> Tracing the code in Github it seems like the main culprits are the >> following: >> >> >> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/io/gcp/bigquery.py#L470 >> >> >> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/transforms/display.py#L244 >> >> >> Do we have any idea when a fix can be made? >> >> -Theo >> >> >> >>