Hi Diego, That's exactly what I thought I could do: Read a variable and use case/switch to decide which output transform to use. It didn't work for me, but maybe there's something else amiss, since not even disconnecting the transform keeps it from getting initialised on pipeline start.
For what it's worth, here's the output of launching the pipeline with the "Beam BigQuery Output" transform, using the local runner: 2022/07/19 14:16:23 - Hop - Pipeline opened. 2022/07/19 14:16:23 - Hop - Launching pipeline [sites]... 2022/07/19 14:16:23 - Hop - Started the pipeline execution. 2022/07/19 14:16:23 - sites - nr of transforms to run : 7 , nr of hops : 6 2022/07/19 14:16:23 - sites - Executing this pipeline using the Local Pipeline Engine with run configuration 'local' 2022/07/19 14:16:23 - sites - Not running a unit test... 2022/07/19 14:16:23 - sites - Execution started for pipeline [sites] 2022/07/19 14:16:23 - sites - I found 7 different transforms to launch. 2022/07/19 14:16:23 - sites - Allocating rowsets... 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 0 --> Avro File Input 2022/07/19 14:16:23 - sites - prevcopies = 1, nextcopies=1 2022/07/19 14:16:23 - sites - Pipeline allocated new rowset [Avro File Input.0 - Avro to site.0] 2022/07/19 14:16:23 - sites - Allocated 1 rowsets for transform 0 --> Avro File Input 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 1 --> Avro to site 2022/07/19 14:16:23 - sites - prevcopies = 1, nextcopies=1 2022/07/19 14:16:23 - sites - Pipeline allocated new rowset [Avro to site.0 - Filter rows.0] 2022/07/19 14:16:23 - sites - Allocated 2 rowsets for transform 1 --> Avro to site 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 2 --> Filter rows 2022/07/19 14:16:23 - sites - prevcopies = 1, nextcopies=1 2022/07/19 14:16:23 - sites - Pipeline allocated new rowset [Filter rows.0 - Select values.0] 2022/07/19 14:16:23 - sites - Allocated 3 rowsets for transform 2 --> Filter rows 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 3 --> Select values 2022/07/19 14:16:23 - sites - prevcopies = 1, nextcopies=1 2022/07/19 14:16:23 - sites - Pipeline allocated new rowset [Select values.0 - Beam BigQuery Output.0] 2022/07/19 14:16:23 - sites - Allocated 4 rowsets for transform 3 --> Select values 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 4 --> Last modified 2022/07/19 14:16:23 - sites - prevcopies = 1, nextcopies=1 2022/07/19 14:16:23 - sites - Pipeline allocated new rowset [Last modified.0 - Avro File Input.0] 2022/07/19 14:16:23 - sites - Allocated 5 rowsets for transform 4 --> Last modified 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 5 --> Get file names 2022/07/19 14:16:23 - sites - prevcopies = 1, nextcopies=1 2022/07/19 14:16:23 - sites - Pipeline allocated new rowset [Get file names.0 - Last modified.0] 2022/07/19 14:16:23 - sites - Allocated 6 rowsets for transform 5 --> Get file names 2022/07/19 14:16:23 - sites - Allocating rowsets for transform 6 --> Beam BigQuery Output 2022/07/19 14:16:23 - sites - Allocated 6 rowsets for transform 6 --> Beam BigQuery Output 2022/07/19 14:16:23 - sites - Allocating Transforms & TransformData... 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Avro File Input] of type [AvroFileInput] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 2022/07/19 14:16:23 - Avro File Input.0 - Starting allocation of buffers & new threads... 2022/07/19 14:16:23 - Avro File Input.0 - Transform info: nrinput=1 nroutput=1 2022/07/19 14:16:23 - Avro File Input.0 - Got previous transform from [Avro File Input] #0 --> Last modified 2022/07/19 14:16:23 - Avro File Input.0 - input rel is 1:1 2022/07/19 14:16:23 - Avro File Input.0 - Found input rowset [Last modified.0 - Avro File Input.0] 2022/07/19 14:16:23 - Avro File Input.0 - output rel. is 1:1 2022/07/19 14:16:23 - Avro File Input.0 - Found output rowset [Avro File Input.0 - Avro to site.0] 2022/07/19 14:16:23 - Avro File Input.0 - Finished dispatching 2022/07/19 14:16:23 - sites - Pipeline has allocated a new transform: [Avro File Input].0 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Avro to site] of type [AvroDecode] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 2022/07/19 14:16:23 - Avro to site.0 - Starting allocation of buffers & new threads... 2022/07/19 14:16:23 - Avro to site.0 - Transform info: nrinput=1 nroutput=1 2022/07/19 14:16:23 - Avro to site.0 - Got previous transform from [Avro to site] #0 --> Avro File Input 2022/07/19 14:16:23 - Avro to site.0 - input rel is 1:1 2022/07/19 14:16:23 - Avro to site.0 - Found input rowset [Avro File Input.0 - Avro to site.0] 2022/07/19 14:16:23 - Avro to site.0 - output rel. is 1:1 2022/07/19 14:16:23 - Avro to site.0 - Found output rowset [Avro to site.0 - Filter rows.0] 2022/07/19 14:16:23 - Avro to site.0 - Finished dispatching 2022/07/19 14:16:23 - sites - Pipeline has allocated a new transform: [Avro to site].0 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Filter rows] of type [FilterRows] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 2022/07/19 14:16:23 - Filter rows.0 - Starting allocation of buffers & new threads... 2022/07/19 14:16:23 - Filter rows.0 - Transform info: nrinput=1 nroutput=1 2022/07/19 14:16:23 - Filter rows.0 - Got previous transform from [Filter rows] #0 --> Avro to site 2022/07/19 14:16:23 - Filter rows.0 - input rel is 1:1 2022/07/19 14:16:23 - Filter rows.0 - Found input rowset [Avro to site.0 - Filter rows.0] 2022/07/19 14:16:23 - Filter rows.0 - output rel. is 1:1 2022/07/19 14:16:23 - Filter rows.0 - Found output rowset [Filter rows.0 - Select values.0] 2022/07/19 14:16:23 - Filter rows.0 - Finished dispatching 2022/07/19 14:16:23 - sites - Pipeline has allocated a new transform: [Filter rows].0 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Select values] of type [SelectValues] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 2022/07/19 14:16:23 - Select values.0 - Starting allocation of buffers & new threads... 2022/07/19 14:16:23 - Select values.0 - Transform info: nrinput=1 nroutput=1 2022/07/19 14:16:23 - Select values.0 - Got previous transform from [Select values] #0 --> Filter rows 2022/07/19 14:16:23 - Select values.0 - input rel is 1:1 2022/07/19 14:16:23 - Select values.0 - Found input rowset [Filter rows.0 - Select values.0] 2022/07/19 14:16:23 - Select values.0 - output rel. is 1:1 2022/07/19 14:16:23 - Select values.0 - Found output rowset [Select values.0 - Beam BigQuery Output.0] 2022/07/19 14:16:23 - Select values.0 - Finished dispatching 2022/07/19 14:16:23 - sites - Pipeline has allocated a new transform: [Select values].0 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Last modified] of type [SampleRows] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 2022/07/19 14:16:23 - Last modified.0 - Starting allocation of buffers & new threads... 2022/07/19 14:16:23 - Last modified.0 - Transform info: nrinput=1 nroutput=1 2022/07/19 14:16:23 - Last modified.0 - Got previous transform from [Last modified] #0 --> Get file names 2022/07/19 14:16:23 - Last modified.0 - input rel is 1:1 2022/07/19 14:16:23 - Last modified.0 - Found input rowset [Get file names.0 - Last modified.0] 2022/07/19 14:16:23 - Last modified.0 - output rel. is 1:1 2022/07/19 14:16:23 - Last modified.0 - Found output rowset [Last modified.0 - Avro File Input.0] 2022/07/19 14:16:23 - Last modified.0 - Finished dispatching 2022/07/19 14:16:23 - sites - Pipeline has allocated a new transform: [Last modified].0 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Get file names] of type [GetFileNames] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 2022/07/19 14:16:23 - Get file names.0 - Starting allocation of buffers & new threads... 2022/07/19 14:16:23 - Get file names.0 - Transform info: nrinput=0 nroutput=1 2022/07/19 14:16:23 - Get file names.0 - output rel. is 1:1 2022/07/19 14:16:23 - Get file names.0 - Found output rowset [Get file names.0 - Last modified.0] 2022/07/19 14:16:23 - Get file names.0 - Finished dispatching 2022/07/19 14:16:23 - sites - Pipeline has allocated a new transform: [Get file names].0 2022/07/19 14:16:23 - sites - Pipeline is about to allocate transform [Beam BigQuery Output] of type [BeamBQOutput] 2022/07/19 14:16:23 - sites - Transform has nrcopies=1 Nothing further happens, i.e. no error is being reported. Once the transform is removed, the pipeline behaves as expected. I would hope for the ability to just ignore the transform when launching locally. Fabian > Am 19.07.2022 um 13:26 schrieb Diego Mainou <[email protected]>: > > Add a get parameter > A filter step > And dump your rows to file, do nothing, etc when your parameter is set to > local and bean when set to that environment. > > > > > > Sent from my Galaxy > > > -------- Original message -------- > From: Fabian Peters <[email protected]> > Date: 19/7/22 9:13 pm (GMT+10:00) > To: [email protected] > Subject: Previewing pipelines with BigQuery output > > Hi all! > > I'm developing a number of pipelines that write data to BigQuery. This works > fine, alas, during development I find I have to entirely remove the "Beam > BigQuery Output" transform to be able to run the pipeline locally, disabling > or deleting the hop to it did not help. Is there a way to keep the transform > around while debugging the pipeline? > > cheers > > Fabian
