Consuming one PCollection before consuming another with Beam

2023-02-23 Thread Sahil Modak via dev
Hi, We have a requirement wherein we are consuming input from pub/sub (PubSubIO) as well as BQ (BQIO) We want to make sure that we consume the BQ stream first before we start consuming the data from pub-sub. Is there a way to achieve this? Can you please help with some code samples? Currently, w

Re: Consuming one PCollection before consuming another with Beam

2023-02-24 Thread Sahil Modak via dev
s this a streaming pipeline? If > so, how are you reading from BigQuery? > > On Thu, Feb 23, 2023 at 10:06 PM Sahil Modak via dev > wrote: > >> Hi, >> >> We have a requirement wherein we are consuming input from pub/sub >> (PubSubIO) as well as BQ (BQIO) >

Re: Consuming one PCollection before consuming another with Beam

2023-02-27 Thread Sahil Modak via dev
; >>> >>> Business Logic: >>> >>> mergedInput.apply("Business Logic", ParDo.of(new BusinessLogic())) >>> >>> >>> >>> Above logic is what we use currently in our pipeline. >>> >>> We want to make sure t

Re: Consuming one PCollection before consuming another with Beam

2023-02-28 Thread Sahil Modak via dev
quot;, BigQueryIO >>>>>> .readTableRows().fromQuery(bqQuery).usingStandardSql()) >>>>>> >>>>>> .apply("JSon Transform", AsJsons.of(TableRow.class)); >>>>>> >>>>>> >>>>>> Merg