Thanks Cham for this. I actually came across this yesterday and am now looking into it. Not sure if it will work but trying it out.
Regards, Pravin On Tue, Nov 7, 2023 at 12:36 PM Chamikara Jayalath via user < user@beam.apache.org> wrote: > Not sure what runner you are using but some runners support updating or > redeploying your pipeline with the changes which might be easier than doing > it manually using your workflow. For example, Dataflow streaming can deploy > a replacement job for you with the updated config for PubSub: > https://cloud.google.com/dataflow/docs/guides/updating-a-pipeline#Launching > > > Thanks, > Cham > > On Tue, Nov 7, 2023 at 9:25 AM Pravin DSouza <pravin...@gmail.com> wrote: > >> Hi Wisniowski, >> >> Thank you for the inputs. >> It is not possible for us to identify all the topics in advance as new >> requirements in future might warrant new topics or may be able to use one >> of existing topics. >> As such, we have decided to proceed with the redeployment option for now >> if the input topic list changes. >> >> Regards, >> Pravin >> >> On Sun, Nov 5, 2023 at 3:10 AM Piotr Wiśniowski < >> contact.wisniowskipi...@gmail.com> wrote: >> >>> Hi, >>> Another idea. Less cost effective, but easier to implement and maintain: >>> 1. Read from all possible topics. >>> 2. 'Map' to add constant topic name to every topic. (Or some other >>> variations of this step) >>> 3. 'Filter' with side input to just filter out all topics that should be >>> silenced. >>> 4. 'Flatten' >>> But this requires that new topics are not created during run of the >>> pipeline and You know them. If not then suggestion with custom 'ParDo' is >>> only option, but I would also suggest rethinking Your infrastructure setup. >>> >>> Best >>> Wiśniowski Piotr >>> >>> śr., 1 lis 2023, 19:06 użytkownik Chamikara Jayalath via user < >>> user@beam.apache.org> napisał: >>> >>>> Currently only some Beam sources are able to consume a configuration >>>> (set of topics here) that is dynamically generated and I don't think >>>> PubSubIO is one of them. So probably you'll have to implement a custom DoFn >>>> that reads from Cloud Pub/Sub to support this. Also, probably you'll have >>>> to constrain reading with size/time since Cloud Pub/Sub is an unbounded >>>> input source. >>>> >>>> Thanks, >>>> Cham >>>> >>>> >>>> >>>> On Tue, Oct 31, 2023 at 2:22 PM Pravin DSouza <pravin...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have a use case where I want to listen to multiple Pub/Sub input >>>>> topics >>>>> and route messages from all multiple input topics to a single >>>>> destination >>>>> output topic. >>>>> The number of input topics can change any time and are stored in a >>>>> table. >>>>> For example when I deploy for the first time, I might be reading from 3 >>>>> topics, but after 2 days the table is updated and now I want to read >>>>> from 4 >>>>> topics.I don't want to redeploy the job because of a change in input >>>>> topics >>>>> list. >>>>> I know how to use SideInputs for refresh, but I am unable to use >>>>> SideInputs >>>>> to read the input topic details and pass to PubSubIO. >>>>> Can you please suggest which of the following ways this can be >>>>> achieved if >>>>> new input topics are added: >>>>> 1. Using SideInput >>>>> 2. Any other approach >>>>> 3. Redeploy the job (this is the last course I want to rely on) >>>>> >>>>> Please suggest. >>>>> >>>>> Thank you, >>>>> Pravin >>>>> >>>>
