Yes, this is not a problem. We do it regularly in our streaming pipelines as a single stream doesn't have enough load for ruining on a Dataflow. So we run different streams in parallel in a single Beam pipelines.
Data wise the streams have nothing to do with each other, but transformation wise they are very similar. On Wed, Apr 29, 2020, 18:19 Marco Mistroni <mmistr...@gmail.com> wrote: > Hi all > Is it possible in beam to create a pipeline where two tasks can run in > parallel as opposed to sequential,? > Simple usecase would be step 3 will generate some data out of which I > generate eg 3 completely different outcomes. ( Eg 3 different files stored > in a bucket) > Thanks > Marco >
