Correct -- Beam pipelines currently have to be DAGs currently and can't support cycles.
There's a jira to look into iterations ( https://issues.apache.org/jira/browse/BEAM-106). The semantics get a little tricky given the unified model, so there's still quite a lot to figure out. In the meantime, you can maintain the loop outside your pipeline -- essentially creating a new pipeline for each iteration. On Wed, Jun 22, 2016 at 11:36 AM, Eswar Reddy <[email protected]> wrote: > Hi Beam Users, > > For my future projects, I anticipate having to create pipelines having > cycle(s).Could see some engines(Flink > <https://ci.apache.org/projects/flink/flink-docs-master/apis/streaming/index.html#iterations> > and Storm > <https://groups.google.com/forum/#!topic/storm-user/EjN1hU58Q_8>) support > this, they also provide flexibility to the user to ensure the loop ends for > a given message by splitting & filtering streams. Could someone share any > pointers to analogus of this in Beam Model? I did see that documentation > <https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/Pipeline#class-pipeline> > says Pipeline is always a DAG though. > > Thanks, > Eswar. >
