Hi all,
In the documentation it says that a pipeline forms a DAG.
For my project however I need to introduce a loop (feed back some results
to an earlier phase of the pipeline).
For now I have solved this using a separate kafka topic, from which I read
somewhere in the beginning of the pipeline, and only write to .
Using an extra kafka topic of course breaks the flow of eventtime and
watermarks. Also I'm achieving at-least-once behavior iso exactly-once.
For my application these behaviors are acceptable (although it could
perhaps come in handy if I would keep control over the flow of watermarks,
but it's not absolutely needed).
Is there a better (more efficient) way to achieve this?
I was thinking of creating a 'pipe'.
interface Pipe<T> {
PTransform<PBegin, PCollection<T>> getSource();
PTransform<PCollection<T>, PDone> getSink();
}
Pipe<T> createPipe(String name);
the source and sink should be internally connected and participate
correctly in the snapshotting flow. (not sure yet about the required
details for this so that it is correct, any help/pointers would be
appreciated!)
However, before I embark on this effort I would like to hear your thoughts
about it.
Thanks, kind regards,
Jan
