Hello,
I'm starting using Beam and I would like to know if there is any
recommended pattern for doing the following:
I have a message coming from Kafka and then I would like to apply two
different transformations and merge them in a single result at the end. I
attached an image that describes the pipeline.
Each message has its own unique key,
What I'm doing is using a Session Window with a trigger elementCountAtLeast
with the number equal to the number of process I expected to generate
results (in the case of the diagram will be 2)
This is the code fragment I used for construct the window:
Window<KV<String, OUTPUT>> joinWindow = Window.<KV<String,
OUTPUT>>into(Sessions.withGapDuration(Duration.standardSeconds(60))).triggering(
Repeatedly.forever(AfterPane.elementCountAtLeast(nProcessWait))
).discardingFiredPanes().withAllowedLateness(Duration.ZERO);
and then a CoGroupKey to join all of the results. Is this a
recommended approach? Or is there a recommended way? What happens if at
some points I have a lot of windows "open"?
Thank you very much!
