How to cogroup multiple streams?

2022-02-14 Thread Will Lauer
OK, here's what I hope is a stupid question: what's the most efficient way to co-group more than 2 DataStreams together? I'm looking at porting a pipeline from pig to flink, and in a couple of places I use Pig's COGROUP functionality to simultaneously group 3 or 4 and sometimes even more datasets o

Re: How to cogroup multiple streams?

2022-02-15 Thread Chesnay Schepler
You could first transform each stream to a common format (in the worst case, an ugly Either-like capturing all possible types), union those streams, and then do a keyBy + window function. This is how coGroup is implemented internally. On 14/02/2022 16:08, Will Lauer wrote: OK, here's what I ho