I'm joining several kafka dstreams using the join operation but you have the limitation that the duration of the batch has to be same,i.e. 1 second window for all dstreams... so it would not work for you.
2014-07-16 18:08 GMT+01:00 Walrus theCat <walrusthe...@gmail.com>: > Hi, > > My application has multiple dstreams on the same inputstream: > > dstream1 // 1 second window > dstream2 // 2 second window > dstream3 // 5 minute window > > > I want to write logic that deals with all three windows (e.g. when the 1 > second window differs from the 2 second window by some delta ...) > > I've found some examples online (there's not much out there!), and I can > only see people transforming a single dstream. In conventional spark, we'd > do this sort of thing with a cartesian on RDDs. > > How can I deal with multiple Dstreams at once? > > Thanks >
