Yeah -- I tried the .union operation and it didn't work for that reason. Surely there has to be a way to do this, as I imagine this is a commonly desired goal in streaming applications?
On Wed, Jul 16, 2014 at 10:10 AM, Luis Ángel Vicente Sánchez < langel.gro...@gmail.com> wrote: > I'm joining several kafka dstreams using the join operation but you have > the limitation that the duration of the batch has to be same,i.e. 1 second > window for all dstreams... so it would not work for you. > > > 2014-07-16 18:08 GMT+01:00 Walrus theCat <walrusthe...@gmail.com>: > > Hi, >> >> My application has multiple dstreams on the same inputstream: >> >> dstream1 // 1 second window >> dstream2 // 2 second window >> dstream3 // 5 minute window >> >> >> I want to write logic that deals with all three windows (e.g. when the 1 >> second window differs from the 2 second window by some delta ...) >> >> I've found some examples online (there's not much out there!), and I can >> only see people transforming a single dstream. In conventional spark, we'd >> do this sort of thing with a cartesian on RDDs. >> >> How can I deal with multiple Dstreams at once? >> >> Thanks >> > >
