Rdd no. File yes, using fileStream. But filestream does not support replay, I think. You need to manage checkpoint yourself. On 16 Sep 2016 16:56, "Udbhav Agarwal" <udbhav.agar...@syncoms.com> wrote:
> That sounds great. Thanks. > > Can I assume that source for a stream in spark can only be some external > source like kafka etc.? Source cannot be some rdd in spark or some external > file ? > > > > Thanks, > > Udbhav > > *From:* ayan guha [mailto:guha.a...@gmail.com] > *Sent:* Friday, September 16, 2016 3:01 AM > *To:* Udbhav Agarwal <udbhav.agar...@syncoms.com> > *Cc:* user <user@spark.apache.org> > *Subject:* RE: Spark processing Multiple Streams from a single stream > > > > You may consider writing back to Kafka from main stream and then have > downstream consumers. > This will keep things modular and independent. > > On 15 Sep 2016 23:29, "Udbhav Agarwal" <udbhav.agar...@syncoms.com> wrote: > > Thank you Ayan for a reply. > > Source is kafka but I am reading from this source in my main stream. I > will perform some operations here. Then I want to send the output of these > operation to 4 parallel tasks. For these 4 parallel tasks I want 4 new > streams. Is such an implementation possible here ? > > > > Thanks, > > Udbhav > > *From:* ayan guha [mailto:guha.a...@gmail.com] > *Sent:* Thursday, September 15, 2016 6:43 PM > *To:* Udbhav Agarwal <udbhav.agar...@syncoms.com> > *Cc:* user <user@spark.apache.org> > *Subject:* Re: Spark processing Multiple Streams from a single stream > > > > Depending on source. For example, if source is Kafka then you can write 4 > streaming consumers. > > On 15 Sep 2016 20:11, "Udbhav Agarwal" <udbhav.agar...@syncoms.com> wrote: > > Hi All, > > I have a scenario where I want to process a message in various ways in > parallel. For instance a message is coming inside spark stream(DStream) and > I want to send this message to 4 different tasks in parallel. I want these > 4 different tasks to be separate streams in the original spark stream and > are always active and waiting for input. Can I implement such a process > with spark streaming ? How ? > > Thanks in advance. > > > > *Thanks,* > > *Udbhav Agarwal* > > > > > >
