In fact you can use rdd as well using queue stream but it is considered for testing, as per documents. On 16 Sep 2016 17:44, "ayan guha" <guha.a...@gmail.com> wrote:
> Rdd no. File yes, using fileStream. But filestream does not support > replay, I think. You need to manage checkpoint yourself. > On 16 Sep 2016 16:56, "Udbhav Agarwal" <udbhav.agar...@syncoms.com> wrote: > >> That sounds great. Thanks. >> >> Can I assume that source for a stream in spark can only be some external >> source like kafka etc.? Source cannot be some rdd in spark or some external >> file ? >> >> >> >> Thanks, >> >> Udbhav >> >> *From:* ayan guha [mailto:guha.a...@gmail.com] >> *Sent:* Friday, September 16, 2016 3:01 AM >> *To:* Udbhav Agarwal <udbhav.agar...@syncoms.com> >> *Cc:* user <user@spark.apache.org> >> *Subject:* RE: Spark processing Multiple Streams from a single stream >> >> >> >> You may consider writing back to Kafka from main stream and then have >> downstream consumers. >> This will keep things modular and independent. >> >> On 15 Sep 2016 23:29, "Udbhav Agarwal" <udbhav.agar...@syncoms.com> >> wrote: >> >> Thank you Ayan for a reply. >> >> Source is kafka but I am reading from this source in my main stream. I >> will perform some operations here. Then I want to send the output of these >> operation to 4 parallel tasks. For these 4 parallel tasks I want 4 new >> streams. Is such an implementation possible here ? >> >> >> >> Thanks, >> >> Udbhav >> >> *From:* ayan guha [mailto:guha.a...@gmail.com] >> *Sent:* Thursday, September 15, 2016 6:43 PM >> *To:* Udbhav Agarwal <udbhav.agar...@syncoms.com> >> *Cc:* user <user@spark.apache.org> >> *Subject:* Re: Spark processing Multiple Streams from a single stream >> >> >> >> Depending on source. For example, if source is Kafka then you can write 4 >> streaming consumers. >> >> On 15 Sep 2016 20:11, "Udbhav Agarwal" <udbhav.agar...@syncoms.com> >> wrote: >> >> Hi All, >> >> I have a scenario where I want to process a message in various ways in >> parallel. For instance a message is coming inside spark stream(DStream) and >> I want to send this message to 4 different tasks in parallel. I want these >> 4 different tasks to be separate streams in the original spark stream and >> are always active and waiting for input. Can I implement such a process >> with spark streaming ? How ? >> >> Thanks in advance. >> >> >> >> *Thanks,* >> >> *Udbhav Agarwal* >> >> >> >> >> >>
