Hi Rohit, Can you point which thread has this statement? Maybe the additional context would help us disambiguate the original idea.
TD On Tue, Jul 29, 2014 at 6:15 PM, Tobias Pfeiffer <[email protected]> wrote: > Hi, > > that quoted statement doesn't make too much sense for me, either. Maybe if > you had a link for us that shows the context (Google doesn't reveal anything > but this conversation), we could evaluate that statement better. > > Tobias > > > On Tue, Jul 29, 2014 at 5:53 PM, Sean Owen <[email protected]> wrote: >> >> I'm not sure I understand this, maybe because the context is missing. >> An RDD is immutable, so there is no such thing as writing to an RDD. >> I'm not sure which aspect is being referred to as single-threaded. Is >> this the Spark Streaming driver? >> >> What is the difference between "streaming into Spark" and "reading >> from the stream"? Streaming data into Spark means Spark reads the >> stream. >> >> A mini batch of data is exposed as an RDD, but the stream processing >> continues while it is operated on. Saving the RDDs is one of the most >> basic operations exposed by streaming: >> >> http://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations >> No, you do not stop the stream processing to persist it. In fact you >> couldn't. >> >> On that basis, no, this sounds fairly wrong. >> >> On Tue, Jul 29, 2014 at 1:37 AM, Rohit Pujari <[email protected]> >> wrote: >> > Hello folks: >> > >> > I came across a thread that said >> > >> > "A Spark RDD read/write access is driven by a context object and is >> > single >> > threaded. You cannot stream into Spark and read from the stream at the >> > same >> > time. You have to stop the stream processing, snapshot the RDD and >> > continue" >> > >> > Can you please offer some insights? >> > >> > >> > Thanks, >> > Rohit Pujari >> > Solutions Engineer, Hortonworks >> > [email protected] >> > 716-430-6899 >> > >> > CONFIDENTIALITY NOTICE >> > NOTICE: This message is intended for the use of the individual or entity >> > to >> > which it is addressed and may contain information that is confidential, >> > privileged and exempt from disclosure under applicable law. If the >> > reader of >> > this message is not the intended recipient, you are hereby notified that >> > any >> > printing, copying, dissemination, distribution, disclosure or forwarding >> > of >> > this communication is strictly prohibited. If you have received this >> > communication in error, please contact the sender immediately and delete >> > it >> > from your system. Thank You. > >
