Re: Spark Streaming - two questions about the streamingcontext
Great. Thank you! Fang, Yan yanfang...@gmail.com +1 (206) 849-4108 On Wed, Jul 9, 2014 at 11:45 AM, Tathagata Das wrote: > 1. Multiple output operations are processed in the order they are defined. > That is because by default each one output operation is processed at a > time. This *can* be parallelized using an undocumented config parameter > "spark.streaming.concurrentJobs" which is by default set to 1. > > 2. Yes, the output operations (and the spark jobs that are involved with > them) gets queued up. > > TD > > > On Wed, Jul 9, 2014 at 11:22 AM, Yan Fang wrote: > >> I am using the Spark Streaming and have the following two questions: >> >> 1. If more than one output operations are put in the same >> StreamingContext (basically, I mean, I put all the output operations in the >> same class), are they processed one by one as the order they appear in the >> class? Or they are actually processes parallely? >> >> 2. If one DStream takes longer than the interval time, does a new DStream >> wait in the queue until the previous DStream is fully processed? Is there >> any parallelism that may process the two DStream at the same time? >> >> Thank you. >> >> Cheers, >> >> Fang, Yan >> yanfang...@gmail.com >> +1 (206) 849-4108 >> > >
Re: Spark Streaming - two questions about the streamingcontext
1. Multiple output operations are processed in the order they are defined. That is because by default each one output operation is processed at a time. This *can* be parallelized using an undocumented config parameter "spark.streaming.concurrentJobs" which is by default set to 1. 2. Yes, the output operations (and the spark jobs that are involved with them) gets queued up. TD On Wed, Jul 9, 2014 at 11:22 AM, Yan Fang wrote: > I am using the Spark Streaming and have the following two questions: > > 1. If more than one output operations are put in the same StreamingContext > (basically, I mean, I put all the output operations in the same class), are > they processed one by one as the order they appear in the class? Or they > are actually processes parallely? > > 2. If one DStream takes longer than the interval time, does a new DStream > wait in the queue until the previous DStream is fully processed? Is there > any parallelism that may process the two DStream at the same time? > > Thank you. > > Cheers, > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 >
Spark Streaming - two questions about the streamingcontext
I am using the Spark Streaming and have the following two questions: 1. If more than one output operations are put in the same StreamingContext (basically, I mean, I put all the output operations in the same class), are they processed one by one as the order they appear in the class? Or they are actually processes parallely? 2. If one DStream takes longer than the interval time, does a new DStream wait in the queue until the previous DStream is fully processed? Is there any parallelism that may process the two DStream at the same time? Thank you. Cheers, Fang, Yan yanfang...@gmail.com +1 (206) 849-4108
