This setting allows multiple spark jobs generated through multiple foreachRDD to run concurrently, even if they are across batches. So output op2 from batch X, can run concurrently with op1 of batch X+1 This is not safe because it breaks the checkpointing logic in subtle ways. Note that this was never documented in the spark online docs.
On Tue, Mar 14, 2017 at 2:29 PM, shyla deshpande <deshpandesh...@gmail.com> wrote: > Thanks TD for the response. Can you please provide more explanation. I am > having multiple streams in the spark streaming application (Spark 2.0.2 > using DStreams). I know many people using this setting. So your > explanation will help a lot of people. > > Thanks > > On Fri, Mar 10, 2017 at 6:24 PM, Tathagata Das <t...@databricks.com> > wrote: > >> That config I not safe. Please do not use it. >> >> On Mar 10, 2017 10:03 AM, "shyla deshpande" <deshpandesh...@gmail.com> >> wrote: >> >>> I have a spark streaming application which processes 3 kafka streams and >>> has 5 output operations. >>> >>> Not sure what should be the setting for spark.streaming.concurrentJobs. >>> >>> 1. If the concurrentJobs setting is 4 does that mean 2 output operations >>> will be run sequentially? >>> >>> 2. If I had 6 cores what would be a ideal setting for concurrentJobs in >>> this situation? >>> >>> I appreciate your input. Thanks >>> >> >
