Thanks Akhil. Does this mean that the executor running in the VM can spawn two concurrent jobs on the same core? If this is the case, this is what we are looking for. Also, which version of Spark is this flag in?
Thanks, Sateesh On Sat, Aug 22, 2015 at 1:44 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > You can look at the spark.streaming.concurrentJobs by default it runs a > single job. If set it to 2 then it can run 2 jobs parallely. Its an > experimental flag, but go ahead and give it a try. > On Aug 21, 2015 3:36 AM, "Sateesh Kavuri" <sateesh.kav...@gmail.com> > wrote: > >> Hi, >> >> My scenario goes like this: >> I have an algorithm running in Spark streaming mode on a 4 core virtual >> machine. Majority of the time, the algorithm does disk I/O and database >> I/O. Question is, during the I/O, where the CPU is not considerably loaded, >> is it possible to run any other task/thread so as to efficiently utilize >> the CPU? >> >> Note that one DStream of the algorithm runs completely on a single CPU >> >> Thank you, >> Sateesh >> >
