If I understand correctly, you could not change the number of executors at runtime right(correct me if am wrong) - its defined when we start the application and fixed. Do you mean number of tasks?
On Fri, Jul 11, 2014 at 6:29 AM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Can you try setting the number-of-partitions in all the shuffle-based > DStream operations, explicitly. It may be the case that the default > parallelism (that is, spark.default.parallelism) is probably not being > respected. > > Regarding the unusual delay, I would look at the task details of that > stage in the Spark web ui. It will show break of time for each task, > including GC times, etc. That might give some indication. > > TD > > > On Thu, Jul 10, 2014 at 5:13 PM, Bill Jay <bill.jaypeter...@gmail.com> > wrote: > >> Hi Tathagata, >> >> I set default parallelism as 300 in my configuration file. Sometimes >> there are more executors in a job. However, it is still slow. And I further >> observed that most executors take less than 20 seconds but two of them take >> much longer such as 2 minutes. The data size is very small (less than 480k >> lines with only 4 fields). I am not sure why the group by operation takes >> more then 3 minutes. Thanks! >> >> Bill >> >> >> On Thu, Jul 10, 2014 at 4:28 PM, Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> Are you specifying the number of reducers in all the DStream.****ByKey >>> operations? If the reduce by key is not set, then the number of reducers >>> used in the stages can keep changing across batches. >>> >>> TD >>> >>> >>> On Wed, Jul 9, 2014 at 4:05 PM, Bill Jay <bill.jaypeter...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> I have a Spark streaming job running on yarn. It consume data from >>>> Kafka and group the data by a certain field. The data size is 480k lines >>>> per minute where the batch size is 1 minute. >>>> >>>> For some batches, the program sometimes take more than 3 minute to >>>> finish the groupBy operation, which seems slow to me. I allocated 300 >>>> workers and specify 300 as the partition number for groupby. When I checked >>>> the slow stage *"combineByKey at ShuffledDStream.scala:42",* there are >>>> sometimes 2 executors allocated for this stage. However, during other >>>> batches, the executors can be several hundred for the same stage, which >>>> means the number of executors for the same operations change. >>>> >>>> Does anyone know how Spark allocate the number of executors for >>>> different stages and how to increase the efficiency for task? Thanks! >>>> >>>> Bill >>>> >>> >>> >> >
