Re: spark launching range is 10 mins

You might wanna try to assign more cores to the driver?! Sent from my iPhone > On 20 Mar 2016, at 07:34, Jialin Liu wrote: > > Hi, > I have set the partitions as 6000, and requested 100 nodes, with 32 > cores each node, > and the number of executors is 32 per node > > spark-submit --master $SP

Re: Saving intermediate results in mapPartitions

Try to set MEMORY_AND_DISK as RDD’s storage persistence level. http://spark.apache.org/docs/latest/programming-guide.html#rdd-persistence > On 19 Mar 2016, at 00:55, Krishna wrote: > > Hi, > > I've a situation where th