Re: spark launching range is 10 mins

You might wanna try to assign more cores to the driver?! Sent from my iPhone > On 20 Mar 2016, at 07:34, Jialin Liu wrote: > > Hi, > I have set the partitions as 6000, and requested 100 nodes, with 32 > cores each node, > and the number of executors is 32 per node > >

Re: spark launching range is 10 mins

Hi, I have set the partitions as 6000, and requested 100 nodes, with 32 cores each node, and the number of executors is 32 per node spark-submit --master $SPARKURL --executor-cores 32 --driver-memory 20G --executor-memory 80G single-file-test.py And I'm reading a 2.2 TB, the code, just has