Thank you, this looks useful indeed for what I have in mind. Saif
From: Jiří Syrový [mailto:syrovy.j...@gmail.com] Sent: Friday, January 15, 2016 12:06 PM To: Ellafi, Saif A. Cc: user@spark.apache.org Subject: Re: jobs much slower in cluster mode vs local Hi, you can try to use spark job server and submit jobs to it. The thing is that the most expensive part is context creation. J. 2016-01-15 15:28 GMT+01:00 <saif.a.ell...@wellsfargo.com<mailto:saif.a.ell...@wellsfargo.com>>: Hello, In general, I am usually able to run spark submit jobs in local mode, in a 32-cores node with plenty of memory ram. The performance is significantly faster in local mode than when using a cluster of spark workers. How can this be explained and what measures can one take in order to improve such performance? Usually a job that takes 35 seconds in local mode takes around 48 seconds in a small cluster. Thanks, Saif
