Hi, I alwayd underestimated the significant of setting spark.driver.memory
According to documents It is the amount of memory to use for the driver process, i.e. where SparkContext is initialized. (e.g. 1g, 2g). I was running my application using Spark Standalone so the argument about Local mode and one JVM do not come into it. As I know: * The driver program is the main program, which coordinates the executors to run the Spark application.* It is not clear to me whether the driver program also allocates the memory to executors that run on workers. I noticed that if you leave this driver memory low you end up with heap space issue and the job crashes. So I had to increase the driver memory from 1G to 8G to make the job run. So in a nutshell how this driver memory is allocated in Standalone mode given that we also have executer memory --executor-memory that I set separately. Thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
