Re: pySpark driver memory limit

This is my experience too when running under yarn at least On Thu, 9 Nov 2017, 07:11 Nicolas Paris, wrote: > Le 06 nov. 2017 à 19:56, Nicolas Paris écrivait : > > Can anyone clarify the driver memory aspects of pySpark? > > According to [1], spark.driver.memory limits JVM +

Re: pySpark driver memory limit

Le 06 nov. 2017 à 19:56, Nicolas Paris écrivait : > Can anyone clarify the driver memory aspects of pySpark? > According to [1], spark.driver.memory limits JVM + python memory. > > In case: > spark.driver.memory=2G > Then does it mean the user won't be able to use more than 2G, whatever > the

pySpark driver memory limit

hi there Can anyone clarify the driver memory aspects of pySpark? According to [1], spark.driver.memory limits JVM + python memory. In case: spark.driver.memory=2G Then does it mean the user won't be able to use more than 2G, whatever the python code + the RDD stuff he is using ? Thanks, [1]: