Re: driver OOM - need recommended memory for driver

Thank you so much Mich, I am using yarn as my master. I found a statement in Spark mentioning the amount of memory depends on individual application. http://spark.apache.org/docs/1.5.2/hardware-provisioning.html#memory I

Re: driver OOM - need recommended memory for driver

If you make your driver memory too low it is likely you are going to hit OOM error. You have not mentioned with Spark mode you are using (Local, Standalone, Yarn etc) HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: driver OOM - need recommended memory for driver

Thank you so much, Kevin. My data size is around 4GB. I am not using collect(), take() or takeSample() At the final job, number of tasks grows up to 200,000 Still the driver crashes with OOM with default —driver-memory 1g but Job succeeds if i specify 2g. Thanks and regards, Anand Viswanathan

Re: driver OOM - need recommended memory for driver

Hi Anand, Unfortunately, there is not really a "one size fits all" answer to this question; however, here are some things that you may want to consider when trying different sizes. - What is the size of the data you are processing? - Whenever you invoke an action that requires ALL of the

driver OOM - need recommended memory for driver

Hi, Spark version :spark-1.5.2-bin-hadoop2.6 ,using pyspark. I am running a machine learning program, which runs perfectly by specifying 2G for —driver-memory. However the program cannot be run with default 1G, driver crashes with OOM error. What is the recommended configuration for