Hi,
How is 78g distributed in driver, daemon, executor ?
Can you please paste the logs regarding that I don't have enough memory to
hold the data in memory
Are you collecting any data in driver ?
Lastly, did you try doing a re-partition to create smaller and evenly
distributed partitions?
i think you can try to set lower spark.storage.memoryFraction,for example 0.4
conf.set(spark.storage.memoryFraction,0.4) //default 0.6
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Help-with-processing-multiple-RDDs-tp18628p18659.html
Sent from the
the logs as soon as this completes!
Thank you, I sincerely appreciate your help!
Regards,
Ami
-Original Message-
From: Kapil Malik [mailto:kma...@adobe.com]
Sent: Tuesday, November 11, 2014 9:05 PM
To: akhandeshi; u...@spark.incubator.apache.org
Subject: RE: Help with processing multiple
