The storage fraction only limits the amount of memory used for storage. It
doesn't actually limit anything else. I.e you can use all the memory if you
want in collect.
On Sunday, September 28, 2014, Brad Miller bmill...@eecs.berkeley.edu
wrote:
Hi All,
I am interested to collect() a large RDD so that I can run a learning
algorithm on it. I've noticed that when I don't increase
SPARK_DRIVER_MEMORY I can run out of memory. I've also noticed that it
looks like the same fraction of memory is reserved for storage on the
driver as on the worker nodes, and that the web UI doesn't show any storage
usage on the driver. Since that memory is reserved for storage, it seems
possible that it is not being used towards the collection of my RDD.
Is there a way to configure the memory management (
spark.storage.memoryFraction, spark.shuffle.memoryFraction) for the
driver separately from the workers?
Is there any reason to leave space for shuffle or storage on the driver?
It seems like I never see either of these used on the web UI, although I
may not be interpreting the UI correctly or my jobs may not trigger the use
case.
For context, I am using PySpark (so much of my processing happens outside
of the allocated memory in Java) and running the Spark 1.1.0 release
binaries.
best,
-Brad
