Hi,
Currently, the memory fraction of shuffle and storage is automatically
tuned by a memory manager.
So, you do not need to care the parameter in most cases.
See
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L24
// maropu
On
Hi,
I am working with Spark 2.0, the job starts by sorting the input data and
storing the output on HDFS.
I am getting Out of memory errors, the solution was to increase the value
of spark.shuffle.memoryFraction from 0.2 to 0.8 and this solves the
problem. But in the documentation I have found
