Hi, Currently, the memory fraction of shuffle and storage is automatically tuned by a memory manager. So, you do not need to care the parameter in most cases. See https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L24
// maropu On Fri, Sep 23, 2016 at 9:06 PM, tan shai <tan.shai...@gmail.com> wrote: > Hi, > > I am working with Spark 2.0, the job starts by sorting the input data and > storing the output on HDFS. > > I am getting Out of memory errors, the solution was to increase the value > of spark.shuffle.memoryFraction from 0.2 to 0.8 and this solves the > problem. But in the documentation I have found that this is a deprecated > parameter. > > As I have understand, It was replaced by spark.memory.fraction. How to > modify this parameter while taking into account the sort and storage on > HDFS? > > Thanks. > -- --- Takeshi Yamamuro