Hi, I am working with Spark 2.0, the job starts by sorting the input data and storing the output on HDFS.
I am getting Out of memory errors, the solution was to increase the value of spark.shuffle.memoryFraction from 0.2 to 0.8 and this solves the problem. But in the documentation I have found that this is a deprecated parameter. As I have understand, It was replaced by spark.memory.fraction. How to modify this parameter while taking into account the sort and storage on HDFS? Thanks.
