Given the following code which just reads from s3, then saves files to s3
val inputFileName: String = s3n://input/file/path
val outputFileName: String = s3n://output/file/path
val conf = new
SparkConf().setAppName(this.getClass.getName).setMaster(local[4])
val
I have tested on another pc which has 8 CPU cores.
But it hangs when defaultParallelismLevel 4, e.g.
sparkConf.setMaster(local[*])
local[1] ~ local[3] work well.
4 is the mysterious boundary.
It seems that I am not the only one encountered this problem:
