Internally, saveAsTextFile uses saveAsHadoopFile:
https://github.com/apache/spark/blob/d5911d1173fe0872f21cae6c47abf8ff479345a4/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala
.
The final bit in the method first creates the output path and then saves
the data set. However, if
Hi all:
I’ve tried to execute something as below:
result.map(transform).saveAsTextFile(hdfsAddress)
Result is a RDD caluculated from mlilib algorithm.
I submit this to yarn, and after two attempts , the application failed.
But the exception in log is very missleading. It said hdfsAddress
