I used some standard Java IO libraries to write files directly to the cluster. It is a little bit trivial tho:
val sc = getSparkContext val hadoopConf = SparkHadoopUtil.get.newConfiguration val hdfsPath = "hdfs://your/path" val fs = FileSystem.get(hadoopConf) val path = new Path(hdfsPath) val os = new DataOutputStream(new BufferedOutputStream(fs.create(path))) val data = List(List(0,2,3),List(1,4,5),List(2,6,9)) // this works os.writeBytes(data(2).mkString(", ")) os.close Hope you find it helpful -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-make-saveAsTextFile-NOT-split-output-into-multiple-file-tp8129p8143.html Sent from the Apache Spark User List mailing list archive at Nabble.com.