Hi, Yash,
It should work.
val df = spark.range(1, 5)
.select('id + 1 as 'p1, 'id + 2 as 'p2, 'id + 3 as 'p3, 'id + 4 as 'p4,
'id + 5 as 'p5, 'id as 'b)
.selectExpr("p1", "p2", "p3", "p4", "p5", "CAST(b AS STRING) AS
s").coalesce(1)
df.write.partitionBy("p1", "p2", "p3", "p4",
"p5").text(dir.getCanonicalPath)
val newDF = spark.read.text(dir.getCanonicalPath)
newDF.show()
df.write.partitionBy("p1", "p2", "p3", "p4", "p5")
.mode(SaveMode.Append).text(dir.getCanonicalPath)
val newDF2 = spark.read.text(dir.getCanonicalPath)
newDF2.show()
I tried it. It works well.
Thanks,
Xiao Li
2016-06-18 8:57 GMT-07:00 Yash Sharma <[email protected]>:
> Hi All,
> I have been using the parquet append mode for write which works just
> fine. Just wanted to check if the same is supported for plain text format.
> The below code blows up with error saying the file already exists.
>
>
>
> {code}
> userEventsDF.write.mode("append").partitionBy("year", "month",
> "date").text(outputDir)
> or,
> userEventsDF.write.mode("append").partitionBy("year", "month",
> "date").format("text").save(outputDir)
> {code}
>