Hi Folks,
Is this issue resolved ? If yes, could you please throw some light on how to
fix this ?
I am facing the same problem during writing to text files.
When I do
stream.foreachRDD(rdd =>{
rdd.saveAsTextFile(<"Some path">)
})
This works fine for me. But it creates multiple text files for each
partition within an RDD.
So I tried with coalesce option to merge my results in a single file for
each RDD as :
stream.foreachRDD(rdd =>{
rdd.coalesce(1, true).saveAsTextFile(<"Some
path">)
})
This fails with :
org.apache.spark.SparkException: Job aborted: Task 75.0:0 failed 1 times
(most recent failure: Exception failure: java.lang.IllegalStateException:
unread block data)
I am using Spark Streaming 0.9.0
Any clue what's going wrong when using coalesce ?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p3238.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
