Re: rdd.saveAsTextFile problem
Hi , I had tried the workaround shared here, but still facing the same issue... Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p22970.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: rdd.saveAsTextFile problem
This thread happened a year back, can you please share what issue you are facing? which version of spark you are using? What is your system environment? Exception stack-trace? Thanks Best Regards On Thu, May 21, 2015 at 12:19 PM, Keerthi keerthi.reddy1...@gmail.com wrote: Hi , I had tried the workaround shared here, but still facing the same issue... Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p22970.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: rdd.saveAsTextFile problem
On Thu, May 21, 2015 at 4:17 PM, Howard Yang howardyang2...@gmail.com wrote: follow http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os to build latest version Hadoop in my windows machine, and Add Environment Variable *HADOOP_HOME* and edit *Path* Variable to add *bin* directory of *HADOOP_HOME* (say*C:\hadoop\bin*). fix this issue in my env 2015-05-21 9:55 GMT+03:00 Akhil Das ak...@sigmoidanalytics.com: This thread happened a year back, can you please share what issue you are facing? which version of spark you are using? What is your system environment? Exception stack-trace? Thanks Best Regards On Thu, May 21, 2015 at 12:19 PM, Keerthi keerthi.reddy1...@gmail.com wrote: Hi , I had tried the workaround shared here, but still facing the same issue... Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p22970.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: rdd.saveAsTextFile problem
Try the workaround for Windows found here: http://qnalist.com/questions/4994960/run-spark-unit-test-on-windows-7. This fix the issue when calling rdd.saveAsTextFile(..) for me with Spark v1.1.0 on windows 8.1 in local mode. Summary of steps: 1) download compiled winutils.exe from http://social.msdn.microsoft.com/Forums/windowsazure/en-US/28a57efb-082b-424b-8d9e-731b1fe135de/please-read-if-experiencing-job-failures?forum=hdinsight 2) put this file into d:\winutil\bin 3) add in code: System.setProperty(hadoop.home.dir, d:\\winutil\\) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p20546.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: rdd.saveAsTextFile problem
Can you give us the more detailed exception + stack trace in the log? It
should be in the driver log. If not, please take a look at the executor
logs, through the web ui to find the stack trace.
TD
On Tue, Mar 25, 2014 at 10:43 PM, gaganbm gagan.mis...@gmail.com wrote:
Hi Folks,
Is this issue resolved ? If yes, could you please throw some light on how
to
fix this ?
I am facing the same problem during writing to text files.
When I do
stream.foreachRDD(rdd ={
rdd.saveAsTextFile(Some path)
})
This works fine for me. But it creates multiple text files for each
partition within an RDD.
So I tried with coalesce option to merge my results in a single file for
each RDD as :
stream.foreachRDD(rdd ={
rdd.coalesce(1,
true).saveAsTextFile(Some path)
})
This fails with :
org.apache.spark.SparkException: Job aborted: Task 75.0:0 failed 1 times
(most recent failure: Exception failure: java.lang.IllegalStateException:
unread block data)
I am using Spark Streaming 0.9.0
Any clue what's going wrong when using coalesce ?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p3238.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: rdd.saveAsTextFile problem
Hi Folks,
Is this issue resolved ? If yes, could you please throw some light on how to
fix this ?
I am facing the same problem during writing to text files.
When I do
stream.foreachRDD(rdd ={
rdd.saveAsTextFile(Some path)
})
This works fine for me. But it creates multiple text files for each
partition within an RDD.
So I tried with coalesce option to merge my results in a single file for
each RDD as :
stream.foreachRDD(rdd ={
rdd.coalesce(1, true).saveAsTextFile(Some
path)
})
This fails with :
org.apache.spark.SparkException: Job aborted: Task 75.0:0 failed 1 times
(most recent failure: Exception failure: java.lang.IllegalStateException:
unread block data)
I am using Spark Streaming 0.9.0
Any clue what's going wrong when using coalesce ?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/rdd-saveAsTextFile-problem-tp176p3238.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
