Hi,
I use Spark 1.4. When saving the model to HDFS, I got error?
Please help!
Regards
my scala command:
sc.makeRDD(model.clusterCenters,10).saveAsObjectFile("/tmp/tweets/model")
The error log:
15/07/14 18:27:40 INFO SequenceFileRDDFunctions: Saving as sequence file of
type (NullWritable,BytesWritable)
15/07/14 18:27:40 INFO SparkContext: Starting job: saveAsObjectFile at
<console>:45
15/07/14 18:27:40 INFO DAGScheduler: Got job 110 (saveAsObjectFile at
<console>:45) with 10 output partitions (allowLocal=false)
15/07/14 18:27:40 INFO DAGScheduler: Final stage: ResultStage
174(saveAsObjectFile at <console>:45)
15/07/14 18:27:40 INFO DAGScheduler: Parents of final stage: List()
15/07/14 18:27:40 INFO DAGScheduler: Missing parents: List()
15/07/14 18:27:40 INFO DAGScheduler: Submitting ResultStage 174
(MapPartitionsRDD[258] at saveAsObjectFile at <console>:45), which has no
missing parents
15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(135360) called with
curMem=14724380, maxMem=280248975
15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256 stored as values in
memory (estimated size 132.2 KB, free 253.1 MB)
15/07/14 18:27:40 INFO MemoryStore: ensureFreeSpace(46231) called with
curMem=14859740, maxMem=280248975
15/07/14 18:27:40 INFO MemoryStore: Block broadcast_256_piece0 stored as
bytes in memory (estimated size 45.1 KB, free 253.1 MB)
15/07/14 18:27:40 INFO BlockManagerInfo: Added broadcast_256_piece0 in
memory on localhost:52681 (size: 45.1 KB, free: 263.1 MB)
15/07/14 18:27:40 INFO SparkContext: Created broadcast 256 from broadcast
at DAGScheduler.scala:874
15/07/14 18:27:40 INFO DAGScheduler: Submitting 10 missing tasks from
ResultStage 174 (MapPartitionsRDD[258] at saveAsObjectFile at <console>:45)
15/07/14 18:27:40 INFO TaskSchedulerImpl: Adding task set 174.0 with 10
tasks
15/07/14 18:27:40 INFO TaskSetManager: Starting task 0.0 in stage 174.0
(TID 4513, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 1.0 in stage 174.0
(TID 4514, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 2.0 in stage 174.0
(TID 4515, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 3.0 in stage 174.0
(TID 4516, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 4.0 in stage 174.0
(TID 4517, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 5.0 in stage 174.0
(TID 4518, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 6.0 in stage 174.0
(TID 4519, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 7.0 in stage 174.0
(TID 4520, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 8.0 in stage 174.0
(TID 4521, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO TaskSetManager: Starting task 9.0 in stage 174.0
(TID 4522, localhost, PROCESS_LOCAL, 9486 bytes)
15/07/14 18:27:40 INFO Executor: Running task 0.0 in stage 174.0 (TID 4513)
15/07/14 18:27:40 INFO Executor: Running task 1.0 in stage 174.0 (TID 4514)
15/07/14 18:27:40 INFO Executor: Running task 2.0 in stage 174.0 (TID 4515)
15/07/14 18:27:40 INFO Executor: Running task 3.0 in stage 174.0 (TID 4516)
15/07/14 18:27:40 INFO Executor: Running task 4.0 in stage 174.0 (TID 4517)
15/07/14 18:27:40 INFO Executor: Running task 5.0 in stage 174.0 (TID 4518)
15/07/14 18:27:40 INFO Executor: Running task 6.0 in stage 174.0 (TID 4519)
15/07/14 18:27:40 INFO Executor: Running task 7.0 in stage 174.0 (TID 4520)
15/07/14 18:27:40 INFO Executor: Running task 8.0 in stage 174.0 (TID 4521)
15/07/14 18:27:40 ERROR Executor: Exception in task 1.0 in stage 174.0 (TID
4514)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 ERROR Executor: Exception in task 3.0 in stage 174.0 (TID
4516)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 ERROR Executor: Exception in task 5.0 in stage 174.0 (TID
4518)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 INFO Executor: Running task 9.0 in stage 174.0 (TID 4522)
15/07/14 18:27:40 WARN TaskSetManager: Lost task 1.0 in stage 174.0 (TID
4514, localhost): java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 ERROR TaskSetManager: Task 1 in stage 174.0 failed 1
times; aborting job
15/07/14 18:27:40 INFO TaskSetManager: Lost task 3.0 in stage 174.0 (TID
4516) on executor localhost: java.lang.IllegalStateException (unread block
data) [duplicate 1]
15/07/14 18:27:40 INFO TaskSchedulerImpl: Cancelling stage 174
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 6.0 in
stage 174.0 (TID 4519)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 0.0 in
stage 174.0 (TID 4513)
15/07/14 18:27:40 INFO TaskSchedulerImpl: Stage 174 was cancelled
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 7.0 in
stage 174.0 (TID 4520)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 4.0 in
stage 174.0 (TID 4517)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 8.0 in
stage 174.0 (TID 4521)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 9.0 in
stage 174.0 (TID 4522)
15/07/14 18:27:40 INFO Executor: Executor is trying to kill task 2.0 in
stage 174.0 (TID 4515)
15/07/14 18:27:40 INFO DAGScheduler: ResultStage 174 (saveAsObjectFile at
<console>:45) failed in 0.005 s
15/07/14 18:27:40 INFO DAGScheduler: Job 110 failed: saveAsObjectFile at
<console>:45, took 0.016422 s
15/07/14 18:27:40 ERROR Executor: Exception in task 8.0 in stage 174.0 (TID
4521)
java.lang.IllegalStateException: unread block data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
15/07/14 18:27:40 INFO TaskSetManager: Lost task 5.0 in stage 174.0 (TID
4518) on executor localhost: java.lang.IllegalStateException (unread block
data) [duplicate 2]
15/07/14 18:27:40 INFO TaskSetManager: Lost task 8.0 in stage 174.0 (TID
4521) on executor localhost: java.lang.IllegalStateException (unread block
data) [duplicate 3]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1
in stage 174.0 failed 1 times, most recent failure: Lost task 1.0 in stage
174.0 (TID 4514, localhost): java.lang.IllegalStateException: unread block
data
at
java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:58)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org
$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1266)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1257)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1256)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1256)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730)
at scala.Option.foreach(Option.scala:257)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1450)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1411)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
