Dear Community,
Please ignore my last post about Spark SQL.
When I run:
val file = sc.textFile("./README.md")
val count = file.flatMap(line => line.split(" ")).map(word => (word,
1)).reduceByKey(_+_)
count.collect()
it happends too.
is there any possible reason for that? we make have some adjustment in
network last night
Chen Weikeng
14/10/09 20:45:23 ERROR Executor: Exception in task ID 1
java.lang.NullPointerException at
org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:571) at
org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:571) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
org.apache.spark.sql.SchemaRDD.compute(SchemaRDD.scala:116) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) at
org.apache.spark.scheduler.Task.run(Task.scala:51) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) 14/10/09 20:45:23 INFO
CoarseGrainedExecutorBackend: Got assigned task 2 14/10/09 20:45:23 INFO
Executor: Running task ID 2 14/10/09 20:45:23 DEBUG BlockManager: Getting local
block broadcast_0 14/10/09 20:45:23 DEBUG BlockManager: Level for block
broadcast_0 is StorageLevel(true, true, false, true, 1) 14/10/09 20:45:23 DEBUG
BlockManager: Getting block broadcast_0 from memory 14/10/09 20:45:23 INFO
BlockManager: Found block broadcast_0 locally 14/10/09 20:45:23 DEBUG Executor:
Task 2's epoch is 0 14/10/09 20:45:23 INFO HadoopRDD: Input split:
file:/public/rdma14/app/spark-rdma/examples/src/main/resources/people.txt:16+16
14/10/09 20:45:23 ERROR Executor: Exception in task ID 2
java.lang.NullPointerException at
org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:571) at
org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:571) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) at
org.apache.spark.sql.SchemaRDD.compute(SchemaRDD.scala:116) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) at
org.apache.spark.rdd.RDD.iterator(RDD.scala:229) at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111) at
org.apache.spark.scheduler.Task.run(Task.scala:51) at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:183) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
