Hi, experts. I wrote a very little program to learn how to use Broadcast Variables, but met an exception. The program and the exception are listed as following. Could anyone help me to solve this problem? Thanks!
**************My Program is as following************** object TestBroadcast02 { var brdcst : Broadcast[Array[Int]] = null def main(args: Array[String]) { val conf = new SparkConf() val sc = new SparkContext(conf) brdcst = sc.broadcast(Array(1,2,3,4,5,6)) val rdd = sc.textFile("hdfs://bgdt-dev-hrb/user/spark/tst/charset/A_utf8.txt") rdd.foreachPartition(fun1) sc.stop() } def fun1(it : Iterator[String]) : Unit = { val v = brdcst.value for(i <- v) println("BroadCast Variable:"+i) for(j <- it) println("Text File Content:"+j) } } **************The Exception is as following************** 15/04/21 17:39:53 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, bgdt01.dev.hrb): java.lang.NullPointerException at dhao.test.BroadCast.TestBroadcast02$.fun1(TestBroadcast02.scala:27) at dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22) at dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806) at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:64) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) By the way, if I use anonymous function instead of 'fun1' in my program, it works. But since I think the readability is not good for anonymous functions, I still prefer to use the 'fun1' .