please make always broadcast variable as final, they cant be changed as per
their property.
Also Does line 27 corresponds to extracting value?
On 21 April 2015 at 19:28, Ted Yu <yuzhih...@gmail.com> wrote:
> Does line 27 correspond to brdcst.value ?
>
> Cheers
>
>
>
> > On Apr 21, 2015, at 3:19 AM, donhoff_h <165612...@qq.com> wrote:
> >
> > Hi, experts.
> >
> > I wrote a very little program to learn how to use Broadcast Variables,
> but met an exception. The program and the exception are listed as
> following. Could anyone help me to solve this problem? Thanks!
> >
> > **************My Program is as following**************
> > object TestBroadcast02 {
> > var brdcst : Broadcast[Array[Int]] = null
> >
> > def main(args: Array[String]) {
> > val conf = new SparkConf()
> > val sc = new SparkContext(conf)
> > brdcst = sc.broadcast(Array(1,2,3,4,5,6))
> > val rdd =
> sc.textFile("hdfs://bgdt-dev-hrb/user/spark/tst/charset/A_utf8.txt")
> > rdd.foreachPartition(fun1)
> > sc.stop()
> > }
> >
> > def fun1(it : Iterator[String]) : Unit = {
> > val v = brdcst.value
> > for(i <- v) println("BroadCast Variable:"+i)
> > for(j <- it) println("Text File Content:"+j)
> > }
> > }
> > **************The Exception is as following**************
> > 15/04/21 17:39:53 WARN scheduler.TaskSetManager: Lost task 0.0 in stage
> 0.0 (TID 0, bgdt01.dev.hrb): java.lang.NullPointerException
> > at
> dhao.test.BroadCast.TestBroadcast02$.fun1(TestBroadcast02.scala:27)
> > at
> dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22)
> > at
> dhao.test.BroadCast.TestBroadcast02$$anonfun$main$1.apply(TestBroadcast02.scala:22)
> > at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806)
> > at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1.apply(RDD.scala:806)
> > at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)
> > at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)
> > at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> > at org.apache.spark.scheduler.Task.run(Task.scala:64)
> > at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > By the way, if I use anonymous function instead of 'fun1' in my program,
> it works. But since I think the readability is not good for anonymous
> functions, I still prefer to use the 'fun1' .
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
