Hi I'm trying to use broadcast variables in my Spark streaming program.
val conf = new
SparkConf().setMaster(SPARK_MASTER).setAppName(APPLICATION_NAME)
val ssc = new StreamingContext(conf, Seconds(1))
val LIMIT = ssc.sparkContext.broadcast(5L)
println(LIMIT.value) // this print 5
val lines = ssc.socketTextStream("localhost", 9999)
val words = lines.flatMap(_.split(" ")) filter (_.size < LIMIT.value)
words.print()
ssc.start()
ssc.awaitTermination()
It throws java.lang.NullPointerException at the line (_.size < LIMIT.value)
so I'm guessing LIMIT is not accessible within the stream
I'm running spark 1.3.1 in standalone mode with 2 nodes cluster. I tried
with spark-shell and it works fine. Please help!
Thanks
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NullPointerException-when-accessing-broadcast-variable-in-DStream-tp22934.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
