I think you’re going to have to make it serializable by registering it with the
Kryo registrator. I think multiple workers are running as separate VMs so it
might need to be able to serialize and deserialize broadcasted variables to the
different executors.
Thanks,
Ron
On Aug 3, 2014, at 6:38 PM, Fengyun RAO <raofeng...@gmail.com> wrote:
> Could anybody help?
>
> I wonder if I asked a stupid question or I didn't make the question clear?
>
>
> 2014-07-31 21:47 GMT+08:00 Fengyun RAO <raofeng...@gmail.com>:
> As shown here:
> 2 - Why Is My Spark Job so Slow and Only Using a Single Thread?
>
>
> 1
> 2
> 3
> 4
> 5
> 6
> 7
> 8
> 9
> 10
> 11
> 12
> 13
> 14
> 15
> object JSONParser {
> def parse(raw: String): String = ...
> }
>
> object MyFirstSparkJob {
> def main(args: Array[String]) {
> val sc = new SparkContext()
>
> val lines = sc.textFileStream("beacons.txt")
> lines.map(line => JSONParser.parse(line))
> lines.foreach(line => println(line))
>
> ssc.start()
> }
> }
> It says " parser instance is now a singleton created in the scope of our
> driver program" which I thought was in the scope of executor. Am I wrong, or
> why?
>
> What if the parser is not serializable, and I want to share it among tasks in
> the same worker node?
>
>
>
>
>
>
