Note that `object` is equivalent to a class full of static fields / methods
(in Java), so the data it holds will not be serialized, ever.
What you want is a config class instead, so you can instantiate it, and
that instance can be serialized. Then you can easily do (1) or (3).
On Mon, May 11, 2015 at 7:55 AM, hotdog <lisend...@163.com> wrote:
> I have a global config Object in my spark app.
>
> Object Config {
> var lambda = 0.01
> }
>
> and I will set the value of lambda according to user's input.
>
> Object MyApp {
> def main(args: String[]) {
> Config.lambda = args(0).toDouble
> ...
> rdd.map(_ * Config.lambda)
> }
> }
>
> and I found that the modification does not take effect in executors. The
> value of lambda is always 0.01. I guess the modification in driver's jvm
> will not effect the executor's.
>
>
> Do you have other solution ?
>
> I found a similar question in stackoverflow :
>
>
> http://stackoverflow.com/questions/29685330/how-to-set-and-get-static-variables-from-spark
>
> in @DanielL. 's answer, he gives three solutions:
>
> 1. Put the value inside a closure to be serialized to the executors to
> perform a task.
>
> **But I wonder how to write the closure and how to serialized it to the
> executors, could any one give me some code example?**
>
>
> 2.If the values are fixed or the configuration is available on the executor
> nodes (lives inside the jar, etc), then you can have a lazy val,
> guaranteeing initialization only once.
>
> **what if I declare the lambda as a lazy val variable? the modification in
> the driver will take effects in the executors? could you give me some code
> example?**
>
>
> 3.Create a broadcast variable with the data. I know this way, but it also
> need a local Broadcast[] variable which wraps the Config Object right? for
> example:
>
> val config = sc.broadcast(Config)
>
> and use `config.value.lambda` in executors , right ?
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-use-the-global-config-variables-in-executors-tp22846.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
--
Marcelo
