Unfortunately, After some research I found its just a side effect of how closure containing var works in scala: http://stackoverflow.com/questions/11657676/how-does-scala-maintains-the-values-of-variable-when-the-closure-was-defined
the closure keep referring var broadcasted wrapper as a pointer, until it is shipped to nodes, which is only triggered lazily. So, you can't do this after shipping already started (e.g. change the broadcasted value in a new thread when an action is running). It's neither a feature or bug, just an illusion. I would really like to see a non-blocking Broadcast.set() being implemented, it makes a lot of stochastic algorithms easier to write. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Bug-or-feature-Overwrite-broadcasted-variables-tp12315p12382.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
