Re: Bug or feature? Overwrite broadcasted variables.
Unfortunately, After some research I found its just a side effect of how closure containing var works in scala: http://stackoverflow.com/questions/11657676/how-does-scala-maintains-the-values-of-variable-when-the-closure-was-defined the closure keep referring var broadcasted wrapper as a pointer, until it is shipped to nodes, which is only triggered lazily. So, you can't do this after shipping already started (e.g. change the broadcasted value in a new thread when an action is running). It's neither a feature or bug, just an illusion. I would really like to see a non-blocking Broadcast.set() being implemented, it makes a lot of stochastic algorithms easier to write. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Bug-or-feature-Overwrite-broadcasted-variables-tp12315p12382.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Bug or feature? Overwrite broadcasted variables.
Yeah, Thanks a lot. I know for people understanding lazy execution this seems straightforward. But for those who don't it may become a liability. I've only tested its stability on a small example (which seems stable), hopefully it's not a serendipity. Can a committer confirm this? Yours Peng -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Bug-or-feature-Overwrite-broadcasted-variables-tp12315p12348.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Bug or feature? Overwrite broadcasted variables.
I think the behavior is by designed. Because if b is not persisted, and in each
call b.collect, broadcasted has point to a new broadcasted variable, serialized
by driver, and fetched by executors.
If you do persist, you don’t expect the RDD get changed due to new broadcasted
variable.
Thanks.
Zhan Zhang
On Aug 18, 2014, at 11:26 AM, Peng Cheng wrote:
> I'm curious to see that if you declare broadcasted wrapper as a var, and
> overwrite it in the driver program, the modification can have stable impact
> on all transformations/actions defined BEFORE the overwrite but was executed
> lazily AFTER the overwrite:
>
> val a = sc.parallelize(1 to 10)
>
>var broadcasted = sc.broadcast("broad")
>
>val b = a.map(_ + broadcasted.value)
> // b.persist()
>for (line <- b.collect()) { print(line) }
>
>println("\n===")
>broadcasted = sc.broadcast("cast")
>
>for (line <- b.collect()) { print(line) }
>
> the result is:
>
> 1broad2broad3broad4broad5broad6broad7broad8broad9broad10broad
> ===
> 1cast2cast3cast4cast5cast6cast7cast8cast9cast10cast
>
> Of course, if you persist b before overwriting it will still get the
> non-surprising result (both are 10broad... because they are persisted). This
> can be useful sometimes but may cause confusion at other times (people can
> no longer add persist at will just for backup because it may change the
> result).
>
> So far I've found no documentation supporting this feature. So can some one
> confirm that its a feature craftly designed?
>
> Yours Peng
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Bug-or-feature-Overwrite-broadcasted-variables-tp12315.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
Bug or feature? Overwrite broadcasted variables.
I'm curious to see that if you declare broadcasted wrapper as a var, and
overwrite it in the driver program, the modification can have stable impact
on all transformations/actions defined BEFORE the overwrite but was executed
lazily AFTER the overwrite:
val a = sc.parallelize(1 to 10)
var broadcasted = sc.broadcast("broad")
val b = a.map(_ + broadcasted.value)
// b.persist()
for (line <- b.collect()) { print(line) }
println("\n===")
broadcasted = sc.broadcast("cast")
for (line <- b.collect()) { print(line) }
the result is:
1broad2broad3broad4broad5broad6broad7broad8broad9broad10broad
===
1cast2cast3cast4cast5cast6cast7cast8cast9cast10cast
Of course, if you persist b before overwriting it will still get the
non-surprising result (both are 10broad... because they are persisted). This
can be useful sometimes but may cause confusion at other times (people can
no longer add persist at will just for backup because it may change the
result).
So far I've found no documentation supporting this feature. So can some one
confirm that its a feature craftly designed?
Yours Peng
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Bug-or-feature-Overwrite-broadcasted-variables-tp12315.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
