You're not using the broadcasted variable within your map operations. You're attempting to modify myObjrct directly which won't work because you are modifying the serialized copy on the executor. You want to do myObjectBroadcasted.value.insert and myObjectBroadcasted.value.lookup.
Sent with Good (www.good.com) -----Original Message----- From: Yiannis Gkoufas [johngou...@gmail.com<mailto:johngou...@gmail.com>] Sent: Tuesday, February 24, 2015 12:12 PM Eastern Standard Time To: user@spark.apache.org Subject: Brodcast Variable updated from one transformation and used from another Hi all, I am trying to do the following. val myObject = new MyObject(); val myObjectBroadcasted = sc.broadcast(myObject); val rdd1 = sc.textFile("/file1").map(e => { myObject.insert(e._1); (e._1,1) }); rdd.cache.count(); //to make sure it is transformed. val rdd2 = sc.textFile("/file2").map(e => { val lookedUp = myObject.lookup(e._1); (e._1, lookedUp) }); When I check the contents of myObject within the map of rdd1 everything seems ok. On the other hand when I check the contents of myObject within the map of rdd2 it seems to be empty. I am doing something wrong? Thanks a lot! ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
