What I think is happening that the map operations are executed concurrently
and the map operation in rdd2 has the initial copy of myObjectBroadcated.
Is there a way to apply the transformations sequentially? First materialize
rdd1 and then rdd2.
Thanks a lot!
On 24 February 2015 at 18:49, Yiannis Gkoufas <johngou...@gmail.com> wrote:
> Sorry for the mistake, I actually have it this way:
>
> val myObject = new MyObject();
> val myObjectBroadcasted = sc.broadcast(myObject);
>
> val rdd1 = sc.textFile("/file1").map(e =>
> {
> myObjectBroadcasted.value.insert(e._1);
> (e._1,1)
> });
> rdd.cache.count(); //to make sure it is transformed.
>
> val rdd2 = sc.textFile("/file2").map(e =>
> {
> val lookedUp = myObjectBroadcasted.value.lookup(e._1);
> (e._1, lookedUp)
> });
>
> On 24 February 2015 at 17:36, Ganelin, Ilya <ilya.gane...@capitalone.com>
> wrote:
>
>> You're not using the broadcasted variable within your map operations.
>> You're attempting to modify myObjrct directly which won't work because you
>> are modifying the serialized copy on the executor. You want to do
>> myObjectBroadcasted.value.insert and myObjectBroadcasted.value.lookup.
>>
>>
>>
>> Sent with Good (www.good.com)
>>
>>
>>
>> -----Original Message-----
>> *From: *Yiannis Gkoufas [johngou...@gmail.com]
>> *Sent: *Tuesday, February 24, 2015 12:12 PM Eastern Standard Time
>> *To: *user@spark.apache.org
>> *Subject: *Brodcast Variable updated from one transformation and used
>> from another
>>
>> Hi all,
>>
>> I am trying to do the following.
>>
>> val myObject = new MyObject();
>> val myObjectBroadcasted = sc.broadcast(myObject);
>>
>> val rdd1 = sc.textFile("/file1").map(e =>
>> {
>> myObject.insert(e._1);
>> (e._1,1)
>> });
>> rdd.cache.count(); //to make sure it is transformed.
>>
>> val rdd2 = sc.textFile("/file2").map(e =>
>> {
>> val lookedUp = myObject.lookup(e._1);
>> (e._1, lookedUp)
>> });
>>
>> When I check the contents of myObject within the map of rdd1 everything
>> seems ok.
>> On the other hand when I check the contents of myObject within the map of
>> rdd2 it seems to be empty.
>> I am doing something wrong?
>>
>> Thanks a lot!
>>
>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the
>> intended recipient, you are hereby notified that any review,
>> retransmission, dissemination, distribution, copying or other use of, or
>> taking of any action in reliance upon this information is strictly
>> prohibited. If you have received this communication in error, please
>> contact the sender and delete the material from your computer.
>>
>
>
