You can't broadcast an RDD to begin with, and can't use RDDs inside
RDDs. They are really driver-side concepts.
Yes that's how you'd use a broadcast of anything else though, though
you need to reference ".value" on the broadcast. The 'if' is redundant
in that example, and if it's a map- or collection-like structure, you
don't even need the arg.
RDD2.filter(broadcasted.value.contains)
On Mon, Dec 7, 2015 at 2:43 PM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
> Something like this?
>
> val broadcasted = sc.broadcast(...)
>
> RDD2.filter(value => {
>
> //simply use broadcasted
> if(broadcasted.contains(value)) true
>
> })
>
>
>
> Thanks
> Best Regards
>
> On Fri, Dec 4, 2015 at 10:43 PM, Abhishek Shivkumar
> <abhishek.shivku...@bigdatapartnership.com> wrote:
>>
>> Hi,
>>
>> I have RDD1 that is broadcasted.
>>
>> I have a user defined method for the filter functionality of RDD2, written
>> as follows:
>>
>> RDD2.filter(my_func)
>>
>>
>> I want to access the values of RDD1 inside my_func. Is that possible?
>> Should I pass RDD1 as a parameter into my_func?
>>
>> Thanks
>> Abhishek S
>>
>> NOTICE AND DISCLAIMER
>>
>> This email (including attachments) is confidential. If you are not the
>> intended recipient, notify the sender immediately, delete this email from
>> your system and do not disclose or use for any purpose.
>>
>> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United
>> Kingdom
>> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United
>> Kingdom
>> Big Data Partnership Limited is a company registered in England & Wales
>> with Company No 7904824
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
