Re: How to access a RDD (that has been broadcasted) inside the filter method of another RDD?
You can't broadcast an RDD to begin with, and can't use RDDs inside RDDs. They are really driver-side concepts. Yes that's how you'd use a broadcast of anything else though, though you need to reference ".value" on the broadcast. The 'if' is redundant in that example, and if it's a map- or collection-like structure, you don't even need the arg. RDD2.filter(broadcasted.value.contains) On Mon, Dec 7, 2015 at 2:43 PM, Akhil Daswrote: > Something like this? > > val broadcasted = sc.broadcast(...) > > RDD2.filter(value => { > > //simply use broadcasted > if(broadcasted.contains(value)) true > > }) > > > > Thanks > Best Regards > > On Fri, Dec 4, 2015 at 10:43 PM, Abhishek Shivkumar > wrote: >> >> Hi, >> >> I have RDD1 that is broadcasted. >> >> I have a user defined method for the filter functionality of RDD2, written >> as follows: >> >> RDD2.filter(my_func) >> >> >> I want to access the values of RDD1 inside my_func. Is that possible? >> Should I pass RDD1 as a parameter into my_func? >> >> Thanks >> Abhishek S >> >> NOTICE AND DISCLAIMER >> >> This email (including attachments) is confidential. If you are not the >> intended recipient, notify the sender immediately, delete this email from >> your system and do not disclose or use for any purpose. >> >> Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United >> Kingdom >> Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United >> Kingdom >> Big Data Partnership Limited is a company registered in England & Wales >> with Company No 7904824 > > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to access a RDD (that has been broadcasted) inside the filter method of another RDD?
Something like this? val broadcasted = sc.broadcast(...) RDD2.filter(value => { //simply use *broadcasted* if(broadcasted.contains(value)) true }) Thanks Best Regards On Fri, Dec 4, 2015 at 10:43 PM, Abhishek Shivkumar < abhishek.shivku...@bigdatapartnership.com> wrote: > Hi, > > I have RDD1 that is broadcasted. > > I have a user defined method for the filter functionality of RDD2, written > as follows: > > RDD2.filter(my_func) > > > I want to access the values of RDD1 inside my_func. Is that possible? > Should I pass RDD1 as a parameter into my_func? > > Thanks > Abhishek S > > *NOTICE AND DISCLAIMER* > > This email (including attachments) is confidential. If you are not the > intended recipient, notify the sender immediately, delete this email from > your system and do not disclose or use for any purpose. > > Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United > Kingdom > Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United > Kingdom > Big Data Partnership Limited is a company registered in England & Wales > with Company No 7904824 >
How to access a RDD (that has been broadcasted) inside the filter method of another RDD?
Hi, I have RDD1 that is broadcasted. I have a user defined method for the filter functionality of RDD2, written as follows: RDD2.filter(my_func) I want to access the values of RDD1 inside my_func. Is that possible? Should I pass RDD1 as a parameter into my_func? Thanks Abhishek S -- *NOTICE AND DISCLAIMER* This email (including attachments) is confidential. If you are not the intended recipient, notify the sender immediately, delete this email from your system and do not disclose or use for any purpose. Business Address: Eagle House, 163 City Road, London, EC1V 1NR. United Kingdom Registered Office: Finsgate, 5-7 Cranwood Street, London, EC1V 9EE. United Kingdom Big Data Partnership Limited is a company registered in England & Wales with Company No 7904824