Hi Sonal Thank you for the response but since we are joining to reference data different partitions of application data would need to join with same reference data and thus I am not sure if spark join would be a good fit for this.
Eg out application data has person with zip code and then the reference data has attributes of zip code (city, state etc), so person objects in different partitions in spark cluster may be referring to same zip and if I partition our application data by zip there will be a lot of shuffling and then latter for our application code we would have to repatriation with another key and another shuffling of whole application data. I think it will not be a good idea. Thanks Ankur On Oct 16, 2014 11:06 PM, "Sonal Goyal" <sonalgoy...@gmail.com> wrote: > Hi Ankur, > > If your rdds have common keys, you can look at partitioning both your > datasets using a custom partitioner based on keys so that you can avoid > shuffling and optimize join performance. > > HTH > > Best Regards, > Sonal > Nube Technologies <http://www.nubetech.co> > > <http://in.linkedin.com/in/sonalgoyal> > > > > On Fri, Oct 17, 2014 at 4:27 AM, Ankur Srivastava < > ankur.srivast...@gmail.com> wrote: > >> Hi, >> >> I have a rdd which is my application data and is huge. I want to join >> this with reference data which is also huge to fit in-memory and thus I do >> not want to use Broadcast variable. >> >> What other options do I have to perform such joins? >> >> I am using Cassandra as my data store, so should I just query cassandra >> to get the reference data needed? >> >> Also when I join two rdds, will it result in rdd scan or would spark do a >> hash partition on the two rdds to get the data with same keys on same node? >> >> Thanks >> Ankur >> > >
