Re: how to broadcast huge lookup table?

have a look at .transformWith, you can specify another RDD. Sent from my iPhone On 02 Oct 2015, at 21:50, "saif.a.ell...@wellsfargo.com" > wrote: I tried broadcasting a key-value rdd, but

RE: how to broadcast huge lookup table?

Hi, thank you I would prefer to leave writing-to-disk as a last resort. Is it a last resort? Saif From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Friday, October 02, 2015 3:54 PM To: Ellafi, Saif A. Cc: user Subject: Re: how to broadcast huge lookup table? Have you considered using external

how to broadcast huge lookup table?

I tried broadcasting a key-value rdd, but then I cannot perform any rdd-actions inside a map/foreach function of another rdd. any tips? If going into scala collections I end up with huge memory bottlenecks. Saif

Re: how to broadcast huge lookup table?

Have you considered using external storage such as hbase for storing the look up table ? Cheers On Fri, Oct 2, 2015 at 11:50 AM, wrote: > I tried broadcasting a key-value rdd, but then I cannot perform any > rdd-actions inside a map/foreach function of another