Did you try enabling spark.memory.useLegacyMode and upping spark.storage.memoryFraction?
Thanks Best Regards On Fri, Jan 22, 2016 at 3:40 AM, Arun Luthra <arun.lut...@gmail.com> wrote: > WARN MemoryStore: Not enough space to cache broadcast_4 in memory! > (computed 60.2 MB so far) > WARN MemoryStore: Persisting block broadcast_4 to disk instead. > > > Can I increase the memory allocation for broadcast variables? > > I have a few broadcast variables that I create with sc.broadcast() . Are > these labeled starting from 0 or from 1 (in reference to "broadcast_N")? I > want to debug/track down which one is offending. > > As a feature request, it would be good if there were an optional argument > (or perhaps a requireed argument) added to sc.broadcast() so that we could > give it an internal label. Then it would work the same as the > sc.accumulator() "name" argument. It would enable more useful warn/error > messages. > > Arun >
