Re: MemoryStore: Not enough space to cache broadcast_N in memory

[Akhil Das]

Did you try enabling spark.memory.useLegacyMode and
upping spark.storage.memoryFraction?

Thanks
Best Regards

On Fri, Jan 22, 2016 at 3:40 AM, Arun Luthra  wrote:

> WARN MemoryStore: Not enough space to cache broadcast_4 in memory!
> (computed 60.2 MB so far)
> WARN MemoryStore: Persisting block broadcast_4 to disk instead.
>
>
> Can I increase the memory allocation for broadcast variables?
>
> I have a few broadcast variables that I create with sc.broadcast() . Are
> these labeled starting from 0 or from 1 (in reference to "broadcast_N")? I
> want to debug/track down which one is offending.
>
> As a feature request, it would be good if there were an optional argument
> (or perhaps a requireed argument) added to sc.broadcast() so that we could
> give it an internal label. Then it would work the same as the
> sc.accumulator() "name" argument. It would enable more useful warn/error
> messages.
>
> Arun
>

MemoryStore: Not enough space to cache broadcast_N in memory

[Arun Luthra]

WARN MemoryStore: Not enough space to cache broadcast_4 in memory!
(computed 60.2 MB so far)
WARN MemoryStore: Persisting block broadcast_4 to disk instead.


Can I increase the memory allocation for broadcast variables?

I have a few broadcast variables that I create with sc.broadcast() . Are
these labeled starting from 0 or from 1 (in reference to "broadcast_N")? I
want to debug/track down which one is offending.

As a feature request, it would be good if there were an optional argument
(or perhaps a requireed argument) added to sc.broadcast() so that we could
give it an internal label. Then it would work the same as the
sc.accumulator() "name" argument. It would enable more useful warn/error
messages.

Arun