If you are launching your application with spark-submit you can manually edit the spark-class file to make it 1g as baseline. It’s pretty easy to do and to figure out how once you open the file. This worked for me even if it’s not a final solution of course.
Gianluca On 12 Jun 2014, at 15:16, ericjohnston1989 <ericjohnston1...@gmail.com> wrote: > Hey everyone, > > I'm having some trouble increasing the default storage size for a broadcast > variable. It looks like it defaults to a little less than 512MB every time, > and I can't figure out which configuration to change to increase this. > > INFO storage.MemoryStore: Block broadcast_0 stored as values to memory > (estimated size 426.5 MB, free 64.2 MB) > > (I'm seeing this in the terminal on my driver computer) > > I can change "spark.executor.memory", and that seems to increase the amount > of RAM available on my nodes, but it doesn't seem to adjust this storage > size for my broadcast variables. Any ideas? > > Thanks, > > Eric > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Increase-storage-MemoryStore-size-tp7516.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
