Re: Understanding Spark Memory distribution

2015-03-28 Thread Wisely Chen
Hi Ankur If your hardware is ok, looks like it is config problem. Can you show me the config of spark-env.sh or JVM config? Thanks Wisely Chen 2015-03-28 15:39 GMT+08:00 Ankur Srivastava ankur.srivast...@gmail.com: Hi Wisely, I have 26gb for driver and the master is running on m3.2xlarge

Re: Understanding Spark Memory distribution

2015-03-28 Thread Wisely Chen
Hi In broadcast, spark will collect the whole 3gb object into master node and broadcast to each slaves. It is very common situation that the master node don't have enough memory . What is your master node settings? Wisely Chen Ankur Srivastava ankur.srivast...@gmail.com 於 2015年3月28日 星期六寫道: I