If the JVM heap size is close to the memory limit the OS sometimes kills the process under memory pressure. I've usually found that lowering the executor memory size helps.
Shivaram On Wed, Aug 13, 2014 at 11:01 AM, Matei Zaharia <matei.zaha...@gmail.com> wrote: > What is your Spark executor memory set to? (You can see it in Spark's web > UI at http://<driver>:4040 under the executors tab). One thing to be > aware of is that the JVM never really releases memory back to the OS, so it > will keep filling up to the maximum heap size you set. Maybe 4 executors > with that much heap are taking a lot of the memory. > > Persist as DISK_ONLY should indeed stream data from disk, so I don't think > that will be a problem. > > Matei > > On August 13, 2014 at 6:49:11 AM, rpandya (r...@iecommerce.com) wrote: > > After a lot of grovelling through logs, I found out that the Nagios > monitor > process detected that the machine was almost out of memory, and killed the > SNAP executor process. > > So why is the machine running out of memory? Each node has 128GB of RAM, 4 > executors, about 40GB of data. It did run out of memory if I tried to > cache() the RDD, but I would hope that persist() is implemented so that it > would stream to disk without trying to materialize too much data in RAM. > > Ravi > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Lost-executors-tp11722p12032.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
