Re: How to avoid being killed by YARN node manager ?
Thank you for your reply. I'm sorry confirmation is slow. I'll try the tuning 'spark.yarn.executor.memoryOverhead'. Thanks, Yuichiro Sakamoto On 2015/03/25 0:56, Sandy Ryza wrote: Hi Yuichiro, The way to avoid this is to boost spark.yarn.executor.memoryOverhead until the executors have enough off-heap memory to avoid going over their limits. -Sandy On Tue, Mar 24, 2015 at 11:49 AM, Yuichiro Sakamoto ks...@muc.biglobe.ne.jp mailto:ks...@muc.biglobe.ne.jp wrote: Hello. We use ALS(Collaborative filtering) of Spark MLlib on YARN. Spark version is 1.2.0 included CDH 5.3.1. 1,000,000,000 records(5,000,000 users data and 5,000,000 items data) are used for machine learning with ALS. These large quantities of data increases virtual memory usage, node manager of YARN kills Spark worker process. Even though Spark run again after killing process, Spark worker process is killed again. As a result, the whole Spark processes are terminated. # Spark worker process is killed, it seems that virtual memory usage increased by # 'Shuffle' or 'Disk writing' gets over the threshold of YARN. To avoid such a case from occurring, we use the method that 'yarn.nodemanager.vmem-check-enabled' is false, then exit successfully. But it does not seem to have an appropriate way. If you know, please let me know about tuning method of Spark. The conditions of machines and Spark settings are as follows. 1)six machines, physical memory is 32GB of each machine. 2)Spark settings - spark.executor.memory=16g - spark.closure.serializer=org.apache.spark.serializer.KryoSerializer - spark.rdd.compress=true - spark.shuffle.memoryFraction=0.4 Thanks, Yuichiro Sakamoto -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-being-killed-by-YARN-node-manager-tp22199.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org mailto:user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org mailto:user-h...@spark.apache.org -- *=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=* Yuichiro SAKAMOTO - ks...@muc.biglobe.ne.jp - phonypian...@gmail.com - http://www2u.biglobe.ne.jp/~yuichi/ *=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=* - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: How to avoid being killed by YARN node manager ?
Hi Yuichiro, The way to avoid this is to boost spark.yarn.executor.memoryOverhead until the executors have enough off-heap memory to avoid going over their limits. -Sandy On Tue, Mar 24, 2015 at 11:49 AM, Yuichiro Sakamoto ks...@muc.biglobe.ne.jp wrote: Hello. We use ALS(Collaborative filtering) of Spark MLlib on YARN. Spark version is 1.2.0 included CDH 5.3.1. 1,000,000,000 records(5,000,000 users data and 5,000,000 items data) are used for machine learning with ALS. These large quantities of data increases virtual memory usage, node manager of YARN kills Spark worker process. Even though Spark run again after killing process, Spark worker process is killed again. As a result, the whole Spark processes are terminated. # Spark worker process is killed, it seems that virtual memory usage increased by # 'Shuffle' or 'Disk writing' gets over the threshold of YARN. To avoid such a case from occurring, we use the method that 'yarn.nodemanager.vmem-check-enabled' is false, then exit successfully. But it does not seem to have an appropriate way. If you know, please let me know about tuning method of Spark. The conditions of machines and Spark settings are as follows. 1)six machines, physical memory is 32GB of each machine. 2)Spark settings - spark.executor.memory=16g - spark.closure.serializer=org.apache.spark.serializer.KryoSerializer - spark.rdd.compress=true - spark.shuffle.memoryFraction=0.4 Thanks, Yuichiro Sakamoto -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-being-killed-by-YARN-node-manager-tp22199.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
How to avoid being killed by YARN node manager ?
Hello. We use ALS(Collaborative filtering) of Spark MLlib on YARN. Spark version is 1.2.0 included CDH 5.3.1. 1,000,000,000 records(5,000,000 users data and 5,000,000 items data) are used for machine learning with ALS. These large quantities of data increases virtual memory usage, node manager of YARN kills Spark worker process. Even though Spark run again after killing process, Spark worker process is killed again. As a result, the whole Spark processes are terminated. # Spark worker process is killed, it seems that virtual memory usage increased by # 'Shuffle' or 'Disk writing' gets over the threshold of YARN. To avoid such a case from occurring, we use the method that 'yarn.nodemanager.vmem-check-enabled' is false, then exit successfully. But it does not seem to have an appropriate way. If you know, please let me know about tuning method of Spark. The conditions of machines and Spark settings are as follows. 1)six machines, physical memory is 32GB of each machine. 2)Spark settings - spark.executor.memory=16g - spark.closure.serializer=org.apache.spark.serializer.KryoSerializer - spark.rdd.compress=true - spark.shuffle.memoryFraction=0.4 Thanks, Yuichiro Sakamoto -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-avoid-being-killed-by-YARN-node-manager-tp22199.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
