To clarify, yarn actually supports excluding node right when requesting resources. It’s spark that doesn’t provide a way to populate such a blacklist.
If you can change yarn config, the equivalent is node label: https://hadoop.apache.org/docs/r2.7.4/hadoop-yarn/hadoop-yarn-site/NodeLabel.html ________________________________ From: Li Gao <ligao...@gmail.com> Sent: Saturday, January 19, 2019 8:43 AM To: Felix Cheung Cc: Serega Sheypak; user Subject: Re: Spark on Yarn, is it possible to manually blacklist nodes before running spark job? on yarn it is impossible afaik. on kubernetes you can use taints to keep certain nodes outside of spark On Fri, Jan 18, 2019 at 9:35 PM Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>> wrote: Not as far as I recall... ________________________________ From: Serega Sheypak <serega.shey...@gmail.com<mailto:serega.shey...@gmail.com>> Sent: Friday, January 18, 2019 3:21 PM To: user Subject: Spark on Yarn, is it possible to manually blacklist nodes before running spark job? Hi, is there any possibility to tell Scheduler to blacklist specific nodes in advance?
