I would disagree. While you can tune the system to not over subscribe, I would rather have it hit swap then fail. Especially on long running jobs.
If we look at oversubscription on Hadoop clusters which are not running HBase… they survive. Its when you have things like HBase that don’t handle swap well… or you don’t allocate enough swap that things go boom. Also consider that you could move swap to something that is faster than spinning rust. > On Sep 22, 2016, at 12:44 PM, Sean Owen <so...@cloudera.com> wrote: > > I don't think I'd enable swap on a cluster. You'd rather processes > fail than grind everything to a halt. You'd buy more memory or > optimize memory before trading it for I/O. > > On Thu, Sep 22, 2016 at 6:29 PM, Michael Segel > <msegel_had...@hotmail.com> wrote: >> Ok… gotcha… wasn’t sure that YARN just looked at the heap size allocation >> and ignored the off heap. >> >> WRT over all OS memory… this would be one reason why I’d keep a decent >> amount of swap around. (Maybe even putting it on a fast device like an .m2 >> or PCIe flash drive….
