On Sun, Dec 9, 2012 at 5:45 AM, a...@hsk.hk <a...@hsk.hk> wrote: > Hi, > > I always set "vm.swappiness = 0" for my hadoop servers (PostgreSQL > servers too). >
I have just done this for that machine. So far, I have not seen a re-occurrence of the strange behavior; it appears this might have solved the problem. > The reason is that Linux moves memory pages to swap space if they have not > been accessed for a period of time (swapping). Java virtual machine (JVM) > does not act well in the case of swapping that will make MapReduce (and > HBase and ZooKeeper) run into trouble. So I would suggest to set > vm.swappiness = 0. > > Thanks > ac > > On 9 Dec 2012, at 12:58 PM, seth wrote: > > > Oracle frequently recommends vm.swappiness = 0 to get well behaved RAC > nodes. Otherwise you start paging out things you don't usually want paged > out in favor of a larger filesystem cache. > > > > There is also a vm parameter that controls the minimum size of the free > chain, might want to increase that a bit. > > > > Also, look into hosting your JVM heap on huge pages, they can't be paged > out and will help the JVM perform better too. > > > > On Dec 8, 2012, at 6:09 PM, Robert Dyer <rd...@iastate.edu> wrote: > > > >> Has anyone experienced a TaskTracker/DataNode behaving like the > attached image? > >> > >> This was during a MR job (which runs often). Note the extremely high > System CPU time. Upon investigating I saw that out of 64GB ram the system > had allocated almost 45GB to cache! > >> > >> I did a sudo sh -c "sync ; echo 3 > /proc/sys/vm/drop_cache ; sync" > which is roughly where the graph goes back to normal (much lower System, > much higher User). > >> > >> This has happened a few times. > >> > >> I have tried playing with the sysctl vm.swappiness value (default of > 60) by setting it to 30 (which it was at when the graph was collected) and > now to 10. I am not sure that helps. > >> > >> Any ideas? Anyone else run into this before? > >> > >> 24 cores > >> 64GB ram > >> 4x2TB sata3 hdd > >> > >> Running Hadoop 1.0.4, with a DataNode (2gb heap), TaskTracker (2gb > heap) on this machine. > >> > >> 24 map slots (1gb heap each), no reducers. > >> > >> Also running HBase 0.94.2 with a RS (8gb ram) on this machine. > >> <cpu-use.png> > > -- Robert Dyer rd...@iastate.edu
