> 1) Fix the direct memory usage to a fixed value -XX:MaxDirectMemorySize=1G
This flag should be in RS ou DN? Thanks, Pablo -----Original Message----- From: Laxman [mailto:lakshman...@huawei.com] Sent: quarta-feira, 11 de julho de 2012 01:41 To: user@hbase.apache.org; Pablo Musa Subject: RE: Hmaster and HRegionServer disappearance reason to ask IIUC, this seems to be a problem with Direct memory cleanup. I've seen this problem mostly with application using lot of NIO (RegionServer, DataNode). Symptoms of this problem 1) Allocated heap is 1 GB 2) VIRT/RES will occupy 10GB or even more 3) GC logs says, it takes more than 1 minute to clean up where allocated heap is just few GB (<4GB) 4) Your application uses lot of NIO/RMI calls(DataNode, RegionServer) How to solve? 1) Fix the direct memory usage to a fixed value -XX:MaxDirectMemorySize=1G 2) DON'T set -XX:+DisableExplicitGC Recent issue which is relevant to this HDFS-3600. Please let us know your findings. -- Regards, Laxman > -----Original Message----- > From: Pablo Musa [mailto:pa...@psafe.com] > Sent: Tuesday, July 10, 2012 7:58 PM > To: user@hbase.apache.org > Subject: RE: Hmaster and HRegionServer disappearance reason to ask > > I tried to change the flag but yesterday it happened again: > > Application time: 0.3025790 seconds > 30013.866: [GC 30013.866: [ParNew: 106069K->989K(118016K), 178.8437590 > secs] 473853K->369013K(1275392K), 178.8438570 secs] [Times: user=0.05 > sys=178.82, real=178.81 secs] Total time for which application threads > were stopped: 178.8441500 seconds > > I also checked the possibility of a swap, but I don't think it is the > problem as vmstat always show clean swap. > > Help guys, please :) > > Abs, > Pablo
