I would second upping the NameNode RAM. Most name nodes have the most ram of any server in the cluster. Make sure you are not storing small files and have a very high block count. From the article that I linked below about 10 million files = 3Gb of JVM heap for the NameNode.
http://www.cloudera.com/blog/2009/02/the-small-files-problem/ ________________________________________ From: Edward Capriolo [[email protected]] Sent: Friday, August 13, 2010 2:16 PM To: [email protected] Subject: Re: Fw: namenode crash On Fri, Aug 13, 2010 at 3:03 PM, Ryan Rawson <[email protected]> wrote: > We don't use centos here at Stumbleupon... your version looks quite > old! Our uname looks like: > > Linux host 2.6.28-14-generic #47-Ubuntu SMP Sat Jul 25 01:19:55 UTC > 2009 x86_64 GNU/Linux > > I'd consider using something newer than 2.6.18! > > On Fri, Aug 13, 2010 at 11:54 AM, Jean-Daniel Cryans > <[email protected]> wrote: >> u18 should never be used. >> >> You say it's crashing on both u17 and u20? How is it crashing? (it's >> kind of a vague word) >> >> Here with use both u14 and u17 on 20 nodes clusters without any issue. >> >> J-D >> >> On Fri, Aug 13, 2010 at 11:27 AM, Jinsong Hu <[email protected]> wrote: >>> >>> >>> Hi, There: >>> does anybody know of a good combination of centos version and jdk version >>> that works stably ? I am using centos version >>> >>> Linux 2.6.18-194.8.1.el5.centos.plus #1 SMP Wed Jul 7 11:45:38 EDT 2010 >>> x86_64 x86_64 x86_64 GNU/Linux >>> >>> jdk version >>> Java(TM) SE Runtime Environment (build 1.6.0_17-b04) >>> Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) >>> >>> and run the namenode with the following jvm config >>> -Xmx1000m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode >>> -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError >>> -XX:+UseCompressedOops -XX:+DoEscapeAnalysis -XX:+AggressiveOpts -Xmx2G >>> >>> but it crashed silently after 16 hours. >>> >>> I used jdk >>> Java(TM) SE Runtime Environment (build 1.6.0_20-b02) >>> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) >>> >>> with the same jvm config, and the namenode crashed in about 1 week. I >>> searched internet and people say 1.6.0_18 is not good. >>> but does anybody can recommend a good combination of jdk and os version >>> that can run stably ? >>> >>> >>> This crashing problem doesn't happen with a small cluster of 4 datanodes. >>> but it happens with a cluster of 17 datanodes. >>> >>> Jimmy. >>> >>> >>> >> > RedHat/CentOS backport kernel patches and attempt to keep the minor number relatively stable. Something like 2.6.18-194 is probably closer to 2.6.28 then 2.6.18. Do you have any more free memory? Maybe for fun raise you -Xmx4G. Edward
