Check how much memory is allocated for the JVM running namenode. In a file HADOOP_INSTALL/conf/hadoop-env.sh you should change a line that starts with "export HADOOP_HEAPSIZE=1000"
It's set to 1GB by default. On Fri, Jul 25, 2008 at 2:51 AM, Gert Pfeifer <[EMAIL PROTECTED]> wrote: > Update on this one... > > I put some more memory in the machine running the name node. Now fsck is > running. Unfortunately ls fails with a time-out. > > I identified one directory that causes the trouble. I can run fsck on it > but not ls. > > What could be the problem? > > Gert > > Gert Pfeifer schrieb: > > Hi, >> I am running a Hadoop DFS on a cluster of 5 data nodes with a name node >> and one secondary name node. >> >> I have 1788874 files and directories, 1465394 blocks = 3254268 total. >> Heap Size max is 3.47 GB. >> >> My problem is that I produce many small files. Therefore I have a cron >> job which just runs daily across the new files and copies them into >> bigger files and deletes the small files. >> >> Apart from this program, even a fsck kills the cluster. >> >> The problem is that, as soon as I start this program, the heap space of >> the name node reaches 100 %. >> >> What could be the problem? There are not many small files right now and >> still it doesn't work. I guess we have this problem since the upgrade to >> 0.17. >> >> Here is some additional data about the DFS: >> Capacity : 2 TB >> DFS Remaining : 1.19 TB >> DFS Used : 719.35 GB >> DFS Used% : 35.16 % >> >> Thanks for hints, >> Gert >> > >
