Hey Mafish,If you are getting 1-2m blocks on a single datanode, you'll have many other problems - especially with regards to periodic block reports.
With regards to memory, have you tried the compressed pointers JDK option (we saw great benefits on the NN)? Java is incredibly hard to get a straight answer from with regards to memory. You need to perform a GC first manually - the actual usage is the amount it reports used post-GC. You can get these details by using JMX.
Brian On Sep 1, 2009, at 4:08 AM, Mafish Liu wrote:
Both NameNode and DataNode will be affected by number of files greatly.In my test, almost 60% memory are used in datanodes while storing 1m files, and the value reach 80% with 2m files. My test best is with 5 nodes, 1 namenode and 4 datanodes. All nodes have 2GB memory and replication is 3. 2009/9/1 Stas Oskin <stas.os...@gmail.com>:Hi. 2009/9/1 Mafish Liu <maf...@gmail.com>Did you have many small files in your system?Yes, quite plenty.But this should influence the Namenode, and not the Datanode, correct?Regards.-- maf...@gmail.com
smime.p7s
Description: S/MIME cryptographic signature
