Namenode in trunk has much slower performance than Namenode in MR-279 branch ----------------------------------------------------------------------------
Key: HDFS-2142 URL: https://issues.apache.org/jira/browse/HDFS-2142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Eric Payne I am measureing the performance of the namenode by running the org.apache.hadoop.fs.loadGenerator.LoadGenerator application. This application shows there is a very large slowdown in the processing of opens, writes, closes, and operations per second in the trunk when compared to the MR-279 branch There have been some race conditions and locking issues fixed in trunk, which is a very good thing because these race conditions were causing the namenode to crash under load conditions (see HDFS:1257). However, the slowdown to the namenode is considerable. I am still trying to verify which changes caused the slowdown. It was originally suggested that the HDFS:988 may have caused the slowdown, but I don't think it was the culprit. I have checked out and built from SVN 3 revisions previous to HDFS988 and they all have about the same performance. Here is my environment: Host0: namenode daemon Host1-9: simulate many datanodes using org.apache.hadoop.hdfs.DataNodeCluster LoadGenerator output on MR-279 branch: Average open execution time: 1.8496516782773909ms Average deletion execution time: 2.956340167046317ms Average create execution time: 3.725259427992913ms Average write_close execution time: 11.151860288534548ms Average operations per second: 1053.3666666666666ops/s LoadGenerator output on trunk: Average open execution time: 28.603515625ms Average deletion execution time: 32.20792079207921ms Average create execution time: 32.37326732673267ms Average write_close execution time: 82.84752475247525ms Average operations per second: 135.13333333333333ops/s -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira