Hi Azuryy,
So you have measurements for hadoop-1.0.4 and hadoop-2.0.3+QJM, but I
think you should also measure hadoop-2.0.3 _wihout_ QJM so you can know
for sure if the performance degrade is actually related to QJM or not.
George
Hi,
HarshJ is a good guy, I've seen this JIRA:
https://issues.apache.org/jira/browse/HDFS-4508
I have a test cluster hadoop-1.0.4, I've upgrade to
hadoop-2.0.3-alpha. mu cluster is very small, four nodes totally.
then I did some test on the original Hadoop and new Hadoop, the
testing is very simple: I have a data file with 450MB, I just put it
on the HDFS.
block size: 128MB, replica: 2
the following is the result:
[root@webdm test]# ll testspeed.tar.gz
-rw-r--r-- 1 root root 452M Feb 18 13:54 testspeed.tar.gz
[root@webdm test]#
//On the hadoop-1.0.4
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hadoop dfs -put
testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
2013-02-18_13:54:24
Warning: $HADOOP_HOME is deprecated.
2013-02-18_13:54:58
//On the hadoop-2.0.3-alpha with QJM
[root@webdm test]# date +%Y-%m-%d_%H:%M:%S; hdfs dfs -put
testspeed.tar.gz / ; date +%Y-%m-%d_%H:%M:%S
2013-02-18_14:13:29
13/02/18 14:13:30 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
2013-02-18_14:14:33
I do think QJM HA feature affect the performance, because each writer
from QJM, it will do: fence old writer; sync in-progress log; start
new log segment; then write. only if writer received a successful
response from a quorum of JNs, writer finished for this time.
But NFS HA just write log segment in the local and NFS, when it
receive successful response from NFS, it finished this time.
So, I just suggest we always keep these two HA features in future,
even in the stable release. which one should be used, which depends on
yourself based on your infrastructure.
Thanks.
